2024-08-20T22:04:54.6605230Z Current runner version: '2.319.1' 2024-08-20T22:04:54.6611544Z Runner name: 'i-0eeb15fe47bdc7534' 2024-08-20T22:04:54.6612331Z Runner group name: 'Default' 2024-08-20T22:04:54.6613291Z Machine name: 'ip-10-0-28-59' 2024-08-20T22:04:54.6630307Z Testing runner upgrade compatibility 2024-08-20T22:04:54.8477214Z ##[group]GITHUB_TOKEN Permissions 2024-08-20T22:04:54.8479248Z Actions: read 2024-08-20T22:04:54.8479772Z Attestations: read 2024-08-20T22:04:54.8480367Z Checks: read 2024-08-20T22:04:54.8480861Z Contents: read 2024-08-20T22:04:54.8481364Z Deployments: read 2024-08-20T22:04:54.8481966Z Discussions: read 2024-08-20T22:04:54.8482440Z Issues: read 2024-08-20T22:04:54.8482918Z Metadata: read 2024-08-20T22:04:54.8483552Z Packages: read 2024-08-20T22:04:54.8484015Z Pages: read 2024-08-20T22:04:54.8484520Z PullRequests: read 2024-08-20T22:04:54.8485123Z RepositoryProjects: read 2024-08-20T22:04:54.8485713Z SecurityEvents: read 2024-08-20T22:04:54.8486233Z Statuses: read 2024-08-20T22:04:54.8486956Z ##[endgroup] 2024-08-20T22:04:54.8490129Z Secret source: Actions 2024-08-20T22:04:54.8490937Z Prepare workflow directory 2024-08-20T22:04:54.9561391Z Prepare all required actions 2024-08-20T22:04:54.9731070Z Getting action download info 2024-08-20T22:04:55.1791145Z Download action repository 'pytorch/test-infra@main' (SHA:0c3a2634aaa2f638c8f640e743f03d696ce1191f) 2024-08-20T22:04:55.8842959Z Download action repository 'pytorch/pytorch@main' (SHA:1ae5d5bb62141d7c9b1b0b66c66a462b4e10b1f2) 2024-08-20T22:04:59.8739622Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-20T22:05:00.0849385Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-20T22:05:00.3484759Z Getting action download info 2024-08-20T22:05:00.4427747Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-20T22:05:00.6406033Z Getting action download info 2024-08-20T22:05:00.7350221Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-20T22:05:00.9040463Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/133712 (40ec5f6ddd9787aca0449b24128343ff4c4a88b3) 2024-08-20T22:05:00.9042935Z ##[group] Inputs 2024-08-20T22:05:00.9043415Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:05:00.9050478Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:05:00.9058533Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:05:00.9059722Z sync-tag: 2024-08-20T22:05:00.9060602Z timeout-minutes: 240 2024-08-20T22:05:00.9060938Z use-gha: 2024-08-20T22:05:00.9061232Z dashboard-tag: 2024-08-20T22:05:00.9061550Z s3-bucket: gha-artifacts 2024-08-20T22:05:00.9061896Z aws-role-to-assume: 2024-08-20T22:05:00.9062227Z ##[endgroup] 2024-08-20T22:05:00.9063308Z Complete job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:05:00.9729521Z A job started hook has been configured by the self-hosted runner administrator 2024-08-20T22:05:00.9872815Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-20T22:05:00.9884899Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:05:00.9885434Z ##[endgroup] 2024-08-20T22:05:02.4094576Z Runner Type: linux.g5.4xlarge.nvidia.gpu 2024-08-20T22:05:02.4095107Z Instance Type: g5.4xlarge 2024-08-20T22:05:02.4095819Z AMI Name: al2023-ami-2023.5.20240701.0-kernel-6.1-x86_64 2024-08-20T22:05:02.4096369Z AMI ID: ami-06c68f701d8090592 2024-08-20T22:05:08.1921334Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-20T22:05:08.1921902Z with: 2024-08-20T22:05:08.1922609Z github-secret: *** 2024-08-20T22:05:08.1923574Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:05:08.1924576Z activate-with-label: false 2024-08-20T22:05:08.1924939Z label: with-ssh 2024-08-20T22:05:08.1925263Z remove-existing-keys: true 2024-08-20T22:05:08.1925621Z fail-silently: true 2024-08-20T22:05:08.1925937Z env: 2024-08-20T22:05:08.1926210Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:05:08.1926648Z ##[endgroup] 2024-08-20T22:05:08.2817057Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-20T22:05:08.2819830Z ciflow reference detected, attempting to extract PR number 2024-08-20T22:05:08.5199953Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-08-20T22:05:08.5912113Z No SSH keys found for user pytorch-bot[bot] 2024-08-20T22:05:08.5912924Z Grabbing public ssh keys from https://github.com/XuehaiPan.keys 2024-08-20T22:05:08.6567271Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-20T22:05:08.6583073Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-20T22:05:08.6608465Z Login using: ssh ec2-user@ec2-3-237-94-168.compute-1.amazonaws.com 2024-08-20T22:05:08.6609241Z All testing is done inside the container, to start an interactive session run: 2024-08-20T22:05:08.6610053Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:05:08.6743210Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-20T22:05:08.6743780Z with: 2024-08-20T22:05:08.6744071Z submodules: recursive 2024-08-20T22:05:08.6744408Z fetch-depth: 0 2024-08-20T22:05:08.6744707Z env: 2024-08-20T22:05:08.6744983Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:05:08.6745328Z ##[endgroup] 2024-08-20T22:05:08.6941235Z ##[group]Run retry () { 2024-08-20T22:05:08.6941618Z retry () { 2024-08-20T22:05:08.6942129Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-20T22:05:08.6942684Z } 2024-08-20T22:05:08.6942991Z echo "${GITHUB_WORKSPACE}" 2024-08-20T22:05:08.6943411Z if [ -z "${NO_SUDO}" ]; then 2024-08-20T22:05:08.6943886Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:05:08.6944338Z else 2024-08-20T22:05:08.6944677Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:05:08.6945097Z fi 2024-08-20T22:05:08.6945650Z mkdir "${GITHUB_WORKSPACE}" 2024-08-20T22:05:08.6957714Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:05:08.6958225Z env: 2024-08-20T22:05:08.6958513Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:05:08.6958860Z NO_SUDO: 2024-08-20T22:05:08.6959139Z ##[endgroup] 2024-08-20T22:05:08.6992503Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:05:08.8255903Z ##[group]Run malfet/checkout@silent-checkout 2024-08-20T22:05:08.8256334Z with: 2024-08-20T22:05:08.8256786Z ref: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:05:08.8257239Z fetch-depth: 0 2024-08-20T22:05:08.8257554Z submodules: recursive 2024-08-20T22:05:08.8257893Z quiet-checkout: true 2024-08-20T22:05:08.8258244Z repository: pytorch/pytorch 2024-08-20T22:05:08.8258735Z token: *** 2024-08-20T22:05:08.8259029Z ssh-strict: true 2024-08-20T22:05:08.8259351Z persist-credentials: true 2024-08-20T22:05:08.8259710Z clean: true 2024-08-20T22:05:08.8260062Z sparse-checkout-cone-mode: true 2024-08-20T22:05:08.8260451Z lfs: false 2024-08-20T22:05:08.8260752Z set-safe-directory: true 2024-08-20T22:05:08.8261092Z env: 2024-08-20T22:05:08.8261369Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:05:08.8261713Z ##[endgroup] 2024-08-20T22:05:08.9240484Z Syncing repository: pytorch/pytorch 2024-08-20T22:05:08.9242060Z ##[group]Getting Git version info 2024-08-20T22:05:08.9242872Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:05:08.9243722Z [command]/usr/bin/git version 2024-08-20T22:05:08.9244084Z git version 2.40.1 2024-08-20T22:05:08.9248571Z ##[endgroup] 2024-08-20T22:05:08.9263702Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/938704c2-7822-45ae-9e7b-b3610937c366' before making global git config changes 2024-08-20T22:05:08.9264937Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T22:05:08.9268878Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:05:08.9321114Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:05:08.9325280Z ##[group]Initializing the repository 2024-08-20T22:05:08.9328799Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:05:08.9375691Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-20T22:05:08.9376828Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-20T22:05:08.9377679Z hint: of your new repositories, which will suppress this warning, call: 2024-08-20T22:05:08.9378238Z hint: 2024-08-20T22:05:08.9378703Z hint: git config --global init.defaultBranch 2024-08-20T22:05:08.9379177Z hint: 2024-08-20T22:05:08.9379706Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-20T22:05:08.9380569Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-20T22:05:08.9381470Z hint: 2024-08-20T22:05:08.9381904Z hint: git branch -m 2024-08-20T22:05:08.9382740Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-20T22:05:08.9389641Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-20T22:05:08.9433736Z ##[endgroup] 2024-08-20T22:05:08.9434393Z ##[group]Disabling automatic garbage collection 2024-08-20T22:05:08.9436798Z [command]/usr/bin/git config --local gc.auto 0 2024-08-20T22:05:08.9480055Z ##[endgroup] 2024-08-20T22:05:08.9480701Z ##[group]Setting up auth 2024-08-20T22:05:08.9486322Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T22:05:08.9530110Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T22:05:08.9924924Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T22:05:08.9967743Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T22:05:09.0324711Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:05:09.0386474Z ##[endgroup] 2024-08-20T22:05:09.0387175Z ##[group]Fetching the repository 2024-08-20T22:05:09.0393160Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-20T22:05:11.8928124Z remote: Enumerating objects: 1008452 2024-08-20T22:05:11.8928818Z remote: Enumerating objects: 1010208, done. 2024-08-20T22:05:11.8929759Z remote: Counting objects: 0% (1/1756) 2024-08-20T22:05:11.8930411Z remote: Counting objects: 1% (18/1756) 2024-08-20T22:05:11.8931088Z remote: Counting objects: 2% (36/1756) 2024-08-20T22:05:11.8931646Z remote: Counting objects: 3% (53/1756) 2024-08-20T22:05:11.8932316Z remote: Counting objects: 4% (71/1756) 2024-08-20T22:05:11.8932956Z remote: Counting objects: 5% (88/1756) 2024-08-20T22:05:11.8933608Z remote: Counting objects: 6% (106/1756) 2024-08-20T22:05:11.8934357Z remote: Counting objects: 7% (123/1756) 2024-08-20T22:05:11.8935052Z remote: Counting objects: 8% (141/1756) 2024-08-20T22:05:11.8935698Z remote: Counting objects: 9% (159/1756) 2024-08-20T22:05:11.8936339Z remote: Counting objects: 10% (176/1756) 2024-08-20T22:05:11.8936989Z remote: Counting objects: 11% (194/1756) 2024-08-20T22:05:11.8937567Z remote: Counting objects: 12% (211/1756) 2024-08-20T22:05:11.8938176Z remote: Counting objects: 13% (229/1756) 2024-08-20T22:05:11.8938830Z remote: Counting objects: 14% (246/1756) 2024-08-20T22:05:11.8939515Z remote: Counting objects: 15% (264/1756) 2024-08-20T22:05:11.8940173Z remote: Counting objects: 16% (281/1756) 2024-08-20T22:05:11.8940808Z remote: Counting objects: 17% (299/1756) 2024-08-20T22:05:11.8941465Z remote: Counting objects: 18% (317/1756) 2024-08-20T22:05:11.8942116Z remote: Counting objects: 19% (334/1756) 2024-08-20T22:05:11.8942760Z remote: Counting objects: 20% (352/1756) 2024-08-20T22:05:11.8943393Z remote: Counting objects: 21% (369/1756) 2024-08-20T22:05:11.8943950Z remote: Counting objects: 22% (387/1756) 2024-08-20T22:05:11.8944571Z remote: Counting objects: 23% (404/1756) 2024-08-20T22:05:11.8945209Z remote: Counting objects: 24% (422/1756) 2024-08-20T22:05:11.8945831Z remote: Counting objects: 25% (439/1756) 2024-08-20T22:05:11.8946474Z remote: Counting objects: 26% (457/1756) 2024-08-20T22:05:11.8947110Z remote: Counting objects: 27% (475/1756) 2024-08-20T22:05:11.8948212Z remote: Counting objects: 28% (492/1756) 2024-08-20T22:05:11.8948883Z remote: Counting objects: 29% (510/1756) 2024-08-20T22:05:11.8949536Z remote: Counting objects: 30% (527/1756) 2024-08-20T22:05:11.8950183Z remote: Counting objects: 31% (545/1756) 2024-08-20T22:05:11.8950760Z remote: Counting objects: 32% (562/1756) 2024-08-20T22:05:11.8951247Z remote: Counting objects: 33% (580/1756) 2024-08-20T22:05:11.8951745Z remote: Counting objects: 34% (598/1756) 2024-08-20T22:05:11.8952239Z remote: Counting objects: 35% (615/1756) 2024-08-20T22:05:11.8952732Z remote: Counting objects: 36% (633/1756) 2024-08-20T22:05:11.8953218Z remote: Counting objects: 37% (650/1756) 2024-08-20T22:05:11.8953722Z remote: Counting objects: 38% (668/1756) 2024-08-20T22:05:11.8954214Z remote: Counting objects: 39% (685/1756) 2024-08-20T22:05:11.8954714Z remote: Counting objects: 40% (703/1756) 2024-08-20T22:05:11.8955397Z remote: Counting objects: 41% (720/1756) 2024-08-20T22:05:11.8955891Z remote: Counting objects: 42% (738/1756) 2024-08-20T22:05:11.8956410Z remote: Counting objects: 43% (756/1756) 2024-08-20T22:05:11.8956909Z remote: Counting objects: 44% (773/1756) 2024-08-20T22:05:11.8957394Z remote: Counting objects: 45% (791/1756) 2024-08-20T22:05:11.8957890Z remote: Counting objects: 46% (808/1756) 2024-08-20T22:05:11.8958379Z remote: Counting objects: 47% (826/1756) 2024-08-20T22:05:11.8958875Z remote: Counting objects: 48% (843/1756) 2024-08-20T22:05:11.8959359Z remote: Counting objects: 49% (861/1756) 2024-08-20T22:05:11.8959861Z remote: Counting objects: 50% (878/1756) 2024-08-20T22:05:11.8960355Z remote: Counting objects: 51% (896/1756) 2024-08-20T22:05:11.8960859Z remote: Counting objects: 52% (914/1756) 2024-08-20T22:05:11.8961356Z remote: Counting objects: 53% (931/1756) 2024-08-20T22:05:11.8961864Z remote: Counting objects: 54% (949/1756) 2024-08-20T22:05:11.8962365Z remote: Counting objects: 55% (966/1756) 2024-08-20T22:05:11.8962863Z remote: Counting objects: 56% (984/1756) 2024-08-20T22:05:11.8963358Z remote: Counting objects: 57% (1001/1756) 2024-08-20T22:05:11.8963883Z remote: Counting objects: 58% (1019/1756) 2024-08-20T22:05:11.8964407Z remote: Counting objects: 59% (1037/1756) 2024-08-20T22:05:11.8964931Z remote: Counting objects: 60% (1054/1756) 2024-08-20T22:05:11.8965448Z remote: Counting objects: 61% (1072/1756) 2024-08-20T22:05:11.8965967Z remote: Counting objects: 62% (1089/1756) 2024-08-20T22:05:11.8966496Z remote: Counting objects: 63% (1107/1756) 2024-08-20T22:05:11.8967016Z remote: Counting objects: 64% (1124/1756) 2024-08-20T22:05:11.8967531Z remote: Counting objects: 65% (1142/1756) 2024-08-20T22:05:11.8968041Z remote: Counting objects: 66% (1159/1756) 2024-08-20T22:05:11.8968555Z remote: Counting objects: 67% (1177/1756) 2024-08-20T22:05:11.8969069Z remote: Counting objects: 68% (1195/1756) 2024-08-20T22:05:11.8969585Z remote: Counting objects: 69% (1212/1756) 2024-08-20T22:05:11.8970097Z remote: Counting objects: 70% (1230/1756) 2024-08-20T22:05:11.8970613Z remote: Counting objects: 71% (1247/1756) 2024-08-20T22:05:11.8971126Z remote: Counting objects: 72% (1265/1756) 2024-08-20T22:05:11.8971640Z remote: Counting objects: 73% (1282/1756) 2024-08-20T22:05:11.8972142Z remote: Counting objects: 74% (1300/1756) 2024-08-20T22:05:11.8972656Z remote: Counting objects: 75% (1317/1756) 2024-08-20T22:05:11.8973171Z remote: Counting objects: 76% (1335/1756) 2024-08-20T22:05:11.8973804Z remote: Counting objects: 77% (1353/1756) 2024-08-20T22:05:11.8974310Z remote: Counting objects: 78% (1370/1756) 2024-08-20T22:05:11.8974915Z remote: Counting objects: 79% (1388/1756) 2024-08-20T22:05:11.8975433Z remote: Counting objects: 80% (1405/1756) 2024-08-20T22:05:11.8975939Z remote: Counting objects: 81% (1423/1756) 2024-08-20T22:05:11.8976448Z remote: Counting objects: 82% (1440/1756) 2024-08-20T22:05:11.8976955Z remote: Counting objects: 83% (1458/1756) 2024-08-20T22:05:11.8977471Z remote: Counting objects: 84% (1476/1756) 2024-08-20T22:05:11.8977978Z remote: Counting objects: 85% (1493/1756) 2024-08-20T22:05:11.8978481Z remote: Counting objects: 86% (1511/1756) 2024-08-20T22:05:11.8978987Z remote: Counting objects: 87% (1528/1756) 2024-08-20T22:05:11.8979500Z remote: Counting objects: 88% (1546/1756) 2024-08-20T22:05:11.8980009Z remote: Counting objects: 89% (1563/1756) 2024-08-20T22:05:11.8980514Z remote: Counting objects: 90% (1581/1756) 2024-08-20T22:05:11.8981034Z remote: Counting objects: 91% (1598/1756) 2024-08-20T22:05:11.8981655Z remote: Counting objects: 92% (1616/1756) 2024-08-20T22:05:11.8982176Z remote: Counting objects: 93% (1634/1756) 2024-08-20T22:05:11.8982682Z remote: Counting objects: 94% (1651/1756) 2024-08-20T22:05:11.8983189Z remote: Counting objects: 95% (1669/1756) 2024-08-20T22:05:11.8983708Z remote: Counting objects: 96% (1686/1756) 2024-08-20T22:05:11.8984223Z remote: Counting objects: 97% (1704/1756) 2024-08-20T22:05:11.8984732Z remote: Counting objects: 98% (1721/1756) 2024-08-20T22:05:11.8985246Z remote: Counting objects: 99% (1739/1756) 2024-08-20T22:05:11.8985762Z remote: Counting objects: 100% (1756/1756) 2024-08-20T22:05:11.8986312Z remote: Counting objects: 100% (1756/1756), done. 2024-08-20T22:05:11.9108956Z remote: Compressing objects: 0% (1/837) 2024-08-20T22:05:11.9372939Z remote: Compressing objects: 1% (9/837) 2024-08-20T22:05:11.9718872Z remote: Compressing objects: 2% (17/837) 2024-08-20T22:05:12.0117252Z remote: Compressing objects: 3% (26/837) 2024-08-20T22:05:12.0827942Z remote: Compressing objects: 4% (34/837) 2024-08-20T22:05:12.1253596Z remote: Compressing objects: 5% (42/837) 2024-08-20T22:05:12.1818424Z remote: Compressing objects: 6% (51/837) 2024-08-20T22:05:12.2124163Z remote: Compressing objects: 7% (59/837) 2024-08-20T22:05:12.2487072Z remote: Compressing objects: 8% (67/837) 2024-08-20T22:05:12.2690896Z remote: Compressing objects: 9% (76/837) 2024-08-20T22:05:12.2942512Z remote: Compressing objects: 10% (84/837) 2024-08-20T22:05:12.3146151Z remote: Compressing objects: 11% (93/837) 2024-08-20T22:05:12.3316101Z remote: Compressing objects: 12% (101/837) 2024-08-20T22:05:12.3471863Z remote: Compressing objects: 13% (109/837) 2024-08-20T22:05:12.3585850Z remote: Compressing objects: 14% (118/837) 2024-08-20T22:05:12.3668971Z remote: Compressing objects: 15% (126/837) 2024-08-20T22:05:12.3755473Z remote: Compressing objects: 16% (134/837) 2024-08-20T22:05:12.3810705Z remote: Compressing objects: 17% (143/837) 2024-08-20T22:05:12.3861412Z remote: Compressing objects: 18% (151/837) 2024-08-20T22:05:12.3899835Z remote: Compressing objects: 19% (160/837) 2024-08-20T22:05:12.3931205Z remote: Compressing objects: 20% (168/837) 2024-08-20T22:05:12.3952706Z remote: Compressing objects: 21% (176/837) 2024-08-20T22:05:12.3956840Z remote: Compressing objects: 22% (185/837) 2024-08-20T22:05:12.3962732Z remote: Compressing objects: 23% (193/837) 2024-08-20T22:05:12.3968685Z remote: Compressing objects: 24% (201/837) 2024-08-20T22:05:12.3973294Z remote: Compressing objects: 25% (210/837) 2024-08-20T22:05:12.3981495Z remote: Compressing objects: 26% (218/837) 2024-08-20T22:05:12.4011855Z remote: Compressing objects: 27% (226/837) 2024-08-20T22:05:12.4024789Z remote: Compressing objects: 28% (235/837) 2024-08-20T22:05:12.4028647Z remote: Compressing objects: 29% (243/837) 2024-08-20T22:05:12.4035918Z remote: Compressing objects: 30% (252/837) 2024-08-20T22:05:12.4044718Z remote: Compressing objects: 31% (260/837) 2024-08-20T22:05:12.4057345Z remote: Compressing objects: 32% (268/837) 2024-08-20T22:05:12.4058433Z remote: Compressing objects: 33% (277/837) 2024-08-20T22:05:12.4063747Z remote: Compressing objects: 34% (285/837) 2024-08-20T22:05:12.4071968Z remote: Compressing objects: 35% (293/837) 2024-08-20T22:05:12.4078842Z remote: Compressing objects: 36% (302/837) 2024-08-20T22:05:12.4089198Z remote: Compressing objects: 37% (310/837) 2024-08-20T22:05:12.4095567Z remote: Compressing objects: 38% (319/837) 2024-08-20T22:05:12.4103165Z remote: Compressing objects: 39% (327/837) 2024-08-20T22:05:12.4112436Z remote: Compressing objects: 40% (335/837) 2024-08-20T22:05:12.4117876Z remote: Compressing objects: 41% (344/837) 2024-08-20T22:05:12.4127098Z remote: Compressing objects: 42% (352/837) 2024-08-20T22:05:12.4131051Z remote: Compressing objects: 43% (360/837) 2024-08-20T22:05:12.4137449Z remote: Compressing objects: 44% (369/837) 2024-08-20T22:05:12.4142847Z remote: Compressing objects: 45% (377/837) 2024-08-20T22:05:12.4145979Z remote: Compressing objects: 46% (386/837) 2024-08-20T22:05:12.4149852Z remote: Compressing objects: 47% (394/837) 2024-08-20T22:05:12.4153641Z remote: Compressing objects: 48% (402/837) 2024-08-20T22:05:12.4156500Z remote: Compressing objects: 49% (411/837) 2024-08-20T22:05:12.4160719Z remote: Compressing objects: 50% (419/837) 2024-08-20T22:05:12.4163647Z remote: Compressing objects: 51% (427/837) 2024-08-20T22:05:12.4164993Z remote: Compressing objects: 52% (436/837) 2024-08-20T22:05:12.4167703Z remote: Compressing objects: 53% (444/837) 2024-08-20T22:05:12.4170492Z remote: Compressing objects: 54% (452/837) 2024-08-20T22:05:12.4173929Z remote: Compressing objects: 55% (461/837) 2024-08-20T22:05:12.4174635Z remote: Compressing objects: 56% (469/837) 2024-08-20T22:05:12.4175311Z remote: Compressing objects: 57% (478/837) 2024-08-20T22:05:12.4176022Z remote: Compressing objects: 58% (486/837) 2024-08-20T22:05:12.4176974Z remote: Compressing objects: 59% (494/837) 2024-08-20T22:05:12.4177674Z remote: Compressing objects: 60% (503/837) 2024-08-20T22:05:12.4181245Z remote: Compressing objects: 61% (511/837) 2024-08-20T22:05:12.4181934Z remote: Compressing objects: 62% (519/837) 2024-08-20T22:05:12.4182507Z remote: Compressing objects: 63% (528/837) 2024-08-20T22:05:12.4185904Z remote: Compressing objects: 64% (536/837) 2024-08-20T22:05:12.4192476Z remote: Compressing objects: 65% (545/837) 2024-08-20T22:05:12.4199615Z remote: Compressing objects: 66% (553/837) 2024-08-20T22:05:12.4203278Z remote: Compressing objects: 67% (561/837) 2024-08-20T22:05:12.4208452Z remote: Compressing objects: 68% (570/837) 2024-08-20T22:05:12.4213083Z remote: Compressing objects: 69% (578/837) 2024-08-20T22:05:12.4217798Z remote: Compressing objects: 70% (586/837) 2024-08-20T22:05:12.4221319Z remote: Compressing objects: 71% (595/837) 2024-08-20T22:05:12.4225349Z remote: Compressing objects: 72% (603/837) 2024-08-20T22:05:12.4229108Z remote: Compressing objects: 73% (612/837) 2024-08-20T22:05:12.4233251Z remote: Compressing objects: 74% (620/837) 2024-08-20T22:05:12.4237169Z remote: Compressing objects: 75% (628/837) 2024-08-20T22:05:12.4241329Z remote: Compressing objects: 76% (637/837) 2024-08-20T22:05:12.4245643Z remote: Compressing objects: 77% (645/837) 2024-08-20T22:05:12.4248517Z remote: Compressing objects: 78% (653/837) 2024-08-20T22:05:12.4251535Z remote: Compressing objects: 79% (662/837) 2024-08-20T22:05:12.4254412Z remote: Compressing objects: 80% (670/837) 2024-08-20T22:05:12.4255546Z remote: Compressing objects: 81% (678/837) 2024-08-20T22:05:12.4259852Z remote: Compressing objects: 82% (687/837) 2024-08-20T22:05:12.4262364Z remote: Compressing objects: 83% (695/837) 2024-08-20T22:05:12.4265387Z remote: Compressing objects: 84% (704/837) 2024-08-20T22:05:12.4267857Z remote: Compressing objects: 85% (712/837) 2024-08-20T22:05:12.4270485Z remote: Compressing objects: 86% (720/837) 2024-08-20T22:05:12.4271969Z remote: Compressing objects: 87% (729/837) 2024-08-20T22:05:12.4275296Z remote: Compressing objects: 88% (737/837) 2024-08-20T22:05:12.4278868Z remote: Compressing objects: 89% (745/837) 2024-08-20T22:05:12.4281100Z remote: Compressing objects: 90% (754/837) 2024-08-20T22:05:12.4283181Z remote: Compressing objects: 91% (762/837) 2024-08-20T22:05:12.4283873Z remote: Compressing objects: 92% (771/837) 2024-08-20T22:05:12.4284396Z remote: Compressing objects: 93% (779/837) 2024-08-20T22:05:12.4287858Z remote: Compressing objects: 94% (787/837) 2024-08-20T22:05:12.4290154Z remote: Compressing objects: 95% (796/837) 2024-08-20T22:05:12.4292130Z remote: Compressing objects: 96% (804/837) 2024-08-20T22:05:12.4294364Z remote: Compressing objects: 97% (812/837) 2024-08-20T22:05:12.4296433Z remote: Compressing objects: 98% (821/837) 2024-08-20T22:05:12.4298316Z remote: Compressing objects: 99% (829/837) 2024-08-20T22:05:12.4298878Z remote: Compressing objects: 100% (837/837) 2024-08-20T22:05:12.4299578Z remote: Compressing objects: 100% (837/837), done. 2024-08-20T22:05:35.6700064Z remote: Total 1010208 (delta 1154), reused 1394 (delta 916), pack-reused 1008452 (from 1) 2024-08-20T22:05:59.7817565Z [command]/usr/bin/git rev-parse --verify --quiet 40ec5f6ddd9787aca0449b24128343ff4c4a88b3^{object} 2024-08-20T22:05:59.7855133Z 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:05:59.7861190Z ##[endgroup] 2024-08-20T22:05:59.7863428Z ##[group]Determining the checkout info 2024-08-20T22:05:59.7864215Z ##[endgroup] 2024-08-20T22:05:59.7867865Z ##[group]Checking out the ref 2024-08-20T22:05:59.7868747Z [command]/usr/bin/git checkout --quiet --force 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:06:01.5186669Z ##[endgroup] 2024-08-20T22:06:01.5187278Z ##[group]Setting up auth for fetching submodules 2024-08-20T22:06:01.5191372Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:06:01.5253727Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-20T22:06:01.5293710Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-20T22:06:01.5334304Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-20T22:06:01.5372042Z ##[endgroup] 2024-08-20T22:06:01.5372574Z ##[group]Fetching submodules 2024-08-20T22:06:01.5376208Z [command]/usr/bin/git submodule sync --recursive 2024-08-20T22:06:01.5754736Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-20T22:06:01.6117048Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-20T22:06:01.6118722Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-20T22:06:01.6121863Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-20T22:06:01.6125301Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-20T22:06:01.6129865Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-20T22:06:01.6132755Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-20T22:06:01.6136401Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-20T22:06:01.6140223Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-20T22:06:01.6144097Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-20T22:06:01.6148254Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-20T22:06:01.6152237Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-20T22:06:01.6156398Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-20T22:06:01.6160611Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-20T22:06:01.6165063Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-20T22:06:01.6169297Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-20T22:06:01.6175647Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:06:01.6183683Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-20T22:06:01.6188278Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-20T22:06:01.6192846Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-20T22:06:01.6197875Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-20T22:06:01.6202704Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-20T22:06:01.6207657Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-20T22:06:01.6212575Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-20T22:06:01.6217820Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-20T22:06:01.6222784Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-20T22:06:01.6228633Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-20T22:06:01.6233777Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-20T22:06:01.6239310Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-20T22:06:01.6244624Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-20T22:06:01.6250239Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-20T22:06:01.6255852Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-20T22:06:01.6261619Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-20T22:06:01.6267040Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-20T22:06:01.6276342Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-20T22:06:01.6310939Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-20T22:06:01.9507199Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-20T22:06:02.1237653Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-20T22:06:02.3064230Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-20T22:06:02.5531351Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-20T22:06:04.4785614Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-20T22:06:15.7138364Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-20T22:06:16.1371445Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-20T22:06:16.6993640Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-20T22:06:17.2783934Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-20T22:06:18.5809250Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-20T22:06:20.7026281Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-20T22:06:26.8656814Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-20T22:06:28.2419293Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-20T22:06:30.0251478Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-20T22:06:31.3438138Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-20T22:06:31.8343418Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-20T22:06:32.1592984Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-20T22:06:33.3034093Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-20T22:06:33.6905651Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-20T22:06:33.9283095Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-20T22:06:35.5560636Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-20T22:06:36.2910836Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-20T22:06:37.0435593Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-20T22:06:43.0980100Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-20T22:06:45.1464070Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-20T22:06:49.4150143Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-20T22:06:49.6617827Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-20T22:06:59.1731479Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-20T22:06:59.3570117Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-20T22:06:59.5889777Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-20T22:07:00.5122096Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-20T22:07:00.8123657Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-20T22:07:01.4200480Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-20T22:07:01.8905947Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-20T22:07:01.9053290Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-20T22:07:01.9174641Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-20T22:07:01.9489802Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-20T22:07:01.9934072Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-20T22:07:03.1632512Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-20T22:07:03.1920597Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-20T22:07:03.2447278Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-20T22:07:03.3577932Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-20T22:07:03.3980895Z Submodule path 'third_party/cudnn_frontend': checked out '23511ba176243f27b3b275da1fb3814ea805a171' 2024-08-20T22:07:04.0225370Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-20T22:07:04.3098409Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-20T22:07:04.4046436Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-20T22:07:04.4066500Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:07:04.4070259Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:07:04.4074100Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:07:04.4078188Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:07:04.4082319Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:07:04.4115339Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-20T22:07:05.4939915Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-20T22:07:06.1294244Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-20T22:07:08.2778168Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-20T22:07:09.3305849Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-20T22:07:09.6900508Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-20T22:07:09.8009872Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-20T22:07:10.3079373Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-20T22:07:10.3784474Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-20T22:07:10.3933587Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-20T22:07:10.5450298Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-20T22:07:10.5907234Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-20T22:07:10.6365001Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-20T22:07:10.6689789Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-20T22:07:10.7219744Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:07:10.7379178Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-20T22:07:10.7397949Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-20T22:07:10.7432659Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-20T22:07:23.7645288Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-20T22:07:23.7859690Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-20T22:07:23.8852872Z Submodule path 'third_party/kineto': checked out 'd9753139d181b9ff42872465aac0e5d3018be415' 2024-08-20T22:07:23.8877444Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:07:23.8880626Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:07:23.8884252Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:07:23.8916526Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-20T22:07:24.4416291Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-20T22:07:25.6082819Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-20T22:07:26.7768979Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-20T22:07:26.7790563Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:07:26.7793855Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:07:26.7797365Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:07:26.7801122Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:07:26.7804841Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:07:26.7808660Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:07:26.7812571Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:07:26.7816512Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:07:26.7849097Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-20T22:07:27.6884931Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-20T22:07:28.0644331Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-20T22:07:29.2455032Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-20T22:07:29.5466761Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-20T22:07:30.1005039Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-20T22:07:31.1524198Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-20T22:07:38.4303714Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-20T22:07:38.8252289Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-20T22:07:38.8487299Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-20T22:07:38.8910232Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-20T22:07:38.9082547Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-20T22:07:38.9102093Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:07:38.9133738Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-20T22:07:39.2167205Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-20T22:07:39.2386968Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-20T22:07:39.2863004Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-20T22:07:39.4078989Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-20T22:07:39.4278338Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-20T22:07:39.4724349Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-20T22:07:39.5377685Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-20T22:07:39.5818912Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-20T22:07:39.6110684Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-20T22:07:39.7336533Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-20T22:07:40.2242849Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-20T22:07:40.2281437Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-20T22:07:40.2284393Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-20T22:07:40.2316286Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-20T22:07:40.7461166Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-20T22:07:41.7361928Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-20T22:07:41.7766858Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-20T22:07:41.8637513Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-20T22:07:41.8662063Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:07:41.8664845Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:07:41.8668117Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:07:41.8671651Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:07:41.8675367Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:07:41.8678854Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:07:41.8682552Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:07:41.8686039Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:07:41.8718060Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-20T22:07:42.2943232Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-20T22:07:43.4048305Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-20T22:07:43.7463200Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-20T22:07:49.9040451Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-20T22:07:50.1703891Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-20T22:07:50.3816824Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-20T22:07:50.6806357Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-20T22:07:57.6444317Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-20T22:07:57.6911568Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-20T22:07:57.7099654Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-20T22:07:57.8333822Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-20T22:07:57.8492678Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-20T22:07:57.8675882Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-20T22:07:57.8877340Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-20T22:07:57.8896714Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:07:57.8899683Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:07:57.8930905Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-20T22:07:59.8200771Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-20T22:08:01.1581924Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-20T22:08:01.2105818Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:08:01.8538104Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-20T22:08:01.8680690Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-20T22:08:02.1755218Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-20T22:08:02.1781278Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:02.1784387Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:02.1815661Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-20T22:08:02.6867310Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-20T22:08:03.7822731Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-20T22:08:03.8634057Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-20T22:08:03.8751960Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-20T22:08:03.8907529Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-20T22:08:03.9343870Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-20T22:08:03.9685551Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-20T22:08:04.0174421Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-20T22:08:04.0517716Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-20T22:08:04.0540031Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:04.0543134Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:04.0546747Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:04.0550031Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:04.0581434Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-20T22:08:05.1684783Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-20T22:08:05.4189087Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-20T22:08:06.6516968Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-20T22:08:07.6268742Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-20T22:08:07.6463627Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-20T22:08:07.7250757Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-20T22:08:07.7593864Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-20T22:08:07.7611499Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:07.7648398Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-20T22:08:07.9989211Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-20T22:08:08.0038187Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-20T22:08:08.0400469Z Entering 'android/libs/fbjni' 2024-08-20T22:08:08.0454206Z Entering 'third_party/FP16' 2024-08-20T22:08:08.0506057Z Entering 'third_party/FXdiv' 2024-08-20T22:08:08.0557186Z Entering 'third_party/NNPACK' 2024-08-20T22:08:08.0609221Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:08.0660772Z Entering 'third_party/XNNPACK' 2024-08-20T22:08:08.0729006Z Entering 'third_party/benchmark' 2024-08-20T22:08:08.0784572Z Entering 'third_party/cpp-httplib' 2024-08-20T22:08:08.0837095Z Entering 'third_party/cpuinfo' 2024-08-20T22:08:08.0890167Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:08:08.0943309Z Entering 'third_party/cutlass' 2024-08-20T22:08:08.1003884Z Entering 'third_party/eigen' 2024-08-20T22:08:08.1059003Z Entering 'third_party/fbgemm' 2024-08-20T22:08:08.1110435Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:08:08.1162554Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:08:08.1213945Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:08:08.1276421Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:08:08.1330905Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:08:08.1383212Z Entering 'third_party/flatbuffers' 2024-08-20T22:08:08.1438548Z Entering 'third_party/fmt' 2024-08-20T22:08:08.1489794Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:08.1541520Z Entering 'third_party/gloo' 2024-08-20T22:08:08.1592765Z Entering 'third_party/googletest' 2024-08-20T22:08:08.1643665Z Entering 'third_party/ideep' 2024-08-20T22:08:08.1693437Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:08:08.1752686Z Entering 'third_party/ittapi' 2024-08-20T22:08:08.1803317Z Entering 'third_party/kineto' 2024-08-20T22:08:08.1854080Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:08:08.1907968Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:08:08.1960455Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:08:08.2010236Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:08:08.2060260Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:08:08.2108158Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:08.2163072Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:08:08.2213267Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:08:08.2263585Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:08:08.2315327Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:08:08.2368179Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:08:08.2417814Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:08:08.2474777Z Entering 'third_party/mimalloc' 2024-08-20T22:08:08.2527381Z Entering 'third_party/nccl/nccl' 2024-08-20T22:08:08.2582833Z Entering 'third_party/nlohmann' 2024-08-20T22:08:08.2636846Z Entering 'third_party/onnx' 2024-08-20T22:08:08.2703198Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:08.2755962Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:08.2814913Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:08:08.2867704Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:08.2916929Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:08.2969313Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:08.3018119Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:08.3070136Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:08.3119373Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:08.3168610Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:08.3216945Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:08.3269994Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:08.3323098Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:08.3399957Z Entering 'third_party/pocketfft' 2024-08-20T22:08:08.3453028Z Entering 'third_party/protobuf' 2024-08-20T22:08:08.3507493Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:08.3558374Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:08.3614449Z Entering 'third_party/psimd' 2024-08-20T22:08:08.3667642Z Entering 'third_party/pthreadpool' 2024-08-20T22:08:08.3720144Z Entering 'third_party/pybind11' 2024-08-20T22:08:08.3772691Z Entering 'third_party/python-peachpy' 2024-08-20T22:08:08.3824631Z Entering 'third_party/sleef' 2024-08-20T22:08:08.3878298Z Entering 'third_party/tensorpipe' 2024-08-20T22:08:08.3930292Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:08.3980280Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:08.4030230Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:08.4080679Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:08.4130020Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:08.4204567Z ##[endgroup] 2024-08-20T22:08:08.4209979Z ##[group]Persisting credentials for submodules 2024-08-20T22:08:08.4212170Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-20T22:08:08.4574233Z Entering 'android/libs/fbjni' 2024-08-20T22:08:08.4645984Z Entering 'third_party/FP16' 2024-08-20T22:08:08.4714181Z Entering 'third_party/FXdiv' 2024-08-20T22:08:08.4788492Z Entering 'third_party/NNPACK' 2024-08-20T22:08:08.4858246Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:08.4924648Z Entering 'third_party/XNNPACK' 2024-08-20T22:08:08.5011596Z Entering 'third_party/benchmark' 2024-08-20T22:08:08.5080005Z Entering 'third_party/cpp-httplib' 2024-08-20T22:08:08.5150934Z Entering 'third_party/cpuinfo' 2024-08-20T22:08:08.5219557Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:08:08.5287977Z Entering 'third_party/cutlass' 2024-08-20T22:08:08.5363762Z Entering 'third_party/eigen' 2024-08-20T22:08:08.5433033Z Entering 'third_party/fbgemm' 2024-08-20T22:08:08.5499538Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:08:08.5566206Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:08:08.5633225Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:08:08.5705683Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:08:08.5775418Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:08:08.5844045Z Entering 'third_party/flatbuffers' 2024-08-20T22:08:08.5913566Z Entering 'third_party/fmt' 2024-08-20T22:08:08.5981915Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:08.6050427Z Entering 'third_party/gloo' 2024-08-20T22:08:08.6118056Z Entering 'third_party/googletest' 2024-08-20T22:08:08.6187571Z Entering 'third_party/ideep' 2024-08-20T22:08:08.6253406Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:08:08.6329488Z Entering 'third_party/ittapi' 2024-08-20T22:08:08.6395977Z Entering 'third_party/kineto' 2024-08-20T22:08:08.6466401Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:08:08.6534164Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:08:08.6602052Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:08:08.6668922Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:08:08.6736521Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:08:08.6800640Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:08.6873264Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:08:08.6939874Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:08:08.7006265Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:08:08.7077904Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:08:08.7148495Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:08:08.7214636Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:08:08.7283648Z Entering 'third_party/mimalloc' 2024-08-20T22:08:08.7351300Z Entering 'third_party/nccl/nccl' 2024-08-20T22:08:08.7419733Z Entering 'third_party/nlohmann' 2024-08-20T22:08:08.7489514Z Entering 'third_party/onnx' 2024-08-20T22:08:08.7572322Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:08.7639697Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:08.7711038Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:08:08.7780336Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:08.7846565Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:08.7912490Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:08.7979509Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:08.8047135Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:08.8114089Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:08.8181271Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:08.8245621Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:08.8318539Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:08.8389004Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:08.8477038Z Entering 'third_party/pocketfft' 2024-08-20T22:08:08.8544960Z Entering 'third_party/protobuf' 2024-08-20T22:08:08.8618016Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:08.8685122Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:08.8755476Z Entering 'third_party/psimd' 2024-08-20T22:08:08.8821985Z Entering 'third_party/pthreadpool' 2024-08-20T22:08:08.8888966Z Entering 'third_party/pybind11' 2024-08-20T22:08:08.8957508Z Entering 'third_party/python-peachpy' 2024-08-20T22:08:08.9025993Z Entering 'third_party/sleef' 2024-08-20T22:08:08.9092975Z Entering 'third_party/tensorpipe' 2024-08-20T22:08:08.9159592Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:08.9225795Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:08.9292455Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:08.9359114Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:08.9422660Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:08.9515248Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-20T22:08:08.9880173Z Entering 'android/libs/fbjni' 2024-08-20T22:08:08.9943439Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-20T22:08:08.9964839Z Entering 'third_party/FP16' 2024-08-20T22:08:09.0026491Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-20T22:08:09.0048350Z Entering 'third_party/FXdiv' 2024-08-20T22:08:09.0110358Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-20T22:08:09.0131452Z Entering 'third_party/NNPACK' 2024-08-20T22:08:09.0191937Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-20T22:08:09.0213387Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:09.0275362Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-20T22:08:09.0296218Z Entering 'third_party/XNNPACK' 2024-08-20T22:08:09.0357360Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-20T22:08:09.0393715Z Entering 'third_party/benchmark' 2024-08-20T22:08:09.0455740Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:08:09.0476739Z Entering 'third_party/cpp-httplib' 2024-08-20T22:08:09.0537880Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-20T22:08:09.0559011Z Entering 'third_party/cpuinfo' 2024-08-20T22:08:09.0619913Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:08:09.0641930Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:08:09.0703348Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-20T22:08:09.0724515Z Entering 'third_party/cutlass' 2024-08-20T22:08:09.0785429Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:08:09.0813321Z Entering 'third_party/eigen' 2024-08-20T22:08:09.0875499Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-20T22:08:09.0898586Z Entering 'third_party/fbgemm' 2024-08-20T22:08:09.0963859Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-20T22:08:09.0983509Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:08:09.1044493Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-20T22:08:09.1065014Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:08:09.1125748Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:08:09.1149154Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:08:09.1209387Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:08:09.1237326Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:08:09.1298197Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:09.1318076Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:08:09.1382555Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-20T22:08:09.1404854Z Entering 'third_party/flatbuffers' 2024-08-20T22:08:09.1467327Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-20T22:08:09.1490838Z Entering 'third_party/fmt' 2024-08-20T22:08:09.1551467Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:08:09.1572783Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:09.1633534Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-20T22:08:09.1654958Z Entering 'third_party/gloo' 2024-08-20T22:08:09.1715224Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-20T22:08:09.1737150Z Entering 'third_party/googletest' 2024-08-20T22:08:09.1798960Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:09.1820506Z Entering 'third_party/ideep' 2024-08-20T22:08:09.1882195Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-20T22:08:09.1901051Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:08:09.1964079Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-20T22:08:09.1996922Z Entering 'third_party/ittapi' 2024-08-20T22:08:09.2058020Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-20T22:08:09.2079068Z Entering 'third_party/kineto' 2024-08-20T22:08:09.2141107Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-20T22:08:09.2160899Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:08:09.2221703Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-20T22:08:09.2240538Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:08:09.2301117Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-20T22:08:09.2323360Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:08:09.2384704Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-20T22:08:09.2404989Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:08:09.2466789Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:08:09.2486703Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:08:09.2547993Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-20T22:08:09.2565628Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:09.2629379Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-20T22:08:09.2651787Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:08:09.2712794Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-20T22:08:09.2732966Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:08:09.2795100Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:09.2817383Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:08:09.2878822Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-20T22:08:09.2899752Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:08:09.2962889Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-20T22:08:09.2985157Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:08:09.3047141Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-20T22:08:09.3066605Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:08:09.3129051Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-20T22:08:09.3153367Z Entering 'third_party/mimalloc' 2024-08-20T22:08:09.3215247Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-20T22:08:09.3237591Z Entering 'third_party/nccl/nccl' 2024-08-20T22:08:09.3300616Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-20T22:08:09.3322236Z Entering 'third_party/nlohmann' 2024-08-20T22:08:09.3385118Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-20T22:08:09.3407819Z Entering 'third_party/onnx' 2024-08-20T22:08:09.3470126Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-20T22:08:09.3503613Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:09.3565872Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:08:09.3586957Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:09.3649355Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:08:09.3675756Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:08:09.3737200Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-20T22:08:09.3758252Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:09.3822000Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:08:09.3842217Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:09.3906323Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:09.3926875Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:09.3990769Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-20T22:08:09.4011020Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:09.4078421Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-20T22:08:09.4101301Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:09.4166401Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-20T22:08:09.4188936Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:09.4251611Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-20T22:08:09.4272655Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:09.4335539Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-20T22:08:09.4354369Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:09.4417772Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-20T22:08:09.4441872Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:09.4504726Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-20T22:08:09.4531121Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:09.4593630Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-20T22:08:09.4636405Z Entering 'third_party/pocketfft' 2024-08-20T22:08:09.4699569Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-20T22:08:09.4721359Z Entering 'third_party/protobuf' 2024-08-20T22:08:09.4783511Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-20T22:08:09.4807606Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:09.4869498Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:08:09.4890123Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:09.4959068Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:09.4984836Z Entering 'third_party/psimd' 2024-08-20T22:08:09.5051383Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-20T22:08:09.5073655Z Entering 'third_party/pthreadpool' 2024-08-20T22:08:09.5138298Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-20T22:08:09.5159270Z Entering 'third_party/pybind11' 2024-08-20T22:08:09.5221411Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:08:09.5243633Z Entering 'third_party/python-peachpy' 2024-08-20T22:08:09.5307176Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-20T22:08:09.5331352Z Entering 'third_party/sleef' 2024-08-20T22:08:09.5393368Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-20T22:08:09.5415134Z Entering 'third_party/tensorpipe' 2024-08-20T22:08:09.5478209Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-20T22:08:09.5498345Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:09.5560404Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:09.5581648Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:09.5643306Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-20T22:08:09.5663416Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:09.5724577Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-20T22:08:09.5747186Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:09.5813394Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:08:09.5831730Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:09.5894185Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-20T22:08:09.7767552Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-20T22:08:09.8131990Z Entering 'android/libs/fbjni' 2024-08-20T22:08:09.8185282Z Entering 'third_party/FP16' 2024-08-20T22:08:09.8239513Z Entering 'third_party/FXdiv' 2024-08-20T22:08:09.8292243Z Entering 'third_party/NNPACK' 2024-08-20T22:08:09.8346123Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:09.8398494Z Entering 'third_party/XNNPACK' 2024-08-20T22:08:09.8468431Z Entering 'third_party/benchmark' 2024-08-20T22:08:09.8521615Z Entering 'third_party/cpp-httplib' 2024-08-20T22:08:09.8573035Z Entering 'third_party/cpuinfo' 2024-08-20T22:08:09.8628907Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:08:09.8680836Z Entering 'third_party/cutlass' 2024-08-20T22:08:09.8741794Z Entering 'third_party/eigen' 2024-08-20T22:08:09.8795889Z Entering 'third_party/fbgemm' 2024-08-20T22:08:09.8847706Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:08:09.8898904Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:08:09.8950247Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:08:09.9009595Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:08:09.9062731Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:08:09.9116747Z Entering 'third_party/flatbuffers' 2024-08-20T22:08:09.9173102Z Entering 'third_party/fmt' 2024-08-20T22:08:09.9225399Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:09.9281447Z Entering 'third_party/gloo' 2024-08-20T22:08:09.9336116Z Entering 'third_party/googletest' 2024-08-20T22:08:09.9387909Z Entering 'third_party/ideep' 2024-08-20T22:08:09.9445004Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:08:09.9505746Z Entering 'third_party/ittapi' 2024-08-20T22:08:09.9560788Z Entering 'third_party/kineto' 2024-08-20T22:08:09.9621954Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:08:09.9672848Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:08:09.9725777Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:08:09.9779418Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:08:09.9832798Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:08:09.9883417Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:09.9939295Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:08:09.9998218Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:08:10.0051003Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:08:10.0105868Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:08:10.0162370Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:08:10.0213419Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:08:10.0267104Z Entering 'third_party/mimalloc' 2024-08-20T22:08:10.0326098Z Entering 'third_party/nccl/nccl' 2024-08-20T22:08:10.0379681Z Entering 'third_party/nlohmann' 2024-08-20T22:08:10.0434631Z Entering 'third_party/onnx' 2024-08-20T22:08:10.0500548Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:10.0554551Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:10.0615028Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:08:10.0668258Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:10.0719514Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:10.0773966Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:10.0828424Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:10.0880088Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:10.0931198Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:10.0988218Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:10.1039359Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:10.1093210Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:10.1154514Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:10.1227981Z Entering 'third_party/pocketfft' 2024-08-20T22:08:10.1280695Z Entering 'third_party/protobuf' 2024-08-20T22:08:10.1335419Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:10.1387675Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:10.1445971Z Entering 'third_party/psimd' 2024-08-20T22:08:10.1499981Z Entering 'third_party/pthreadpool' 2024-08-20T22:08:10.1554556Z Entering 'third_party/pybind11' 2024-08-20T22:08:10.1607841Z Entering 'third_party/python-peachpy' 2024-08-20T22:08:10.1662263Z Entering 'third_party/sleef' 2024-08-20T22:08:10.1718937Z Entering 'third_party/tensorpipe' 2024-08-20T22:08:10.1771062Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:10.1823941Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:10.1877434Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:10.1930601Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:10.1985817Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:10.2061398Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-20T22:08:10.2423086Z Entering 'android/libs/fbjni' 2024-08-20T22:08:10.2475724Z Entering 'third_party/FP16' 2024-08-20T22:08:10.2532457Z Entering 'third_party/FXdiv' 2024-08-20T22:08:10.2590691Z Entering 'third_party/NNPACK' 2024-08-20T22:08:10.2646652Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:10.2701987Z Entering 'third_party/XNNPACK' 2024-08-20T22:08:10.2772120Z Entering 'third_party/benchmark' 2024-08-20T22:08:10.2826879Z Entering 'third_party/cpp-httplib' 2024-08-20T22:08:10.2883606Z Entering 'third_party/cpuinfo' 2024-08-20T22:08:10.2939683Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:08:10.2993072Z Entering 'third_party/cutlass' 2024-08-20T22:08:10.3061791Z Entering 'third_party/eigen' 2024-08-20T22:08:10.3119058Z Entering 'third_party/fbgemm' 2024-08-20T22:08:10.3175120Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:08:10.3230032Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:08:10.3281080Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:08:10.3340117Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:08:10.3391518Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:08:10.3444949Z Entering 'third_party/flatbuffers' 2024-08-20T22:08:10.3502869Z Entering 'third_party/fmt' 2024-08-20T22:08:10.3555619Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:10.3613168Z Entering 'third_party/gloo' 2024-08-20T22:08:10.3667143Z Entering 'third_party/googletest' 2024-08-20T22:08:10.3722243Z Entering 'third_party/ideep' 2024-08-20T22:08:10.3774473Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:08:10.3833690Z Entering 'third_party/ittapi' 2024-08-20T22:08:10.3885471Z Entering 'third_party/kineto' 2024-08-20T22:08:10.3936783Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:08:10.3989273Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:08:10.4041404Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:08:10.4091107Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:08:10.4143851Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:08:10.4191905Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:10.4247495Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:08:10.4298148Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:08:10.4349329Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:08:10.4401831Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:08:10.4458776Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:08:10.4510689Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:08:10.4563975Z Entering 'third_party/mimalloc' 2024-08-20T22:08:10.4619156Z Entering 'third_party/nccl/nccl' 2024-08-20T22:08:10.4672349Z Entering 'third_party/nlohmann' 2024-08-20T22:08:10.4727333Z Entering 'third_party/onnx' 2024-08-20T22:08:10.4793440Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:10.4844066Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:10.4901663Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:08:10.4958318Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:10.5007976Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:10.5067014Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:10.5117029Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:10.5169254Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:10.5218986Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:10.5268329Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:10.5316409Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:10.5369097Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:10.5421223Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:10.5495006Z Entering 'third_party/pocketfft' 2024-08-20T22:08:10.5547934Z Entering 'third_party/protobuf' 2024-08-20T22:08:10.5602720Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:10.5653684Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:10.5708045Z Entering 'third_party/psimd' 2024-08-20T22:08:10.5761206Z Entering 'third_party/pthreadpool' 2024-08-20T22:08:10.5813806Z Entering 'third_party/pybind11' 2024-08-20T22:08:10.5867105Z Entering 'third_party/python-peachpy' 2024-08-20T22:08:10.5920363Z Entering 'third_party/sleef' 2024-08-20T22:08:10.5973092Z Entering 'third_party/tensorpipe' 2024-08-20T22:08:10.6024625Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:10.6074904Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:10.6124582Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:10.6178167Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:10.6228147Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:10.6300958Z ##[endgroup] 2024-08-20T22:08:10.6351127Z [command]/usr/bin/git log -1 --format='%H' 2024-08-20T22:08:10.6387513Z '40ec5f6ddd9787aca0449b24128343ff4c4a88b3' 2024-08-20T22:08:10.6589416Z Prepare all required actions 2024-08-20T22:08:10.6589901Z Getting action download info 2024-08-20T22:08:10.8183231Z ##[group]Run ./.github/actions/setup-linux 2024-08-20T22:08:10.8183653Z env: 2024-08-20T22:08:10.8183921Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:10.8184272Z ##[endgroup] 2024-08-20T22:08:10.8244012Z ##[group]Run set -euo pipefail 2024-08-20T22:08:10.8244460Z set -euo pipefail 2024-08-20T22:08:10.8244900Z function get_ec2_metadata() { 2024-08-20T22:08:10.8245433Z  # Pulled from instance metadata endpoint for EC2 2024-08-20T22:08:10.8246317Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-20T22:08:10.8247079Z  category=$1 2024-08-20T22:08:10.8247611Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-20T22:08:10.8248255Z  runner_name_str=i-0eeb15fe47bdc7534 2024-08-20T22:08:10.8248763Z  if [[ -f /.inarc ]]; then 2024-08-20T22:08:10.8249256Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-20T22:08:10.8249822Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-20T22:08:10.8250509Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-20T22:08:10.8251122Z  else 2024-08-20T22:08:10.8251608Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-20T22:08:10.8252186Z  fi 2024-08-20T22:08:10.8252478Z } 2024-08-20T22:08:10.8252835Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-20T22:08:10.8253419Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-20T22:08:10.8254144Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-20T22:08:10.8254729Z echo "system info $(uname -a)" 2024-08-20T22:08:10.8265924Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:10.8266419Z env: 2024-08-20T22:08:10.8266705Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:10.8267055Z ##[endgroup] 2024-08-20T22:08:10.8372050Z ami-id: ami-06c68f701d8090592 2024-08-20T22:08:10.8436273Z instance-id: i-0eeb15fe47bdc7534 2024-08-20T22:08:10.8494425Z instance-type: g5.4xlarge 2024-08-20T22:08:10.8508469Z system info Linux ip-10-0-28-59.ec2.internal 6.1.94-99.176.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-20T22:08:10.8531739Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:08:10.8532808Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:08:10.8541683Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:10.8542185Z env: 2024-08-20T22:08:10.8542469Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:10.8542818Z ##[endgroup] 2024-08-20T22:08:10.8619326Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-20T22:08:10.8619935Z if systemctl is-active --quiet docker; then 2024-08-20T22:08:10.8620475Z  echo "Docker daemon is running..."; 2024-08-20T22:08:10.8620921Z else 2024-08-20T22:08:10.8621403Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-20T22:08:10.8622146Z fi 2024-08-20T22:08:10.8630607Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:10.8631090Z env: 2024-08-20T22:08:10.8631362Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:10.8631700Z ##[endgroup] 2024-08-20T22:08:10.8723864Z Docker daemon is running... 2024-08-20T22:08:10.8771919Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:08:10.8772480Z with: 2024-08-20T22:08:10.8772753Z shell: bash 2024-08-20T22:08:10.8773048Z timeout_minutes: 5 2024-08-20T22:08:10.8773372Z max_attempts: 3 2024-08-20T22:08:10.8773770Z retry_wait_seconds: 30 2024-08-20T22:08:10.8776942Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-08-20T22:08:10.8779921Z polling_interval_seconds: 1 2024-08-20T22:08:10.8780296Z warning_on_retry: true 2024-08-20T22:08:10.8780644Z continue_on_error: false 2024-08-20T22:08:10.8780984Z env: 2024-08-20T22:08:10.8781254Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:10.8781610Z AWS_RETRY_MODE: standard 2024-08-20T22:08:10.8781959Z AWS_MAX_ATTEMPTS: 5 2024-08-20T22:08:10.8782301Z AWS_DEFAULT_REGION: us-east-1 2024-08-20T22:08:10.8782676Z ##[endgroup] 2024-08-20T22:08:12.0550119Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:08:12.0550955Z Configure a credential helper to remove this warning. See 2024-08-20T22:08:12.0551807Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:08:12.0552353Z 2024-08-20T22:08:12.0552470Z Login Succeeded 2024-08-20T22:08:12.9344129Z Command completed after 1 attempt(s). 2024-08-20T22:08:12.9406133Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:08:12.9406850Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:08:12.9407495Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:08:12.9417248Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:12.9417743Z env: 2024-08-20T22:08:12.9418015Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:12.9418357Z ##[endgroup] 2024-08-20T22:08:12.9511332Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:08:12.9512092Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:08:12.9512685Z # shellcheck disable=SC2046 2024-08-20T22:08:12.9513145Z docker stop $(docker ps -q) || true 2024-08-20T22:08:12.9513632Z # Prune all of the docker images 2024-08-20T22:08:12.9514082Z docker system prune -af 2024-08-20T22:08:12.9522246Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:12.9522741Z env: 2024-08-20T22:08:12.9523027Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:12.9523377Z ##[endgroup] 2024-08-20T22:08:12.9834754Z "docker stop" requires at least 1 argument. 2024-08-20T22:08:12.9835493Z See 'docker stop --help'. 2024-08-20T22:08:12.9835739Z 2024-08-20T22:08:12.9835963Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-20T22:08:12.9836337Z 2024-08-20T22:08:12.9836494Z Stop one or more running containers 2024-08-20T22:08:16.2699329Z Deleted Images: 2024-08-20T22:08:16.2700040Z untagged: pytorch/manylinux-builder:cuda12.1 2024-08-20T22:08:16.2701006Z untagged: pytorch/manylinux-builder@sha256:1ceaa79868583367287b01793bb482334e53f31ed3974a78cd271b02247df451 2024-08-20T22:08:16.2702006Z deleted: sha256:f4fc3a9f2fd055e5b72e4feda5bbbd8c6e79ef22b6071d45945eb8ec64042f41 2024-08-20T22:08:16.2703420Z deleted: sha256:70032a09fef4ce88f555f69ad94ca1943d316b6f3b82da8dc7c2821690e88eed 2024-08-20T22:08:16.2704433Z deleted: sha256:a56cfbfa766fde9a9ee399928a535662d561613c3e54f9bf7d42a22557e05941 2024-08-20T22:08:16.2705427Z deleted: sha256:26c2625d88a3265ee12a5179363793d84e44778efa3b0565f571020d370121a9 2024-08-20T22:08:16.2706396Z deleted: sha256:00f9eb5fad579947541dd876721cd7e111bcbd38ca057b4c9c7c5d5130b45561 2024-08-20T22:08:16.2707395Z deleted: sha256:4bce2aee5079b7a763b5022d7617acfa9dc05c721b799345424836fff1bec24f 2024-08-20T22:08:16.2708407Z deleted: sha256:f572e0d9d5a866e08e3e512558dc935b53506c01916dcf5eb41c562cafca97c4 2024-08-20T22:08:16.2723980Z deleted: sha256:276ca07dabc1b6ec9a5577d3fe2fa1b0912f2c14eb1135fbfd5e883dadf96a55 2024-08-20T22:08:16.2725124Z deleted: sha256:413d9646e2cdd6bb7d142b7e2b44e80b8e55b0ea3101a06d2ae462bd98d41b53 2024-08-20T22:08:16.2726246Z deleted: sha256:eb7e8fa961c4c9ce0d409c41e2db44a5abf11078c04dcd727065808670502776 2024-08-20T22:08:16.2727585Z deleted: sha256:f052aaceea687692e6797adac86f83463ca13c7acb13399f3fe60671d914d837 2024-08-20T22:08:16.2728653Z deleted: sha256:95abcbbd6fe3402acb416254069895c430be984f650e0beb61e0b96dc9d4e3a1 2024-08-20T22:08:16.2729744Z deleted: sha256:1f1575a0dc7eafbe0bc24349a498188dc66b0c7afe0a92ad3177982dc53d1d7f 2024-08-20T22:08:16.2730826Z deleted: sha256:e8516cf5f8eaea7208825a0e74bc6ccc2425b2eeb87bf835fba068f096435065 2024-08-20T22:08:16.2731916Z deleted: sha256:50e753ec0ac354845824b7647dcb5c80fc1ecdf20246a62ef10afe70d2cba914 2024-08-20T22:08:16.2732988Z deleted: sha256:9a3a953a80dac55ead243068e4801c591294ce2467a7f2ab37acf37e7b50623d 2024-08-20T22:08:16.2734183Z deleted: sha256:8b589e158e3986bebcf7bfa8daf21e07ea295d0129753a23b59dc6016212d07a 2024-08-20T22:08:16.2735232Z deleted: sha256:9780567f56682e7ccb6519d6a582a4233a94a473e8319fdce963a938e8348437 2024-08-20T22:08:16.2736285Z deleted: sha256:83f38e31da6d72b6c7a862161962449d865590ecee2da9f77ce4a146acc998b0 2024-08-20T22:08:16.2737360Z deleted: sha256:b23603a0fe1921a9b378cd4f49db9da250b4f8026d73c027f355725059ea9ea8 2024-08-20T22:08:16.2738428Z deleted: sha256:6fc31f41ca816c5630174d83c904113d618ef54ee97610240e0f5f74a7b05ddb 2024-08-20T22:08:16.2739509Z deleted: sha256:b1fc1302c8cd0f46187e82d20d88b8202fea499b545d365200cdf56252b58512 2024-08-20T22:08:16.2740573Z deleted: sha256:53b918ca72cbb58e7562e0c9dc2babc7518b268d0ec7536af7f2606e1dff95cd 2024-08-20T22:08:16.2741671Z deleted: sha256:4913bc3e68cdc276b1bdc93abfb9d4fe26c4edc0e02f58b976716f71c7cff383 2024-08-20T22:08:16.2742761Z deleted: sha256:12c37601a9c0fafe2eace184744245d4aacfb250075ccfea1318049fac59673c 2024-08-20T22:08:16.2743829Z deleted: sha256:76c1229c2e89d5a621f482e8ad46f1ab304af8101f6dc0c94e7a2d968d0e5dd1 2024-08-20T22:08:16.2744896Z deleted: sha256:8a0b862a9ff3652d53bc44d54db195bccd91824ae70c438bf13eeb29a262dae6 2024-08-20T22:08:16.2745994Z deleted: sha256:36bbacb75b1fc6aa5f9dd973767ed867786a74144cb10b11b3b5459b5a36453c 2024-08-20T22:08:16.2747084Z deleted: sha256:601fbf08be902110317ee896327aedc5f0d4f0bb2c68e3be8a9afab4d7e375ac 2024-08-20T22:08:16.2748192Z deleted: sha256:ebfd2b021b5e26b3ac1c81ab75ee84007f8ae82fc5329d1bb797589046ed7e3a 2024-08-20T22:08:16.2749295Z deleted: sha256:7f450dd7de6a0c0234061a4d6d27513dbd48d3af305d48ac844be6f213c9f484 2024-08-20T22:08:16.2750397Z deleted: sha256:0afb7f31e1ac43cfddfdc4034a31fa4ededc1d057ba90aa605105d5004d09ff6 2024-08-20T22:08:16.2751476Z deleted: sha256:83933310864f0db0b6271f6484fe2a2ec4c4aa107dc676707aab1a86e84e78b8 2024-08-20T22:08:16.2752550Z deleted: sha256:5a8dd28488303431ac01ac5ef13992e7bf21d61c6e94702046235c8e1892c01c 2024-08-20T22:08:16.2753622Z deleted: sha256:0d1932c19f3321c16b8b0febc6905cf104fe67e6cc7901d220de7f2899e7336c 2024-08-20T22:08:16.2754705Z deleted: sha256:701fa06264450ce0aa66e58d35aad330e7bd68b0fbac8949d8e2d7587340a139 2024-08-20T22:08:16.2755811Z deleted: sha256:f89a6b18745ab169f86e3cfa97cd85d114b480b991fc3d9c5be888a9c29bed91 2024-08-20T22:08:16.2756940Z deleted: sha256:174f5685490326fc0a1c0f5570b8663732189b327007e47ff13d2ca59673db02 2024-08-20T22:08:16.2757783Z 2024-08-20T22:08:16.3560966Z Total reclaimed space: 12.33GB 2024-08-20T22:08:16.3617275Z ##[group]Run set +e 2024-08-20T22:08:16.3617774Z set +e 2024-08-20T22:08:16.3618220Z set -x 2024-08-20T22:08:16.3618646Z  2024-08-20T22:08:16.3619115Z PT_DOMAIN=download.pytorch.org 2024-08-20T22:08:16.3620313Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-20T22:08:16.3621991Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-20T22:08:16.3623142Z # one is returned at random 2024-08-20T22:08:16.3624209Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-20T22:08:16.3625274Z  2024-08-20T22:08:16.3625737Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:08:16.3626652Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-20T22:08:16.3628107Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-20T22:08:16.3628933Z  2024-08-20T22:08:16.3629415Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:08:16.3630236Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-20T22:08:16.3630983Z  exit 1 2024-08-20T22:08:16.3631426Z  fi 2024-08-20T22:08:16.3631829Z fi 2024-08-20T22:08:16.3632215Z  2024-08-20T22:08:16.3632719Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-20T22:08:16.3633494Z  # Clean up any old records first 2024-08-20T22:08:16.3634267Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-20T22:08:16.3634955Z fi 2024-08-20T22:08:16.3635353Z  2024-08-20T22:08:16.3635964Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-20T22:08:16.3636783Z cat /etc/hosts 2024-08-20T22:08:16.3648422Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:16.3649174Z env: 2024-08-20T22:08:16.3649562Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:16.3650053Z ##[endgroup] 2024-08-20T22:08:16.3684432Z + PT_DOMAIN=download.pytorch.org 2024-08-20T22:08:16.3690932Z ++ dig -4 +short download.pytorch.org 2024-08-20T22:08:16.3691419Z ++ tail -n1 2024-08-20T22:08:16.4185622Z + RESOLVED_IP=18.160.10.36 2024-08-20T22:08:16.4186167Z + '[' -z 18.160.10.36 ']' 2024-08-20T22:08:16.4186732Z + grep -r download.pytorch.org /etc/hosts 2024-08-20T22:08:16.4203902Z + echo '18.160.10.36 download.pytorch.org' 2024-08-20T22:08:16.4204907Z + sudo tee -a /etc/hosts 2024-08-20T22:08:16.6797811Z 18.160.10.36 download.pytorch.org 2024-08-20T22:08:16.6820004Z + cat /etc/hosts 2024-08-20T22:08:16.6830382Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-20T22:08:16.6843497Z ::1 localhost6 localhost6.localdomain6 2024-08-20T22:08:16.6844178Z 18.160.10.36 download.pytorch.org 2024-08-20T22:08:16.6997203Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-20T22:08:16.6997807Z with: 2024-08-20T22:08:16.6998916Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:16.7000153Z docker-build-dir: .ci/docker 2024-08-20T22:08:16.7000536Z working-directory: . 2024-08-20T22:08:16.7001013Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:16.7001537Z force-push: false 2024-08-20T22:08:16.7001845Z env: 2024-08-20T22:08:16.7002123Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:16.7002464Z ##[endgroup] 2024-08-20T22:08:16.7021140Z ##[group]Run set -ex 2024-08-20T22:08:16.7021502Z set -ex 2024-08-20T22:08:16.7021792Z  2024-08-20T22:08:16.7022345Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-20T22:08:16.7023315Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-20T22:08:16.7024293Z # job could then download the pre-built image as usual 2024-08-20T22:08:16.7025039Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-20T22:08:16.7025817Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:16.7026696Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:16.7027794Z  2024-08-20T22:08:16.7028406Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-20T22:08:16.7029183Z  exit 0 2024-08-20T22:08:16.7029564Z else 2024-08-20T22:08:16.7030027Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:16.7030639Z fi 2024-08-20T22:08:16.7031009Z  2024-08-20T22:08:16.7031537Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-20T22:08:16.7032558Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-20T22:08:16.7033426Z  # use it as it is, but first let's extract the tag 2024-08-20T22:08:16.7034223Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-20T22:08:16.7035098Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:16.7035853Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:16.7036643Z else 2024-08-20T22:08:16.7037218Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-20T22:08:16.7037971Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:16.7038941Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:16.7039824Z fi 2024-08-20T22:08:16.7050965Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:16.7051578Z env: 2024-08-20T22:08:16.7051943Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:16.7052414Z REPO_NAME: pytorch 2024-08-20T22:08:16.7053735Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:16.7055030Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:08:16.7055673Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:16.7056266Z ##[endgroup] 2024-08-20T22:08:16.7086673Z + [[ ! -d .ci/docker ]] 2024-08-20T22:08:16.7087372Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-20T22:08:16.7088032Z + echo skip=false 2024-08-20T22:08:16.7090136Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-20T22:08:16.7094333Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:16.7096058Z ++ awk -F '[:,]' '{print $2}' 2024-08-20T22:08:16.7120801Z + DOCKER_TAG=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:16.7121928Z + echo docker-tag=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:16.7123814Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:16.7157001Z ##[group]Run set +e 2024-08-20T22:08:16.7157541Z set +e 2024-08-20T22:08:16.7157954Z set -x 2024-08-20T22:08:16.7158351Z  2024-08-20T22:08:16.7158744Z login() { 2024-08-20T22:08:16.7159509Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:08:16.7160532Z } 2024-08-20T22:08:16.7160961Z  2024-08-20T22:08:16.7161329Z retry () { 2024-08-20T22:08:16.7161903Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:08:16.7162415Z } 2024-08-20T22:08:16.7162782Z  2024-08-20T22:08:16.7163256Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:08:16.7163772Z  2024-08-20T22:08:16.7164308Z # Check if image already exists, if it does then skip building it 2024-08-20T22:08:16.7165181Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-20T22:08:16.7165780Z  exit 0 2024-08-20T22:08:16.7166170Z fi 2024-08-20T22:08:16.7166569Z  2024-08-20T22:08:16.7167231Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-20T22:08:16.7168178Z # be empty. The default action would be to continue rebuild the image 2024-08-20T22:08:16.7169070Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-20T22:08:16.7169815Z  # if we're on the base branch then use the parent commit 2024-08-20T22:08:16.7170522Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-20T22:08:16.7171099Z else 2024-08-20T22:08:16.7171666Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-20T22:08:16.7172438Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-20T22:08:16.7173096Z fi 2024-08-20T22:08:16.7173464Z  2024-08-20T22:08:16.7174053Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-20T22:08:16.7174689Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:16.7175261Z  2024-08-20T22:08:16.7176051Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-20T22:08:16.7176903Z  exit 0 2024-08-20T22:08:16.7177281Z fi 2024-08-20T22:08:16.7177694Z  2024-08-20T22:08:16.7178229Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-20T22:08:16.7179316Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-20T22:08:16.7180221Z  exit 1 2024-08-20T22:08:16.7180617Z fi 2024-08-20T22:08:16.7181007Z  2024-08-20T22:08:16.7181697Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-20T22:08:16.7182731Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-20T22:08:16.7183689Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-20T22:08:16.7184792Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-20T22:08:16.7186003Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-20T22:08:16.7186950Z fi 2024-08-20T22:08:16.7187372Z  2024-08-20T22:08:16.7187822Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:16.7196322Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:16.7196910Z env: 2024-08-20T22:08:16.7197238Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:16.7197855Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:08:16.7198416Z BASE_REVISION: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:08:16.7199853Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:16.7201175Z DOCKER_TAG: f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:16.7201866Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:16.7202541Z ##[endgroup] 2024-08-20T22:08:16.7232329Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:16.7233324Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:16.7234686Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:08:16.7236392Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:17.2586293Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:08:17.2587680Z Configure a credential helper to remove this warning. See 2024-08-20T22:08:17.2588778Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:08:17.2589375Z 2024-08-20T22:08:17.2589532Z Login Succeeded 2024-08-20T22:08:17.2612117Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:17.4922570Z { 2024-08-20T22:08:17.4923063Z "schemaVersion": 2, 2024-08-20T22:08:17.4924256Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-20T22:08:17.4924953Z "config": { 2024-08-20T22:08:17.4925875Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-20T22:08:17.4927192Z "size": 48711, 2024-08-20T22:08:17.4928267Z "digest": "sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb" 2024-08-20T22:08:17.4929099Z }, 2024-08-20T22:08:17.4929737Z "layers": [ 2024-08-20T22:08:17.4930610Z { 2024-08-20T22:08:17.4931475Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.4932497Z "size": 28580681, 2024-08-20T22:08:17.4933304Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-08-20T22:08:17.4934373Z }, 2024-08-20T22:08:17.4935060Z { 2024-08-20T22:08:17.4935728Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.4936520Z "size": 7943451, 2024-08-20T22:08:17.4938523Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-08-20T22:08:17.4939478Z }, 2024-08-20T22:08:17.4939981Z { 2024-08-20T22:08:17.4940653Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.4941958Z "size": 55728572, 2024-08-20T22:08:17.4942800Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-08-20T22:08:17.4943741Z }, 2024-08-20T22:08:17.4944498Z { 2024-08-20T22:08:17.4945029Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.4946364Z "size": 186, 2024-08-20T22:08:17.4947670Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-08-20T22:08:17.4948759Z }, 2024-08-20T22:08:17.4949334Z { 2024-08-20T22:08:17.4950092Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.4950916Z "size": 6886, 2024-08-20T22:08:17.4951777Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-08-20T22:08:17.4952670Z }, 2024-08-20T22:08:17.4953316Z { 2024-08-20T22:08:17.4953821Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.4954536Z "size": 1329001756, 2024-08-20T22:08:17.4955240Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-08-20T22:08:17.4956054Z }, 2024-08-20T22:08:17.4956358Z { 2024-08-20T22:08:17.4956921Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.4957618Z "size": 62501, 2024-08-20T22:08:17.4958537Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-08-20T22:08:17.4959392Z }, 2024-08-20T22:08:17.4959752Z { 2024-08-20T22:08:17.4960316Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.4961020Z "size": 1684, 2024-08-20T22:08:17.4961607Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-08-20T22:08:17.4962324Z }, 2024-08-20T22:08:17.4962725Z { 2024-08-20T22:08:17.4963286Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.4964095Z "size": 1523, 2024-08-20T22:08:17.4964812Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-08-20T22:08:17.4965563Z }, 2024-08-20T22:08:17.4965866Z { 2024-08-20T22:08:17.4966475Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.4967195Z "size": 2528295403, 2024-08-20T22:08:17.4967952Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-08-20T22:08:17.4968671Z }, 2024-08-20T22:08:17.4969037Z { 2024-08-20T22:08:17.4969555Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.4970316Z "size": 86016, 2024-08-20T22:08:17.4970927Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-08-20T22:08:17.4971653Z }, 2024-08-20T22:08:17.4972065Z { 2024-08-20T22:08:17.4972595Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.4973203Z "size": 1822, 2024-08-20T22:08:17.4974041Z "digest": "sha256:ac23c5fcc1ef310e9cd0d138b180281188e20a3e28c51cd14751622e3c7d92ea" 2024-08-20T22:08:17.4974747Z }, 2024-08-20T22:08:17.4975063Z { 2024-08-20T22:08:17.4975659Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.4976301Z "size": 246761563, 2024-08-20T22:08:17.4977057Z "digest": "sha256:ca426ffc9d2bd051ad1ff5c4f02fd5d9be161e70953cc37882c815080c708f4d" 2024-08-20T22:08:17.4977740Z }, 2024-08-20T22:08:17.4978085Z { 2024-08-20T22:08:17.4978680Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.4979327Z "size": 544, 2024-08-20T22:08:17.4979910Z "digest": "sha256:4373aa743caf6660c671295b837048c8350a2165f2336dc3b7b6c96462579e50" 2024-08-20T22:08:17.4980680Z }, 2024-08-20T22:08:17.4981022Z { 2024-08-20T22:08:17.4981507Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.4982238Z "size": 1258, 2024-08-20T22:08:17.4982867Z "digest": "sha256:75b2abb16c685854c017d0cf75c626526c956734c656d9e78fdd4520317e9b3c" 2024-08-20T22:08:17.4983587Z }, 2024-08-20T22:08:17.4983967Z { 2024-08-20T22:08:17.4984494Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.4985160Z "size": 484, 2024-08-20T22:08:17.4985833Z "digest": "sha256:ef1e3ef1e85f0124270b8506721219150e230184181f8207d64463bf5c67e656" 2024-08-20T22:08:17.4986486Z }, 2024-08-20T22:08:17.4986844Z { 2024-08-20T22:08:17.4987419Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.4988062Z "size": 91711766, 2024-08-20T22:08:17.4988687Z "digest": "sha256:0bf61114d37b2d09065aa6852b1e78a32580a1ff011dd5d23f03b0f5c4e710af" 2024-08-20T22:08:17.4989466Z }, 2024-08-20T22:08:17.4989825Z { 2024-08-20T22:08:17.4990308Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.4991035Z "size": 3390, 2024-08-20T22:08:17.4991808Z "digest": "sha256:2df849f72ba4258df10c9c45ace21a66d93b73196e78e96f12c13098822f60e2" 2024-08-20T22:08:17.4992520Z }, 2024-08-20T22:08:17.4992872Z { 2024-08-20T22:08:17.4993409Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.4994042Z "size": 1909, 2024-08-20T22:08:17.4994742Z "digest": "sha256:34c47d63c5de27418739aa048b4dd781c3006d65691e9b903c95304d0509d89c" 2024-08-20T22:08:17.4995402Z }, 2024-08-20T22:08:17.4995744Z { 2024-08-20T22:08:17.4996360Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.4997064Z "size": 701, 2024-08-20T22:08:17.4997662Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:08:17.4998443Z }, 2024-08-20T22:08:17.4998783Z { 2024-08-20T22:08:17.4999309Z + exit 0 2024-08-20T22:08:17.4999848Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5000493Z "size": 2864254879, 2024-08-20T22:08:17.5001344Z "digest": "sha256:6efb96fc87ea1bca54876d85331e5585e28f325a1d60ee1aef7811eabb74f933" 2024-08-20T22:08:17.5002018Z }, 2024-08-20T22:08:17.5002378Z { 2024-08-20T22:08:17.5002972Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5003643Z "size": 380, 2024-08-20T22:08:17.5004253Z "digest": "sha256:b119c4fc6c58d9e72744201ff4850b34184a94eb3566031b7d36e87ecce950a6" 2024-08-20T22:08:17.5005009Z }, 2024-08-20T22:08:17.5005349Z { 2024-08-20T22:08:17.5005893Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5006545Z "size": 12900, 2024-08-20T22:08:17.5007191Z "digest": "sha256:33b639226eeefa41d7bb26a8ecc20d271ceef45571dabfd80aaee3634926381b" 2024-08-20T22:08:17.5007918Z }, 2024-08-20T22:08:17.5008264Z { 2024-08-20T22:08:17.5008810Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5009456Z "size": 803, 2024-08-20T22:08:17.5010128Z "digest": "sha256:d3547bd43146930aa830d54562c98aa678e355cf5f84527e9717eb72fffe8ca3" 2024-08-20T22:08:17.5010896Z }, 2024-08-20T22:08:17.5011286Z { 2024-08-20T22:08:17.5011773Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5012441Z "size": 106, 2024-08-20T22:08:17.5013110Z "digest": "sha256:5645520256fa6d7f2270608eaed53849a131d03fbf4c5d4472c9cfde1a439e47" 2024-08-20T22:08:17.5013935Z }, 2024-08-20T22:08:17.5014239Z { 2024-08-20T22:08:17.5014819Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5015490Z "size": 503, 2024-08-20T22:08:17.5016082Z "digest": "sha256:829ed7d91e3818d0e11834c71551117c7aeec0a51d798e93d641faa0c02582cb" 2024-08-20T22:08:17.5016852Z }, 2024-08-20T22:08:17.5017240Z { 2024-08-20T22:08:17.5017770Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5018438Z "size": 121477267, 2024-08-20T22:08:17.5019108Z "digest": "sha256:385d0421d67c533d628985c453a186028a61f6ad4209ae519ec3d4ecea661d3e" 2024-08-20T22:08:17.5019800Z }, 2024-08-20T22:08:17.5020209Z { 2024-08-20T22:08:17.5020698Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5021338Z "size": 109, 2024-08-20T22:08:17.5022053Z "digest": "sha256:7cbfce999d76851e07dd71056a6d852629d789ccfd46be44cc405b451fb8bb58" 2024-08-20T22:08:17.5022770Z }, 2024-08-20T22:08:17.5023078Z { 2024-08-20T22:08:17.5023678Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5024357Z "size": 490, 2024-08-20T22:08:17.5024950Z "digest": "sha256:6658e558ebe496f0d628947534b0ea1e44808f84f2dfd8a9a704e8aa5329d791" 2024-08-20T22:08:17.5025727Z }, 2024-08-20T22:08:17.5026067Z { 2024-08-20T22:08:17.5026687Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5027754Z "size": 297, 2024-08-20T22:08:17.5028388Z "digest": "sha256:33444a5a7e01a3010fd1cc16b32a66892a08600d7f700747d23faf3c36624f3a" 2024-08-20T22:08:17.5029284Z }, 2024-08-20T22:08:17.5029689Z { 2024-08-20T22:08:17.5030200Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5030854Z "size": 103, 2024-08-20T22:08:17.5031563Z "digest": "sha256:7c47122f4e09b47e891b4ae9e499a35a4eb46faacae87d0e946615adda6e0706" 2024-08-20T22:08:17.5032256Z }, 2024-08-20T22:08:17.5032585Z { 2024-08-20T22:08:17.5033157Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5033815Z "size": 1473, 2024-08-20T22:08:17.5034412Z "digest": "sha256:1811ff4d3556ac255604f346d879528aff2aadd7652c2d39baf4fe6eaa60aec3" 2024-08-20T22:08:17.5035185Z }, 2024-08-20T22:08:17.5035518Z { 2024-08-20T22:08:17.5036098Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5036721Z "size": 594729516, 2024-08-20T22:08:17.5037362Z "digest": "sha256:7754ee068b33f760d9c320c26a1a3935247ace0c8c463a976105c1586a43fb1c" 2024-08-20T22:08:17.5038144Z }, 2024-08-20T22:08:17.5038504Z { 2024-08-20T22:08:17.5039112Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5039845Z "size": 164, 2024-08-20T22:08:17.5040475Z "digest": "sha256:0479fbac3361713c075bf788e778117917dcacdd1ff71ed79a789dd2016a1714" 2024-08-20T22:08:17.5041139Z }, 2024-08-20T22:08:17.5041548Z { 2024-08-20T22:08:17.5042065Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5042698Z "size": 564, 2024-08-20T22:08:17.5043347Z "digest": "sha256:f1d6874dee9c34d154278d2b6935d04447f07fa89d25d4a09398234040d8a7e8" 2024-08-20T22:08:17.5044040Z }, 2024-08-20T22:08:17.5044422Z { 2024-08-20T22:08:17.5044992Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5045594Z "size": 35874203, 2024-08-20T22:08:17.5046273Z "digest": "sha256:afc429e87b593a64f3e0a1be51e09e966ea1aa71e6e8a0bec4edc9f2f7bc1c05" 2024-08-20T22:08:17.5047040Z }, 2024-08-20T22:08:17.5047338Z { 2024-08-20T22:08:17.5047888Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5048573Z "size": 104, 2024-08-20T22:08:17.5049188Z "digest": "sha256:019620471c2592cd22437fe8ed1a94c784114cf7445c5c2ab36dd0bae727fc66" 2024-08-20T22:08:17.5049869Z }, 2024-08-20T22:08:17.5050252Z { 2024-08-20T22:08:17.5050793Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5051458Z "size": 425, 2024-08-20T22:08:17.5052110Z "digest": "sha256:27aeb1b2cd5e28d8f0e54fc1090baf88abc29ebf5fce088b493fcae99c9550d1" 2024-08-20T22:08:17.5052861Z }, 2024-08-20T22:08:17.5053192Z { 2024-08-20T22:08:17.5053861Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5054492Z "size": 20262183, 2024-08-20T22:08:17.5055131Z "digest": "sha256:829c0e19f4b7797e9120a86172fabfd710f9b3971ffd714a616931a3fadc7192" 2024-08-20T22:08:17.5055874Z }, 2024-08-20T22:08:17.5056193Z { 2024-08-20T22:08:17.5056709Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5057417Z "size": 438, 2024-08-20T22:08:17.5058073Z "digest": "sha256:560cd6122a3868e7e82e75eb103d182edf6512114d61ccdfae099af00cb16b4c" 2024-08-20T22:08:17.5058735Z }, 2024-08-20T22:08:17.5059147Z { 2024-08-20T22:08:17.5059663Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5060349Z "size": 701, 2024-08-20T22:08:17.5060957Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:08:17.5061654Z }, 2024-08-20T22:08:17.5062061Z { 2024-08-20T22:08:17.5062564Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5063198Z "size": 141, 2024-08-20T22:08:17.5063902Z "digest": "sha256:5a4eb19d72198a01f3babd4711c40fcf7c6e5cb9d5cfedc507b0061408d47097" 2024-08-20T22:08:17.5064607Z }, 2024-08-20T22:08:17.5064939Z { 2024-08-20T22:08:17.5065524Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5066276Z "size": 135, 2024-08-20T22:08:17.5066960Z "digest": "sha256:da6a838a66c72b5de9a5590c233779880ea901c1264940a1c618825d2501ffd1" 2024-08-20T22:08:17.5067664Z }, 2024-08-20T22:08:17.5067999Z { 2024-08-20T22:08:17.5068538Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5069223Z "size": 4838068375, 2024-08-20T22:08:17.5069829Z "digest": "sha256:cbb79a7053537bed2d80b77198ea552b985563b11d3f8616593c6ab575a3a090" 2024-08-20T22:08:17.5070548Z }, 2024-08-20T22:08:17.5070930Z { 2024-08-20T22:08:17.5071437Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5072096Z "size": 194, 2024-08-20T22:08:17.5072772Z "digest": "sha256:4d4fa6802ab3e6b36d8d4adb650046e9370c4a16a19876970559ef8a312ca855" 2024-08-20T22:08:17.5073492Z }, 2024-08-20T22:08:17.5073787Z { 2024-08-20T22:08:17.5074367Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5075021Z "size": 564, 2024-08-20T22:08:17.5075744Z "digest": "sha256:0f085d4fa279366fcd52d303b7ad8abddc5962a8872174b8109f3f5d5ca5f2cf" 2024-08-20T22:08:17.5076479Z }, 2024-08-20T22:08:17.5076842Z { 2024-08-20T22:08:17.5077367Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5078049Z "size": 43162921, 2024-08-20T22:08:17.5078714Z "digest": "sha256:4a0412373305517489d891a4749e6d2b80b4aeb7df3b8adca0ed0abb2f3e5300" 2024-08-20T22:08:17.5079409Z }, 2024-08-20T22:08:17.5079812Z { 2024-08-20T22:08:17.5080290Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5080925Z "size": 106, 2024-08-20T22:08:17.5081614Z "digest": "sha256:302c8c1779444e54919f83e91c6d5e1fdea653fc899be802b3a2a34d34d99d5d" 2024-08-20T22:08:17.5082305Z }, 2024-08-20T22:08:17.5082602Z { 2024-08-20T22:08:17.5083184Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5083823Z "size": 1380, 2024-08-20T22:08:17.5084510Z "digest": "sha256:02970347dff622ee45629bc28788716d00528464f71f88ec7b74ae5f44a486e2" 2024-08-20T22:08:17.5085194Z }, 2024-08-20T22:08:17.5085528Z { 2024-08-20T22:08:17.5086125Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5086719Z "size": 701, 2024-08-20T22:08:17.5087348Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:08:17.5088128Z }, 2024-08-20T22:08:17.5088472Z { 2024-08-20T22:08:17.5088951Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5089665Z "size": 139, 2024-08-20T22:08:17.5090284Z "digest": "sha256:916c5e8580164926a3aec87e88a5d1c2d02756e49fdeed4198e1e6304efa2fd3" 2024-08-20T22:08:17.5091003Z }, 2024-08-20T22:08:17.5091390Z { 2024-08-20T22:08:17.5091909Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5092571Z "size": 118, 2024-08-20T22:08:17.5093225Z "digest": "sha256:c42b3602a1fb4df5111224650f15adffb025979ae9c614807379ea84e189533b" 2024-08-20T22:08:17.5094014Z }, 2024-08-20T22:08:17.5094379Z { 2024-08-20T22:08:17.5094960Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5095574Z "size": 4324672728, 2024-08-20T22:08:17.5096268Z "digest": "sha256:11fad972ef6cb9d2dc236504b0ee5b6e5ea996e4576ece0ed3262fbb9ee2de60" 2024-08-20T22:08:17.5097087Z }, 2024-08-20T22:08:17.5097453Z { 2024-08-20T22:08:17.5097946Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5098666Z "size": 174, 2024-08-20T22:08:17.5099305Z "digest": "sha256:f957c16244819132b478272aaf24231699c4fcbd495c229a574f94daab8d6b86" 2024-08-20T22:08:17.5100003Z }, 2024-08-20T22:08:17.5100355Z { 2024-08-20T22:08:17.5100899Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5101537Z "size": 907, 2024-08-20T22:08:17.5102193Z "digest": "sha256:768c31dca016b80bbdc973c8ee818034bdbd17b7ea9bdadbf0ec45ea51005720" 2024-08-20T22:08:17.5103038Z }, 2024-08-20T22:08:17.5103386Z { 2024-08-20T22:08:17.5103986Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5104589Z "size": 701, 2024-08-20T22:08:17.5105250Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:08:17.5106020Z }, 2024-08-20T22:08:17.5106360Z { 2024-08-20T22:08:17.5106880Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5107612Z "size": 134, 2024-08-20T22:08:17.5108249Z "digest": "sha256:8c32d0ecebbe2054c96a3a5c1f3d6c8b87c2e826686b369e5242cd55d0f01c2d" 2024-08-20T22:08:17.5109030Z }, 2024-08-20T22:08:17.5109332Z { 2024-08-20T22:08:17.5109859Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5110570Z "size": 32, 2024-08-20T22:08:17.5111166Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:08:17.5111879Z }, 2024-08-20T22:08:17.5112332Z { 2024-08-20T22:08:17.5112951Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5113579Z "size": 158, 2024-08-20T22:08:17.5114267Z "digest": "sha256:78ee77eddef23f20afc78e2924f4467041cadb754634ee3df431c68211cf676d" 2024-08-20T22:08:17.5114983Z }, 2024-08-20T22:08:17.5115342Z { 2024-08-20T22:08:17.5115894Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5116534Z "size": 1841, 2024-08-20T22:08:17.5117185Z "digest": "sha256:37513e9b43d7123e7ddee09100f62a72b57af7b3833eaecd515e247e2f63e13f" 2024-08-20T22:08:17.5117898Z }, 2024-08-20T22:08:17.5118253Z { 2024-08-20T22:08:17.5118804Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5119486Z "size": 7529776, 2024-08-20T22:08:17.5120115Z "digest": "sha256:a7a5392e2bf6b73d9be292cc22a99ad4637636c15733beb7afbe6403b5ba21a8" 2024-08-20T22:08:17.5120830Z }, 2024-08-20T22:08:17.5121218Z { 2024-08-20T22:08:17.5121774Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5122388Z "size": 163, 2024-08-20T22:08:17.5123065Z "digest": "sha256:b10de02255c021b2366b85bf12ddf593aac89f88964201c455b71a145bc5ca2e" 2024-08-20T22:08:17.5123789Z }, 2024-08-20T22:08:17.5124091Z { 2024-08-20T22:08:17.5124677Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5125346Z "size": 7943, 2024-08-20T22:08:17.5125972Z "digest": "sha256:e56eeff2db93ffe5002a2bb764795b7333321c6802cc28bef0801a619fe6a07e" 2024-08-20T22:08:17.5126894Z }, 2024-08-20T22:08:17.5127300Z { 2024-08-20T22:08:17.5127819Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5128539Z "size": 8068, 2024-08-20T22:08:17.5129130Z "digest": "sha256:70ac3e38d0fd1f0f46ad710c3fc485fd01406606054e8cf84a63af1570bc9f3a" 2024-08-20T22:08:17.5129838Z }, 2024-08-20T22:08:17.5130241Z { 2024-08-20T22:08:17.5130765Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5131391Z "size": 302, 2024-08-20T22:08:17.5132053Z "digest": "sha256:eb09cbf3e4168749cc8605e3188ed35070f6037d13930e443c38d8941734488f" 2024-08-20T22:08:17.5132777Z }, 2024-08-20T22:08:17.5133088Z { 2024-08-20T22:08:17.5133722Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5134364Z "size": 7629638, 2024-08-20T22:08:17.5135089Z "digest": "sha256:ee1273153675e6543b58f5be50321abee5e91f7351701d5d501a6837eaf676ef" 2024-08-20T22:08:17.5135745Z }, 2024-08-20T22:08:17.5136082Z { 2024-08-20T22:08:17.5136664Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5137297Z "size": 108, 2024-08-20T22:08:17.5137909Z "digest": "sha256:e22979f9e0eedfa16f98eb4fa26f30a50c237f5ffd2f6e5b898f2b908058496d" 2024-08-20T22:08:17.5138657Z }, 2024-08-20T22:08:17.5139028Z { 2024-08-20T22:08:17.5139523Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5140349Z "size": 54145661, 2024-08-20T22:08:17.5141023Z "digest": "sha256:6c1f0aabce7c84f9b0b9ab3b96325778dba1c1465b7463c5b4c6be1cf52b74db" 2024-08-20T22:08:17.5141738Z }, 2024-08-20T22:08:17.5142102Z { 2024-08-20T22:08:17.5142642Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5143273Z "size": 473, 2024-08-20T22:08:17.5143949Z "digest": "sha256:7ebe49c3a23e71b289ce012b48dad35791a2f2aa7fcee2326a64d763b0d14df8" 2024-08-20T22:08:17.5144642Z }, 2024-08-20T22:08:17.5144977Z { 2024-08-20T22:08:17.5145567Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5146225Z "size": 1374858766, 2024-08-20T22:08:17.5146840Z "digest": "sha256:5d115b29b5d53e5418762bc41ae975dd245bcffc3d058b8902c58146d505cb37" 2024-08-20T22:08:17.5147607Z }, 2024-08-20T22:08:17.5147950Z { 2024-08-20T22:08:17.5148433Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5149144Z "size": 106, 2024-08-20T22:08:17.5149786Z "digest": "sha256:7f2d7a04fbe65bbe1e895138e0cfeeef53ad926c890bab8e7e132ad1e655e48f" 2024-08-20T22:08:17.5150648Z }, 2024-08-20T22:08:17.5151000Z { 2024-08-20T22:08:17.5151526Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5152196Z "size": 559, 2024-08-20T22:08:17.5152893Z "digest": "sha256:d79fd7d526989e554455eda7d459f2c88444030f61710101e52ec8c7a49b7b79" 2024-08-20T22:08:17.5153539Z }, 2024-08-20T22:08:17.5153898Z { 2024-08-20T22:08:17.5154469Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5155075Z "size": 46248488, 2024-08-20T22:08:17.5155733Z "digest": "sha256:9752d6fbbcab4e7403f26777c5b65911320de6389c81b833a47ba45c4f3354b4" 2024-08-20T22:08:17.5156508Z }, 2024-08-20T22:08:17.5156899Z { 2024-08-20T22:08:17.5157382Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5158069Z "size": 111, 2024-08-20T22:08:17.5158717Z "digest": "sha256:83808b2eff4d71ed46db7c29f66fe918f328405c25b314225dfad01b4868d1f4" 2024-08-20T22:08:17.5159497Z }, 2024-08-20T22:08:17.5346839Z { 2024-08-20T22:08:17.5347364Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5347925Z "size": 32, 2024-08-20T22:08:17.5348474Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:08:17.5349090Z }, 2024-08-20T22:08:17.5349365Z { 2024-08-20T22:08:17.5349797Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5350343Z "size": 32, 2024-08-20T22:08:17.5350871Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:08:17.5351482Z }, 2024-08-20T22:08:17.5351726Z { 2024-08-20T22:08:17.5352147Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5352694Z "size": 32, 2024-08-20T22:08:17.5353248Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:08:17.5353867Z }, 2024-08-20T22:08:17.5354104Z { 2024-08-20T22:08:17.5354530Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:17.5355082Z "size": 32, 2024-08-20T22:08:17.5355615Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:08:17.5356220Z } 2024-08-20T22:08:17.5356466Z ] 2024-08-20T22:08:17.5356707Z } 2024-08-20T22:08:17.5458652Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:08:17.5459161Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:08:17.5459737Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-20T22:08:17.5469579Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:17.5470071Z env: 2024-08-20T22:08:17.5470346Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:17.5471497Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:17.5472669Z ##[endgroup] 2024-08-20T22:08:17.5507988Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:17.5559743Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-20T22:08:17.5560319Z with: 2024-08-20T22:08:17.5561387Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:17.5562705Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:17.5563223Z env: 2024-08-20T22:08:17.5563497Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:17.5563848Z ##[endgroup] 2024-08-20T22:08:17.5580963Z ##[group]Run set -x 2024-08-20T22:08:17.5581306Z set -x 2024-08-20T22:08:17.5581611Z set +e 2024-08-20T22:08:17.5581909Z  2024-08-20T22:08:17.5582183Z login() { 2024-08-20T22:08:17.5583024Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:08:17.5583745Z } 2024-08-20T22:08:17.5584027Z  2024-08-20T22:08:17.5584334Z retry () { 2024-08-20T22:08:17.5584747Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:08:17.5585196Z } 2024-08-20T22:08:17.5585473Z  2024-08-20T22:08:17.5585796Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:08:17.5586207Z  2024-08-20T22:08:17.5586481Z set -e 2024-08-20T22:08:17.5586956Z # ignore output since only exit code is used for conditional 2024-08-20T22:08:17.5587659Z # only pull docker image if it's not available locally 2024-08-20T22:08:17.5588426Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-20T22:08:17.5589118Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-20T22:08:17.5589564Z fi 2024-08-20T22:08:17.5597552Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:17.5598051Z env: 2024-08-20T22:08:17.5598327Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:17.5599498Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:17.5600818Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:17.5601344Z ##[endgroup] 2024-08-20T22:08:17.5629318Z + set +e 2024-08-20T22:08:17.5629847Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:17.5630486Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:17.5632743Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:08:17.5634692Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:18.1058457Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:08:18.1059321Z Configure a credential helper to remove this warning. See 2024-08-20T22:08:18.1060164Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:08:18.1060694Z 2024-08-20T22:08:18.1060828Z Login Succeeded 2024-08-20T22:08:18.1081822Z + set -e 2024-08-20T22:08:18.1083566Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:18.1241081Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:18.1243252Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:18.3599369Z f6d216893d65c7b8ae43df4daaf247db808378e9: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-08-20T22:08:18.3602322Z 7a2c55901189: Pulling fs layer 2024-08-20T22:08:18.3602859Z 224fe954d725: Pulling fs layer 2024-08-20T22:08:18.3603718Z 75722010b82e: Pulling fs layer 2024-08-20T22:08:18.3604256Z d527cbbb87e3: Pulling fs layer 2024-08-20T22:08:18.3604773Z b57676e46aee: Pulling fs layer 2024-08-20T22:08:18.3605282Z a8c1e85b5e14: Pulling fs layer 2024-08-20T22:08:18.3605797Z a41a8d1c11c8: Pulling fs layer 2024-08-20T22:08:18.3606202Z 0c1227890755: Pulling fs layer 2024-08-20T22:08:18.3606662Z d8d1234baab3: Pulling fs layer 2024-08-20T22:08:18.3607212Z 7ed32bc8e469: Pulling fs layer 2024-08-20T22:08:18.3607716Z ec1e7978c1fe: Pulling fs layer 2024-08-20T22:08:18.3608217Z ac23c5fcc1ef: Pulling fs layer 2024-08-20T22:08:18.3608727Z ca426ffc9d2b: Pulling fs layer 2024-08-20T22:08:18.3609233Z 4373aa743caf: Pulling fs layer 2024-08-20T22:08:18.3609749Z 75b2abb16c68: Pulling fs layer 2024-08-20T22:08:18.3610263Z ef1e3ef1e85f: Pulling fs layer 2024-08-20T22:08:18.3610932Z 0bf61114d37b: Pulling fs layer 2024-08-20T22:08:18.3611427Z 2df849f72ba4: Pulling fs layer 2024-08-20T22:08:18.3611937Z 34c47d63c5de: Pulling fs layer 2024-08-20T22:08:18.3612433Z 0c1227890755: Waiting 2024-08-20T22:08:18.3612883Z 1fef4bd19792: Pulling fs layer 2024-08-20T22:08:18.3613376Z b57676e46aee: Waiting 2024-08-20T22:08:18.3613977Z 6efb96fc87ea: Pulling fs layer 2024-08-20T22:08:18.3614478Z a8c1e85b5e14: Waiting 2024-08-20T22:08:18.3614910Z a41a8d1c11c8: Waiting 2024-08-20T22:08:18.3615295Z b119c4fc6c58: Pulling fs layer 2024-08-20T22:08:18.3615670Z 33b639226eee: Pulling fs layer 2024-08-20T22:08:18.3616029Z d3547bd43146: Pulling fs layer 2024-08-20T22:08:18.3616393Z 5645520256fa: Pulling fs layer 2024-08-20T22:08:18.3616753Z d8d1234baab3: Waiting 2024-08-20T22:08:18.3617086Z 829ed7d91e38: Pulling fs layer 2024-08-20T22:08:18.3617433Z 7ed32bc8e469: Waiting 2024-08-20T22:08:18.3617789Z ec1e7978c1fe: Waiting 2024-08-20T22:08:18.3618104Z ca426ffc9d2b: Waiting 2024-08-20T22:08:18.3618430Z ac23c5fcc1ef: Waiting 2024-08-20T22:08:18.3618760Z 385d0421d67c: Pulling fs layer 2024-08-20T22:08:18.3619112Z 4373aa743caf: Waiting 2024-08-20T22:08:18.3619425Z d527cbbb87e3: Waiting 2024-08-20T22:08:18.3619764Z 7cbfce999d76: Pulling fs layer 2024-08-20T22:08:18.3620118Z 75b2abb16c68: Waiting 2024-08-20T22:08:18.3620445Z 6658e558ebe4: Pulling fs layer 2024-08-20T22:08:18.3620818Z 33444a5a7e01: Pulling fs layer 2024-08-20T22:08:18.3621184Z 7c47122f4e09: Pulling fs layer 2024-08-20T22:08:18.3621530Z 0bf61114d37b: Waiting 2024-08-20T22:08:18.3621856Z 1811ff4d3556: Pulling fs layer 2024-08-20T22:08:18.3622223Z 7754ee068b33: Pulling fs layer 2024-08-20T22:08:18.3622572Z 2df849f72ba4: Waiting 2024-08-20T22:08:18.3622881Z 34c47d63c5de: Waiting 2024-08-20T22:08:18.3623193Z 6efb96fc87ea: Waiting 2024-08-20T22:08:18.3623515Z 385d0421d67c: Waiting 2024-08-20T22:08:18.3623823Z d3547bd43146: Waiting 2024-08-20T22:08:18.3624133Z ef1e3ef1e85f: Waiting 2024-08-20T22:08:18.3624445Z 7cbfce999d76: Waiting 2024-08-20T22:08:18.3624761Z 5645520256fa: Waiting 2024-08-20T22:08:18.3625064Z 6658e558ebe4: Waiting 2024-08-20T22:08:18.3625375Z 1fef4bd19792: Waiting 2024-08-20T22:08:18.3625681Z 33444a5a7e01: Waiting 2024-08-20T22:08:18.3625988Z b119c4fc6c58: Waiting 2024-08-20T22:08:18.3626294Z 7c47122f4e09: Waiting 2024-08-20T22:08:18.3626600Z 1811ff4d3556: Waiting 2024-08-20T22:08:18.3627250Z 33b639226eee: Waiting 2024-08-20T22:08:18.3627561Z 829ed7d91e38: Waiting 2024-08-20T22:08:18.3627884Z 0479fbac3361: Pulling fs layer 2024-08-20T22:08:18.3628257Z f1d6874dee9c: Pulling fs layer 2024-08-20T22:08:18.3628632Z afc429e87b59: Pulling fs layer 2024-08-20T22:08:18.3628980Z 7754ee068b33: Waiting 2024-08-20T22:08:18.3629289Z 0479fbac3361: Waiting 2024-08-20T22:08:18.3629606Z f1d6874dee9c: Waiting 2024-08-20T22:08:18.3629930Z 019620471c25: Pulling fs layer 2024-08-20T22:08:18.3630299Z 27aeb1b2cd5e: Pulling fs layer 2024-08-20T22:08:18.3630671Z 829c0e19f4b7: Pulling fs layer 2024-08-20T22:08:18.3631040Z 560cd6122a38: Pulling fs layer 2024-08-20T22:08:18.3631394Z afc429e87b59: Waiting 2024-08-20T22:08:18.3631705Z 019620471c25: Waiting 2024-08-20T22:08:18.3632014Z 27aeb1b2cd5e: Waiting 2024-08-20T22:08:18.3632540Z 5a4eb19d7219: Pulling fs layer 2024-08-20T22:08:18.3632895Z 829c0e19f4b7: Waiting 2024-08-20T22:08:18.3633224Z da6a838a66c7: Pulling fs layer 2024-08-20T22:08:18.3633595Z cbb79a705353: Pulling fs layer 2024-08-20T22:08:18.3633959Z 4d4fa6802ab3: Pulling fs layer 2024-08-20T22:08:18.3634327Z 0f085d4fa279: Pulling fs layer 2024-08-20T22:08:18.3634682Z 560cd6122a38: Waiting 2024-08-20T22:08:18.3634989Z da6a838a66c7: Waiting 2024-08-20T22:08:18.3635305Z 4a0412373305: Pulling fs layer 2024-08-20T22:08:18.3635655Z cbb79a705353: Waiting 2024-08-20T22:08:18.3635968Z 5a4eb19d7219: Waiting 2024-08-20T22:08:18.3636275Z 0f085d4fa279: Waiting 2024-08-20T22:08:18.3636627Z 4d4fa6802ab3: Waiting 2024-08-20T22:08:18.3636953Z 302c8c177944: Pulling fs layer 2024-08-20T22:08:18.3637304Z 4a0412373305: Waiting 2024-08-20T22:08:18.3637749Z 02970347dff6: Pulling fs layer 2024-08-20T22:08:18.3638099Z 302c8c177944: Waiting 2024-08-20T22:08:18.3638419Z 916c5e858016: Pulling fs layer 2024-08-20T22:08:18.3638786Z c42b3602a1fb: Pulling fs layer 2024-08-20T22:08:18.3639156Z 11fad972ef6c: Pulling fs layer 2024-08-20T22:08:18.3639510Z 02970347dff6: Waiting 2024-08-20T22:08:18.3639832Z f957c1624481: Pulling fs layer 2024-08-20T22:08:18.3640174Z 916c5e858016: Waiting 2024-08-20T22:08:18.3640485Z c42b3602a1fb: Waiting 2024-08-20T22:08:18.3640797Z 11fad972ef6c: Waiting 2024-08-20T22:08:18.3641122Z 768c31dca016: Pulling fs layer 2024-08-20T22:08:18.3641471Z f957c1624481: Waiting 2024-08-20T22:08:18.3641801Z 8c32d0ecebbe: Pulling fs layer 2024-08-20T22:08:18.3642173Z 4f4fb700ef54: Pulling fs layer 2024-08-20T22:08:18.3642522Z 768c31dca016: Waiting 2024-08-20T22:08:18.3642843Z 78ee77eddef2: Pulling fs layer 2024-08-20T22:08:18.3643199Z 4f4fb700ef54: Waiting 2024-08-20T22:08:18.3643512Z 78ee77eddef2: Waiting 2024-08-20T22:08:18.3643834Z 37513e9b43d7: Pulling fs layer 2024-08-20T22:08:18.3644207Z a7a5392e2bf6: Pulling fs layer 2024-08-20T22:08:18.3644560Z 37513e9b43d7: Waiting 2024-08-20T22:08:18.3644885Z b10de02255c0: Pulling fs layer 2024-08-20T22:08:18.3645236Z a7a5392e2bf6: Waiting 2024-08-20T22:08:18.3645564Z e56eeff2db93: Pulling fs layer 2024-08-20T22:08:18.3645917Z b10de02255c0: Waiting 2024-08-20T22:08:18.3646239Z 70ac3e38d0fd: Pulling fs layer 2024-08-20T22:08:18.3646611Z eb09cbf3e416: Pulling fs layer 2024-08-20T22:08:18.3646973Z e56eeff2db93: Waiting 2024-08-20T22:08:18.3647299Z ee1273153675: Pulling fs layer 2024-08-20T22:08:18.3647646Z 70ac3e38d0fd: Waiting 2024-08-20T22:08:18.3647974Z e22979f9e0ee: Pulling fs layer 2024-08-20T22:08:18.3648329Z ee1273153675: Waiting 2024-08-20T22:08:18.3648638Z eb09cbf3e416: Waiting 2024-08-20T22:08:18.3648962Z 6c1f0aabce7c: Pulling fs layer 2024-08-20T22:08:18.3649316Z e22979f9e0ee: Waiting 2024-08-20T22:08:18.3649641Z 7ebe49c3a23e: Pulling fs layer 2024-08-20T22:08:18.3650005Z 5d115b29b5d5: Pulling fs layer 2024-08-20T22:08:18.3650380Z 7f2d7a04fbe6: Pulling fs layer 2024-08-20T22:08:18.3650753Z d79fd7d52698: Pulling fs layer 2024-08-20T22:08:18.3651125Z 9752d6fbbcab: Pulling fs layer 2024-08-20T22:08:18.3651495Z 83808b2eff4d: Pulling fs layer 2024-08-20T22:08:18.3651930Z 5d115b29b5d5: Waiting 2024-08-20T22:08:18.3652245Z 7ebe49c3a23e: Waiting 2024-08-20T22:08:18.3727411Z 83808b2eff4d: Waiting 2024-08-20T22:08:18.3727871Z d79fd7d52698: Waiting 2024-08-20T22:08:18.3728206Z 6c1f0aabce7c: Waiting 2024-08-20T22:08:18.3728535Z 7f2d7a04fbe6: Waiting 2024-08-20T22:08:18.4984059Z 224fe954d725: Verifying Checksum 2024-08-20T22:08:18.4984601Z 224fe954d725: Download complete 2024-08-20T22:08:18.5845928Z d527cbbb87e3: Download complete 2024-08-20T22:08:18.7027029Z 7a2c55901189: Verifying Checksum 2024-08-20T22:08:18.7027567Z 7a2c55901189: Download complete 2024-08-20T22:08:18.7359729Z b57676e46aee: Verifying Checksum 2024-08-20T22:08:18.7360297Z b57676e46aee: Download complete 2024-08-20T22:08:18.8040678Z a41a8d1c11c8: Download complete 2024-08-20T22:08:18.8883612Z 0c1227890755: Verifying Checksum 2024-08-20T22:08:18.8884068Z 0c1227890755: Download complete 2024-08-20T22:08:18.9639123Z d8d1234baab3: Download complete 2024-08-20T22:08:18.9840476Z 75722010b82e: Verifying Checksum 2024-08-20T22:08:18.9840940Z 75722010b82e: Download complete 2024-08-20T22:08:19.0684586Z ec1e7978c1fe: Verifying Checksum 2024-08-20T22:08:19.1407786Z ac23c5fcc1ef: Download complete 2024-08-20T22:08:19.8815464Z 7a2c55901189: Pull complete 2024-08-20T22:08:20.1931293Z 224fe954d725: Pull complete 2024-08-20T22:08:21.0982315Z 75722010b82e: Pull complete 2024-08-20T22:08:21.1156707Z d527cbbb87e3: Pull complete 2024-08-20T22:08:21.1340158Z b57676e46aee: Pull complete 2024-08-20T22:08:21.7002241Z ca426ffc9d2b: Verifying Checksum 2024-08-20T22:08:21.7002849Z ca426ffc9d2b: Download complete 2024-08-20T22:08:21.7688970Z 4373aa743caf: Download complete 2024-08-20T22:08:21.8567486Z 75b2abb16c68: Verifying Checksum 2024-08-20T22:08:21.8568043Z 75b2abb16c68: Download complete 2024-08-20T22:08:21.9423371Z ef1e3ef1e85f: Verifying Checksum 2024-08-20T22:08:21.9423895Z ef1e3ef1e85f: Download complete 2024-08-20T22:08:22.9082636Z 0bf61114d37b: Verifying Checksum 2024-08-20T22:08:22.9083154Z 0bf61114d37b: Download complete 2024-08-20T22:08:22.9949192Z 2df849f72ba4: Download complete 2024-08-20T22:08:23.0733142Z 34c47d63c5de: Verifying Checksum 2024-08-20T22:08:23.0733748Z 34c47d63c5de: Download complete 2024-08-20T22:08:23.1462477Z 1fef4bd19792: Download complete 2024-08-20T22:08:32.0652690Z a8c1e85b5e14: Verifying Checksum 2024-08-20T22:08:32.0653270Z a8c1e85b5e14: Download complete 2024-08-20T22:08:32.1422295Z b119c4fc6c58: Download complete 2024-08-20T22:08:32.2142560Z 33b639226eee: Download complete 2024-08-20T22:08:32.2952052Z d3547bd43146: Verifying Checksum 2024-08-20T22:08:32.2952657Z d3547bd43146: Download complete 2024-08-20T22:08:32.3662911Z 5645520256fa: Download complete 2024-08-20T22:08:32.4399695Z 829ed7d91e38: Verifying Checksum 2024-08-20T22:08:32.4400337Z 829ed7d91e38: Download complete 2024-08-20T22:08:33.7071032Z 385d0421d67c: Verifying Checksum 2024-08-20T22:08:33.7071620Z 385d0421d67c: Download complete 2024-08-20T22:08:33.7795459Z 7cbfce999d76: Download complete 2024-08-20T22:08:33.8500647Z 6658e558ebe4: Verifying Checksum 2024-08-20T22:08:33.8501297Z 6658e558ebe4: Download complete 2024-08-20T22:08:33.9233904Z 33444a5a7e01: Verifying Checksum 2024-08-20T22:08:33.9234500Z 33444a5a7e01: Download complete 2024-08-20T22:08:33.9918531Z 7c47122f4e09: Verifying Checksum 2024-08-20T22:08:33.9919180Z 7c47122f4e09: Download complete 2024-08-20T22:08:34.0600890Z 1811ff4d3556: Verifying Checksum 2024-08-20T22:08:34.0601525Z 1811ff4d3556: Download complete 2024-08-20T22:08:40.0616152Z 7754ee068b33: Verifying Checksum 2024-08-20T22:08:40.0616749Z 7754ee068b33: Download complete 2024-08-20T22:08:40.1333426Z 0479fbac3361: Verifying Checksum 2024-08-20T22:08:40.1334095Z 0479fbac3361: Download complete 2024-08-20T22:08:40.2032309Z f1d6874dee9c: Verifying Checksum 2024-08-20T22:08:40.2032925Z f1d6874dee9c: Download complete 2024-08-20T22:08:40.6170651Z afc429e87b59: Verifying Checksum 2024-08-20T22:08:40.6171217Z afc429e87b59: Download complete 2024-08-20T22:08:40.6843456Z 019620471c25: Download complete 2024-08-20T22:08:40.7629515Z 27aeb1b2cd5e: Verifying Checksum 2024-08-20T22:08:40.7630172Z 27aeb1b2cd5e: Download complete 2024-08-20T22:08:41.0272802Z 829c0e19f4b7: Verifying Checksum 2024-08-20T22:08:41.0273506Z 829c0e19f4b7: Download complete 2024-08-20T22:08:41.0992445Z 560cd6122a38: Verifying Checksum 2024-08-20T22:08:41.0993002Z 560cd6122a38: Download complete 2024-08-20T22:08:41.1622172Z 5a4eb19d7219: Verifying Checksum 2024-08-20T22:08:41.1622804Z 5a4eb19d7219: Download complete 2024-08-20T22:08:41.2470564Z da6a838a66c7: Download complete 2024-08-20T22:08:44.3021227Z 7ed32bc8e469: Download complete 2024-08-20T22:08:44.3719751Z 4d4fa6802ab3: Verifying Checksum 2024-08-20T22:08:44.3720321Z 4d4fa6802ab3: Download complete 2024-08-20T22:08:44.4458061Z 0f085d4fa279: Download complete 2024-08-20T22:08:44.9335661Z 4a0412373305: Verifying Checksum 2024-08-20T22:08:44.9336243Z 4a0412373305: Download complete 2024-08-20T22:08:45.0322428Z 302c8c177944: Verifying Checksum 2024-08-20T22:08:45.0323194Z 302c8c177944: Download complete 2024-08-20T22:08:45.1032126Z 02970347dff6: Verifying Checksum 2024-08-20T22:08:45.1032712Z 02970347dff6: Download complete 2024-08-20T22:08:45.1752590Z a8c1e85b5e14: Pull complete 2024-08-20T22:08:45.1799739Z 916c5e858016: Verifying Checksum 2024-08-20T22:08:45.1800222Z 916c5e858016: Download complete 2024-08-20T22:08:45.2564774Z c42b3602a1fb: Download complete 2024-08-20T22:08:45.4067964Z a41a8d1c11c8: Pull complete 2024-08-20T22:08:45.6414658Z 0c1227890755: Pull complete 2024-08-20T22:08:45.8409061Z d8d1234baab3: Pull complete 2024-08-20T22:08:51.8468946Z 6efb96fc87ea: Verifying Checksum 2024-08-20T22:08:51.8469575Z 6efb96fc87ea: Download complete 2024-08-20T22:08:51.9333870Z f957c1624481: Verifying Checksum 2024-08-20T22:08:51.9334438Z f957c1624481: Download complete 2024-08-20T22:08:52.0134341Z 768c31dca016: Download complete 2024-08-20T22:08:52.0956495Z 8c32d0ecebbe: Verifying Checksum 2024-08-20T22:08:52.0957157Z 8c32d0ecebbe: Download complete 2024-08-20T22:08:52.1027217Z 4f4fb700ef54: Verifying Checksum 2024-08-20T22:08:52.1027822Z 4f4fb700ef54: Download complete 2024-08-20T22:08:52.1657925Z 78ee77eddef2: Verifying Checksum 2024-08-20T22:08:52.1658497Z 78ee77eddef2: Download complete 2024-08-20T22:08:52.2577259Z 37513e9b43d7: Verifying Checksum 2024-08-20T22:08:52.2577712Z 37513e9b43d7: Download complete 2024-08-20T22:08:52.3844890Z a7a5392e2bf6: Verifying Checksum 2024-08-20T22:08:52.3845396Z a7a5392e2bf6: Download complete 2024-08-20T22:08:52.4755708Z b10de02255c0: Verifying Checksum 2024-08-20T22:08:52.4756326Z b10de02255c0: Download complete 2024-08-20T22:08:52.5555430Z e56eeff2db93: Verifying Checksum 2024-08-20T22:08:52.5556047Z e56eeff2db93: Download complete 2024-08-20T22:08:52.6311860Z 70ac3e38d0fd: Verifying Checksum 2024-08-20T22:08:52.6312347Z 70ac3e38d0fd: Download complete 2024-08-20T22:08:52.6983359Z eb09cbf3e416: Verifying Checksum 2024-08-20T22:08:52.6983943Z eb09cbf3e416: Download complete 2024-08-20T22:08:52.8378799Z ee1273153675: Verifying Checksum 2024-08-20T22:08:52.8379466Z ee1273153675: Download complete 2024-08-20T22:08:52.8936718Z e22979f9e0ee: Verifying Checksum 2024-08-20T22:08:52.8937331Z e22979f9e0ee: Download complete 2024-08-20T22:08:53.4847023Z 6c1f0aabce7c: Verifying Checksum 2024-08-20T22:08:53.4847628Z 6c1f0aabce7c: Download complete 2024-08-20T22:08:53.5618511Z 7ebe49c3a23e: Verifying Checksum 2024-08-20T22:08:53.5619127Z 7ebe49c3a23e: Download complete 2024-08-20T22:09:21.0229489Z 5d115b29b5d5: Verifying Checksum 2024-08-20T22:09:21.0231791Z 5d115b29b5d5: Download complete 2024-08-20T22:09:21.1048886Z 7f2d7a04fbe6: Download complete 2024-08-20T22:09:21.1830099Z d79fd7d52698: Verifying Checksum 2024-08-20T22:09:21.1830577Z d79fd7d52698: Download complete 2024-08-20T22:09:22.7148715Z 9752d6fbbcab: Verifying Checksum 2024-08-20T22:09:22.7149280Z 9752d6fbbcab: Download complete 2024-08-20T22:09:22.7850818Z 83808b2eff4d: Verifying Checksum 2024-08-20T22:09:22.7851261Z 83808b2eff4d: Download complete 2024-08-20T22:10:00.7979477Z 11fad972ef6c: Download complete 2024-08-20T22:10:01.9846006Z cbb79a705353: Download complete 2024-08-20T22:10:17.5928697Z 7ed32bc8e469: Pull complete 2024-08-20T22:10:17.8108549Z ec1e7978c1fe: Pull complete 2024-08-20T22:10:18.0230887Z ac23c5fcc1ef: Pull complete 2024-08-20T22:10:26.5486507Z ca426ffc9d2b: Pull complete 2024-08-20T22:10:26.6074973Z 4373aa743caf: Pull complete 2024-08-20T22:10:26.6875821Z 75b2abb16c68: Pull complete 2024-08-20T22:10:26.8052852Z ef1e3ef1e85f: Pull complete 2024-08-20T22:10:29.4033011Z 0bf61114d37b: Pull complete 2024-08-20T22:10:29.6270258Z 2df849f72ba4: Pull complete 2024-08-20T22:10:29.8426378Z 34c47d63c5de: Pull complete 2024-08-20T22:10:30.0587855Z 1fef4bd19792: Pull complete 2024-08-20T22:11:28.1319898Z 6efb96fc87ea: Pull complete 2024-08-20T22:11:28.3546646Z b119c4fc6c58: Pull complete 2024-08-20T22:11:28.5856947Z 33b639226eee: Pull complete 2024-08-20T22:11:28.8050590Z d3547bd43146: Pull complete 2024-08-20T22:11:29.0345577Z 5645520256fa: Pull complete 2024-08-20T22:11:29.2602621Z 829ed7d91e38: Pull complete 2024-08-20T22:11:32.8071469Z 385d0421d67c: Pull complete 2024-08-20T22:11:32.9293127Z 7cbfce999d76: Pull complete 2024-08-20T22:11:33.1535802Z 6658e558ebe4: Pull complete 2024-08-20T22:11:33.3605390Z 33444a5a7e01: Pull complete 2024-08-20T22:11:33.5796159Z 7c47122f4e09: Pull complete 2024-08-20T22:11:33.7901523Z 1811ff4d3556: Pull complete 2024-08-20T22:11:44.4148328Z 7754ee068b33: Pull complete 2024-08-20T22:11:44.6171250Z 0479fbac3361: Pull complete 2024-08-20T22:11:44.8021756Z f1d6874dee9c: Pull complete 2024-08-20T22:11:45.7237523Z afc429e87b59: Pull complete 2024-08-20T22:11:45.8629005Z 019620471c25: Pull complete 2024-08-20T22:11:45.9973552Z 27aeb1b2cd5e: Pull complete 2024-08-20T22:11:46.4355949Z 829c0e19f4b7: Pull complete 2024-08-20T22:11:46.5685800Z 560cd6122a38: Pull complete 2024-08-20T22:11:46.8252139Z 5a4eb19d7219: Pull complete 2024-08-20T22:11:46.9654956Z da6a838a66c7: Pull complete 2024-08-20T22:12:25.5924620Z cbb79a705353: Pull complete 2024-08-20T22:12:25.8183301Z 4d4fa6802ab3: Pull complete 2024-08-20T22:12:26.0438984Z 0f085d4fa279: Pull complete 2024-08-20T22:12:28.4445107Z 4a0412373305: Pull complete 2024-08-20T22:12:28.4629900Z 302c8c177944: Pull complete 2024-08-20T22:12:28.4822994Z 02970347dff6: Pull complete 2024-08-20T22:12:28.5200205Z 916c5e858016: Pull complete 2024-08-20T22:12:28.5394346Z c42b3602a1fb: Pull complete 2024-08-20T22:13:50.4038426Z 11fad972ef6c: Pull complete 2024-08-20T22:13:50.6291426Z f957c1624481: Pull complete 2024-08-20T22:13:50.8622060Z 768c31dca016: Pull complete 2024-08-20T22:13:51.2588643Z 8c32d0ecebbe: Pull complete 2024-08-20T22:13:51.4779774Z 4f4fb700ef54: Pull complete 2024-08-20T22:13:51.7002080Z 78ee77eddef2: Pull complete 2024-08-20T22:13:51.9215024Z 37513e9b43d7: Pull complete 2024-08-20T22:13:52.3169689Z a7a5392e2bf6: Pull complete 2024-08-20T22:13:52.5398957Z b10de02255c0: Pull complete 2024-08-20T22:13:52.7561174Z e56eeff2db93: Pull complete 2024-08-20T22:13:52.9226055Z 70ac3e38d0fd: Pull complete 2024-08-20T22:13:53.1028809Z eb09cbf3e416: Pull complete 2024-08-20T22:13:54.5095537Z ee1273153675: Pull complete 2024-08-20T22:13:54.7315853Z e22979f9e0ee: Pull complete 2024-08-20T22:13:56.9358003Z 6c1f0aabce7c: Pull complete 2024-08-20T22:13:57.1573242Z 7ebe49c3a23e: Pull complete 2024-08-20T22:14:10.5748565Z 5d115b29b5d5: Pull complete 2024-08-20T22:14:10.8000239Z 7f2d7a04fbe6: Pull complete 2024-08-20T22:14:11.0157543Z d79fd7d52698: Pull complete 2024-08-20T22:14:11.8966042Z 9752d6fbbcab: Pull complete 2024-08-20T22:14:12.1279615Z 83808b2eff4d: Pull complete 2024-08-20T22:14:13.1350303Z Digest: sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T22:14:13.1830761Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:14:13.2024136Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:14:13.2116219Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:14:13.2117145Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:14:13.2129161Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:14:13.2129668Z env: 2024-08-20T22:14:13.2129939Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:14:13.2130293Z ##[endgroup] 2024-08-20T22:14:13.2314759Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-08-20T22:14:13.2315306Z with: 2024-08-20T22:14:13.2315588Z driver-version: 550.54.15 2024-08-20T22:14:13.2315947Z env: 2024-08-20T22:14:13.2316241Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:14:13.2316597Z ##[endgroup] 2024-08-20T22:14:13.2397536Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:14:13.2398082Z with: 2024-08-20T22:14:13.2398356Z timeout_minutes: 10 2024-08-20T22:14:13.2398677Z max_attempts: 3 2024-08-20T22:14:13.2432014Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-08-20T22:14:13.2465271Z retry_wait_seconds: 10 2024-08-20T22:14:13.2465632Z polling_interval_seconds: 1 2024-08-20T22:14:13.2466001Z warning_on_retry: true 2024-08-20T22:14:13.2466351Z continue_on_error: false 2024-08-20T22:14:13.2466681Z env: 2024-08-20T22:14:13.2466950Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:14:13.2467306Z DRIVER_VERSION: 550.54.15 2024-08-20T22:14:13.2467653Z ##[endgroup] 2024-08-20T22:14:13.3281141Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-08-20T22:14:13.3282232Z + pre_install_nvidia_driver_amzn2 2024-08-20T22:14:13.3283547Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-08-20T22:14:13.7762571Z No match for argument: nvidia-driver-latest-dkms 2024-08-20T22:14:13.7763200Z No packages marked for removal. 2024-08-20T22:14:13.7825127Z Dependencies resolved. 2024-08-20T22:14:13.7836331Z Nothing to do. 2024-08-20T22:14:13.7836768Z Complete! 2024-08-20T22:14:13.8792408Z + install_nvidia_driver_common 2024-08-20T22:14:13.8796365Z + echo 'Before installing NVIDIA driver' 2024-08-20T22:14:13.8797015Z + lspci 2024-08-20T22:14:13.8797416Z Before installing NVIDIA driver 2024-08-20T22:14:13.9796342Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:14:13.9797110Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:14:13.9798245Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:14:13.9799292Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:14:13.9800130Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:14:13.9800928Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:14:13.9801660Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:14:13.9802466Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:14:13.9803051Z + lsmod 2024-08-20T22:14:13.9845823Z Module Size Used by 2024-08-20T22:14:13.9846385Z xt_nat 16384 0 2024-08-20T22:14:13.9846834Z nvidia_modeset 1351680 0 2024-08-20T22:14:13.9847500Z video 65536 1 nvidia_modeset 2024-08-20T22:14:13.9847991Z wmi 36864 1 video 2024-08-20T22:14:13.9848394Z nvidia_uvm 4706304 0 2024-08-20T22:14:13.9848846Z nvidia 54071296 2 nvidia_uvm,nvidia_modeset 2024-08-20T22:14:13.9849370Z drm 602112 1 nvidia 2024-08-20T22:14:13.9849980Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T22:14:13.9850656Z backlight 24576 3 video,drm,nvidia_modeset 2024-08-20T22:14:13.9851179Z i2c_core 106496 2 nvidia,drm 2024-08-20T22:14:13.9851595Z veth 36864 0 2024-08-20T22:14:13.9851969Z xt_conntrack 16384 1 2024-08-20T22:14:13.9852343Z nft_chain_nat 16384 3 2024-08-20T22:14:13.9852740Z xt_MASQUERADE 20480 1 2024-08-20T22:14:13.9853502Z nf_nat 57344 3 xt_nat,nft_chain_nat,xt_MASQUERADE 2024-08-20T22:14:13.9854178Z nf_conntrack_netlink 57344 0 2024-08-20T22:14:13.9854980Z nf_conntrack 184320 5 xt_conntrack,nf_nat,xt_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:14:13.9855840Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:14:13.9856677Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:14:13.9857228Z xfrm_user 57344 1 2024-08-20T22:14:13.9857692Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:14:13.9858109Z xt_addrtype 16384 2 2024-08-20T22:14:13.9858489Z nft_compat 20480 4 2024-08-20T22:14:13.9858943Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:14:13.9859551Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:14:13.9860084Z br_netfilter 36864 0 2024-08-20T22:14:13.9860504Z bridge 307200 1 br_netfilter 2024-08-20T22:14:13.9860939Z stp 16384 1 bridge 2024-08-20T22:14:13.9861368Z llc 16384 2 bridge,stp 2024-08-20T22:14:13.9861780Z overlay 167936 0 2024-08-20T22:14:13.9862154Z tls 114688 0 2024-08-20T22:14:13.9862523Z nls_ascii 16384 1 2024-08-20T22:14:13.9862896Z nls_cp437 20480 1 2024-08-20T22:14:13.9863262Z vfat 24576 1 2024-08-20T22:14:13.9863631Z fat 86016 1 vfat 2024-08-20T22:14:13.9864021Z ena 167936 0 2024-08-20T22:14:13.9864396Z ptp 36864 1 ena 2024-08-20T22:14:13.9864785Z pps_core 24576 1 ptp 2024-08-20T22:14:13.9865187Z ghash_clmulni_intel 16384 0 2024-08-20T22:14:13.9865566Z aesni_intel 393216 0 2024-08-20T22:14:13.9865929Z i8042 45056 0 2024-08-20T22:14:13.9866453Z serio 28672 3 i8042 2024-08-20T22:14:13.9867077Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:14:13.9867864Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:14:13.9868476Z button 24576 0 2024-08-20T22:14:13.9868984Z sunrpc 692224 1 2024-08-20T22:14:13.9869570Z sch_fq_codel 20480 17 2024-08-20T22:14:13.9869984Z dm_mod 188416 0 2024-08-20T22:14:13.9870343Z fuse 163840 1 2024-08-20T22:14:13.9870726Z dax 45056 1 dm_mod 2024-08-20T22:14:13.9871127Z loop 36864 0 2024-08-20T22:14:13.9871496Z configfs 57344 1 2024-08-20T22:14:13.9871887Z dmi_sysfs 20480 0 2024-08-20T22:14:13.9872423Z crc32_pclmul 16384 0 2024-08-20T22:14:13.9872974Z crc32c_intel 24576 0 2024-08-20T22:14:13.9873505Z efivarfs 24576 1 2024-08-20T22:14:13.9873882Z + modinfo nvidia 2024-08-20T22:14:13.9874588Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:14:13.9875606Z alias: char-major-195-* 2024-08-20T22:14:13.9876209Z version: 550.54.15 2024-08-20T22:14:13.9876614Z supported: external 2024-08-20T22:14:13.9876955Z license: NVIDIA 2024-08-20T22:14:13.9877332Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:14:13.9877809Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:14:13.9878410Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:14:13.9879048Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:14:13.9879553Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:14:13.9880031Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:14:13.9880565Z depends: i2c-core,drm 2024-08-20T22:14:13.9880925Z retpoline: Y 2024-08-20T22:14:13.9881232Z name: nvidia 2024-08-20T22:14:13.9881840Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:14:13.9882542Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:14:13.9883207Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:14:13.9883804Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:14:13.9884264Z parm: NVreg_RmLogonRC:int 2024-08-20T22:14:13.9884708Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:14:13.9885164Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:14:13.9885724Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:14:13.9886163Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:14:13.9886684Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:14:13.9887248Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:14:13.9887728Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:14:13.9888251Z parm: NVreg_EnableMSI:int 2024-08-20T22:14:13.9888663Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:14:13.9889119Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:14:13.9889646Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:14:13.9890237Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:14:13.9890793Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:14:13.9891400Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:14:13.9892006Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:14:13.9892634Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:14:13.9893223Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:14:13.9893802Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:14:13.9894332Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:14:13.9894876Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:14:13.9895370Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:14:13.9895839Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:14:13.9896327Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:14:13.9896805Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:14:13.9897260Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:14:13.9897763Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:14:13.9898285Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:14:13.9898756Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:14:13.9899241Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:14:13.9899750Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:14:13.9900234Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:14:13.9900723Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:14:13.9901208Z parm: NVreg_RmMsg:charp 2024-08-20T22:14:13.9901618Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:14:13.9902086Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:14:13.9902552Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:14:13.9903005Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:14:13.9903486Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:14:13.9904068Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:14:13.9904515Z parm: rm_firmware_active:charp 2024-08-20T22:14:13.9904927Z + HAS_NVIDIA_DRIVER=0 2024-08-20T22:14:13.9905323Z ++ command -v nvidia-smi 2024-08-20T22:14:13.9905733Z + '[' -x /usr/bin/nvidia-smi ']' 2024-08-20T22:14:13.9906097Z + set +e 2024-08-20T22:14:13.9906613Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-08-20T22:14:16.0365332Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-08-20T22:14:16.0365980Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:14:16.0366634Z + '[' 0 -ne 0 ']' 2024-08-20T22:14:16.0367180Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-08-20T22:14:16.0367708Z + HAS_NVIDIA_DRIVER=1 2024-08-20T22:14:16.0368469Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-08-20T22:14:16.0369181Z + set -e 2024-08-20T22:14:16.0369486Z + '[' 1 -eq 0 ']' 2024-08-20T22:14:16.0370184Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-08-20T22:14:16.0371153Z + post_install_nvidia_driver_common 2024-08-20T22:14:16.0371727Z + sudo modprobe nvidia 2024-08-20T22:14:16.1726620Z + echo 'After installing NVIDIA driver' 2024-08-20T22:14:16.1727344Z + lspci 2024-08-20T22:14:16.1727641Z After installing NVIDIA driver 2024-08-20T22:14:16.1833947Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:14:16.1834811Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:14:16.1835945Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:14:16.1836884Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:14:16.1837719Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:14:16.1838502Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:14:16.1839240Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:14:16.1840061Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:14:16.1840651Z + lsmod 2024-08-20T22:14:16.1867366Z Module Size Used by 2024-08-20T22:14:16.1867825Z xt_nat 16384 0 2024-08-20T22:14:16.1868252Z nvidia_modeset 1351680 0 2024-08-20T22:14:16.1868847Z video 65536 1 nvidia_modeset 2024-08-20T22:14:16.1869414Z wmi 36864 1 video 2024-08-20T22:14:16.1869910Z nvidia_uvm 4706304 0 2024-08-20T22:14:16.1870467Z nvidia 54071296 2 nvidia_uvm,nvidia_modeset 2024-08-20T22:14:16.1870950Z drm 602112 1 nvidia 2024-08-20T22:14:16.1871394Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T22:14:16.1871918Z backlight 24576 3 video,drm,nvidia_modeset 2024-08-20T22:14:16.1872430Z i2c_core 106496 2 nvidia,drm 2024-08-20T22:14:16.1872849Z veth 36864 0 2024-08-20T22:14:16.1873218Z xt_conntrack 16384 1 2024-08-20T22:14:16.1873602Z nft_chain_nat 16384 3 2024-08-20T22:14:16.1874009Z xt_MASQUERADE 20480 1 2024-08-20T22:14:16.1874485Z nf_nat 57344 3 xt_nat,nft_chain_nat,xt_MASQUERADE 2024-08-20T22:14:16.1874995Z nf_conntrack_netlink 57344 0 2024-08-20T22:14:16.1875608Z nf_conntrack 184320 5 xt_conntrack,nf_nat,xt_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:14:16.1876268Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:14:16.1876740Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:14:16.1877199Z xfrm_user 57344 1 2024-08-20T22:14:16.1877609Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:14:16.1878025Z xt_addrtype 16384 2 2024-08-20T22:14:16.1878403Z nft_compat 20480 4 2024-08-20T22:14:16.1878851Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:14:16.1879465Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:14:16.1880221Z br_netfilter 36864 0 2024-08-20T22:14:16.1880638Z bridge 307200 1 br_netfilter 2024-08-20T22:14:16.1881074Z stp 16384 1 bridge 2024-08-20T22:14:16.1881494Z llc 16384 2 bridge,stp 2024-08-20T22:14:16.1881911Z overlay 167936 0 2024-08-20T22:14:16.1882282Z tls 114688 0 2024-08-20T22:14:16.1882645Z nls_ascii 16384 1 2024-08-20T22:14:16.1883017Z nls_cp437 20480 1 2024-08-20T22:14:16.1883380Z vfat 24576 1 2024-08-20T22:14:16.1883745Z fat 86016 1 vfat 2024-08-20T22:14:16.1884126Z ena 167936 0 2024-08-20T22:14:16.1884495Z ptp 36864 1 ena 2024-08-20T22:14:16.1884889Z pps_core 24576 1 ptp 2024-08-20T22:14:16.1885305Z ghash_clmulni_intel 16384 0 2024-08-20T22:14:16.1885805Z aesni_intel 393216 0 2024-08-20T22:14:16.1886282Z i8042 45056 0 2024-08-20T22:14:16.1886772Z serio 28672 3 i8042 2024-08-20T22:14:16.1887355Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:14:16.1887937Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:14:16.1888432Z button 24576 0 2024-08-20T22:14:16.1888798Z sunrpc 692224 1 2024-08-20T22:14:16.1889318Z sch_fq_codel 20480 17 2024-08-20T22:14:16.1889701Z dm_mod 188416 0 2024-08-20T22:14:16.1890067Z fuse 163840 1 2024-08-20T22:14:16.1890439Z dax 45056 1 dm_mod 2024-08-20T22:14:16.1890838Z loop 36864 0 2024-08-20T22:14:16.1891209Z configfs 57344 1 2024-08-20T22:14:16.1891588Z dmi_sysfs 20480 0 2024-08-20T22:14:16.1891954Z crc32_pclmul 16384 0 2024-08-20T22:14:16.1892330Z crc32c_intel 24576 0 2024-08-20T22:14:16.1892702Z efivarfs 24576 1 2024-08-20T22:14:16.1893064Z + modinfo nvidia 2024-08-20T22:14:16.1893882Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:14:16.1894577Z alias: char-major-195-* 2024-08-20T22:14:16.1894952Z version: 550.54.15 2024-08-20T22:14:16.1895299Z supported: external 2024-08-20T22:14:16.1895632Z license: NVIDIA 2024-08-20T22:14:16.1896011Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:14:16.1896490Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:14:16.1896943Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:14:16.1897423Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:14:16.1897943Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:14:16.1898429Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:14:16.1898925Z depends: i2c-core,drm 2024-08-20T22:14:16.1899274Z retpoline: Y 2024-08-20T22:14:16.1899578Z name: nvidia 2024-08-20T22:14:16.1900185Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:14:16.1900877Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:14:16.1901514Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:14:16.1902099Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:14:16.1902539Z parm: NVreg_RmLogonRC:int 2024-08-20T22:14:16.1902966Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:14:16.1903421Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:14:16.1903856Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:14:16.1904295Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:14:16.1904811Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:14:16.1905363Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:14:16.1905841Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:14:16.1906266Z parm: NVreg_EnableMSI:int 2024-08-20T22:14:16.1906670Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:14:16.1907285Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:14:16.1907808Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:14:16.1908379Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:14:16.1908927Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:14:16.1909536Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:14:16.1910128Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:14:16.1910734Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:14:16.1911314Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:14:16.1911802Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:14:16.1912395Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:14:16.1912979Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:14:16.1913463Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:14:16.1913921Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:14:16.1914405Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:14:16.1914864Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:14:16.1915306Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:14:16.1915803Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:14:16.1916424Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:14:16.1916941Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:14:16.1917465Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:14:16.1917958Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:14:16.1918436Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:14:16.1918920Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:14:16.1919392Z parm: NVreg_RmMsg:charp 2024-08-20T22:14:16.1919802Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:14:16.1920260Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:14:16.1920717Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:14:16.1921179Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:14:16.1921651Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:14:16.1922119Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:14:16.1922561Z parm: rm_firmware_active:charp 2024-08-20T22:14:16.1922960Z + set +e 2024-08-20T22:14:16.1923274Z + nvidia-smi 2024-08-20T22:14:17.7723368Z Tue Aug 20 22:14:17 2024 2024-08-20T22:14:17.7724394Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:14:17.7725241Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:14:17.7726026Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:14:17.7727128Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:14:17.7728078Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:14:17.7728774Z | | | MIG M. | 2024-08-20T22:14:17.7729315Z |=========================================+========================+======================| 2024-08-20T22:14:17.7822185Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:14:17.7823159Z | 0% 30C P0 57W / 300W | 0MiB / 23028MiB | 5% Default | 2024-08-20T22:14:17.7823951Z | | | N/A | 2024-08-20T22:14:17.7824856Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:14:17.7825534Z 2024-08-20T22:14:17.7826209Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:14:17.7827411Z | Processes: | 2024-08-20T22:14:17.7828199Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:14:17.7828895Z | ID ID Usage | 2024-08-20T22:14:17.7829461Z |=========================================================================================| 2024-08-20T22:14:17.7830116Z | No running processes found | 2024-08-20T22:14:17.7830896Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:14:18.4005612Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-08-20T22:14:19.9802128Z NVIDIA A10G 2024-08-20T22:14:20.4394218Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:14:20.4394803Z + '[' 0 -eq 0 ']' 2024-08-20T22:14:20.4395220Z + echo 'INFO: Ignoring allowed status 0' 2024-08-20T22:14:20.4395659Z + set -e 2024-08-20T22:14:20.4395983Z INFO: Ignoring allowed status 0 2024-08-20T22:14:20.4402739Z == Installing nvidia container toolkit for amzn2023 == 2024-08-20T22:14:20.4406348Z + sudo yum install -y yum-utils 2024-08-20T22:14:20.8162364Z Last metadata expiration check: 0:29:45 ago on Tue Aug 20 21:44:35 2024. 2024-08-20T22:14:20.8387504Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-08-20T22:14:20.8693349Z Dependencies resolved. 2024-08-20T22:14:20.8821051Z Nothing to do. 2024-08-20T22:14:20.8821379Z Complete! 2024-08-20T22:14:20.9649663Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-08-20T22:14:20.9650848Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:14:20.9652155Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:14:21.2317541Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:14:21.3031760Z + sudo yum install -y nvidia-docker2 2024-08-20T22:14:21.8189960Z nvidia-container-toolkit 10 kB/s | 833 B 00:00 2024-08-20T22:14:21.8416310Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-08-20T22:14:21.8715545Z Dependencies resolved. 2024-08-20T22:14:21.8842014Z Nothing to do. 2024-08-20T22:14:21.8842334Z Complete! 2024-08-20T22:14:21.9869143Z + sudo systemctl restart docker 2024-08-20T22:15:16.2669460Z Tue Aug 20 22:15:16 2024 2024-08-20T22:15:16.2670550Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:15:16.2671408Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:15:16.2672207Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:15:16.2673022Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:15:16.2673948Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:15:16.2674689Z | | | MIG M. | 2024-08-20T22:15:16.2675256Z |=========================================+========================+======================| 2024-08-20T22:15:16.2784174Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:15:16.2784921Z | 0% 29C P0 59W / 300W | 0MiB / 23028MiB | 5% Default | 2024-08-20T22:15:16.2785694Z | | | N/A | 2024-08-20T22:15:16.2786611Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:15:16.2787208Z 2024-08-20T22:15:16.2788608Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:15:16.2789403Z | Processes: | 2024-08-20T22:15:16.2790236Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:15:16.2790941Z | ID ID Usage | 2024-08-20T22:15:16.2791506Z |=========================================================================================| 2024-08-20T22:15:16.2792165Z | No running processes found | 2024-08-20T22:15:16.2792948Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:15:17.3658754Z Command completed after 1 attempt(s). 2024-08-20T22:15:17.3742112Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:15:17.3742884Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:15:17.3743572Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-20T22:15:17.3744218Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:15:17.3757821Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:17.3758532Z env: 2024-08-20T22:15:17.3758812Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:17.3759270Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:17.3759759Z ##[endgroup] 2024-08-20T22:15:17.6863001Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:15:18.1029192Z Collecting psutil==5.9.1 2024-08-20T22:15:18.1425402Z Downloading psutil-5.9.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2024-08-20T22:15:18.1950642Z Collecting nvidia-ml-py==11.525.84 2024-08-20T22:15:18.2016008Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2024-08-20T22:15:18.2890194Z Installing collected packages: psutil, nvidia-ml-py 2024-08-20T22:15:18.4457500Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1 2024-08-20T22:15:18.6466335Z Prepare all required actions 2024-08-20T22:15:18.6467053Z Getting action download info 2024-08-20T22:15:18.7668161Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-20T22:15:18.9698810Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-20T22:15:19.0936018Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-20T22:15:19.0936504Z with: 2024-08-20T22:15:19.0936837Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:15:19.0937291Z s3-bucket: gha-artifacts 2024-08-20T22:15:19.0937626Z env: 2024-08-20T22:15:19.0937899Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:19.0938354Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:19.0938841Z ##[endgroup] 2024-08-20T22:15:19.0972506Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:15:19.0972958Z with: 2024-08-20T22:15:19.0973335Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:15:19.0973900Z s3-bucket: gha-artifacts 2024-08-20T22:15:19.0974249Z region: us-east-1 2024-08-20T22:15:19.0974558Z env: 2024-08-20T22:15:19.0974852Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:19.0975345Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:19.0975815Z ##[endgroup] 2024-08-20T22:15:19.5711915Z (node:57513) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:15:19.5712731Z 2024-08-20T22:15:19.5712989Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:15:19.5713717Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:15:19.5714592Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:15:19.6488572Z Found 1 objects with prefix pytorch/pytorch/10479310721/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-08-20T22:15:19.6490749Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:15:27.1780412Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:15:27.1787088Z Artifact download has finished successfully 2024-08-20T22:15:27.2143450Z ##[group]Run unzip -o artifacts.zip 2024-08-20T22:15:27.2143886Z unzip -o artifacts.zip 2024-08-20T22:15:27.2153822Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:27.2154320Z env: 2024-08-20T22:15:27.2154600Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:27.2155056Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:27.2155536Z ##[endgroup] 2024-08-20T22:15:27.2243778Z Archive: artifacts.zip 2024-08-20T22:15:27.2245400Z creating: dist/ 2024-08-20T22:15:29.3593890Z inflating: dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:15:29.3595232Z creating: build/custom_test_artifacts/ 2024-08-20T22:15:29.3596421Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-20T22:15:29.3597854Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-20T22:15:29.3598691Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:15:29.3602933Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:15:29.3603889Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-20T22:15:29.3604917Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:15:29.3605980Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:15:29.3606955Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:15:29.3608512Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:15:29.3611133Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:15:29.3612171Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:15:29.3613194Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:15:29.3614884Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:15:29.3616973Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:15:29.3619424Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:15:29.3620595Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:15:29.3622360Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:15:29.3624075Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:15:29.3625122Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:15:29.3626141Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:15:29.3667544Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:15:29.3707276Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:15:29.3708805Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:15:29.3755181Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:15:29.3756634Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:15:29.3758134Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:15:29.3759640Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:15:29.3761073Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:15:29.3762476Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:15:29.3763925Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:15:29.3765557Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:15:29.3766941Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:15:29.3768260Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:15:29.3770074Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:15:29.3771310Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:15:29.3772543Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:15:29.3773823Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:15:29.3775040Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:15:29.3835418Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:15:29.3896108Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:15:29.3897311Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:15:29.3898308Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:15:29.3899241Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:15:29.3900155Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-20T22:15:29.3901106Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-20T22:15:29.3902134Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-20T22:15:29.3903305Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-20T22:15:29.3904419Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-20T22:15:29.3905466Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-20T22:15:29.3906546Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-20T22:15:29.3907650Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-20T22:15:29.3908737Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-20T22:15:29.3909829Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-20T22:15:29.3910892Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-20T22:15:29.3932052Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-20T22:15:29.4076419Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-20T22:15:29.4077427Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-20T22:15:29.4079482Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-20T22:15:29.4081892Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-20T22:15:29.4084209Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-20T22:15:29.4086373Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-20T22:15:29.4088182Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-20T22:15:29.4089321Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-20T22:15:29.4090476Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-20T22:15:29.4091787Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-20T22:15:29.4092908Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-20T22:15:29.4108847Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-20T22:15:29.4193973Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-20T22:15:29.4196364Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:15:29.4198209Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:15:29.4199371Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-20T22:15:29.4200299Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-20T22:15:29.4201215Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-20T22:15:29.4202120Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-08-20T22:15:29.4202972Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-20T22:15:29.4204006Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-20T22:15:29.4205072Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-20T22:15:29.4325983Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-20T22:15:29.4390310Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-20T22:15:29.4391785Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-20T22:15:29.4393199Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-20T22:15:29.4394826Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:15:29.4399116Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:15:29.4400048Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-20T22:15:29.4400988Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:15:29.4401977Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:15:29.4402942Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:15:29.4404326Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:15:29.4406700Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:15:29.4407721Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:15:29.4408764Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:15:29.4410460Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:15:29.4412540Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:15:29.4414970Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:15:29.4416127Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:15:29.4418011Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:15:29.4419572Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:15:29.4420623Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:15:29.4421633Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:15:29.4462598Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:15:29.4502620Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:15:29.4504069Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:15:29.4550776Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:15:29.4552180Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:15:29.4553778Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:15:29.4555262Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:15:29.4556688Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:15:29.4558123Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:15:29.4559522Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:15:29.4560906Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:15:29.4562257Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:15:29.4563547Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:15:29.4564796Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:15:29.4566009Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:15:29.4567236Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:15:29.4568472Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:15:29.4569698Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:15:29.4630679Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:15:29.4691856Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:15:29.4693051Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:15:29.4694103Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:15:29.4694976Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:15:29.4695880Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-20T22:15:29.4696841Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-20T22:15:29.4697918Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-20T22:15:29.4699163Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-20T22:15:29.4700319Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-20T22:15:29.4701408Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-20T22:15:29.4702698Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-20T22:15:29.4703834Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-20T22:15:29.4704956Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-20T22:15:29.4706075Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-20T22:15:29.4707181Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-20T22:15:29.4728886Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-20T22:15:29.4793652Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-20T22:15:29.4794847Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:15:29.4795923Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:15:29.4796899Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-20T22:15:29.4797949Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-20T22:15:29.4799532Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-20T22:15:29.4800554Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-08-20T22:15:29.4803783Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-20T22:15:29.4804750Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-20T22:15:29.4805872Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-20T22:15:29.4857775Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-20T22:15:29.4858545Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-20T22:15:29.4859302Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-20T22:15:29.4860189Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:15:29.4866474Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:15:29.4867472Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-20T22:15:29.4868534Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:15:29.4869587Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:15:29.4870632Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:15:29.4872083Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:15:29.4874375Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:15:29.4875460Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:15:29.4876516Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:15:29.4878260Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:15:29.4880145Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:15:29.4882564Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:15:29.4883785Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:15:29.4885577Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:15:29.4887193Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:15:29.4888354Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:15:29.4889432Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:15:29.4930220Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:15:29.4970435Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:15:29.4972121Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:15:29.5018447Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:15:29.5019935Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:15:29.5021446Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:15:29.5023010Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:15:29.5024492Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:15:29.5025949Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:15:29.5027577Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:15:29.5029091Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:15:29.5030513Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:15:29.5031863Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:15:29.5033169Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:15:29.5034454Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:15:29.5035766Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:15:29.5037038Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:15:29.5038380Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:15:29.5098091Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:15:29.5160480Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:15:29.5162956Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:15:29.5165021Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:15:29.5166840Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:15:29.5168837Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-20T22:15:29.5169962Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-20T22:15:29.5171263Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-20T22:15:29.5172514Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-20T22:15:29.5173772Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-20T22:15:29.5174907Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-20T22:15:29.5176086Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-20T22:15:29.5177402Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-20T22:15:29.5178598Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-20T22:15:29.5179787Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-20T22:15:29.5180954Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-20T22:15:29.5182201Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-20T22:15:29.5306175Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-20T22:15:29.5307345Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-20T22:15:29.5308529Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-20T22:15:29.5309848Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-20T22:15:29.5311113Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-20T22:15:29.5312322Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-20T22:15:29.5313553Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-20T22:15:29.5314795Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-20T22:15:29.5316022Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-20T22:15:29.5317264Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-20T22:15:29.5318490Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-20T22:15:29.5339117Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-20T22:15:29.5394783Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-20T22:15:29.5396069Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:15:29.5397214Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:15:29.5398248Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-20T22:15:29.5399343Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-20T22:15:29.5400929Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-20T22:15:29.5402011Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-08-20T22:15:29.5405332Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-20T22:15:29.5406475Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-20T22:15:29.5407559Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-20T22:15:29.5513416Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-20T22:15:29.5556399Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-20T22:15:29.5557017Z creating: build/lib/ 2024-08-20T22:15:29.5646632Z inflating: build/lib/libprotobuf-lite.a 2024-08-20T22:15:29.6105065Z inflating: build/lib/libprotobuf.a 2024-08-20T22:15:29.6115209Z inflating: build/lib/libpthreadpool.a 2024-08-20T22:15:29.6123742Z inflating: build/lib/libcpuinfo.a 2024-08-20T22:15:29.6132534Z inflating: build/lib/libcpuinfo_internals.a 2024-08-20T22:15:29.6133324Z inflating: build/lib/libclog.a 2024-08-20T22:15:29.6152242Z inflating: build/lib/libnnpack.a 2024-08-20T22:15:29.6155261Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-20T22:15:29.6220955Z inflating: build/lib/libgtest.a 2024-08-20T22:15:29.6296927Z inflating: build/lib/libbenchmark.a 2024-08-20T22:15:29.6361944Z inflating: build/lib/libasmjit.a 2024-08-20T22:15:29.6369520Z inflating: build/lib/libittnotify.a 2024-08-20T22:15:29.6397754Z inflating: build/lib/libtensorpipe_uv.a 2024-08-20T22:15:29.6526103Z inflating: build/lib/libgloo.a 2024-08-20T22:15:29.6547310Z inflating: build/lib/libfmt.a 2024-08-20T22:15:29.6643417Z inflating: build/lib/libc10.so 2024-08-20T22:15:29.6645453Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-20T22:15:29.6647197Z inflating: build/lib/libtorch_global_deps.so 2024-08-20T22:15:29.7156171Z inflating: build/lib/libprotoc.a 2024-08-20T22:15:29.7175642Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-20T22:15:29.7194052Z inflating: build/lib/libgmock.a 2024-08-20T22:15:29.7195286Z inflating: build/lib/libgtest_main.a 2024-08-20T22:15:29.7196095Z inflating: build/lib/libbenchmark_main.a 2024-08-20T22:15:29.7589621Z inflating: build/lib/libgloo_cuda.a 2024-08-20T22:15:30.7809564Z inflating: build/lib/libdnnl.a 2024-08-20T22:15:30.8386806Z inflating: build/lib/libtensorpipe.a 2024-08-20T22:15:30.8445037Z inflating: build/lib/libc10_cuda.so 2024-08-20T22:15:30.8445948Z inflating: build/lib/libgmock_main.a 2024-08-20T22:15:30.9712689Z inflating: build/lib/libfbgemm.a 2024-08-20T22:15:30.9971056Z inflating: build/lib/libtensorpipe_cuda.a 2024-08-20T22:15:31.0482381Z inflating: build/lib/libkineto.a 2024-08-20T22:15:31.0667535Z inflating: build/lib/libXNNPACK.a 2024-08-20T22:15:31.0710407Z inflating: build/lib/libonnx_proto.a 2024-08-20T22:15:31.1409419Z inflating: build/lib/libonnx.a 2024-08-20T22:15:33.6030160Z inflating: build/lib/libtorch_cpu.so 2024-08-20T22:15:33.6035011Z inflating: build/lib/libunbox_lib.a 2024-08-20T22:15:33.6039798Z inflating: build/lib/libshm.so 2024-08-20T22:15:35.6831238Z inflating: build/lib/libtorch_cuda.so 2024-08-20T22:15:35.6834096Z inflating: build/lib/libtorch.so 2024-08-20T22:15:35.6835863Z inflating: build/lib/libc10d_cuda_test.so 2024-08-20T22:15:36.5078133Z inflating: build/lib/libtorch_cuda_linalg.so 2024-08-20T22:15:36.7088123Z inflating: build/lib/libtorch_python.so 2024-08-20T22:15:36.7160167Z inflating: build/lib/libtorchbind_test.so 2024-08-20T22:15:36.7181799Z inflating: build/lib/libjitbackend_test.so 2024-08-20T22:15:36.7207846Z inflating: build/lib/libbackend_with_compiler.so 2024-08-20T22:15:36.7234865Z inflating: build/lib/libaoti_custom_ops.so 2024-08-20T22:15:36.7269926Z inflating: build/lib/libnnapi_backend.so 2024-08-20T22:15:36.7270408Z creating: build/bin/ 2024-08-20T22:15:36.7321469Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-20T22:15:36.7374358Z inflating: build/bin/c10_DeviceGuard_test 2024-08-20T22:15:36.7426111Z inflating: build/bin/c10_Device_test 2024-08-20T22:15:36.7486690Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-20T22:15:36.7540466Z inflating: build/bin/c10_Scalar_test 2024-08-20T22:15:36.7590093Z inflating: build/bin/c10_StreamGuard_test 2024-08-20T22:15:36.7642390Z inflating: build/bin/c10_SymInt_test 2024-08-20T22:15:36.7699170Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-20T22:15:36.7755995Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-20T22:15:36.7813137Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-20T22:15:36.7885370Z inflating: build/bin/c10_cow_test 2024-08-20T22:15:36.7939066Z inflating: build/bin/c10_Bitset_test 2024-08-20T22:15:36.7988531Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-20T22:15:36.8039434Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-20T22:15:36.8090977Z inflating: build/bin/c10_Half_test 2024-08-20T22:15:36.8148462Z inflating: build/bin/c10_LeftRight_test 2024-08-20T22:15:36.8204390Z inflating: build/bin/c10_Metaprogramming_test 2024-08-20T22:15:36.8256225Z inflating: build/bin/c10_Synchronized_test 2024-08-20T22:15:36.8312195Z inflating: build/bin/c10_ThreadLocal_test 2024-08-20T22:15:36.8365132Z inflating: build/bin/c10_TypeIndex_test 2024-08-20T22:15:36.8416586Z inflating: build/bin/c10_TypeList_test 2024-08-20T22:15:36.8466786Z inflating: build/bin/c10_TypeTraits_test 2024-08-20T22:15:36.8519692Z inflating: build/bin/c10_accumulate_test 2024-08-20T22:15:36.8576463Z inflating: build/bin/c10_bfloat16_test 2024-08-20T22:15:36.8628045Z inflating: build/bin/c10_bit_cast_test 2024-08-20T22:15:36.8685509Z inflating: build/bin/c10_complex_math_test 2024-08-20T22:15:36.8742169Z inflating: build/bin/c10_complex_test 2024-08-20T22:15:36.8795765Z inflating: build/bin/c10_exception_test 2024-08-20T22:15:36.8847645Z inflating: build/bin/c10_flags_test 2024-08-20T22:15:36.8898685Z inflating: build/bin/c10_generic_math_test 2024-08-20T22:15:36.9065347Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-20T22:15:36.9117049Z inflating: build/bin/c10_irange_test 2024-08-20T22:15:36.9171573Z inflating: build/bin/c10_lazy_test 2024-08-20T22:15:36.9230016Z inflating: build/bin/c10_logging_test 2024-08-20T22:15:36.9306725Z inflating: build/bin/c10_optional_test 2024-08-20T22:15:36.9370289Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-20T22:15:36.9425250Z inflating: build/bin/c10_registry_test 2024-08-20T22:15:36.9577655Z inflating: build/bin/c10_small_vector_test 2024-08-20T22:15:36.9630854Z inflating: build/bin/c10_ssize_test 2024-08-20T22:15:36.9683083Z inflating: build/bin/c10_string_util_test 2024-08-20T22:15:36.9734871Z inflating: build/bin/c10_tempfile_test 2024-08-20T22:15:36.9795617Z inflating: build/bin/c10_string_view_test 2024-08-20T22:15:36.9845343Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-20T22:15:36.9902547Z inflating: build/bin/c10_typeid_test 2024-08-20T22:15:37.0355369Z inflating: build/bin/protoc-3.13.0.0 2024-08-20T22:15:37.0806192Z inflating: build/bin/protoc 2024-08-20T22:15:37.0859934Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-08-20T22:15:37.0913785Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-08-20T22:15:37.0967832Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-08-20T22:15:37.1020818Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-08-20T22:15:37.1075284Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-20T22:15:37.1128398Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-20T22:15:37.1178954Z inflating: build/bin/c10_cuda_CUDATest 2024-08-20T22:15:37.1232793Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-08-20T22:15:37.1566942Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-20T22:15:37.1915215Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-20T22:15:37.2282539Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-20T22:15:37.2337001Z inflating: build/bin/BackoffTest 2024-08-20T22:15:37.2390193Z inflating: build/bin/FileStoreTest 2024-08-20T22:15:37.2447059Z inflating: build/bin/TCPStoreTest 2024-08-20T22:15:37.2501999Z inflating: build/bin/HashStoreTest 2024-08-20T22:15:37.2516254Z inflating: build/bin/ProcessGroupMPITest 2024-08-20T22:15:37.2571273Z inflating: build/bin/test_edge_op_registration 2024-08-20T22:15:37.2575991Z inflating: build/bin/torch_shm_manager 2024-08-20T22:15:37.2579214Z inflating: build/bin/example_allreduce 2024-08-20T22:15:37.2635473Z inflating: build/bin/test_dist_autograd 2024-08-20T22:15:37.2705361Z inflating: build/bin/test_cpp_rpc 2024-08-20T22:15:37.2708544Z inflating: build/bin/parallel_benchmark 2024-08-20T22:15:37.2776942Z inflating: build/bin/test_mobile_nnc 2024-08-20T22:15:37.2786414Z inflating: build/bin/aot_model_compiler_test 2024-08-20T22:15:37.3137151Z inflating: build/bin/test_lazy 2024-08-20T22:15:37.4331498Z inflating: build/bin/test_api 2024-08-20T22:15:37.4406258Z inflating: build/bin/Dict_test 2024-08-20T22:15:37.4459605Z inflating: build/bin/Dimname_test 2024-08-20T22:15:37.4526185Z inflating: build/bin/MaybeOwned_test 2024-08-20T22:15:37.4584780Z inflating: build/bin/NamedTensor_test 2024-08-20T22:15:37.4645192Z inflating: build/bin/apply_utils_test 2024-08-20T22:15:37.4705256Z inflating: build/bin/atest 2024-08-20T22:15:37.4770103Z inflating: build/bin/basic 2024-08-20T22:15:37.4825836Z inflating: build/bin/broadcast_test 2024-08-20T22:15:37.4877708Z inflating: build/bin/cpu_allocator_test 2024-08-20T22:15:37.4937124Z inflating: build/bin/cpu_generator_test 2024-08-20T22:15:37.4992131Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-20T22:15:37.5086353Z inflating: build/bin/cpu_rng_test 2024-08-20T22:15:37.5137295Z inflating: build/bin/dispatch_key_set_test 2024-08-20T22:15:37.5188589Z inflating: build/bin/dlconvertor_test 2024-08-20T22:15:37.5248086Z inflating: build/bin/extension_backend_test 2024-08-20T22:15:37.5304296Z inflating: build/bin/half_test 2024-08-20T22:15:37.5402177Z inflating: build/bin/ivalue_test 2024-08-20T22:15:37.5452709Z inflating: build/bin/lazy_tensor_test 2024-08-20T22:15:37.5507967Z inflating: build/bin/math_kernel_test 2024-08-20T22:15:37.5563136Z inflating: build/bin/memory_format_test 2024-08-20T22:15:37.5617940Z inflating: build/bin/memory_overlapping_test 2024-08-20T22:15:37.5672173Z inflating: build/bin/mobile_memory_cleanup 2024-08-20T22:15:37.5729214Z inflating: build/bin/native_test 2024-08-20T22:15:37.5780195Z inflating: build/bin/operator_name_test 2024-08-20T22:15:37.5833056Z inflating: build/bin/operators_test 2024-08-20T22:15:37.5885699Z inflating: build/bin/packedtensoraccessor_test 2024-08-20T22:15:37.5954634Z inflating: build/bin/pow_test 2024-08-20T22:15:37.6013078Z inflating: build/bin/quantized_test 2024-08-20T22:15:37.6064218Z inflating: build/bin/reduce_ops_test 2024-08-20T22:15:37.6117155Z inflating: build/bin/reportMemoryUsage_test 2024-08-20T22:15:37.6175050Z inflating: build/bin/scalar_tensor_test 2024-08-20T22:15:37.6235764Z inflating: build/bin/scalar_test 2024-08-20T22:15:37.6288299Z inflating: build/bin/StorageUtils_test 2024-08-20T22:15:37.6341907Z inflating: build/bin/stride_properties_test 2024-08-20T22:15:37.6421730Z inflating: build/bin/tensor_iterator_test 2024-08-20T22:15:37.6477003Z inflating: build/bin/test_parallel 2024-08-20T22:15:37.6481221Z inflating: build/bin/thread_init_test 2024-08-20T22:15:37.6537993Z inflating: build/bin/type_ptr_test 2024-08-20T22:15:37.6598784Z inflating: build/bin/type_test 2024-08-20T22:15:37.6651924Z inflating: build/bin/undefined_tensor_test 2024-08-20T22:15:37.6653857Z inflating: build/bin/verify_api_visibility 2024-08-20T22:15:37.6724779Z inflating: build/bin/legacy_vmap_test 2024-08-20T22:15:37.6777479Z inflating: build/bin/weakref_test 2024-08-20T22:15:37.6829737Z inflating: build/bin/wrapdim_test 2024-08-20T22:15:37.6883222Z inflating: build/bin/xla_tensor_test 2024-08-20T22:15:37.6944136Z inflating: build/bin/IListRef_test 2024-08-20T22:15:37.7052430Z inflating: build/bin/List_test 2024-08-20T22:15:37.7120034Z inflating: build/bin/KernelFunction_test 2024-08-20T22:15:37.7243964Z inflating: build/bin/kernel_function_legacy_test 2024-08-20T22:15:37.7342032Z inflating: build/bin/kernel_function_test 2024-08-20T22:15:37.7473281Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-20T22:15:37.7578189Z inflating: build/bin/kernel_lambda_test 2024-08-20T22:15:37.7640057Z inflating: build/bin/kernel_stackbased_test 2024-08-20T22:15:37.7738381Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-20T22:15:37.7789809Z inflating: build/bin/CppSignature_test 2024-08-20T22:15:37.7839940Z inflating: build/bin/op_allowlist_test 2024-08-20T22:15:37.7896663Z inflating: build/bin/backend_fallback_test 2024-08-20T22:15:37.8204662Z inflating: build/bin/op_registration_test 2024-08-20T22:15:37.8269427Z inflating: build/bin/inline_container_test 2024-08-20T22:15:37.8322965Z inflating: build/bin/cuda_apply_test 2024-08-20T22:15:37.8375867Z inflating: build/bin/cuda_allocator_test 2024-08-20T22:15:37.8436081Z inflating: build/bin/cuda_atomic_ops_test 2024-08-20T22:15:37.8491679Z inflating: build/bin/cuda_caching_host_allocator_test 2024-08-20T22:15:37.8563686Z inflating: build/bin/cuda_complex_math_test 2024-08-20T22:15:37.8623370Z inflating: build/bin/cuda_complex_test 2024-08-20T22:15:37.8673862Z inflating: build/bin/cuda_device_test 2024-08-20T22:15:37.8733745Z inflating: build/bin/cuda_cub_test 2024-08-20T22:15:37.8785208Z inflating: build/bin/cuda_dlconvertor_test 2024-08-20T22:15:37.8851073Z inflating: build/bin/cuda_distributions_test 2024-08-20T22:15:37.8908786Z inflating: build/bin/cuda_generator_test 2024-08-20T22:15:37.8959321Z inflating: build/bin/cuda_half_test 2024-08-20T22:15:37.9011352Z inflating: build/bin/cuda_integer_divider_test 2024-08-20T22:15:37.9062057Z inflating: build/bin/cuda_optional_test 2024-08-20T22:15:37.9114048Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-08-20T22:15:37.9167804Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-08-20T22:15:37.9219093Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-08-20T22:15:37.9280226Z inflating: build/bin/cuda_stream_test 2024-08-20T22:15:37.9330788Z inflating: build/bin/cuda_cudnn_test 2024-08-20T22:15:37.9383877Z inflating: build/bin/cuda_vectorized_test 2024-08-20T22:15:37.9398706Z inflating: build/bin/tutorial_tensorexpr 2024-08-20T22:15:37.9466397Z inflating: build/bin/ProcessGroupGlooTest 2024-08-20T22:15:37.9524493Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-08-20T22:15:37.9590258Z inflating: build/bin/ProcessGroupNCCLTest 2024-08-20T22:15:37.9653354Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-08-20T22:15:38.0491362Z inflating: build/bin/test_tensorexpr 2024-08-20T22:15:38.1078734Z inflating: build/bin/test_jit 2024-08-20T22:15:38.1079176Z creating: .additional_ci_files/ 2024-08-20T22:15:38.1141238Z inflating: .additional_ci_files/test-times.json 2024-08-20T22:15:38.1385291Z inflating: .additional_ci_files/test-class-times.json 2024-08-20T22:15:38.1420486Z ##[group]Run rm artifacts.zip 2024-08-20T22:15:38.1420884Z rm artifacts.zip 2024-08-20T22:15:38.1430865Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:38.1431373Z env: 2024-08-20T22:15:38.1431657Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:38.1432112Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:38.1432818Z ##[endgroup] 2024-08-20T22:15:38.2533246Z ##[group]Run df -H 2024-08-20T22:15:38.2533643Z df -H 2024-08-20T22:15:38.2542693Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:38.2543195Z env: 2024-08-20T22:15:38.2543482Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:38.2543934Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:38.2544411Z ##[endgroup] 2024-08-20T22:15:38.2593218Z Filesystem Size Used Avail Use% Mounted on 2024-08-20T22:15:38.2593935Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-08-20T22:15:38.2594530Z tmpfs 34G 0 34G 0% /dev/shm 2024-08-20T22:15:38.2595009Z tmpfs 14G 553k 14G 1% /run 2024-08-20T22:15:38.2595463Z /dev/nvme0n1p1 161G 57G 105G 36% / 2024-08-20T22:15:38.2596191Z tmpfs 34G 4.1k 34G 1% /tmp 2024-08-20T22:15:38.2596688Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-08-20T22:15:38.2597446Z tmpfs 6.7G 0 6.7G 0% /run/user/0 2024-08-20T22:15:38.2637346Z Prepare all required actions 2024-08-20T22:15:38.2637796Z Getting action download info 2024-08-20T22:15:38.4090765Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-20T22:15:38.4091232Z with: 2024-08-20T22:15:38.4091493Z env: 2024-08-20T22:15:38.4091769Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:38.4092208Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:38.4092698Z ##[endgroup] 2024-08-20T22:15:38.4126126Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:15:38.4126566Z with: 2024-08-20T22:15:38.4127196Z name: td_results 2024-08-20T22:15:38.4127519Z s3-bucket: gha-artifacts 2024-08-20T22:15:38.4127867Z region: us-east-1 2024-08-20T22:15:38.4128171Z env: 2024-08-20T22:15:38.4128437Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:38.4128897Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:38.4129383Z ##[endgroup] 2024-08-20T22:15:38.9021895Z (node:57532) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:15:38.9023153Z 2024-08-20T22:15:38.9023564Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:15:38.9024664Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:15:38.9025997Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:15:38.9645367Z Found 0 objects with prefix pytorch/pytorch/10479310721/td_results/ 2024-08-20T22:15:38.9652671Z Artifact download has finished successfully 2024-08-20T22:15:38.9975075Z ##[group]Run mkdir -p .additional_ci_files 2024-08-20T22:15:38.9975564Z mkdir -p .additional_ci_files 2024-08-20T22:15:38.9976124Z mv td_results.json .additional_ci_files/td_results.json 2024-08-20T22:15:38.9985286Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:38.9985780Z env: 2024-08-20T22:15:38.9986053Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:38.9986529Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:38.9987010Z ##[endgroup] 2024-08-20T22:15:39.0046110Z mv: cannot stat 'td_results.json': No such file or directory 2024-08-20T22:15:39.0059596Z ##[error]Process completed with exit code 1. 2024-08-20T22:15:39.0094743Z ##[group]Run .github/scripts/parse_ref.py 2024-08-20T22:15:39.0095228Z .github/scripts/parse_ref.py 2024-08-20T22:15:39.0104044Z shell: /usr/bin/bash -e {0} 2024-08-20T22:15:39.0104401Z env: 2024-08-20T22:15:39.0104681Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:39.0105141Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:39.0105625Z ##[endgroup] 2024-08-20T22:15:39.0385926Z Prepare all required actions 2024-08-20T22:15:39.0424707Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-20T22:15:39.0425162Z with: 2024-08-20T22:15:39.0425726Z github-token: *** 2024-08-20T22:15:39.0426032Z env: 2024-08-20T22:15:39.0426299Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:39.0427329Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:39.0427820Z ##[endgroup] 2024-08-20T22:15:39.0444562Z ##[group]Run set -eux 2024-08-20T22:15:39.0444903Z set -eux 2024-08-20T22:15:39.0445493Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-20T22:15:39.0454198Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:39.0454694Z env: 2024-08-20T22:15:39.0454976Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:39.0455431Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:39.0456074Z GITHUB_TOKEN: *** 2024-08-20T22:15:39.0456381Z ##[endgroup] 2024-08-20T22:15:39.0488719Z + python3 .github/scripts/get_workflow_job_id.py 10479310721 i-0eeb15fe47bdc7534 2024-08-20T22:15:39.6953222Z setting job-id=29026300491 2024-08-20T22:15:39.6954285Z setting job-name=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:15:39.7173779Z Prepare all required actions 2024-08-20T22:15:39.7174231Z Getting action download info 2024-08-20T22:15:39.8289603Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-20T22:15:39.8290057Z with: 2024-08-20T22:15:39.8290530Z github-token: *** 2024-08-20T22:15:39.8297207Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:15:39.8304638Z job-name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:15:39.8305540Z env: 2024-08-20T22:15:39.8305814Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:39.8306273Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:39.8306755Z ##[endgroup] 2024-08-20T22:15:39.8349952Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:15:39.8350502Z with: 2024-08-20T22:15:39.8350775Z shell: bash 2024-08-20T22:15:39.8351078Z timeout_minutes: 10 2024-08-20T22:15:39.8351400Z max_attempts: 5 2024-08-20T22:15:39.8351735Z retry_wait_seconds: 30 2024-08-20T22:15:39.8352884Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:15:39.8354046Z polling_interval_seconds: 1 2024-08-20T22:15:39.8354421Z warning_on_retry: true 2024-08-20T22:15:39.8354768Z continue_on_error: false 2024-08-20T22:15:39.8355306Z env: 2024-08-20T22:15:39.8355612Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:39.8356058Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:39.8356712Z GITHUB_TOKEN: *** 2024-08-20T22:15:39.8357032Z ##[endgroup] 2024-08-20T22:15:39.9107189Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:15:40.1477818Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:15:40.2951667Z Collecting requests==2.27.1 2024-08-20T22:15:40.3319076Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2024-08-20T22:15:40.5302352Z Collecting pyyaml==6.0.1 2024-08-20T22:15:40.5352983Z Downloading PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (738 kB) 2024-08-20T22:15:40.5801310Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-08-20T22:15:40.5809911Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-08-20T22:15:40.8823350Z Collecting charset-normalizer~=2.0.0 2024-08-20T22:15:40.8871827Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-08-20T22:15:40.9399444Z Collecting certifi>=2017.4.17 2024-08-20T22:15:40.9446164Z Downloading certifi-2024.7.4-py3-none-any.whl (162 kB) 2024-08-20T22:15:41.0274187Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2024-08-20T22:15:41.1493477Z Successfully installed certifi-2024.7.4 charset-normalizer-2.0.12 pyyaml-6.0.1 requests-2.27.1 2024-08-20T22:15:41.8895068Z Command completed after 1 attempt(s). 2024-08-20T22:15:41.8955759Z ##[group]Run set -x 2024-08-20T22:15:41.8956100Z set -x 2024-08-20T22:15:41.8956406Z  2024-08-20T22:15:41.8956967Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:15:41.8957651Z # in runner workspace 2024-08-20T22:15:41.8958205Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-20T22:15:41.8967364Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:41.8967880Z env: 2024-08-20T22:15:41.8968165Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:41.8968627Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:41.8969119Z ##[endgroup] 2024-08-20T22:15:41.8999199Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-20T22:15:41.9242794Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:15:41.9243350Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:15:41.9243825Z echo "Job name: ${JOB_NAME}" 2024-08-20T22:15:41.9244230Z  2024-08-20T22:15:41.9244777Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:15:41.9245474Z # in runner workspace 2024-08-20T22:15:41.9246075Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-20T22:15:41.9246737Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-20T22:15:41.9247204Z  --job-name "${JOB_NAME}" \ 2024-08-20T22:15:41.9254602Z  --test-matrix "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-08-20T22:15:41.9261964Z  --selected-test-configs "" \ 2024-08-20T22:15:41.9262436Z  --pr-number "${PR_NUMBER}" \ 2024-08-20T22:15:41.9263026Z  --tag "${TAG}" \ 2024-08-20T22:15:41.9263446Z  --event-name "${EVENT_NAME}" \ 2024-08-20T22:15:41.9263899Z  --schedule "${SCHEDULE}" \ 2024-08-20T22:15:41.9264337Z  --branch "${HEAD_BRANCH}" 2024-08-20T22:15:41.9273167Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:41.9273677Z env: 2024-08-20T22:15:41.9273954Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:41.9274417Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:41.9275137Z GITHUB_TOKEN: *** 2024-08-20T22:15:41.9275990Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:15:41.9276899Z PR_NUMBER: 2024-08-20T22:15:41.9277221Z TAG: ciflow/inductor/133712 2024-08-20T22:15:41.9277592Z EVENT_NAME: push 2024-08-20T22:15:41.9277912Z SCHEDULE: 2024-08-20T22:15:41.9278202Z HEAD_BRANCH: 2024-08-20T22:15:41.9278508Z ##[endgroup] 2024-08-20T22:15:41.9307959Z Workflow: inductor-periodic 2024-08-20T22:15:41.9309100Z Job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:15:42.1591518Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-20T22:15:42.3153206Z ##[group]Run echo "Filtered matrix:" 2024-08-20T22:15:42.3153662Z echo "Filtered matrix:" 2024-08-20T22:15:42.3160322Z echo "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-08-20T22:15:42.3167479Z  2024-08-20T22:15:42.3167759Z echo 2024-08-20T22:15:42.3168137Z echo "Is the current job unstable? False" 2024-08-20T22:15:42.3168591Z  2024-08-20T22:15:42.3168870Z echo 2024-08-20T22:15:42.3169240Z echo "Is keep-going label set? False" 2024-08-20T22:15:42.3169683Z  2024-08-20T22:15:42.3169952Z echo 2024-08-20T22:15:42.3170271Z echo "Renabled issues? " 2024-08-20T22:15:42.3179380Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:42.3179882Z env: 2024-08-20T22:15:42.3180154Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:42.3180613Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:42.3181096Z ##[endgroup] 2024-08-20T22:15:42.3210483Z Filtered matrix: 2024-08-20T22:15:42.3218615Z {include: [{config: dynamo_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-08-20T22:15:42.3225349Z 2024-08-20T22:15:42.3225508Z Is the current job unstable? False 2024-08-20T22:15:42.3225794Z 2024-08-20T22:15:42.3226159Z Is keep-going label set? False 2024-08-20T22:15:42.3226411Z 2024-08-20T22:15:42.3226543Z Renabled issues? 2024-08-20T22:15:42.3269998Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:15:42.3270711Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:15:42.3278931Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:42.3279464Z env: 2024-08-20T22:15:42.3279758Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:42.3280243Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:42.3280780Z JOB_TIMEOUT: 240 2024-08-20T22:15:42.3281095Z ##[endgroup] 2024-08-20T22:15:42.3367712Z ##[group]Run set -x 2024-08-20T22:15:42.3368111Z set -x 2024-08-20T22:15:42.3368409Z  2024-08-20T22:15:42.3368759Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-20T22:15:42.3369304Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-20T22:15:42.3369862Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-20T22:15:42.3370378Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-20T22:15:42.3370792Z else 2024-08-20T22:15:42.3371133Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:15:42.3371559Z fi 2024-08-20T22:15:42.3371842Z  2024-08-20T22:15:42.3372313Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-20T22:15:42.3373063Z # TODO: Stop building test binaries as part of the build phase 2024-08-20T22:15:42.3373842Z # Used for GPU_FLAG since that doesn't play nice 2024-08-20T22:15:42.3374627Z # shellcheck disable=SC2086,SC2090 2024-08-20T22:15:42.3375080Z container_name=$(docker run \ 2024-08-20T22:15:42.3375494Z  ${GPU_FLAG:-} \ 2024-08-20T22:15:42.3375863Z  -e BUILD_ENVIRONMENT \ 2024-08-20T22:15:42.3376259Z  -e PR_NUMBER \ 2024-08-20T22:15:42.3376612Z  -e GITHUB_ACTIONS \ 2024-08-20T22:15:42.3377012Z  -e GITHUB_REPOSITORY \ 2024-08-20T22:15:42.3377413Z  -e GITHUB_WORKFLOW \ 2024-08-20T22:15:42.3377793Z  -e GITHUB_JOB \ 2024-08-20T22:15:42.3378149Z  -e GITHUB_RUN_ID \ 2024-08-20T22:15:42.3378530Z  -e GITHUB_RUN_NUMBER \ 2024-08-20T22:15:42.3378932Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-20T22:15:42.3379328Z  -e JOB_ID \ 2024-08-20T22:15:42.3379662Z  -e JOB_NAME \ 2024-08-20T22:15:42.3380108Z  -e BASE_SHA \ 2024-08-20T22:15:42.3380546Z  -e BRANCH \ 2024-08-20T22:15:42.3380965Z  -e SHA1 \ 2024-08-20T22:15:42.3381521Z  -e AWS_DEFAULT_REGION \ 2024-08-20T22:15:42.3382021Z  -e IN_WHEEL_TEST \ 2024-08-20T22:15:42.3382475Z  -e SHARD_NUMBER \ 2024-08-20T22:15:42.3382984Z  -e TEST_CONFIG \ 2024-08-20T22:15:42.3383443Z  -e NUM_TEST_SHARDS \ 2024-08-20T22:15:42.3383991Z  -e REENABLED_ISSUES \ 2024-08-20T22:15:42.3384459Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-20T22:15:42.3384990Z  -e VERBOSE_TEST_LOGS \ 2024-08-20T22:15:42.3385521Z  -e TEST_SHOWLOCALS \ 2024-08-20T22:15:42.3385996Z  -e NO_TEST_TIMEOUT \ 2024-08-20T22:15:42.3386440Z  -e NO_TD \ 2024-08-20T22:15:42.3386907Z  -e TD_DISTRIBUTED \ 2024-08-20T22:15:42.3387446Z  -e PR_LABELS \ 2024-08-20T22:15:42.3387934Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-20T22:15:42.3388547Z  -e SCCACHE_BUCKET \ 2024-08-20T22:15:42.3389037Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-20T22:15:42.3389572Z  -e XLA_CUDA \ 2024-08-20T22:15:42.3390089Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-20T22:15:42.3390718Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-20T22:15:42.3391336Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-20T22:15:42.3391993Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-20T22:15:42.3392537Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-20T22:15:42.3393080Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-08-20T22:15:42.3393718Z  -e DASHBOARD_TAG \ 2024-08-20T22:15:42.3394311Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-20T22:15:42.3394921Z  --security-opt seccomp=unconfined \ 2024-08-20T22:15:42.3395517Z  --cap-add=SYS_PTRACE \ 2024-08-20T22:15:42.3396022Z  --ipc=host \ 2024-08-20T22:15:42.3396472Z  --shm-size="${SHM_SIZE}" \ 2024-08-20T22:15:42.3396988Z  --tty \ 2024-08-20T22:15:42.3397395Z  --detach \ 2024-08-20T22:15:42.3397990Z  --name="${container_name}" \ 2024-08-20T22:15:42.3398584Z  --user jenkins \ 2024-08-20T22:15:42.3399124Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-20T22:15:42.3399846Z  -w /var/lib/jenkins/workspace \ 2024-08-20T22:15:42.3400428Z  "${DOCKER_IMAGE}" 2024-08-20T22:15:42.3400872Z ) 2024-08-20T22:15:42.3401356Z # Propagate download.pytorch.org IP to container 2024-08-20T22:15:42.3402411Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-20T22:15:42.3431230Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-20T22:15:42.3432164Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-20T22:15:42.3441512Z shell: /usr/bin/bash -e {0} 2024-08-20T22:15:42.3441869Z env: 2024-08-20T22:15:42.3442344Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:42.3442801Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:42.3443404Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:15:42.3443909Z PR_NUMBER: 2024-08-20T22:15:42.3444233Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-20T22:15:42.3444670Z GITHUB_WORKFLOW: inductor-periodic 2024-08-20T22:15:42.3445067Z GITHUB_JOB: test 2024-08-20T22:15:42.3445398Z GITHUB_RUN_ID: 10479310721 2024-08-20T22:15:42.3445766Z GITHUB_RUN_NUMBER: 54565 2024-08-20T22:15:42.3446124Z GITHUB_RUN_ATTEMPT: 1 2024-08-20T22:15:42.3446453Z JOB_ID: 29026300491 2024-08-20T22:15:42.3447296Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:15:42.3448185Z BRANCH: 2024-08-20T22:15:42.3448529Z SHA1: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:15:42.3449050Z BASE_SHA: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:15:42.3449560Z TEST_CONFIG: dynamic_aot_eager_timm 2024-08-20T22:15:42.3449970Z SHARD_NUMBER: 1 2024-08-20T22:15:42.3450279Z NUM_TEST_SHARDS: 2 2024-08-20T22:15:42.3450607Z REENABLED_ISSUES: 2024-08-20T22:15:42.3450948Z CONTINUE_THROUGH_ERROR: False 2024-08-20T22:15:42.3451337Z VERBOSE_TEST_LOGS: False 2024-08-20T22:15:42.3451693Z TEST_SHOWLOCALS: False 2024-08-20T22:15:42.3452041Z NO_TEST_TIMEOUT: False 2024-08-20T22:15:42.3452377Z NO_TD: False 2024-08-20T22:15:42.3452686Z TD_DISTRIBUTED: False 2024-08-20T22:15:42.3453096Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-20T22:15:42.3453679Z SCCACHE_S3_KEY_PREFIX: inductor-periodic 2024-08-20T22:15:42.3454105Z SHM_SIZE: 2g 2024-08-20T22:15:42.3455191Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:15:42.3456356Z XLA_CUDA: 2024-08-20T22:15:42.3456833Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:15:42.3457451Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-20T22:15:42.3457888Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-20T22:15:42.3458287Z DASHBOARD_TAG: 2024-08-20T22:15:42.3458815Z HUGGING_FACE_HUB_TOKEN: *** 2024-08-20T22:15:42.3459200Z SCRIBE_GRAPHQL_ACCESS_TOKEN: 2024-08-20T22:15:42.3459570Z ##[endgroup] 2024-08-20T22:15:42.3489921Z + [[ dynamic_aot_eager_timm == \m\u\l\t\i\g\p\u ]] 2024-08-20T22:15:42.3490716Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-08-20T22:15:42.3491239Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:15:42.3498932Z +++ nproc --ignore=2 2024-08-20T22:15:42.3529621Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10479310721 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:15:53.7735185Z + container_name=6fb7c9a8debe00f72ec404e19beb53e46f5d9d9b4ed5e85a10f4882d30d1f88a 2024-08-20T22:15:53.7742364Z + grep download.pytorch.org /etc/hosts 2024-08-20T22:15:53.7743786Z + docker exec -i 6fb7c9a8debe00f72ec404e19beb53e46f5d9d9b4ed5e85a10f4882d30d1f88a sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-20T22:15:53.9198666Z + echo DOCKER_CONTAINER_ID=6fb7c9a8debe00f72ec404e19beb53e46f5d9d9b4ed5e85a10f4882d30d1f88a 2024-08-20T22:15:53.9202452Z ++ echo dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:15:53.9205491Z + docker exec -t 6fb7c9a8debe00f72ec404e19beb53e46f5d9d9b4ed5e85a10f4882d30d1f88a sh -c 'pip install dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-20T22:15:54.3265676Z Processing ./dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+git40ec5f6) 2024-08-20T22:15:54.6516333Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.13.1) 2024-08-20T22:15:54.6519072Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (4.12.2) 2024-08-20T22:15:54.6521638Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.8.8) 2024-08-20T22:15:54.6525024Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.1.4) 2024-08-20T22:15:54.6527651Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2024.6.1) 2024-08-20T22:15:54.6533774Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.13.1) 2024-08-20T22:15:54.6549451Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.3.0) 2024-08-20T22:15:54.6562497Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.3.0) 2024-08-20T22:15:54.6579909Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.22.4) 2024-08-20T22:15:54.6965311Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.1.5) 2024-08-20T22:15:55.2624302Z Installing collected packages: torch 2024-08-20T22:16:05.0495983Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-08-20T22:16:05.0498840Z timm 1.0.9.dev0 requires torchvision, which is not installed. 2024-08-20T22:16:05.0499669Z Successfully installed torch-2.5.0a0+git40ec5f6 2024-08-20T22:16:05.1316598Z + export TERM=vt100 2024-08-20T22:16:05.1316966Z + TERM=vt100 2024-08-20T22:16:05.1320510Z ++ dirname .ci/pytorch/test.sh 2024-08-20T22:16:05.1331902Z + source .ci/pytorch/common.sh 2024-08-20T22:16:05.1335062Z +++ dirname .ci/pytorch/common.sh 2024-08-20T22:16:05.1344259Z ++ source .ci/pytorch/common_utils.sh 2024-08-20T22:16:05.1345651Z +++ declare -f -t trap_add 2024-08-20T22:16:05.1352356Z ++ set -ex 2024-08-20T22:16:05.1352952Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:16:05.1353487Z ++ BUILD_TEST_LIBTORCH=0 2024-08-20T22:16:05.1354785Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-08-20T22:16:05.1357795Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-20T22:16:05.1378956Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-20T22:16:05.1379401Z + trap_add cleanup_workspace EXIT 2024-08-20T22:16:05.1380210Z + trap_add_cmd=cleanup_workspace 2024-08-20T22:16:05.1380588Z + shift 2024-08-20T22:16:05.1380874Z + for trap_add_name in "$@" 2024-08-20T22:16:05.1387629Z +++ trap -p EXIT 2024-08-20T22:16:05.1390991Z ++ eval 'extract_trap_cmd ' 2024-08-20T22:16:05.1391394Z +++ extract_trap_cmd 2024-08-20T22:16:05.1391779Z +++ printf '%s\n' '' 2024-08-20T22:16:05.1392190Z ++ printf '%s\n' cleanup_workspace 2024-08-20T22:16:05.1394739Z + trap -- ' 2024-08-20T22:16:05.1395093Z cleanup_workspace' EXIT 2024-08-20T22:16:05.1395573Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-20T22:16:05.8536635Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-20T22:16:05.8559836Z + echo 'Environment variables:' 2024-08-20T22:16:05.8560304Z Environment variables: 2024-08-20T22:16:05.8560726Z + env 2024-08-20T22:16:05.8571008Z INSTALLED_DB=yes 2024-08-20T22:16:05.8571547Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:16:05.8572043Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:16:05.8574575Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:16:05.8576593Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:16:05.8577315Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:16:05.8577929Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:16:05.8578470Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:16:05.8579120Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:16:05.8579692Z HOSTNAME=6fb7c9a8debe 2024-08-20T22:16:05.8580835Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_121f2339-1ab8-441b-a993-10a9d014ff2a 2024-08-20T22:16:05.8581719Z GITHUB_ACTION=__self 2024-08-20T22:16:05.8582136Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:16:05.8586600Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:16:05.8590730Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:16:05.8591238Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:16:05.8591587Z GITHUB_RUN_NUMBER=54565 2024-08-20T22:16:05.8591936Z TEST_CONFIG=dynamic_aot_eager_timm 2024-08-20T22:16:05.8592373Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:16:05.8592842Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:16:05.8593299Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:16:05.8593745Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:16:05.8594432Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:16:05.8594832Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:16:05.8595275Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:16:05.8595767Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:16:05.8596225Z GITHUB_REF_TYPE=tag 2024-08-20T22:16:05.8596547Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:16:05.8596920Z NCCL_VERSION=2.17.1-1 2024-08-20T22:16:05.8597313Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:05.8597748Z XLA_CUDA= 2024-08-20T22:16:05.8598248Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:16:05.8600577Z *** 2024-08-20T22:16:05.8600873Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:16:05.8601264Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:16:05.8601627Z GITHUB_ACTIONS=true 2024-08-20T22:16:05.8601961Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:16:05.8602486Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:16:05.8603020Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:16:05.8603695Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:05.8604214Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:16:05.8604702Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:05.8605642Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:16:05.8606398Z UCC_HOME=/usr 2024-08-20T22:16:05.8606743Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:16:05.8607138Z VERBOSE_TEST_LOGS=False 2024-08-20T22:16:05.8607472Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:16:05.8607933Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:16:05.8608423Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:16:05.8608880Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:16:05.8609255Z SHARD_NUMBER=1 2024-08-20T22:16:05.8609562Z GITHUB_REF_PROTECTED=false 2024-08-20T22:16:05.8609915Z HOME=/var/lib/jenkins 2024-08-20T22:16:05.8610272Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:16:05.8610724Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:16:05.8611200Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:16:05.8611756Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-08-20T22:16:05.8612164Z CUDA_VERSION=12.1.1 2024-08-20T22:16:05.8612589Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:16:05.8613025Z NUM_TEST_SHARDS=2 2024-08-20T22:16:05.8613318Z UCX_HOME=/usr 2024-08-20T22:16:05.8613955Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:16:05.8615033Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_121f2339-1ab8-441b-a993-10a9d014ff2a 2024-08-20T22:16:05.8616462Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:16:05.8617868Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_121f2339-1ab8-441b-a993-10a9d014ff2a 2024-08-20T22:16:05.8618988Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:16:05.8619647Z GITHUB_EVENT_NAME=push 2024-08-20T22:16:05.8619975Z DASHBOARD_TAG= 2024-08-20T22:16:05.8620285Z GITHUB_RUN_ID=10479310721 2024-08-20T22:16:05.8620746Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:16:05.8621285Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:16:05.8622279Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_121f2339-1ab8-441b-a993-10a9d014ff2a 2024-08-20T22:16:05.8623186Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:16:05.8623589Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:16:05.8623963Z PR_NUMBER= 2024-08-20T22:16:05.8624250Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:16:05.8624759Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:16:05.8625204Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:16:05.8625659Z TERM=vt100 2024-08-20T22:16:05.8625998Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:16:05.8626410Z INSTALLED_VISION=yes 2024-08-20T22:16:05.8627403Z BRANCH= 2024-08-20T22:16:05.8627797Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:16:05.8628356Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:16:05.8628875Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:16:05.8629740Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:16:05.8630483Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:16:05.8630960Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:16:05.8631416Z REENABLED_ISSUES= 2024-08-20T22:16:05.8631709Z SHLVL=1 2024-08-20T22:16:05.8631965Z MAX_JOBS=14 2024-08-20T22:16:05.8632272Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:16:05.8632619Z NVARCH=x86_64 2024-08-20T22:16:05.8632911Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:16:05.8633359Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:05.8633879Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:16:05.8634367Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:16:05.8635193Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:16:05.8635745Z GITHUB_JOB=test 2024-08-20T22:16:05.8636145Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:16:05.8636696Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:16:05.8637197Z NO_TEST_TIMEOUT=False 2024-08-20T22:16:05.8637512Z TD_DISTRIBUTED=False 2024-08-20T22:16:05.8637884Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:16:05.8638314Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:16:05.8638733Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:16:05.8639082Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:16:05.8639430Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:16:05.8639775Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:16:05.8640804Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:16:05.8641855Z GITHUB_BASE_REF= 2024-08-20T22:16:05.8642189Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:16:05.8642551Z CI=true 2024-08-20T22:16:05.8642857Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:16:05.8643258Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:16:05.8643620Z JOB_ID=29026300491 2024-08-20T22:16:05.8643928Z INSTALLED_PROTOBUF=yes 2024-08-20T22:16:05.8644243Z GITHUB_HEAD_REF= 2024-08-20T22:16:05.8644547Z GITHUB_ACTION_REF= 2024-08-20T22:16:05.8644979Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:16:05.8645437Z TEST_SHOWLOCALS=False 2024-08-20T22:16:05.8645817Z GITHUB_WORKFLOW=inductor-periodic 2024-08-20T22:16:05.8646216Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:16:05.8647112Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_121f2339-1ab8-441b-a993-10a9d014ff2a 2024-08-20T22:16:05.8647921Z NO_TD=False 2024-08-20T22:16:05.8648214Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:16:05.8648570Z _=/usr/bin/env 2024-08-20T22:16:05.8649041Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-20T22:16:05.8806923Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-08-20T22:16:05.8808103Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T22:16:05.8809134Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-08-20T22:16:05.8810015Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-08-20T22:16:05.8810632Z + BUILD_DIR=build 2024-08-20T22:16:05.8810963Z + BUILD_RENAMED_DIR=build_renamed 2024-08-20T22:16:05.8811356Z + BUILD_BIN_DIR=build/bin 2024-08-20T22:16:05.8811693Z + SHARD_NUMBER=1 2024-08-20T22:16:05.8811996Z + NUM_TEST_SHARDS=2 2024-08-20T22:16:05.8812315Z + export VALGRIND=ON 2024-08-20T22:16:05.8812729Z + VALGRIND=ON 2024-08-20T22:16:05.8813287Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-08-20T22:16:05.8813893Z + [[ 0 == \1 ]] 2024-08-20T22:16:05.8814186Z + [[ False == \1 ]] 2024-08-20T22:16:05.8814954Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-08-20T22:16:05.8815490Z ++ realpath build/custom_test_artifacts 2024-08-20T22:16:05.8828271Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-20T22:16:05.8829064Z + [[ -n '' ]] 2024-08-20T22:16:05.8829421Z + echo 'Environment variables' 2024-08-20T22:16:05.8829812Z Environment variables 2024-08-20T22:16:05.8830130Z + env 2024-08-20T22:16:05.8838490Z INSTALLED_DB=yes 2024-08-20T22:16:05.8838955Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:16:05.8839407Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:16:05.8839916Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:16:05.8840732Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:16:05.8841496Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:16:05.8842220Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:16:05.8842783Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:16:05.8843448Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:16:05.8844249Z HOSTNAME=6fb7c9a8debe 2024-08-20T22:16:05.8845196Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_121f2339-1ab8-441b-a993-10a9d014ff2a 2024-08-20T22:16:05.8846056Z GITHUB_ACTION=__self 2024-08-20T22:16:05.8846403Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:16:05.8850728Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:16:05.8855191Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:16:05.8855702Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:16:05.8856056Z GITHUB_RUN_NUMBER=54565 2024-08-20T22:16:05.8856408Z TEST_CONFIG=dynamic_aot_eager_timm 2024-08-20T22:16:05.8856828Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:16:05.8857299Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:16:05.8857759Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:16:05.8858237Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:16:05.8858659Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:16:05.8859046Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:16:05.8859496Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:16:05.8859986Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:16:05.8860450Z GITHUB_REF_TYPE=tag 2024-08-20T22:16:05.8860770Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:16:05.8861156Z NCCL_VERSION=2.17.1-1 2024-08-20T22:16:05.8861550Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:05.8861982Z XLA_CUDA= 2024-08-20T22:16:05.8862497Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:16:05.8862895Z *** 2024-08-20T22:16:05.8863172Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:16:05.8863564Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:16:05.8863922Z GITHUB_ACTIONS=true 2024-08-20T22:16:05.8864247Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:16:05.8864749Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:16:05.8865278Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:16:05.8865748Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:05.8866258Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:16:05.8866734Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:05.8867668Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:16:05.8868441Z UCC_HOME=/usr 2024-08-20T22:16:05.8868945Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:16:05.8869347Z VERBOSE_TEST_LOGS=False 2024-08-20T22:16:05.8869682Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:16:05.8870141Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:16:05.8870636Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:16:05.8871102Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:16:05.8871469Z SHARD_NUMBER=1 2024-08-20T22:16:05.8871775Z GITHUB_REF_PROTECTED=false 2024-08-20T22:16:05.8872120Z HOME=/var/lib/jenkins 2024-08-20T22:16:05.8872476Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:16:05.8872909Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:16:05.8873378Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:16:05.8873936Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-08-20T22:16:05.8874340Z CUDA_VERSION=12.1.1 2024-08-20T22:16:05.8874758Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:16:05.8875191Z NUM_TEST_SHARDS=2 2024-08-20T22:16:05.8875579Z UCX_HOME=/usr 2024-08-20T22:16:05.8876094Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:16:05.8877178Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_121f2339-1ab8-441b-a993-10a9d014ff2a 2024-08-20T22:16:05.8878613Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:16:05.8880017Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_121f2339-1ab8-441b-a993-10a9d014ff2a 2024-08-20T22:16:05.8881128Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:16:05.8881781Z GITHUB_EVENT_NAME=push 2024-08-20T22:16:05.8882104Z DASHBOARD_TAG= 2024-08-20T22:16:05.8882404Z GITHUB_RUN_ID=10479310721 2024-08-20T22:16:05.8882861Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:16:05.8883399Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:16:05.8884407Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_121f2339-1ab8-441b-a993-10a9d014ff2a 2024-08-20T22:16:05.8885315Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:16:05.8885712Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:16:05.8886089Z PR_NUMBER= 2024-08-20T22:16:05.8886368Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:16:05.8886677Z VALGRIND=ON 2024-08-20T22:16:05.8886970Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:16:05.8887404Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:16:05.8887857Z TERM=vt100 2024-08-20T22:16:05.8888183Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:16:05.8888591Z INSTALLED_VISION=yes 2024-08-20T22:16:05.8888897Z BRANCH= 2024-08-20T22:16:05.8889178Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:16:05.8889565Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:16:05.8889984Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:16:05.8890740Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:16:05.8891500Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:16:05.8891976Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:16:05.8892442Z REENABLED_ISSUES= 2024-08-20T22:16:05.8892737Z SHLVL=1 2024-08-20T22:16:05.8892986Z MAX_JOBS=14 2024-08-20T22:16:05.8893295Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:16:05.8893766Z NVARCH=x86_64 2024-08-20T22:16:05.8894058Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:16:05.8894502Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:05.8895029Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:16:05.8895517Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:16:05.8896175Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:16:05.8896737Z GITHUB_JOB=test 2024-08-20T22:16:05.8897135Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:16:05.8897711Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:16:05.8898345Z NO_TEST_TIMEOUT=False 2024-08-20T22:16:05.8898788Z TD_DISTRIBUTED=False 2024-08-20T22:16:05.8899177Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:16:05.8899612Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:16:05.8900031Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:16:05.8900396Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:16:05.8900743Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:16:05.8901093Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:16:05.8902156Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:16:05.8903235Z GITHUB_BASE_REF= 2024-08-20T22:16:05.8903562Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:16:05.8903934Z CI=true 2024-08-20T22:16:05.8904247Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:16:05.8904665Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:16:05.8905038Z JOB_ID=29026300491 2024-08-20T22:16:05.8905351Z INSTALLED_PROTOBUF=yes 2024-08-20T22:16:05.8905672Z GITHUB_HEAD_REF= 2024-08-20T22:16:05.8906080Z GITHUB_ACTION_REF= 2024-08-20T22:16:05.8906522Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:16:05.8906982Z TEST_SHOWLOCALS=False 2024-08-20T22:16:05.8907376Z GITHUB_WORKFLOW=inductor-periodic 2024-08-20T22:16:05.8907787Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:16:05.8908699Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_121f2339-1ab8-441b-a993-10a9d014ff2a 2024-08-20T22:16:05.8909502Z NO_TD=False 2024-08-20T22:16:05.8909809Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:16:05.8910171Z _=/usr/bin/env 2024-08-20T22:16:05.8910521Z + echo 'Testing pytorch' 2024-08-20T22:16:05.8910855Z Testing pytorch 2024-08-20T22:16:05.8911188Z + export LANG=C.UTF-8 2024-08-20T22:16:05.8911534Z + LANG=C.UTF-8 2024-08-20T22:16:05.8911816Z + PR_NUMBER= 2024-08-20T22:16:05.8912171Z + [[ dynamic_aot_eager_timm == \d\e\f\a\u\l\t ]] 2024-08-20T22:16:05.8912700Z + [[ dynamic_aot_eager_timm == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-20T22:16:05.8913217Z + [[ dynamic_aot_eager_timm == \s\l\o\w ]] 2024-08-20T22:16:05.8913865Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-08-20T22:16:05.8914573Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-08-20T22:16:05.8915125Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:16:05.8915598Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:16:05.8916049Z + [[ dynamic_aot_eager_timm == *crossref* ]] 2024-08-20T22:16:05.8916638Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:16:05.8917276Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-08-20T22:16:05.8917936Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:16:05.8918498Z + pip_install --user ninja==1.10.2 2024-08-20T22:16:05.8919036Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-20T22:16:07.1144227Z Collecting ninja==1.10.2 2024-08-20T22:16:07.1321211Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-20T22:16:07.1865875Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-20T22:16:07.7732339Z Installing collected packages: ninja 2024-08-20T22:16:07.7811905Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-20T22:16:07.7813197Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-20T22:16:07.8273217Z Successfully installed ninja-1.10.2 2024-08-20T22:16:07.9058635Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:16:07.9060959Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:16:07.9062939Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:16:07.9063448Z + install_tlparse 2024-08-20T22:16:07.9063837Z + pip_install --user tlparse==0.3.25 2024-08-20T22:16:07.9064392Z + pip install --progress-bar off --user tlparse==0.3.25 2024-08-20T22:16:08.3325886Z Collecting tlparse==0.3.25 2024-08-20T22:16:08.3489415Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-08-20T22:16:08.3982691Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-20T22:16:09.0378594Z Installing collected packages: tlparse 2024-08-20T22:16:09.1144392Z Successfully installed tlparse-0.3.25 2024-08-20T22:16:09.1930407Z ++ python -m site --user-base 2024-08-20T22:16:09.2174507Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:16:09.2176669Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-08-20T22:16:09.2177334Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-08-20T22:16:09.2177996Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:16:09.2178952Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-08-20T22:16:09.2180235Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-08-20T22:16:09.2180989Z + cd test 2024-08-20T22:16:09.2181530Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-20T22:16:10.8429048Z + [[ dynamic_aot_eager_timm == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-20T22:16:10.8429758Z + [[ dynamic_aot_eager_timm == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-20T22:16:10.8432648Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-20T22:16:10.8433234Z + [[ dynamic_aot_eager_timm == *pr_time_benchmarks* ]] 2024-08-20T22:16:10.8433935Z + [[ dynamic_aot_eager_timm == *dynamo_eager* ]] 2024-08-20T22:16:10.8434559Z + [[ dynamic_aot_eager_timm == *aot_eager* ]] 2024-08-20T22:16:10.8435321Z + DYNAMO_BENCHMARK_FLAGS+=(--backend aot_eager) 2024-08-20T22:16:10.8435801Z + [[ dynamic_aot_eager_timm == *dynamic* ]] 2024-08-20T22:16:10.8436438Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only) 2024-08-20T22:16:10.8437002Z + [[ dynamic_aot_eager_timm == *cpu* ]] 2024-08-20T22:16:10.8437500Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-20T22:16:10.8465906Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:16:10.8466590Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-08-20T22:16:10.8469276Z + cd test 2024-08-20T22:16:10.8469806Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-20T22:16:12.3058048Z PyTorch built with: 2024-08-20T22:16:12.3058795Z - GCC 9.4 2024-08-20T22:16:12.3059253Z - C++ Version: 201703 2024-08-20T22:16:12.3060393Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:16:12.3061498Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:16:12.3062171Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:16:12.3062694Z - LAPACK is enabled (usually provided by MKL) 2024-08-20T22:16:12.3063182Z - NNPACK is enabled 2024-08-20T22:16:12.3063564Z - CPU capability usage: AVX2 2024-08-20T22:16:12.3063977Z - CUDA Runtime 12.1 2024-08-20T22:16:12.3064502Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-08-20T22:16:12.3065110Z - CuDNN 90.1 (built against CUDA 12.4) 2024-08-20T22:16:12.3065554Z - Magma 2.6.1 2024-08-20T22:16:12.3072776Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-20T22:16:12.3079066Z 2024-08-20T22:16:12.6327274Z + cd test 2024-08-20T22:16:12.6328494Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-20T22:16:13.9701268Z ATen/Parallel: 2024-08-20T22:16:13.9701804Z at::get_num_threads() : 8 2024-08-20T22:16:13.9702298Z at::get_num_interop_threads() : 16 2024-08-20T22:16:13.9702801Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:16:13.9703204Z omp_get_max_threads() : 8 2024-08-20T22:16:13.9704279Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:16:13.9705080Z mkl_get_max_threads() : 8 2024-08-20T22:16:13.9705707Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:16:13.9706344Z std::thread::hardware_concurrency() : 16 2024-08-20T22:16:13.9706775Z Environment variables: 2024-08-20T22:16:13.9707135Z OMP_NUM_THREADS : [not set] 2024-08-20T22:16:13.9707507Z MKL_NUM_THREADS : [not set] 2024-08-20T22:16:13.9707891Z ATen parallel backend: OpenMP 2024-08-20T22:16:13.9708145Z 2024-08-20T22:16:14.2485707Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:16:14.2486476Z + [[ dynamic_aot_eager_timm == *backward* ]] 2024-08-20T22:16:14.2487026Z + [[ dynamic_aot_eager_timm == *xla* ]] 2024-08-20T22:16:14.2487538Z + [[ dynamic_aot_eager_timm == *executorch* ]] 2024-08-20T22:16:14.2488048Z + [[ dynamic_aot_eager_timm == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-20T22:16:14.2488710Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:16:14.2489258Z + [[ dynamic_aot_eager_timm == distributed ]] 2024-08-20T22:16:14.2489778Z + [[ dynamic_aot_eager_timm == *inductor_distributed* ]] 2024-08-20T22:16:14.2490379Z + [[ dynamic_aot_eager_timm == *inductor-halide* ]] 2024-08-20T22:16:14.2491147Z + [[ dynamic_aot_eager_timm == *inductor-micro-benchmark* ]] 2024-08-20T22:16:14.2491857Z + [[ dynamic_aot_eager_timm == *huggingface* ]] 2024-08-20T22:16:14.2492393Z + [[ dynamic_aot_eager_timm == *timm* ]] 2024-08-20T22:16:14.2492809Z + install_torchvision 2024-08-20T22:16:14.2493155Z + local orig_preload 2024-08-20T22:16:14.2493472Z + local commit 2024-08-20T22:16:14.2493908Z ++ get_pinned_commit vision 2024-08-20T22:16:14.2494305Z ++ cat .github/ci_commit_pins/vision.txt 2024-08-20T22:16:14.2509024Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:16:14.2509624Z + orig_preload= 2024-08-20T22:16:14.2510082Z + '[' -n '' ']' 2024-08-20T22:16:14.2510995Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:16:14.2512444Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:16:14.5863979Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:16:14.5867288Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-syvihmyg 2024-08-20T22:16:14.5896665Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-syvihmyg 2024-08-20T22:16:16.1331802Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-08-20T22:16:16.1360571Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:16:17.4624041Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:16:17.7790637Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:16:20.2682262Z Preparing metadata (setup.py) ... [?25l- \ done 2024-08-20T22:16:20.2714309Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4) 2024-08-20T22:16:20.2718190Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+git40ec5f6) 2024-08-20T22:16:20.2721923Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-08-20T22:16:20.2791254Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-08-20T22:16:20.2795310Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-08-20T22:16:20.2797739Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-08-20T22:16:20.2800949Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-08-20T22:16:20.2803582Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.6.1) 2024-08-20T22:16:20.2808575Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.1) 2024-08-20T22:16:20.2822304Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-08-20T22:16:20.3296398Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-08-20T22:16:20.3361714Z Building wheels for collected packages: torchvision 2024-08-20T22:17:37.9282080Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-08-20T22:17:37.9315323Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2115988 sha256=6502c4a35a5c38d39ea16854443c395cde78ceb75a3bb0fb49392ff660142ec5 2024-08-20T22:17:37.9318345Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-08-20T22:17:37.9353500Z Successfully built torchvision 2024-08-20T22:17:38.4096772Z Installing collected packages: torchvision 2024-08-20T22:17:38.8257205Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-08-20T22:17:38.9657481Z + '[' -n '' ']' 2024-08-20T22:17:38.9657861Z + id=0 2024-08-20T22:17:38.9660447Z + test_dynamo_benchmark timm_models 0 2024-08-20T22:17:38.9661892Z ++ pwd 2024-08-20T22:17:38.9665274Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:17:38.9665896Z + local suite=timm_models 2024-08-20T22:17:38.9666272Z + shift 2024-08-20T22:17:38.9666550Z + local shard_id=0 2024-08-20T22:17:38.9666923Z + shift 2024-08-20T22:17:38.9667307Z + [[ dynamic_aot_eager_timm == *perf_compare* ]] 2024-08-20T22:17:38.9669222Z + [[ dynamic_aot_eager_timm == *perf* ]] 2024-08-20T22:17:38.9669720Z + [[ dynamic_aot_eager_timm == *cpu* ]] 2024-08-20T22:17:38.9673162Z + [[ dynamic_aot_eager_timm == *aot_inductor* ]] 2024-08-20T22:17:38.9674042Z + test_single_dynamo_benchmark inference timm_models 0 --inference --bfloat16 2024-08-20T22:17:38.9674631Z ++ pwd 2024-08-20T22:17:38.9675101Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:17:38.9675771Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:17:38.9737080Z + local name=inference 2024-08-20T22:17:38.9737565Z + shift 2024-08-20T22:17:38.9737928Z + local suite=timm_models 2024-08-20T22:17:38.9738359Z + shift 2024-08-20T22:17:38.9738632Z + local shard_id=0 2024-08-20T22:17:38.9738929Z + shift 2024-08-20T22:17:38.9739201Z + partition_flags=() 2024-08-20T22:17:38.9742127Z + local partition_flags 2024-08-20T22:17:38.9742686Z + [[ -n 2 ]] 2024-08-20T22:17:38.9743014Z + [[ -n 0 ]] 2024-08-20T22:17:38.9744310Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-08-20T22:17:38.9744999Z + [[ dynamic_aot_eager_timm == *perf_compare* ]] 2024-08-20T22:17:38.9745478Z + [[ dynamic_aot_eager_timm == *perf* ]] 2024-08-20T22:17:38.9745939Z + [[ dynamic_aot_eager_timm == *aot_inductor* ]] 2024-08-20T22:17:38.9746407Z + [[ dynamic_aot_eager_timm == *_avx2* ]] 2024-08-20T22:17:38.9746852Z + [[ dynamic_aot_eager_timm == *_avx512* ]] 2024-08-20T22:17:38.9748706Z + python benchmarks/dynamo/timm_models.py --ci --accuracy --timing --explain --backend aot_eager --dynamic-shapes --dynamic-batch-only --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_timm_models.csv 2024-08-20T22:17:45.4515387Z 2024-08-20T22:17:45.8081481Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:17:45.8081816Z 2024-08-20T22:17:45.9137307Z model.safetensors: 0% 0.00/95.5M [00:00 will be ignored 2024-08-20T22:30:34.5767310Z pass 2024-08-20T22:30:34.6721808Z TIMING: entire_frame_compile:43.06133 backend_compile:25.52249 total_wall_time:43.06133 2024-08-20T22:30:34.6723394Z STATS: call_* op count: 989 | FakeTensorMode.__torch_dispatch__:50729 | attempt fast:1731 | fast is_contiguous:1731 | FakeTensor.__torch_dispatch__:3842 | ProxyTorchDispatchMode.__torch_dispatch__:18627 2024-08-20T22:30:34.6724722Z Dynamo produced 3 graphs covering 989 ops with 7 graph breaks (5 unique) 2024-08-20T22:30:40.1924942Z 2024-08-20T22:30:41.1428052Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:30:41.1428708Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:30:41.1431888Z cuda train botnet26t_256 2024-08-20T22:31:08.1781079Z pass 2024-08-20T22:31:08.1962050Z TIMING: entire_frame_compile:25.51697 backend_compile:14.64751 total_wall_time:25.51697 2024-08-20T22:31:08.1966701Z STATS: call_* op count: 265 | FakeTensorMode.__torch_dispatch__:23304 | attempt fast:839 | fast is_contiguous:824 | FakeTensor.__torch_dispatch__:40 | slow no contiguity match:15 | ProxyTorchDispatchMode.__torch_dispatch__:5527 2024-08-20T22:31:08.1969629Z Dynamo produced 2 graphs covering 265 ops with 6 graph breaks (5 unique) 2024-08-20T22:31:14.7133774Z 2024-08-20T22:31:18.6865814Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:31:18.6866402Z loading model: 0it [00:03, ?it/s] 2024-08-20T22:31:18.6866922Z cuda train cait_m36_384 2024-08-20T22:31:18.6872823Z Traceback (most recent call last): 2024-08-20T22:31:18.6873705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-20T22:31:18.6874505Z self.model_iter_fn(model, example_inputs) 2024-08-20T22:31:18.6875360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 404, in forward_and_backward_pass 2024-08-20T22:31:18.6876243Z pred = mod(*cloned_inputs) 2024-08-20T22:31:18.6877620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:31:18.6878838Z return self._call_impl(*args, **kwargs) 2024-08-20T22:31:18.6879849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:31:18.6880689Z return forward_call(*args, **kwargs) 2024-08-20T22:31:18.6881545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 438, in forward 2024-08-20T22:31:18.6882424Z x = self.forward_features(x) 2024-08-20T22:31:18.6883892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 423, in forward_features 2024-08-20T22:31:18.6884967Z x = self.blocks(x) 2024-08-20T22:31:18.6885892Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:31:18.6886780Z return self._call_impl(*args, **kwargs) 2024-08-20T22:31:18.6887945Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:31:18.6889046Z return forward_call(*args, **kwargs) 2024-08-20T22:31:18.6890122Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/container.py", line 250, in forward 2024-08-20T22:31:18.6890929Z input = module(input) 2024-08-20T22:31:18.6891837Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:31:18.6892730Z return self._call_impl(*args, **kwargs) 2024-08-20T22:31:18.6894076Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:31:18.6894912Z return forward_call(*args, **kwargs) 2024-08-20T22:31:18.6895763Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 199, in forward 2024-08-20T22:31:18.6896633Z x = x + self.drop_path(self.gamma_1 * self.attn(self.norm1(x))) 2024-08-20T22:31:18.6897724Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:31:18.6898628Z return self._call_impl(*args, **kwargs) 2024-08-20T22:31:18.6899569Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:31:18.6900404Z return forward_call(*args, **kwargs) 2024-08-20T22:31:18.6901253Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 153, in forward 2024-08-20T22:31:18.6902059Z x = (attn @ v).transpose(1, 2).reshape(B, N, C) 2024-08-20T22:31:18.6905382Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 16.44 MiB is free. Process 63283 has 21.95 GiB memory in use. Of the allocated memory 21.21 GiB is allocated by PyTorch, and 430.66 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-08-20T22:31:18.6908160Z 2024-08-20T22:31:18.6908456Z The above exception was the direct cause of the following exception: 2024-08-20T22:31:18.6908905Z 2024-08-20T22:31:18.6909049Z Traceback (most recent call last): 2024-08-20T22:31:18.6909709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T22:31:18.6910369Z ) = runner.load_model( 2024-08-20T22:31:18.6911046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 328, in load_model 2024-08-20T22:31:18.6911813Z self.validate_model(model, example_inputs) 2024-08-20T22:31:18.6912574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-20T22:31:18.6913344Z raise RuntimeError("Eager run failed") from e 2024-08-20T22:31:18.6913812Z RuntimeError: Eager run failed 2024-08-20T22:31:18.6914071Z 2024-08-20T22:31:18.6914185Z eager_fail_to_run 2024-08-20T22:31:22.5528292Z 2024-08-20T22:31:23.4574550Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:31:23.4575038Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:31:23.4575472Z cuda train coat_lite_mini 2024-08-20T22:32:10.3532813Z pass 2024-08-20T22:32:10.3730832Z TIMING: entire_frame_compile:45.50049 backend_compile:29.03888 total_wall_time:45.50049 2024-08-20T22:32:10.3734181Z STATS: call_* op count: 456 | FakeTensorMode.__torch_dispatch__:37605 | attempt fast:1467 | fast is_contiguous:1355 | slow no contiguity match:112 | ProxyTorchDispatchMode.__torch_dispatch__:10667 | FakeTensor.__torch_dispatch__:22 2024-08-20T22:32:10.3735706Z Dynamo produced 2 graphs covering 456 ops with 6 graph breaks (5 unique) 2024-08-20T22:32:21.9094902Z 2024-08-20T22:32:23.5535474Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:32:23.5535967Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:32:23.5536399Z cuda train convit_base 2024-08-20T22:32:23.5860544Z WARNING:common:fp64 golden ref were not generated for convit_base. Setting accuracy check to cosine 2024-08-20T22:33:45.0080924Z pass 2024-08-20T22:33:45.0575140Z TIMING: entire_frame_compile:80.43791 backend_compile:52.83143 total_wall_time:80.43791 2024-08-20T22:33:45.0578934Z STATS: call_* op count: 1142 | FakeTensorMode.__torch_dispatch__:91036 | attempt fast:4254 | slow no contiguity match:190 | fast is_contiguous:4064 | ProxyTorchDispatchMode.__torch_dispatch__:28022 | FakeTensor.__torch_dispatch__:24 2024-08-20T22:33:45.0580859Z Dynamo produced 3 graphs covering 1142 ops with 7 graph breaks (5 unique) 2024-08-20T22:33:53.6461886Z 2024-08-20T22:33:54.5171590Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:33:54.5172572Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:33:54.5173535Z cuda train convmixer_768_32 2024-08-20T22:34:33.6258192Z pass 2024-08-20T22:34:33.6943435Z TIMING: entire_frame_compile:32.9496 backend_compile:18.1429 total_wall_time:32.9496 2024-08-20T22:34:33.6946415Z STATS: call_* op count: 300 | FakeTensorMode.__torch_dispatch__:38594 | attempt fast:1663 | fast is_contiguous:1663 | FakeTensor.__torch_dispatch__:26 | ProxyTorchDispatchMode.__torch_dispatch__:9010 2024-08-20T22:34:33.6947766Z Dynamo produced 2 graphs covering 300 ops with 5 graph breaks (4 unique) 2024-08-20T22:34:39.0305289Z 2024-08-20T22:34:40.7283890Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:34:40.7284380Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:34:40.7284819Z cuda train convnext_base 2024-08-20T22:35:33.5262664Z W0820 22:35:33.525000 5197 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T22:36:05.2211145Z pass 2024-08-20T22:36:05.3070945Z TIMING: entire_frame_compile:79.19866 backend_compile:46.25909 total_wall_time:79.19866 2024-08-20T22:36:05.3073458Z STATS: call_* op count: 1508 | FakeTensorMode.__torch_dispatch__:80864 | attempt fast:3151 | fast is_contiguous:2758 | fast channels_last:196 | slow no contiguity match:197 | ProxyTorchDispatchMode.__torch_dispatch__:29746 | FakeTensor.__torch_dispatch__:6313 2024-08-20T22:36:05.3075914Z Dynamo produced 3 graphs covering 1508 ops with 7 graph breaks (5 unique) 2024-08-20T22:36:12.4509848Z 2024-08-20T22:36:13.2500940Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:36:13.2501425Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:36:13.2501867Z cuda train crossvit_9_240 2024-08-20T22:37:16.0935105Z W0820 22:37:16.092000 5295 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T22:37:39.0922609Z pass 2024-08-20T22:37:39.1103769Z TIMING: entire_frame_compile:83.76799 backend_compile:51.97359 total_wall_time:83.76799 2024-08-20T22:37:39.1105617Z STATS: call_* op count: 1312 | FakeTensorMode.__torch_dispatch__:78877 | attempt fast:3070 | fast is_contiguous:3046 | FakeTensor.__torch_dispatch__:5076 | slow no contiguity match:24 | ProxyTorchDispatchMode.__torch_dispatch__:27501 2024-08-20T22:37:39.1107162Z Dynamo produced 3 graphs covering 1312 ops with 7 graph breaks (5 unique) 2024-08-20T22:37:53.5056994Z 2024-08-20T22:37:54.6830514Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:37:54.6830994Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:37:54.6831441Z cuda train cspdarknet53 2024-08-20T22:38:37.3589327Z W0820 22:38:37.358000 5393 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T22:38:58.3389717Z pass 2024-08-20T22:38:58.3681309Z TIMING: entire_frame_compile:60.37279 backend_compile:34.09815 total_wall_time:60.37279 2024-08-20T22:38:58.3682854Z STATS: call_* op count: 950 | FakeTensorMode.__torch_dispatch__:57692 | attempt fast:1999 | fast is_contiguous:1979 | FakeTensor.__torch_dispatch__:4052 | slow no contiguity match:20 | ProxyTorchDispatchMode.__torch_dispatch__:17131 2024-08-20T22:38:58.3686134Z Dynamo produced 3 graphs covering 950 ops with 7 graph breaks (5 unique) 2024-08-20T22:39:04.5431686Z 2024-08-20T22:39:06.1047455Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:39:06.1047935Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:39:06.1048396Z cuda train deit_base_distilled_patch16_224 2024-08-20T22:39:33.9178905Z W0820 22:39:33.916000 5491 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T22:39:47.6273866Z pass 2024-08-20T22:39:47.7172888Z TIMING: entire_frame_compile:36.56089 backend_compile:22.1705 total_wall_time:36.56089 2024-08-20T22:39:47.7175683Z STATS: call_* op count: 753 | FakeTensorMode.__torch_dispatch__:41964 | attempt fast:1609 | fast is_contiguous:1609 | FakeTensor.__torch_dispatch__:3006 | ProxyTorchDispatchMode.__torch_dispatch__:14631 2024-08-20T22:39:47.7177035Z Dynamo produced 3 graphs covering 753 ops with 7 graph breaks (5 unique) 2024-08-20T22:39:52.9881000Z 2024-08-20T22:39:54.2909469Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:39:54.2909952Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:39:54.2910391Z cuda train dla102 2024-08-20T22:40:47.8692255Z W0820 22:40:47.868000 5589 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T22:41:21.5230529Z pass 2024-08-20T22:41:21.5540301Z TIMING: entire_frame_compile:83.34709 backend_compile:46.41665 total_wall_time:83.34709 2024-08-20T22:41:21.5543448Z STATS: call_* op count: 1457 | FakeTensorMode.__torch_dispatch__:84574 | attempt fast:2568 | fast is_contiguous:2529 | FakeTensor.__torch_dispatch__:5872 | ProxyTorchDispatchMode.__torch_dispatch__:26824 | slow no contiguity match:39 2024-08-20T22:41:21.5545006Z Dynamo produced 3 graphs covering 1457 ops with 7 graph breaks (5 unique) 2024-08-20T22:41:28.4084018Z 2024-08-20T22:41:30.1479634Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:41:30.1482172Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:41:30.1482643Z cuda train dm_nfnet_f0 2024-08-20T22:42:05.2285569Z pass 2024-08-20T22:42:05.2797076Z TIMING: entire_frame_compile:31.90834 backend_compile:18.65114 total_wall_time:31.90834 2024-08-20T22:42:05.2798529Z STATS: call_* op count: 594 | FakeTensorMode.__torch_dispatch__:35663 | attempt fast:1872 | fast is_contiguous:1872 | ProxyTorchDispatchMode.__torch_dispatch__:10519 | FakeTensor.__torch_dispatch__:22 2024-08-20T22:42:05.2799847Z Dynamo produced 2 graphs covering 594 ops with 6 graph breaks (5 unique) 2024-08-20T22:42:10.6466504Z 2024-08-20T22:42:12.1852563Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:42:12.1853037Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:42:12.1856808Z cuda train dpn107 2024-08-20T22:43:36.5888371Z pass 2024-08-20T22:43:36.6671279Z TIMING: entire_frame_compile:77.55674 backend_compile:49.16112 total_wall_time:77.55674 2024-08-20T22:43:36.6674050Z STATS: call_* op count: 634 | FakeTensorMode.__torch_dispatch__:71285 | attempt fast:2798 | fast is_contiguous:2596 | FakeTensor.__torch_dispatch__:100 | slow no contiguity match:202 | ProxyTorchDispatchMode.__torch_dispatch__:16915 2024-08-20T22:43:36.6675566Z Dynamo produced 2 graphs covering 634 ops with 6 graph breaks (5 unique) 2024-08-20T22:43:48.3654795Z 2024-08-20T22:43:49.3034350Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:43:49.3036689Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:43:49.3037246Z cuda train eca_botnext26ts_256 2024-08-20T22:44:23.5800574Z W0820 22:44:23.579000 5883 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T22:44:35.4458500Z pass 2024-08-20T22:44:35.4614605Z TIMING: entire_frame_compile:44.54309 backend_compile:25.82275 total_wall_time:44.54309 2024-08-20T22:44:35.4616142Z STATS: call_* op count: 635 | FakeTensorMode.__torch_dispatch__:38418 | attempt fast:1105 | fast is_contiguous:1065 | FakeTensor.__torch_dispatch__:2190 | slow no contiguity match:40 | ProxyTorchDispatchMode.__torch_dispatch__:11488 2024-08-20T22:44:35.4620609Z Dynamo produced 3 graphs covering 635 ops with 7 graph breaks (5 unique) 2024-08-20T22:44:43.0407868Z 2024-08-20T22:44:43.9940608Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:44:43.9941210Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:44:43.9941669Z cuda train eca_halonext26ts 2024-08-20T22:45:23.8406535Z W0820 22:45:23.839000 5981 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T22:45:36.7160134Z pass 2024-08-20T22:45:36.7306305Z TIMING: entire_frame_compile:51.09706 backend_compile:29.20288 total_wall_time:51.09706 2024-08-20T22:45:36.7307850Z STATS: call_* op count: 668 | FakeTensorMode.__torch_dispatch__:39940 | attempt fast:1108 | fast is_contiguous:1068 | FakeTensor.__torch_dispatch__:2240 | slow no contiguity match:40 | ProxyTorchDispatchMode.__torch_dispatch__:11921 2024-08-20T22:45:36.7309382Z Dynamo produced 3 graphs covering 668 ops with 7 graph breaks (5 unique) 2024-08-20T22:45:46.5861283Z 2024-08-20T22:45:47.4163727Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:45:47.4164404Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:45:47.4165098Z cuda train ese_vovnet19b_dw 2024-08-20T22:46:04.0377389Z W0820 22:46:04.036000 6079 torch/_logging/_internal.py:1058] [7/0] Profiler function will be ignored 2024-08-20T22:46:14.4192757Z pass 2024-08-20T22:46:14.4272469Z TIMING: entire_frame_compile:25.98291 backend_compile:14.26569 total_wall_time:25.98291 2024-08-20T22:46:14.4275433Z STATS: call_* op count: 434 | FakeTensorMode.__torch_dispatch__:24285 | attempt fast:613 | fast is_contiguous:601 | FakeTensor.__torch_dispatch__:1842 | ProxyTorchDispatchMode.__torch_dispatch__:7887 | slow no contiguity match:12 2024-08-20T22:46:14.4277029Z Dynamo produced 4 graphs covering 434 ops with 7 graph breaks (5 unique) 2024-08-20T22:46:19.3393898Z 2024-08-20T22:46:20.2016084Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:46:20.2016676Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:46:20.2018294Z cuda train fbnetc_100 2024-08-20T22:46:51.0463696Z W0820 22:46:51.045000 6177 torch/_logging/_internal.py:1058] [7/0] Profiler function will be ignored 2024-08-20T22:47:11.3735590Z pass 2024-08-20T22:47:11.3809357Z TIMING: entire_frame_compile:49.85283 backend_compile:26.77572 total_wall_time:49.85283 2024-08-20T22:47:11.3812497Z STATS: call_* op count: 870 | FakeTensorMode.__torch_dispatch__:50625 | attempt fast:1578 | fast is_contiguous:1578 | FakeTensor.__torch_dispatch__:3794 | ProxyTorchDispatchMode.__torch_dispatch__:16037 2024-08-20T22:47:11.3814162Z Dynamo produced 4 graphs covering 870 ops with 7 graph breaks (5 unique) 2024-08-20T22:47:17.2677182Z 2024-08-20T22:47:18.3423563Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:47:18.3424233Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:47:18.3427576Z cuda train fbnetv3_b 2024-08-20T22:48:37.3990335Z pass 2024-08-20T22:48:37.4126168Z TIMING: entire_frame_compile:77.60462 backend_compile:46.26923 total_wall_time:77.60462 2024-08-20T22:48:37.4129133Z STATS: call_* op count: 459 | FakeTensorMode.__torch_dispatch__:65089 | attempt fast:3567 | fast is_contiguous:3567 | FakeTensor.__torch_dispatch__:364 | ProxyTorchDispatchMode.__torch_dispatch__:11990 2024-08-20T22:48:37.4130475Z Dynamo produced 2 graphs covering 459 ops with 6 graph breaks (5 unique) 2024-08-20T22:48:44.3118925Z 2024-08-20T22:48:45.4928161Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:48:45.4928634Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:48:45.4929072Z cuda train gernet_l 2024-08-20T22:49:14.6966603Z pass 2024-08-20T22:49:14.7193543Z TIMING: entire_frame_compile:27.10396 backend_compile:15.17488 total_wall_time:27.10396 2024-08-20T22:49:14.7196434Z STATS: call_* op count: 250 | FakeTensorMode.__torch_dispatch__:29086 | attempt fast:1409 | fast is_contiguous:1409 | FakeTensor.__torch_dispatch__:38 | ProxyTorchDispatchMode.__torch_dispatch__:6598 2024-08-20T22:49:14.7197785Z Dynamo produced 2 graphs covering 250 ops with 6 graph breaks (5 unique) 2024-08-20T22:49:19.9536294Z 2024-08-20T22:49:20.8983635Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:49:20.8984136Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:49:20.8987781Z cuda train ghostnet_100 2024-08-20T22:50:26.9986984Z pass 2024-08-20T22:50:27.0057070Z TIMING: entire_frame_compile:64.89765 backend_compile:40.13891 total_wall_time:64.89765 2024-08-20T22:50:27.0060443Z STATS: call_* op count: 405 | FakeTensorMode.__torch_dispatch__:49399 | attempt fast:2040 | fast is_contiguous:2008 | FakeTensor.__torch_dispatch__:100 | ProxyTorchDispatchMode.__torch_dispatch__:11967 | slow no contiguity match:32 2024-08-20T22:50:27.0061929Z Dynamo produced 2 graphs covering 405 ops with 6 graph breaks (5 unique) 2024-08-20T22:50:43.9019397Z 2024-08-20T22:50:45.1499826Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:50:45.1500384Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:50:45.1500825Z cuda train gluon_inception_v3 2024-08-20T22:51:32.3375524Z W0820 22:51:32.336000 6569 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T22:52:01.4396570Z pass 2024-08-20T22:52:01.4635913Z TIMING: entire_frame_compile:73.22056 backend_compile:41.08003 total_wall_time:73.22056 2024-08-20T22:52:01.4638872Z STATS: call_* op count: 1281 | FakeTensorMode.__torch_dispatch__:75528 | attempt fast:2205 | fast is_contiguous:2205 | FakeTensor.__torch_dispatch__:5358 | ProxyTorchDispatchMode.__torch_dispatch__:24031 2024-08-20T22:52:01.4640250Z Dynamo produced 3 graphs covering 1281 ops with 7 graph breaks (5 unique) 2024-08-20T22:52:08.3184004Z 2024-08-20T22:52:09.2494476Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:52:09.2496594Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:52:09.2497065Z cuda train gmixer_24_224 2024-08-20T22:53:07.8659446Z pass 2024-08-20T22:53:07.8838845Z TIMING: entire_frame_compile:56.198 backend_compile:36.23849 total_wall_time:56.198 2024-08-20T22:53:07.8841139Z STATS: call_* op count: 587 | FakeTensorMode.__torch_dispatch__:62895 | attempt fast:3111 | fast is_contiguous:2631 | slow no contiguity match:480 | ProxyTorchDispatchMode.__torch_dispatch__:16426 | FakeTensor.__torch_dispatch__:22 2024-08-20T22:53:07.8843348Z Dynamo produced 2 graphs covering 587 ops with 6 graph breaks (5 unique) 2024-08-20T22:53:14.4252104Z 2024-08-20T22:53:15.2742696Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:53:15.2743315Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:53:15.2743817Z cuda train gmlp_s16_224 2024-08-20T22:54:06.3746469Z W0820 22:54:06.373000 6765 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T22:54:33.6194186Z pass 2024-08-20T22:54:33.6453272Z TIMING: entire_frame_compile:75.53998 backend_compile:45.5319 total_wall_time:75.53998 2024-08-20T22:54:33.6456454Z STATS: call_* op count: 1396 | FakeTensorMode.__torch_dispatch__:83898 | attempt fast:3579 | fast is_contiguous:3279 | slow no contiguity match:300 | ProxyTorchDispatchMode.__torch_dispatch__:28113 | FakeTensor.__torch_dispatch__:5572 2024-08-20T22:54:33.6457945Z Dynamo produced 3 graphs covering 1396 ops with 7 graph breaks (5 unique) 2024-08-20T22:54:40.3240657Z 2024-08-20T22:54:41.9998548Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:54:42.0000093Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:54:42.0002146Z cuda train hrnet_w18 2024-08-20T22:57:20.5420516Z pass 2024-08-20T22:57:20.5847165Z TIMING: entire_frame_compile:153.08736 backend_compile:88.61971 total_wall_time:153.08736 2024-08-20T22:57:20.5850998Z STATS: call_* op count: 1475 | FakeTensorMode.__torch_dispatch__:170198 | attempt fast:8144 | fast is_contiguous:8144 | FakeTensor.__torch_dispatch__:46 | ProxyTorchDispatchMode.__torch_dispatch__:39732 2024-08-20T22:57:20.5852346Z Dynamo produced 2 graphs covering 1475 ops with 5 graph breaks (4 unique) 2024-08-20T22:57:33.0319276Z 2024-08-20T22:57:34.2873864Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:57:34.2874463Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:57:34.2878014Z cuda train inception_v3 2024-08-20T22:58:21.8198477Z pass 2024-08-20T22:58:21.8437530Z TIMING: entire_frame_compile:44.93833 backend_compile:26.28163 total_wall_time:44.93833 2024-08-20T22:58:21.8440900Z STATS: call_* op count: 412 | FakeTensorMode.__torch_dispatch__:48768 | attempt fast:2205 | fast is_contiguous:2205 | FakeTensor.__torch_dispatch__:46 | ProxyTorchDispatchMode.__torch_dispatch__:11234 2024-08-20T22:58:21.8442326Z Dynamo produced 2 graphs covering 412 ops with 6 graph breaks (5 unique) 2024-08-20T22:58:28.0862458Z 2024-08-20T22:58:29.6206136Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:58:29.6206779Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:58:29.6207333Z cuda train jx_nest_base 2024-08-20T22:59:57.8502616Z W0820 22:59:57.849000 7059 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:00:26.2808248Z pass 2024-08-20T23:00:26.3481782Z TIMING: entire_frame_compile:110.35984 backend_compile:68.98642 total_wall_time:110.35984 2024-08-20T23:00:26.3485419Z STATS: call_* op count: 1671 | FakeTensorMode.__torch_dispatch__:108192 | attempt fast:3795 | fast is_contiguous:3795 | ProxyTorchDispatchMode.__torch_dispatch__:34323 | FakeTensor.__torch_dispatch__:5627 2024-08-20T23:00:26.3486815Z Dynamo produced 3 graphs covering 1671 ops with 7 graph breaks (5 unique) 2024-08-20T23:00:36.6621690Z 2024-08-20T23:00:37.3667884Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:00:37.3668382Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:00:37.3673287Z cuda train lcnet_050 2024-08-20T23:01:01.7357409Z E0820 23:01:01.734000 7157 torch/_dynamo/utils.py:1688] RMSE (res-fp64): 0.00769, (ref-fp64): 0.00017 and shape=torch.Size([32]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.010000, use_larger_multiplier_for_smaller_tensor: 0 2024-08-20T23:01:01.7359512Z E0820 23:01:01.735000 7157 torch/_dynamo/utils.py:1580] Accuracy failed for key name blocks.1.0.bn2.weight.grad 2024-08-20T23:01:01.7375206Z fail_accuracy 2024-08-20T23:01:01.7397021Z TIMING: entire_frame_compile:23.88107 backend_compile:14.57379 total_wall_time:23.88107 2024-08-20T23:01:01.7399462Z STATS: call_* op count: 129 | FakeTensorMode.__torch_dispatch__:19611 | attempt fast:1081 | fast is_contiguous:1081 | FakeTensor.__torch_dispatch__:148 | ProxyTorchDispatchMode.__torch_dispatch__:3329 2024-08-20T23:01:01.7400792Z Dynamo produced 2 graphs covering 129 ops with 6 graph breaks (5 unique) 2024-08-20T23:01:06.8768184Z 2024-08-20T23:01:07.9275480Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:01:07.9275980Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:01:07.9276438Z cuda train levit_128 2024-08-20T23:02:10.6062113Z W0820 23:02:10.605000 7255 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:02:28.6399725Z pass 2024-08-20T23:02:28.6457145Z TIMING: entire_frame_compile:79.14968 backend_compile:48.24048 total_wall_time:79.14968 2024-08-20T23:02:28.6461422Z STATS: call_* op count: 1261 | FakeTensorMode.__torch_dispatch__:76085 | attempt fast:2375 | fast is_contiguous:2339 | FakeTensor.__torch_dispatch__:3902 | slow no contiguity match:35 | ProxyTorchDispatchMode.__torch_dispatch__:23437 | fast channels_last:1 2024-08-20T23:02:28.6463089Z Dynamo produced 3 graphs covering 1261 ops with 7 graph breaks (5 unique) 2024-08-20T23:02:40.1716799Z 2024-08-20T23:02:41.3748149Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:02:41.3748629Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:02:41.3749069Z cuda train mixer_b16_224 2024-08-20T23:03:08.8321619Z W0820 23:03:08.831000 7353 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:03:21.8849332Z pass 2024-08-20T23:03:21.9541526Z TIMING: entire_frame_compile:36.67392 backend_compile:21.99616 total_wall_time:36.67392 2024-08-20T23:03:21.9543101Z STATS: call_* op count: 670 | FakeTensorMode.__torch_dispatch__:40955 | attempt fast:1839 | fast is_contiguous:1743 | slow no contiguity match:96 | ProxyTorchDispatchMode.__torch_dispatch__:13749 | FakeTensor.__torch_dispatch__:2768 2024-08-20T23:03:21.9545509Z Dynamo produced 3 graphs covering 670 ops with 7 graph breaks (5 unique) 2024-08-20T23:03:24.1074563Z accuracy pass_rate=93.33% 2024-08-20T23:03:24.1077989Z calls_captured gmean=0.00x mean=779.167x 2024-08-20T23:03:24.1081696Z unique_graphs gmean=0.00x mean=2.533x 2024-08-20T23:03:24.1085036Z graph_breaks gmean=0.00x mean=6.267x 2024-08-20T23:03:24.1088723Z unique_graph_breaks gmean=0.00x mean=4.767x 2024-08-20T23:03:24.1092255Z autograd_captures gmean=0.00x mean=0.000x 2024-08-20T23:03:24.1096533Z autograd_compiles gmean=0.00x mean=0.000x 2024-08-20T23:03:24.1100159Z cudagraph_skips gmean=0.00x mean=0.000x 2024-08-20T23:03:24.7679727Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_timm_models.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_aot_eager_timm_training.csv 2024-08-20T23:03:25.0309861Z adv_inception_v3 PASS 2024-08-20T23:03:25.0314432Z beit_base_patch16_224 PASS 2024-08-20T23:03:25.0319817Z botnet26t_256 PASS 2024-08-20T23:03:25.0325077Z cait_m36_384 XFAIL 2024-08-20T23:03:25.0330703Z coat_lite_mini PASS 2024-08-20T23:03:25.0336104Z convit_base PASS 2024-08-20T23:03:25.0341364Z convmixer_768_32 PASS 2024-08-20T23:03:25.0346379Z convnext_base PASS 2024-08-20T23:03:25.0351660Z crossvit_9_240 PASS 2024-08-20T23:03:25.0356739Z cspdarknet53 PASS 2024-08-20T23:03:25.0361980Z deit_base_distilled_patch16_224 PASS 2024-08-20T23:03:25.0367065Z dla102 PASS 2024-08-20T23:03:25.0372121Z dm_nfnet_f0 PASS 2024-08-20T23:03:25.0377484Z dpn107 PASS 2024-08-20T23:03:25.0382579Z eca_botnext26ts_256 PASS 2024-08-20T23:03:25.0387760Z eca_halonext26ts PASS 2024-08-20T23:03:25.0392806Z ese_vovnet19b_dw PASS 2024-08-20T23:03:25.0397697Z fbnetc_100 PASS 2024-08-20T23:03:25.0402773Z fbnetv3_b PASS 2024-08-20T23:03:25.0407817Z gernet_l PASS 2024-08-20T23:03:25.0412987Z ghostnet_100 PASS 2024-08-20T23:03:25.0418265Z gluon_inception_v3 PASS 2024-08-20T23:03:25.0423331Z gmixer_24_224 PASS 2024-08-20T23:03:25.0429101Z gmlp_s16_224 PASS 2024-08-20T23:03:25.0434277Z hrnet_w18 PASS 2024-08-20T23:03:25.0439262Z inception_v3 PASS 2024-08-20T23:03:25.0444516Z jx_nest_base PASS 2024-08-20T23:03:25.0449676Z lcnet_050 XFAIL 2024-08-20T23:03:25.0455148Z levit_128 PASS 2024-08-20T23:03:25.0460555Z mixer_b16_224 PASS 2024-08-20T23:03:25.0892404Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_timm_models.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_aot_eager_timm_training.csv 2024-08-20T23:03:25.3503790Z adv_inception_v3 PASS 2024-08-20T23:03:25.3507905Z beit_base_patch16_224 PASS 2024-08-20T23:03:25.3512738Z botnet26t_256 PASS 2024-08-20T23:03:25.3517611Z cait_m36_384 PASS 2024-08-20T23:03:25.3522269Z coat_lite_mini PASS 2024-08-20T23:03:25.3527472Z convit_base PASS 2024-08-20T23:03:25.3532408Z convmixer_768_32 PASS 2024-08-20T23:03:25.3537440Z convnext_base PASS 2024-08-20T23:03:25.3542053Z crossvit_9_240 PASS 2024-08-20T23:03:25.3546893Z cspdarknet53 PASS 2024-08-20T23:03:25.3551533Z deit_base_distilled_patch16_224 PASS 2024-08-20T23:03:25.3556282Z dla102 PASS 2024-08-20T23:03:25.3560859Z dm_nfnet_f0 PASS 2024-08-20T23:03:25.3565522Z dpn107 PASS 2024-08-20T23:03:25.3570139Z eca_botnext26ts_256 PASS 2024-08-20T23:03:25.3575598Z eca_halonext26ts PASS 2024-08-20T23:03:25.3580223Z ese_vovnet19b_dw PASS 2024-08-20T23:03:25.3585004Z fbnetc_100 PASS 2024-08-20T23:03:25.3589844Z fbnetv3_b PASS 2024-08-20T23:03:25.3594739Z gernet_l PASS 2024-08-20T23:03:25.3599545Z ghostnet_100 PASS 2024-08-20T23:03:25.3604487Z gluon_inception_v3 PASS 2024-08-20T23:03:25.3609236Z gmixer_24_224 PASS 2024-08-20T23:03:25.3614206Z gmlp_s16_224 PASS 2024-08-20T23:03:25.3619126Z hrnet_w18 PASS 2024-08-20T23:03:25.3623969Z inception_v3 PASS 2024-08-20T23:03:25.3629171Z jx_nest_base PASS 2024-08-20T23:03:25.3634128Z lcnet_050 PASS 2024-08-20T23:03:25.3638882Z levit_128 PASS 2024-08-20T23:03:25.3643721Z mixer_b16_224 PASS 2024-08-20T23:03:25.4074527Z + cleanup_workspace 2024-08-20T23:03:25.4075384Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-20T23:03:25.4076467Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-20T23:03:25.4077390Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-20T23:03:25.4079891Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-20T23:03:25.4080949Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-20T23:03:25.4081856Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-20T23:03:25.4082572Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-20T23:03:26.1562040Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-20T23:03:26.1562565Z cat test/**/*_toprint.log || true 2024-08-20T23:03:26.1576197Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:03:26.1576702Z env: 2024-08-20T23:03:26.1576976Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:03:26.1577431Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:03:26.1578184Z DOCKER_CONTAINER_ID: 6fb7c9a8debe00f72ec404e19beb53e46f5d9d9b4ed5e85a10f4882d30d1f88a 2024-08-20T23:03:26.1578845Z ##[endgroup] 2024-08-20T23:03:26.1672637Z cat: 'test/**/*_toprint.log': No such file or directory 2024-08-20T23:03:26.1706504Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-20T23:03:26.1706971Z kill "$MONITOR_SCRIPT_PID" 2024-08-20T23:03:26.1715979Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:03:26.1716489Z env: 2024-08-20T23:03:26.1716785Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:03:26.1717239Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:03:26.1717987Z DOCKER_CONTAINER_ID: 6fb7c9a8debe00f72ec404e19beb53e46f5d9d9b4ed5e85a10f4882d30d1f88a 2024-08-20T23:03:26.1718660Z MONITOR_SCRIPT_PID: 57492 2024-08-20T23:03:26.1719027Z ##[endgroup] 2024-08-20T23:03:26.1918385Z Prepare all required actions 2024-08-20T23:03:26.1918896Z Getting action download info 2024-08-20T23:03:26.3869462Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-20T23:03:26.5676366Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-20T23:03:26.5676844Z with: 2024-08-20T23:03:26.5677378Z file-suffix: test-dynamic_aot_eager_timm-1-2-linux.g5.4xlarge.nvidia.gpu_29026300491 2024-08-20T23:03:26.5678043Z s3-bucket: gha-artifacts 2024-08-20T23:03:26.5678380Z env: 2024-08-20T23:03:26.5678666Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:03:26.5679122Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:03:26.5680101Z DOCKER_CONTAINER_ID: 6fb7c9a8debe00f72ec404e19beb53e46f5d9d9b4ed5e85a10f4882d30d1f88a 2024-08-20T23:03:26.5680755Z ##[endgroup] 2024-08-20T23:03:26.5708060Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-20T23:03:26.5708714Z # Remove any previous test jsons if they exist 2024-08-20T23:03:26.5709225Z rm -f test-jsons-*.zip 2024-08-20T23:03:26.5709737Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-20T23:03:26.5718784Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:03:26.5719285Z env: 2024-08-20T23:03:26.5719570Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:03:26.5720013Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:03:26.5720761Z DOCKER_CONTAINER_ID: 6fb7c9a8debe00f72ec404e19beb53e46f5d9d9b4ed5e85a10f4882d30d1f88a 2024-08-20T23:03:26.5721664Z FILE_SUFFIX: test-dynamic_aot_eager_timm-1-2-linux.g5.4xlarge.nvidia.gpu_29026300491 2024-08-20T23:03:26.5722319Z ##[endgroup] 2024-08-20T23:03:26.6115925Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-20T23:03:26.6148094Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-20T23:03:26.6148802Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-20T23:03:26.6154918Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-20T23:03:26.6160651Z adding: test/slow_tests.json (deflated 82%) 2024-08-20T23:03:26.6194096Z ##[group]Run # Remove any previous test reports if they exist 2024-08-20T23:03:26.6194751Z # Remove any previous test reports if they exist 2024-08-20T23:03:26.6195292Z rm -f test-reports-*.zip 2024-08-20T23:03:26.6195897Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-20T23:03:26.6204993Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:03:26.6205483Z env: 2024-08-20T23:03:26.6205768Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:03:26.6206247Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:03:26.6207042Z DOCKER_CONTAINER_ID: 6fb7c9a8debe00f72ec404e19beb53e46f5d9d9b4ed5e85a10f4882d30d1f88a 2024-08-20T23:03:26.6207959Z FILE_SUFFIX: test-dynamic_aot_eager_timm-1-2-linux.g5.4xlarge.nvidia.gpu_29026300491 2024-08-20T23:03:26.6208623Z ##[endgroup] 2024-08-20T23:03:26.6472347Z adding: test/test-reports/inference_timm_models.csv (deflated 68%) 2024-08-20T23:03:26.6473172Z adding: test/test-reports/training_timm_models.csv (deflated 65%) 2024-08-20T23:03:26.6477136Z adding: test/test-reports/training_timm_model_graph_breaks.csv (deflated 97%) 2024-08-20T23:03:26.6478090Z adding: test/test-reports/training_timm_model_graph_break_deduped.csv (deflated 73%) 2024-08-20T23:03:26.6510856Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-20T23:03:26.6511483Z # Remove any previous usage logs if they exist 2024-08-20T23:03:26.6511985Z rm -f logs-*.zip 2024-08-20T23:03:26.6512673Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-20T23:03:26.6513460Z # so check to see if the file exists first 2024-08-20T23:03:26.6513961Z if [ -f 'usage_log.txt' ]; then 2024-08-20T23:03:26.6514713Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-20T23:03:26.6515212Z fi 2024-08-20T23:03:26.6515576Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-20T23:03:26.6516152Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-20T23:03:26.6516652Z fi 2024-08-20T23:03:26.6525347Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:03:26.6525845Z env: 2024-08-20T23:03:26.6526116Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:03:26.6526573Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:03:26.6527729Z DOCKER_CONTAINER_ID: 6fb7c9a8debe00f72ec404e19beb53e46f5d9d9b4ed5e85a10f4882d30d1f88a 2024-08-20T23:03:26.6528833Z FILE_SUFFIX: test-dynamic_aot_eager_timm-1-2-linux.g5.4xlarge.nvidia.gpu_29026300491 2024-08-20T23:03:26.6529473Z ##[endgroup] 2024-08-20T23:03:26.6604004Z adding: usage_log.txt (deflated 92%) 2024-08-20T23:03:26.6693196Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-20T23:03:26.6694004Z # Remove any previous debugging artifacts if they exist 2024-08-20T23:03:26.6694550Z rm -f debug-*.zip 2024-08-20T23:03:26.6694928Z if [ -d 'test/debug' ]; then 2024-08-20T23:03:26.6695423Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-20T23:03:26.6695898Z fi 2024-08-20T23:03:26.6704530Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:03:26.6705037Z env: 2024-08-20T23:03:26.6705311Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:03:26.6705763Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:03:26.6706516Z DOCKER_CONTAINER_ID: 6fb7c9a8debe00f72ec404e19beb53e46f5d9d9b4ed5e85a10f4882d30d1f88a 2024-08-20T23:03:26.6707489Z FILE_SUFFIX: test-dynamic_aot_eager_timm-1-2-linux.g5.4xlarge.nvidia.gpu_29026300491 2024-08-20T23:03:26.6708134Z ##[endgroup] 2024-08-20T23:03:26.6835850Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:03:26.6836301Z with: 2024-08-20T23:03:26.6836595Z s3-bucket: gha-artifacts 2024-08-20T23:03:26.6837037Z s3-prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T23:03:26.6837561Z retention-days: 14 2024-08-20T23:03:26.6837902Z if-no-files-found: warn 2024-08-20T23:03:26.6838255Z path: test-jsons-*.zip 2024-08-20T23:03:26.6838604Z name: artifact 2024-08-20T23:03:26.6838915Z region: us-east-1 2024-08-20T23:03:26.6839222Z env: 2024-08-20T23:03:26.6839493Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:03:26.6839952Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:03:26.6840707Z DOCKER_CONTAINER_ID: 6fb7c9a8debe00f72ec404e19beb53e46f5d9d9b4ed5e85a10f4882d30d1f88a 2024-08-20T23:03:26.6841367Z ##[endgroup] 2024-08-20T23:03:27.0500345Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T23:03:27.0501239Z With the provided path, there will be 1 file uploaded 2024-08-20T23:03:27.0502031Z Uploading to s3 prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T23:03:27.2143920Z Starting upload of test-jsons-test-dynamic_aot_eager_timm-1-2-linux.g5.4xlarge.nvidia.gpu_29026300491.zip 2024-08-20T23:03:27.3274796Z Finished upload of test-jsons-test-dynamic_aot_eager_timm-1-2-linux.g5.4xlarge.nvidia.gpu_29026300491.zip 2024-08-20T23:03:27.3558865Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:03:27.3559320Z with: 2024-08-20T23:03:27.3559601Z s3-bucket: gha-artifacts 2024-08-20T23:03:27.3560038Z s3-prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T23:03:27.3560514Z retention-days: 14 2024-08-20T23:03:27.3560862Z if-no-files-found: error 2024-08-20T23:03:27.3561227Z path: test-reports-*.zip 2024-08-20T23:03:27.3561581Z name: artifact 2024-08-20T23:03:27.3561892Z region: us-east-1 2024-08-20T23:03:27.3562211Z env: 2024-08-20T23:03:27.3562500Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:03:27.3562958Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:03:27.3563714Z DOCKER_CONTAINER_ID: 6fb7c9a8debe00f72ec404e19beb53e46f5d9d9b4ed5e85a10f4882d30d1f88a 2024-08-20T23:03:27.3564372Z ##[endgroup] 2024-08-20T23:03:27.7196116Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T23:03:27.7196745Z With the provided path, there will be 1 file uploaded 2024-08-20T23:03:27.7197378Z Uploading to s3 prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T23:03:27.7254222Z Starting upload of test-reports-test-dynamic_aot_eager_timm-1-2-linux.g5.4xlarge.nvidia.gpu_29026300491.zip 2024-08-20T23:03:27.8366695Z Finished upload of test-reports-test-dynamic_aot_eager_timm-1-2-linux.g5.4xlarge.nvidia.gpu_29026300491.zip 2024-08-20T23:03:27.8649219Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:03:27.8649670Z with: 2024-08-20T23:03:27.8649950Z s3-bucket: gha-artifacts 2024-08-20T23:03:27.8650576Z s3-prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T23:03:27.8651063Z retention-days: 14 2024-08-20T23:03:27.8651412Z if-no-files-found: ignore 2024-08-20T23:03:27.8651772Z path: logs-*.zip 2024-08-20T23:03:27.8652093Z name: artifact 2024-08-20T23:03:27.8652406Z region: us-east-1 2024-08-20T23:03:27.8652727Z env: 2024-08-20T23:03:27.8653002Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:03:27.8653537Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:03:27.8654297Z DOCKER_CONTAINER_ID: 6fb7c9a8debe00f72ec404e19beb53e46f5d9d9b4ed5e85a10f4882d30d1f88a 2024-08-20T23:03:27.8654960Z ##[endgroup] 2024-08-20T23:03:28.2251642Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T23:03:28.2252375Z With the provided path, there will be 1 file uploaded 2024-08-20T23:03:28.2253006Z Uploading to s3 prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T23:03:28.2308431Z Starting upload of logs-test-dynamic_aot_eager_timm-1-2-linux.g5.4xlarge.nvidia.gpu_29026300491.zip 2024-08-20T23:03:28.3492149Z Finished upload of logs-test-dynamic_aot_eager_timm-1-2-linux.g5.4xlarge.nvidia.gpu_29026300491.zip 2024-08-20T23:03:28.3774660Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:03:28.3775109Z with: 2024-08-20T23:03:28.3775399Z s3-bucket: gha-artifacts 2024-08-20T23:03:28.3775856Z s3-prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T23:03:28.3776341Z retention-days: 14 2024-08-20T23:03:28.3776691Z if-no-files-found: ignore 2024-08-20T23:03:28.3777051Z path: debug-*.zip 2024-08-20T23:03:28.3777371Z name: artifact 2024-08-20T23:03:28.3777686Z region: us-east-1 2024-08-20T23:03:28.3778002Z env: 2024-08-20T23:03:28.3778276Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:03:28.3778742Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:03:28.3779490Z DOCKER_CONTAINER_ID: 6fb7c9a8debe00f72ec404e19beb53e46f5d9d9b4ed5e85a10f4882d30d1f88a 2024-08-20T23:03:28.3780147Z ##[endgroup] 2024-08-20T23:03:28.7274797Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-20T23:03:28.7562602Z ##[group]Run # shellcheck disable=SC2156 2024-08-20T23:03:28.7563084Z # shellcheck disable=SC2156 2024-08-20T23:03:28.7563881Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-20T23:03:28.7573213Z shell: /usr/bin/bash -e {0} 2024-08-20T23:03:28.7573654Z env: 2024-08-20T23:03:28.7573943Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:03:28.7574403Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:03:28.7575153Z DOCKER_CONTAINER_ID: 6fb7c9a8debe00f72ec404e19beb53e46f5d9d9b4ed5e85a10f4882d30d1f88a 2024-08-20T23:03:28.7575798Z ##[endgroup] 2024-08-20T23:03:29.0222283Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-20T23:03:29.0222853Z with: 2024-08-20T23:03:29.0223103Z env: 2024-08-20T23:03:29.0223378Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:03:29.0223848Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:03:29.0224599Z DOCKER_CONTAINER_ID: 6fb7c9a8debe00f72ec404e19beb53e46f5d9d9b4ed5e85a10f4882d30d1f88a 2024-08-20T23:03:29.0225252Z ##[endgroup] 2024-08-20T23:03:29.0245515Z ##[group]Run set -eou pipefail 2024-08-20T23:03:29.0245918Z set -eou pipefail 2024-08-20T23:03:29.0246271Z  2024-08-20T23:03:29.0246791Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-20T23:03:29.0247473Z for _ in $(seq 1440); do 2024-08-20T23:03:29.0247971Z  # Break if no ssh session exists anymore 2024-08-20T23:03:29.0248479Z  if [ "$(who)" = "" ]; then 2024-08-20T23:03:29.0248893Z  break 2024-08-20T23:03:29.0249240Z  fi 2024-08-20T23:03:29.0249550Z  echo "." 2024-08-20T23:03:29.0249892Z  sleep 5 2024-08-20T23:03:29.0250214Z done 2024-08-20T23:03:29.0259228Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:03:29.0259935Z env: 2024-08-20T23:03:29.0260224Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:03:29.0260689Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:03:29.0261456Z DOCKER_CONTAINER_ID: 6fb7c9a8debe00f72ec404e19beb53e46f5d9d9b4ed5e85a10f4882d30d1f88a 2024-08-20T23:03:29.0262138Z ##[endgroup] 2024-08-20T23:03:29.0291892Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-20T23:03:29.0395565Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T23:03:29.0396334Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T23:03:29.0396940Z # shellcheck disable=SC2046 2024-08-20T23:03:29.0397421Z docker stop $(docker ps -q) || true 2024-08-20T23:03:29.0397903Z # Prune all of the docker images 2024-08-20T23:03:29.0398358Z docker system prune -af 2024-08-20T23:03:29.0406443Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:03:29.0406961Z env: 2024-08-20T23:03:29.0407236Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:03:29.0407696Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:03:29.0408463Z DOCKER_CONTAINER_ID: 6fb7c9a8debe00f72ec404e19beb53e46f5d9d9b4ed5e85a10f4882d30d1f88a 2024-08-20T23:03:29.0409115Z ##[endgroup] 2024-08-20T23:03:29.6210266Z 6fb7c9a8debe 2024-08-20T23:03:31.0726878Z Deleted Containers: 2024-08-20T23:03:31.0727583Z 6fb7c9a8debe00f72ec404e19beb53e46f5d9d9b4ed5e85a10f4882d30d1f88a 2024-08-20T23:03:31.0728147Z 2024-08-20T23:03:41.7232419Z Deleted Images: 2024-08-20T23:03:41.7234624Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T23:03:41.7237286Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T23:03:41.7238863Z deleted: sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb 2024-08-20T23:03:41.7239731Z deleted: sha256:c2ce10014dd98f6abbfbf2265f9780a8204c2a7a27463e61ee565573a23d141d 2024-08-20T23:03:41.7240590Z deleted: sha256:ae249c0e6e6013bd6ffea2ca0b0160c015c9ebf7c03070bc2f983a9ff2415bf5 2024-08-20T23:03:41.7241434Z deleted: sha256:8a73883c1067c0e29ff128f553518682b47ed9707d2f35e7d5138e2caf029d61 2024-08-20T23:03:41.7242250Z deleted: sha256:9c040db729fa084584aaf5573082408a2196bc64e5952270c79dc610ed4680b6 2024-08-20T23:03:41.7243077Z deleted: sha256:17311eeeb8b71b16699880c0f646e185107496c4d41da5d23370603768ec030c 2024-08-20T23:03:41.7243949Z deleted: sha256:07cc10bece8501b1c8035d4385ad9e9867dd6e92fa32218a4dcb3e42af7e0411 2024-08-20T23:03:41.7245123Z deleted: sha256:f3e32d05099d1a3115a4722bc2e94a48752c3e965446773673808252671bb967 2024-08-20T23:03:41.7245969Z deleted: sha256:be7c3f41475e3e72ffe3ac2f92ab2f4cf49d5496b2aae738942ac3bb687d742c 2024-08-20T23:03:41.7246821Z deleted: sha256:4fcac490be825129b468cf458089f1fcf8fd56afc44ef4ab27c2eff41ab1ac90 2024-08-20T23:03:41.7247658Z deleted: sha256:31c66d80a2a4792897f267d363c0cf84fb821c5d4a275ab1636f778fd0284c08 2024-08-20T23:03:41.7248490Z deleted: sha256:8a8bcb6085e7e438e5533e8a5613e2ae5888543853568d023af3e2623a1593a9 2024-08-20T23:03:41.7249334Z deleted: sha256:2f6cb60029c79d11567b738f5bdbfb9cb1838cc0f194c4a5e6ae685178d2ed34 2024-08-20T23:03:41.7250179Z deleted: sha256:e546391804d3313f558e1ac16e93cb29a417a92e4eccaf0d4bd12af3bbcf39e3 2024-08-20T23:03:41.7251012Z deleted: sha256:1623b2aa16f2bdc6b572f6799498d9a92fa8114d07c440599b17aea72e22f8b1 2024-08-20T23:03:41.7251859Z deleted: sha256:c0386ca58b7cb7afee015b7e58d2f70525eaf73d6da2cc63868b20e14b1f9497 2024-08-20T23:03:41.7252704Z deleted: sha256:dcbf534ff289186e246531925fed90f72516eb3d63663bad234d611cdc025d62 2024-08-20T23:03:41.7253834Z deleted: sha256:0bab537af579f56c9cd51c7073fd53635fffb424b363236e0de38a7dd0e60ae8 2024-08-20T23:03:41.7254754Z deleted: sha256:d85a852ae1ee027e2fc17240ed68b87957b0a6d210c9119752d8979a0dcd55cf 2024-08-20T23:03:41.7255876Z deleted: sha256:f3821d761b555362e44cf9a1447d8fd2d95b37d54aea67c7519cd9e5db0c4d7e 2024-08-20T23:03:41.7256808Z deleted: sha256:c05e84a545db5dfcb0847c73d2b0f4749fa67aabce05705632ff7f76c84d4688 2024-08-20T23:03:41.7257746Z deleted: sha256:27054eae7ca67b2bd5ca0fd3f4488bf5c5f78e95487d8c45616f3d753d48a8b5 2024-08-20T23:03:41.7258669Z deleted: sha256:9804139f6ae97796aaa993fe4b89ea0cf111eb4d30a79f4c87c339c2e1e04fa0 2024-08-20T23:03:41.7259587Z deleted: sha256:ad6ae2fc594aaa359ff4e287323836570454eb24dd162930655e414d1d77c6dd 2024-08-20T23:03:41.7260510Z deleted: sha256:9cfbb36e328af52a248853a46aab88e266c09e2b44a03e0a42194a031b9b9453 2024-08-20T23:03:41.7261448Z deleted: sha256:a8a842bd7f37b8d50b6d12d6f95f46bbba57953db466fcbde9ba522625f024ef 2024-08-20T23:03:41.7262375Z deleted: sha256:da9738654b3c86f89ebeed2560f1bae093f1219f1eff90b0394b7f211ecabd04 2024-08-20T23:03:41.7263310Z deleted: sha256:cc683bf0cb245394e0f568984b2dd4fce1b6a3a7fe7c44734bae1aa1caaa3251 2024-08-20T23:03:41.7264252Z deleted: sha256:965f220e71c7d1ec19aaf41b1529ab287beb9d86eb8482c4c14094485ab34475 2024-08-20T23:03:41.7265178Z deleted: sha256:da32ecec41a73ce7dd29464f90c7951510a18cd378753659a78cd7fabd10dc36 2024-08-20T23:03:41.7266111Z deleted: sha256:d828d187cee9ad48dbc3702fb61d359619dd8a1ed582878680b51686bf117e5a 2024-08-20T23:03:41.7267046Z deleted: sha256:712111f4b12ff8e2787597d5fdcb043bd0824e55ddd97f262ef550506ca195f0 2024-08-20T23:03:41.7267898Z deleted: sha256:a064eea405958ad14dc6c794e5df9fd1d4515424272801ec559d45a59317309a 2024-08-20T23:03:41.7268724Z deleted: sha256:e124382d136bd5e9a8b409ca79c1f1ef76458deaf444b1b2060d3fa99dbeb506 2024-08-20T23:03:41.7269549Z deleted: sha256:6e10fe757cc190a1b31b47010f4f7610e02cb1b4b438f190dcd301aa4fe2278e 2024-08-20T23:03:41.7270363Z deleted: sha256:4cea352e10f46ca4729f96ac6c4600fc53d4c36e7658c2e0f19f4f309cd4a2a7 2024-08-20T23:03:41.7271191Z deleted: sha256:518bba00fe1d0933053066d21cac0235c9fb485abbc41557ec184b293095c924 2024-08-20T23:03:41.7272027Z deleted: sha256:a64f9ca42e9f27ff2690c15a5c84312e6efaccad45668d9b496cb542165cfb5a 2024-08-20T23:03:41.7272855Z deleted: sha256:096ea0fa14a6c420658f5276b8a8f5d264c0de7e18161ceda2cfc2f4a92cd9bf 2024-08-20T23:03:41.7273674Z deleted: sha256:6f91c94ce6e3f8034c91c66ee25c66a3d85dba9d813b749bd6d2000942a99edc 2024-08-20T23:03:41.7274502Z deleted: sha256:8f8d255eeae546cead8f9a94d237fa57a99940c0f5d51bc29a9d41c7df9cffd9 2024-08-20T23:03:41.7275342Z deleted: sha256:fe05276f4a49f91eef56bfbc3da87cfff5eb116a827fa2d30efe42ce1c134b40 2024-08-20T23:03:41.7276167Z deleted: sha256:5f83cd75d0669c813d4f58978c92e14305cd8f6a07eb9a60665a583460fa7b59 2024-08-20T23:03:41.7277059Z deleted: sha256:8f7ab6366169d39d8953129f650f1927d2fb1328b3f11e2fd9076adb5f46b93a 2024-08-20T23:03:41.7278264Z deleted: sha256:4146cfd5dba0f5308954749f98596b14a1d8f6d23bd93f7ad86c6c3d72239328 2024-08-20T23:03:41.7279353Z deleted: sha256:c6d83cb9ccc606ff63bb3c4d5a5e1ff1450c619b5aae494303a94ae25971cca0 2024-08-20T23:03:41.7280431Z deleted: sha256:71e037e0a3e0963c4565534a7387c082afcb04475d3ac629acf7a0ab2f49fadc 2024-08-20T23:03:41.7281526Z deleted: sha256:4b31135c564686bdaf23262b901237880aea635655721cc7e8c95dc42b05e06d 2024-08-20T23:03:41.7282395Z deleted: sha256:d09bd5c2f35cedcb846543941ee64ce1f684b52e4285696c93b3fb2032da8579 2024-08-20T23:03:41.7283232Z deleted: sha256:4de7fb7d4fc22212aae5fc1ceb637526916d0cced240d7c93d5edd6b0ea28013 2024-08-20T23:03:41.7284055Z deleted: sha256:69890e3e82746e10aff2f1084f11afb4b95a4a6c26b30f64b7001bc0a5fe32bf 2024-08-20T23:03:41.7284861Z deleted: sha256:0156df53b3224e645336ea52b4bb816f1e8d8c9b591748cd645e5e5649fa507b 2024-08-20T23:03:41.7285665Z deleted: sha256:0664624cb9cc4a386f31bf49919a8d8a11d9fea428d05585d9e6228f754bc7cd 2024-08-20T23:03:41.7286489Z deleted: sha256:8c2ddd5fa535c768c058a23f5cc4b6c64cdd71bb7458491282aabc7f7bb94d69 2024-08-20T23:03:41.7287316Z deleted: sha256:8dd03dbe0e1c5f47dd333074dff4bd00415f2f63727a65bf17866774e2ebbd0d 2024-08-20T23:03:41.7288136Z deleted: sha256:3c56b63e094bd18fad9da7a541d537210bca3b730c590d3641213259f6908d6f 2024-08-20T23:03:41.7289053Z deleted: sha256:51484fef07bd1266d4d6ff3d767751172660fc4cc6a1ffe883523a63c985e31e 2024-08-20T23:03:41.7289867Z deleted: sha256:03e83c94023982ab7dd8dc65a6df2024e342f1b760c91ec7a680ee45824869c7 2024-08-20T23:03:41.7290684Z deleted: sha256:58440ab3e8175b5de94a76485c8689a6dc186015475ebf4b386ebba309c5da34 2024-08-20T23:03:41.7291502Z deleted: sha256:9a980f3383e8aa511f0ef68982cb6dfc53b38d1ac8c3d673cb3ba3d45158c91e 2024-08-20T23:03:41.7292331Z deleted: sha256:cb0269c9cf1c47b6c4188ca7be4db261b5d3fc4b4a5153629e222dba7b1f9f98 2024-08-20T23:03:41.7293161Z deleted: sha256:41f1ba870cdffc05e0bf8d87b640fb9d4c153d7f4f3b775f37f94e9f3d584aad 2024-08-20T23:03:41.7294148Z deleted: sha256:7b502ab352f48e8b6eb5e6824a29f308952a5ee7c7d987562e1df7e47639d7f1 2024-08-20T23:03:41.7294961Z deleted: sha256:6ba5217e18738660e6ca8b7a7ec88845e70b0fd781301520bf46c71d99df3144 2024-08-20T23:03:41.7295781Z deleted: sha256:dd5a1c233594780163726bdbbf5de00f7b6b8dca1dace8a001340714c54baedb 2024-08-20T23:03:41.7296603Z deleted: sha256:c27f0db27da73e01f67f6cb78d10f355e350d829479ba4351d82055bb29f6597 2024-08-20T23:03:41.7297427Z deleted: sha256:4e3ce3c854f35c08be3dd95e0d6e718512a885e4614291d7d8b7cd23f48b7dd8 2024-08-20T23:03:41.7298256Z deleted: sha256:c2d6fb088bbec16e60cf0ad74b016f7287c92e659ec2db01271bc309b4dc6d83 2024-08-20T23:03:41.7299095Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-08-20T23:03:41.7299911Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-08-20T23:03:41.7300720Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-08-20T23:03:41.7301535Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-08-20T23:03:41.7302365Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-08-20T23:03:41.7303184Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-08-20T23:03:41.7304008Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-08-20T23:03:41.7304863Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-08-20T23:03:41.7305704Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-08-20T23:03:41.7306538Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-08-20T23:03:41.7307358Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-08-20T23:03:41.7307852Z 2024-08-20T23:03:41.7307998Z Total reclaimed space: 44.92GB 2024-08-20T23:03:41.7374343Z Post job cleanup. 2024-08-20T23:03:41.7424895Z Post job cleanup. 2024-08-20T23:03:41.8312249Z [command]/usr/bin/git version 2024-08-20T23:03:41.8364359Z git version 2.40.1 2024-08-20T23:03:41.8402062Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/b5170738-d799-495a-9d09-2f9bc5a6adae' before making global git config changes 2024-08-20T23:03:41.8403311Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T23:03:41.8407266Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T23:03:41.8455185Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T23:03:41.8499347Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T23:03:41.8902212Z Entering 'android/libs/fbjni' 2024-08-20T23:03:41.8972445Z Entering 'third_party/FP16' 2024-08-20T23:03:41.9041780Z Entering 'third_party/FXdiv' 2024-08-20T23:03:41.9110475Z Entering 'third_party/NNPACK' 2024-08-20T23:03:41.9185210Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T23:03:41.9254520Z Entering 'third_party/XNNPACK' 2024-08-20T23:03:41.9339741Z Entering 'third_party/benchmark' 2024-08-20T23:03:41.9414147Z Entering 'third_party/cpp-httplib' 2024-08-20T23:03:41.9482174Z Entering 'third_party/cpuinfo' 2024-08-20T23:03:41.9551217Z Entering 'third_party/cudnn_frontend' 2024-08-20T23:03:41.9619819Z Entering 'third_party/cutlass' 2024-08-20T23:03:41.9697180Z Entering 'third_party/eigen' 2024-08-20T23:03:41.9767764Z Entering 'third_party/fbgemm' 2024-08-20T23:03:41.9834766Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T23:03:41.9899809Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T23:03:41.9966664Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T23:03:42.0040293Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T23:03:42.0105473Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T23:03:42.0175453Z Entering 'third_party/flatbuffers' 2024-08-20T23:03:42.0247653Z Entering 'third_party/fmt' 2024-08-20T23:03:42.0317086Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T23:03:42.0387603Z Entering 'third_party/gloo' 2024-08-20T23:03:42.0457660Z Entering 'third_party/googletest' 2024-08-20T23:03:42.0526345Z Entering 'third_party/ideep' 2024-08-20T23:03:42.0592706Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T23:03:42.0667881Z Entering 'third_party/ittapi' 2024-08-20T23:03:42.0738562Z Entering 'third_party/kineto' 2024-08-20T23:03:42.0805545Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T23:03:42.0870548Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T23:03:42.0941342Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T23:03:42.1008974Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T23:03:42.1076565Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T23:03:42.1143206Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T23:03:42.1215744Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T23:03:42.1282155Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T23:03:42.1349584Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T23:03:42.1416988Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T23:03:42.1487582Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T23:03:42.1552852Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T23:03:42.1623303Z Entering 'third_party/mimalloc' 2024-08-20T23:03:42.1692897Z Entering 'third_party/nccl/nccl' 2024-08-20T23:03:42.1762374Z Entering 'third_party/nlohmann' 2024-08-20T23:03:42.1832808Z Entering 'third_party/onnx' 2024-08-20T23:03:42.1914222Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T23:03:42.1986499Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T23:03:42.2061218Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T23:03:42.2135407Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T23:03:42.2202397Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T23:03:42.2268901Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T23:03:42.2335658Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T23:03:42.2404555Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T23:03:42.2469491Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T23:03:42.2534872Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T23:03:42.2598986Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T23:03:42.2669197Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T23:03:42.2741535Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T23:03:42.2829290Z Entering 'third_party/pocketfft' 2024-08-20T23:03:42.2895397Z Entering 'third_party/protobuf' 2024-08-20T23:03:42.2962690Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T23:03:42.3032827Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T23:03:42.3101730Z Entering 'third_party/psimd' 2024-08-20T23:03:42.3167296Z Entering 'third_party/pthreadpool' 2024-08-20T23:03:42.3232161Z Entering 'third_party/pybind11' 2024-08-20T23:03:42.3297041Z Entering 'third_party/python-peachpy' 2024-08-20T23:03:42.3361966Z Entering 'third_party/sleef' 2024-08-20T23:03:42.3433870Z Entering 'third_party/tensorpipe' 2024-08-20T23:03:42.3497139Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T23:03:42.3562265Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T23:03:42.3632448Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T23:03:42.3696893Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T23:03:42.3758172Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T23:03:42.3864957Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T23:03:42.3901092Z http.https://github.com/.extraheader 2024-08-20T23:03:42.3910787Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-20T23:03:42.3958376Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T23:03:42.4324519Z Entering 'android/libs/fbjni' 2024-08-20T23:03:42.4368632Z http.https://github.com/.extraheader 2024-08-20T23:03:42.4411836Z Entering 'third_party/FP16' 2024-08-20T23:03:42.4455428Z http.https://github.com/.extraheader 2024-08-20T23:03:42.4498466Z Entering 'third_party/FXdiv' 2024-08-20T23:03:42.4544378Z http.https://github.com/.extraheader 2024-08-20T23:03:42.4588376Z Entering 'third_party/NNPACK' 2024-08-20T23:03:42.4633011Z http.https://github.com/.extraheader 2024-08-20T23:03:42.4677653Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T23:03:42.4721849Z http.https://github.com/.extraheader 2024-08-20T23:03:42.4766611Z Entering 'third_party/XNNPACK' 2024-08-20T23:03:42.4810234Z http.https://github.com/.extraheader 2024-08-20T23:03:42.4870368Z Entering 'third_party/benchmark' 2024-08-20T23:03:42.4913950Z http.https://github.com/.extraheader 2024-08-20T23:03:42.4959286Z Entering 'third_party/cpp-httplib' 2024-08-20T23:03:42.5003486Z http.https://github.com/.extraheader 2024-08-20T23:03:42.5047721Z Entering 'third_party/cpuinfo' 2024-08-20T23:03:42.5092253Z http.https://github.com/.extraheader 2024-08-20T23:03:42.5138955Z Entering 'third_party/cudnn_frontend' 2024-08-20T23:03:42.5183056Z http.https://github.com/.extraheader 2024-08-20T23:03:42.5227264Z Entering 'third_party/cutlass' 2024-08-20T23:03:42.5272753Z http.https://github.com/.extraheader 2024-08-20T23:03:42.5323750Z Entering 'third_party/eigen' 2024-08-20T23:03:42.5367928Z http.https://github.com/.extraheader 2024-08-20T23:03:42.5413714Z Entering 'third_party/fbgemm' 2024-08-20T23:03:42.5458057Z http.https://github.com/.extraheader 2024-08-20T23:03:42.5501524Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T23:03:42.5545919Z http.https://github.com/.extraheader 2024-08-20T23:03:42.5589441Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T23:03:42.5632436Z http.https://github.com/.extraheader 2024-08-20T23:03:42.5676009Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T23:03:42.5719685Z http.https://github.com/.extraheader 2024-08-20T23:03:42.5771811Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T23:03:42.5814038Z http.https://github.com/.extraheader 2024-08-20T23:03:42.5857838Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T23:03:42.5900436Z http.https://github.com/.extraheader 2024-08-20T23:03:42.5947188Z Entering 'third_party/flatbuffers' 2024-08-20T23:03:42.5991326Z http.https://github.com/.extraheader 2024-08-20T23:03:42.6038207Z Entering 'third_party/fmt' 2024-08-20T23:03:42.6081784Z http.https://github.com/.extraheader 2024-08-20T23:03:42.6125904Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T23:03:42.6170187Z http.https://github.com/.extraheader 2024-08-20T23:03:42.6213849Z Entering 'third_party/gloo' 2024-08-20T23:03:42.6258036Z http.https://github.com/.extraheader 2024-08-20T23:03:42.6302046Z Entering 'third_party/googletest' 2024-08-20T23:03:42.6346068Z http.https://github.com/.extraheader 2024-08-20T23:03:42.6389150Z Entering 'third_party/ideep' 2024-08-20T23:03:42.6433167Z http.https://github.com/.extraheader 2024-08-20T23:03:42.6475434Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T23:03:42.6518238Z http.https://github.com/.extraheader 2024-08-20T23:03:42.6571166Z Entering 'third_party/ittapi' 2024-08-20T23:03:42.6614010Z http.https://github.com/.extraheader 2024-08-20T23:03:42.6658134Z Entering 'third_party/kineto' 2024-08-20T23:03:42.6704072Z http.https://github.com/.extraheader 2024-08-20T23:03:42.6747252Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T23:03:42.6790131Z http.https://github.com/.extraheader 2024-08-20T23:03:42.6833604Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T23:03:42.6876355Z http.https://github.com/.extraheader 2024-08-20T23:03:42.6922784Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T23:03:42.6965498Z http.https://github.com/.extraheader 2024-08-20T23:03:42.7009023Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T23:03:42.7052671Z http.https://github.com/.extraheader 2024-08-20T23:03:42.7098144Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T23:03:42.7141105Z http.https://github.com/.extraheader 2024-08-20T23:03:42.7184063Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T23:03:42.7227483Z http.https://github.com/.extraheader 2024-08-20T23:03:42.7275759Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T23:03:42.7318806Z http.https://github.com/.extraheader 2024-08-20T23:03:42.7364013Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T23:03:42.7410350Z http.https://github.com/.extraheader 2024-08-20T23:03:42.7456259Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T23:03:42.7498908Z http.https://github.com/.extraheader 2024-08-20T23:03:42.7546174Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T23:03:42.7589313Z http.https://github.com/.extraheader 2024-08-20T23:03:42.7636954Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T23:03:42.7679750Z http.https://github.com/.extraheader 2024-08-20T23:03:42.7722582Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T23:03:42.7765759Z http.https://github.com/.extraheader 2024-08-20T23:03:42.7812545Z Entering 'third_party/mimalloc' 2024-08-20T23:03:42.7856824Z http.https://github.com/.extraheader 2024-08-20T23:03:42.7901168Z Entering 'third_party/nccl/nccl' 2024-08-20T23:03:42.7944913Z http.https://github.com/.extraheader 2024-08-20T23:03:42.7988738Z Entering 'third_party/nlohmann' 2024-08-20T23:03:42.8032291Z http.https://github.com/.extraheader 2024-08-20T23:03:42.8077597Z Entering 'third_party/onnx' 2024-08-20T23:03:42.8121072Z http.https://github.com/.extraheader 2024-08-20T23:03:42.8181907Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T23:03:42.8224549Z http.https://github.com/.extraheader 2024-08-20T23:03:42.8269359Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T23:03:42.8312209Z http.https://github.com/.extraheader 2024-08-20T23:03:42.8365516Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T23:03:42.8412770Z http.https://github.com/.extraheader 2024-08-20T23:03:42.8457932Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T23:03:42.8500391Z http.https://github.com/.extraheader 2024-08-20T23:03:42.8543512Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T23:03:42.8585243Z http.https://github.com/.extraheader 2024-08-20T23:03:42.8628948Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T23:03:42.8670752Z http.https://github.com/.extraheader 2024-08-20T23:03:42.8713293Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T23:03:42.8755278Z http.https://github.com/.extraheader 2024-08-20T23:03:42.8799431Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T23:03:42.8842527Z http.https://github.com/.extraheader 2024-08-20T23:03:42.8885239Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T23:03:42.8927056Z http.https://github.com/.extraheader 2024-08-20T23:03:42.8969532Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T23:03:42.9012077Z http.https://github.com/.extraheader 2024-08-20T23:03:42.9057507Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T23:03:42.9103171Z http.https://github.com/.extraheader 2024-08-20T23:03:42.9149843Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T23:03:42.9194336Z http.https://github.com/.extraheader 2024-08-20T23:03:42.9242890Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T23:03:42.9289162Z http.https://github.com/.extraheader 2024-08-20T23:03:42.9354843Z Entering 'third_party/pocketfft' 2024-08-20T23:03:42.9400982Z http.https://github.com/.extraheader 2024-08-20T23:03:42.9444770Z Entering 'third_party/protobuf' 2024-08-20T23:03:42.9488436Z http.https://github.com/.extraheader 2024-08-20T23:03:42.9534300Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T23:03:42.9576430Z http.https://github.com/.extraheader 2024-08-20T23:03:42.9619628Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T23:03:42.9663660Z http.https://github.com/.extraheader 2024-08-20T23:03:42.9709915Z Entering 'third_party/psimd' 2024-08-20T23:03:42.9753336Z http.https://github.com/.extraheader 2024-08-20T23:03:42.9797733Z Entering 'third_party/pthreadpool' 2024-08-20T23:03:42.9841161Z http.https://github.com/.extraheader 2024-08-20T23:03:42.9884768Z Entering 'third_party/pybind11' 2024-08-20T23:03:42.9927505Z http.https://github.com/.extraheader 2024-08-20T23:03:42.9970763Z Entering 'third_party/python-peachpy' 2024-08-20T23:03:43.0018320Z http.https://github.com/.extraheader 2024-08-20T23:03:43.0061737Z Entering 'third_party/sleef' 2024-08-20T23:03:43.0105104Z http.https://github.com/.extraheader 2024-08-20T23:03:43.0148922Z Entering 'third_party/tensorpipe' 2024-08-20T23:03:43.0190926Z http.https://github.com/.extraheader 2024-08-20T23:03:43.0237443Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T23:03:43.0279618Z http.https://github.com/.extraheader 2024-08-20T23:03:43.0326942Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T23:03:43.0368507Z http.https://github.com/.extraheader 2024-08-20T23:03:43.0410924Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T23:03:43.0453622Z http.https://github.com/.extraheader 2024-08-20T23:03:43.0498803Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T23:03:43.0541397Z http.https://github.com/.extraheader 2024-08-20T23:03:43.0582590Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T23:03:43.0625669Z http.https://github.com/.extraheader 2024-08-20T23:03:43.0786048Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-20T23:03:43.0806445Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-20T23:03:43.0814464Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:03:43.0814999Z ##[endgroup] 2024-08-20T23:03:50.2291738Z Cleaning up orphan processes