2024-08-20T22:04:58.0946913Z Current runner version: '2.319.1'
2024-08-20T22:04:58.0953110Z Runner name: 'i-0479ed8028e52c92f'
2024-08-20T22:04:58.0953871Z Runner group name: 'Default'
2024-08-20T22:04:58.0954747Z Machine name: 'ip-10-0-77-54'
2024-08-20T22:04:58.0971226Z Testing runner upgrade compatibility
2024-08-20T22:04:58.3114560Z ##[group]GITHUB_TOKEN Permissions
2024-08-20T22:04:58.3116596Z Actions: read
2024-08-20T22:04:58.3117050Z Attestations: read
2024-08-20T22:04:58.3117605Z Checks: read
2024-08-20T22:04:58.3118068Z Contents: read
2024-08-20T22:04:58.3118529Z Deployments: read
2024-08-20T22:04:58.3119022Z Discussions: read
2024-08-20T22:04:58.3119503Z Issues: read
2024-08-20T22:04:58.3119944Z Metadata: read
2024-08-20T22:04:58.3120414Z Packages: read
2024-08-20T22:04:58.3120890Z Pages: read
2024-08-20T22:04:58.3121329Z PullRequests: read
2024-08-20T22:04:58.3121951Z RepositoryProjects: read
2024-08-20T22:04:58.3122455Z SecurityEvents: read
2024-08-20T22:04:58.3122939Z Statuses: read
2024-08-20T22:04:58.3123640Z ##[endgroup]
2024-08-20T22:04:58.3126711Z Secret source: Actions
2024-08-20T22:04:58.3127462Z Prepare workflow directory
2024-08-20T22:04:58.4034091Z Prepare all required actions
2024-08-20T22:04:58.4197411Z Getting action download info
2024-08-20T22:04:58.5995231Z Download action repository 'pytorch/test-infra@main' (SHA:0c3a2634aaa2f638c8f640e743f03d696ce1191f)
2024-08-20T22:04:59.1669360Z Download action repository 'pytorch/pytorch@main' (SHA:1ae5d5bb62141d7c9b1b0b66c66a462b4e10b1f2)
2024-08-20T22:05:03.1866971Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0)
2024-08-20T22:05:03.4525452Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2024-08-20T22:05:03.7426510Z Getting action download info
2024-08-20T22:05:03.8368620Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c)
2024-08-20T22:05:04.0336637Z Getting action download info
2024-08-20T22:05:04.1365648Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482)
2024-08-20T22:05:04.2806793Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/133712 (40ec5f6ddd9787aca0449b24128343ff4c4a88b3)
2024-08-20T22:05:04.2808926Z ##[group] Inputs
2024-08-20T22:05:04.2809446Z   build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86
2024-08-20T22:05:04.2816492Z   test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}
2024-08-20T22:05:04.2824376Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:05:04.2825539Z   sync-tag: 
2024-08-20T22:05:04.2826370Z   timeout-minutes: 240
2024-08-20T22:05:04.2826704Z   use-gha: 
2024-08-20T22:05:04.2826990Z   dashboard-tag: 
2024-08-20T22:05:04.2827314Z   s3-bucket: gha-artifacts
2024-08-20T22:05:04.2827663Z   aws-role-to-assume: 
2024-08-20T22:05:04.2828003Z ##[endgroup]
2024-08-20T22:05:04.2829052Z Complete job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu)
2024-08-20T22:05:04.3554348Z A job started hook has been configured by the self-hosted runner administrator
2024-08-20T22:05:04.3699970Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh'
2024-08-20T22:05:04.3711670Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:05:04.3712251Z ##[endgroup]
2024-08-20T22:05:05.5345951Z Runner Type: linux.g5.4xlarge.nvidia.gpu
2024-08-20T22:05:05.5346491Z Instance Type: g5.4xlarge
2024-08-20T22:05:05.5347173Z AMI Name: al2023-ami-2023.5.20240701.0-kernel-6.1-x86_64
2024-08-20T22:05:05.5347700Z AMI ID: ami-06c68f701d8090592
2024-08-20T22:05:11.3073519Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main
2024-08-20T22:05:11.3074064Z with:
2024-08-20T22:05:11.3074769Z   github-secret: ***
2024-08-20T22:05:11.3075682Z   instructions: All testing is done inside the container, to start an interactive session run:
  docker exec -it $(docker container ps --format '{{.ID}}') bash

2024-08-20T22:05:11.3076645Z   activate-with-label: false
2024-08-20T22:05:11.3076999Z   label: with-ssh
2024-08-20T22:05:11.3077313Z   remove-existing-keys: true
2024-08-20T22:05:11.3077670Z   fail-silently: true
2024-08-20T22:05:11.3077970Z env:
2024-08-20T22:05:11.3078244Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:05:11.3078590Z ##[endgroup]
2024-08-20T22:05:11.3971501Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info.
2024-08-20T22:05:11.3974778Z ciflow reference detected, attempting to extract PR number
2024-08-20T22:05:11.6601929Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys
2024-08-20T22:05:11.7357831Z No SSH keys found for user pytorch-bot[bot]
2024-08-20T22:05:11.7358447Z Grabbing public ssh keys from https://github.com/XuehaiPan.keys
2024-08-20T22:05:11.8120472Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh
2024-08-20T22:05:11.8135586Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys
2024-08-20T22:05:11.8160446Z Login using: ssh ec2-user@ec2-3-236-222-218.compute-1.amazonaws.com
2024-08-20T22:05:11.8161209Z All testing is done inside the container, to start an interactive session run:
2024-08-20T22:05:11.8162007Z   docker exec -it $(docker container ps --format '{{.ID}}') bash
2024-08-20T22:05:11.8292784Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main
2024-08-20T22:05:11.8293343Z with:
2024-08-20T22:05:11.8293621Z   submodules: recursive
2024-08-20T22:05:11.8293943Z   fetch-depth: 0
2024-08-20T22:05:11.8294231Z env:
2024-08-20T22:05:11.8294503Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:05:11.8294834Z ##[endgroup]
2024-08-20T22:05:11.8483951Z ##[group]Run retry () {
2024-08-20T22:05:11.8484334Z retry () {
2024-08-20T22:05:11.8484855Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2024-08-20T22:05:11.8485410Z }
2024-08-20T22:05:11.8485716Z echo "${GITHUB_WORKSPACE}"
2024-08-20T22:05:11.8486140Z if [ -z "${NO_SUDO}" ]; then
2024-08-20T22:05:11.8486615Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2024-08-20T22:05:11.8487076Z else
2024-08-20T22:05:11.8487420Z   retry rm -rf "${GITHUB_WORKSPACE}"
2024-08-20T22:05:11.8487848Z fi
2024-08-20T22:05:11.8488394Z mkdir "${GITHUB_WORKSPACE}"
2024-08-20T22:05:11.8500632Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:05:11.8501148Z env:
2024-08-20T22:05:11.8501435Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:05:11.8501786Z   NO_SUDO: 
2024-08-20T22:05:11.8502069Z ##[endgroup]
2024-08-20T22:05:11.8534103Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-08-20T22:05:12.0086731Z ##[group]Run malfet/checkout@silent-checkout
2024-08-20T22:05:12.0087194Z with:
2024-08-20T22:05:12.0087537Z   ref: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:05:12.0087984Z   fetch-depth: 0
2024-08-20T22:05:12.0088302Z   submodules: recursive
2024-08-20T22:05:12.0088635Z   quiet-checkout: true
2024-08-20T22:05:12.0088988Z   repository: pytorch/pytorch
2024-08-20T22:05:12.0089462Z   token: ***
2024-08-20T22:05:12.0089755Z   ssh-strict: true
2024-08-20T22:05:12.0090082Z   persist-credentials: true
2024-08-20T22:05:12.0090434Z   clean: true
2024-08-20T22:05:12.0090788Z   sparse-checkout-cone-mode: true
2024-08-20T22:05:12.0091177Z   lfs: false
2024-08-20T22:05:12.0091478Z   set-safe-directory: true
2024-08-20T22:05:12.0091810Z env:
2024-08-20T22:05:12.0092091Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:05:12.0092427Z ##[endgroup]
2024-08-20T22:05:12.1068391Z Syncing repository: pytorch/pytorch
2024-08-20T22:05:12.1070387Z ##[group]Getting Git version info
2024-08-20T22:05:12.1071200Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2024-08-20T22:05:12.1072070Z [command]/usr/bin/git version
2024-08-20T22:05:12.1072425Z git version 2.40.1
2024-08-20T22:05:12.1077721Z ##[endgroup]
2024-08-20T22:05:12.1091728Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/586da30d-e198-480c-89ca-74415b55bd5d' before making global git config changes
2024-08-20T22:05:12.1092961Z Adding repository directory to the temporary git global config as a safe directory
2024-08-20T22:05:12.1096828Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-08-20T22:05:12.1147589Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2024-08-20T22:05:12.1151017Z ##[group]Initializing the repository
2024-08-20T22:05:12.1153888Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-08-20T22:05:12.1197688Z hint: Using 'master' as the name for the initial branch. This default branch name
2024-08-20T22:05:12.1198744Z hint: is subject to change. To configure the initial branch name to use in all
2024-08-20T22:05:12.1199547Z hint: of your new repositories, which will suppress this warning, call:
2024-08-20T22:05:12.1200108Z hint: 
2024-08-20T22:05:12.1200866Z hint: 	git config --global init.defaultBranch <name>
2024-08-20T22:05:12.1201330Z hint: 
2024-08-20T22:05:12.1201854Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2024-08-20T22:05:12.1202694Z hint: 'development'. The just-created branch can be renamed via this command:
2024-08-20T22:05:12.1203582Z hint: 
2024-08-20T22:05:12.1203917Z hint: 	git branch -m <name>
2024-08-20T22:05:12.1204675Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2024-08-20T22:05:12.1211327Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2024-08-20T22:05:12.1253582Z ##[endgroup]
2024-08-20T22:05:12.1254202Z ##[group]Disabling automatic garbage collection
2024-08-20T22:05:12.1255547Z [command]/usr/bin/git config --local gc.auto 0
2024-08-20T22:05:12.1296686Z ##[endgroup]
2024-08-20T22:05:12.1297202Z ##[group]Setting up auth
2024-08-20T22:05:12.1302873Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2024-08-20T22:05:12.1343294Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2024-08-20T22:05:12.1710956Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2024-08-20T22:05:12.1752013Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2024-08-20T22:05:12.2099558Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2024-08-20T22:05:12.2151722Z ##[endgroup]
2024-08-20T22:05:12.2152267Z ##[group]Fetching the repository
2024-08-20T22:05:12.2158306Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2024-08-20T22:05:15.2861220Z remote: Enumerating objects: 1008509        
2024-08-20T22:05:15.2861757Z remote: Enumerating objects: 1010208, done.        
2024-08-20T22:05:15.2862569Z remote: Counting objects:   0% (1/1699)        
2024-08-20T22:05:15.2863069Z remote: Counting objects:   1% (17/1699)        
2024-08-20T22:05:15.2863603Z remote: Counting objects:   2% (34/1699)        
2024-08-20T22:05:15.2864103Z remote: Counting objects:   3% (51/1699)        
2024-08-20T22:05:15.2864752Z remote: Counting objects:   4% (68/1699)        
2024-08-20T22:05:15.2865311Z remote: Counting objects:   5% (85/1699)        
2024-08-20T22:05:15.2865991Z remote: Counting objects:   6% (102/1699)        
2024-08-20T22:05:15.2866644Z remote: Counting objects:   7% (119/1699)        
2024-08-20T22:05:15.2867285Z remote: Counting objects:   8% (136/1699)        
2024-08-20T22:05:15.2867891Z remote: Counting objects:   9% (153/1699)        
2024-08-20T22:05:15.2868568Z remote: Counting objects:  10% (170/1699)        
2024-08-20T22:05:15.2869448Z remote: Counting objects:  11% (187/1699)        
2024-08-20T22:05:15.2870008Z remote: Counting objects:  12% (204/1699)        
2024-08-20T22:05:15.2870665Z remote: Counting objects:  13% (221/1699)        
2024-08-20T22:05:15.2871309Z remote: Counting objects:  14% (238/1699)        
2024-08-20T22:05:15.2871871Z remote: Counting objects:  15% (255/1699)        
2024-08-20T22:05:15.2872365Z remote: Counting objects:  16% (272/1699)        
2024-08-20T22:05:15.2872898Z remote: Counting objects:  17% (289/1699)        
2024-08-20T22:05:15.2873567Z remote: Counting objects:  18% (306/1699)        
2024-08-20T22:05:15.2874211Z remote: Counting objects:  19% (323/1699)        
2024-08-20T22:05:15.2874845Z remote: Counting objects:  20% (340/1699)        
2024-08-20T22:05:15.2875435Z remote: Counting objects:  21% (357/1699)        
2024-08-20T22:05:15.2876089Z remote: Counting objects:  22% (374/1699)        
2024-08-20T22:05:15.2876736Z remote: Counting objects:  23% (391/1699)        
2024-08-20T22:05:15.2877376Z remote: Counting objects:  24% (408/1699)        
2024-08-20T22:05:15.2878035Z remote: Counting objects:  25% (425/1699)        
2024-08-20T22:05:15.2878672Z remote: Counting objects:  26% (442/1699)        
2024-08-20T22:05:15.2879324Z remote: Counting objects:  27% (459/1699)        
2024-08-20T22:05:15.2880250Z remote: Counting objects:  28% (476/1699)        
2024-08-20T22:05:15.2880771Z remote: Counting objects:  29% (493/1699)        
2024-08-20T22:05:15.2881339Z remote: Counting objects:  30% (510/1699)        
2024-08-20T22:05:15.2881977Z remote: Counting objects:  31% (527/1699)        
2024-08-20T22:05:15.2882568Z remote: Counting objects:  32% (544/1699)        
2024-08-20T22:05:15.2883148Z remote: Counting objects:  33% (561/1699)        
2024-08-20T22:05:15.2883789Z remote: Counting objects:  34% (578/1699)        
2024-08-20T22:05:15.2884416Z remote: Counting objects:  35% (595/1699)        
2024-08-20T22:05:15.2884901Z remote: Counting objects:  36% (612/1699)        
2024-08-20T22:05:15.2885500Z remote: Counting objects:  37% (629/1699)        
2024-08-20T22:05:15.2886142Z remote: Counting objects:  38% (646/1699)        
2024-08-20T22:05:15.2886780Z remote: Counting objects:  39% (663/1699)        
2024-08-20T22:05:15.2887474Z remote: Counting objects:  40% (680/1699)        
2024-08-20T22:05:15.2888357Z remote: Counting objects:  41% (697/1699)        
2024-08-20T22:05:15.2888917Z remote: Counting objects:  42% (714/1699)        
2024-08-20T22:05:15.2889404Z remote: Counting objects:  43% (731/1699)        
2024-08-20T22:05:15.2889885Z remote: Counting objects:  44% (748/1699)        
2024-08-20T22:05:15.2890370Z remote: Counting objects:  45% (765/1699)        
2024-08-20T22:05:15.2890860Z remote: Counting objects:  46% (782/1699)        
2024-08-20T22:05:15.2891357Z remote: Counting objects:  47% (799/1699)        
2024-08-20T22:05:15.2891837Z remote: Counting objects:  48% (816/1699)        
2024-08-20T22:05:15.2892324Z remote: Counting objects:  49% (833/1699)        
2024-08-20T22:05:15.2892824Z remote: Counting objects:  50% (850/1699)        
2024-08-20T22:05:15.2893321Z remote: Counting objects:  51% (867/1699)        
2024-08-20T22:05:15.2893804Z remote: Counting objects:  52% (884/1699)        
2024-08-20T22:05:15.2894339Z remote: Counting objects:  53% (901/1699)        
2024-08-20T22:05:15.2894861Z remote: Counting objects:  54% (918/1699)        
2024-08-20T22:05:15.2895355Z remote: Counting objects:  55% (935/1699)        
2024-08-20T22:05:15.2895838Z remote: Counting objects:  56% (952/1699)        
2024-08-20T22:05:15.2896345Z remote: Counting objects:  57% (969/1699)        
2024-08-20T22:05:15.2896845Z remote: Counting objects:  58% (986/1699)        
2024-08-20T22:05:15.2897357Z remote: Counting objects:  59% (1003/1699)        
2024-08-20T22:05:15.2897869Z remote: Counting objects:  60% (1020/1699)        
2024-08-20T22:05:15.2898521Z remote: Counting objects:  61% (1037/1699)        
2024-08-20T22:05:15.2899042Z remote: Counting objects:  62% (1054/1699)        
2024-08-20T22:05:15.2899564Z remote: Counting objects:  63% (1071/1699)        
2024-08-20T22:05:15.2900085Z remote: Counting objects:  64% (1088/1699)        
2024-08-20T22:05:15.2900595Z remote: Counting objects:  65% (1105/1699)        
2024-08-20T22:05:15.2901117Z remote: Counting objects:  66% (1122/1699)        
2024-08-20T22:05:15.2901634Z remote: Counting objects:  67% (1139/1699)        
2024-08-20T22:05:15.2902150Z remote: Counting objects:  68% (1156/1699)        
2024-08-20T22:05:15.2902661Z remote: Counting objects:  69% (1173/1699)        
2024-08-20T22:05:15.2903182Z remote: Counting objects:  70% (1190/1699)        
2024-08-20T22:05:15.2903689Z remote: Counting objects:  71% (1207/1699)        
2024-08-20T22:05:15.2904200Z remote: Counting objects:  72% (1224/1699)        
2024-08-20T22:05:15.2904699Z remote: Counting objects:  73% (1241/1699)        
2024-08-20T22:05:15.2905209Z remote: Counting objects:  74% (1258/1699)        
2024-08-20T22:05:15.2905728Z remote: Counting objects:  75% (1275/1699)        
2024-08-20T22:05:15.2906240Z remote: Counting objects:  76% (1292/1699)        
2024-08-20T22:05:15.2906744Z remote: Counting objects:  77% (1309/1699)        
2024-08-20T22:05:15.2907258Z remote: Counting objects:  78% (1326/1699)        
2024-08-20T22:05:15.2907873Z remote: Counting objects:  79% (1343/1699)        
2024-08-20T22:05:15.2908388Z remote: Counting objects:  80% (1360/1699)        
2024-08-20T22:05:15.2908883Z remote: Counting objects:  81% (1377/1699)        
2024-08-20T22:05:15.2909390Z remote: Counting objects:  82% (1394/1699)        
2024-08-20T22:05:15.2909902Z remote: Counting objects:  83% (1411/1699)        
2024-08-20T22:05:15.2910420Z remote: Counting objects:  84% (1428/1699)        
2024-08-20T22:05:15.2910927Z remote: Counting objects:  85% (1445/1699)        
2024-08-20T22:05:15.2911432Z remote: Counting objects:  86% (1462/1699)        
2024-08-20T22:05:15.2911940Z remote: Counting objects:  87% (1479/1699)        
2024-08-20T22:05:15.2912454Z remote: Counting objects:  88% (1496/1699)        
2024-08-20T22:05:15.2912956Z remote: Counting objects:  89% (1513/1699)        
2024-08-20T22:05:15.2913462Z remote: Counting objects:  90% (1530/1699)        
2024-08-20T22:05:15.2913972Z remote: Counting objects:  91% (1547/1699)        
2024-08-20T22:05:15.2914586Z remote: Counting objects:  92% (1564/1699)        
2024-08-20T22:05:15.2915088Z remote: Counting objects:  93% (1581/1699)        
2024-08-20T22:05:15.2915601Z remote: Counting objects:  94% (1598/1699)        
2024-08-20T22:05:15.2916115Z remote: Counting objects:  95% (1615/1699)        
2024-08-20T22:05:15.2916626Z remote: Counting objects:  96% (1632/1699)        
2024-08-20T22:05:15.2917136Z remote: Counting objects:  97% (1649/1699)        
2024-08-20T22:05:15.2917651Z remote: Counting objects:  98% (1666/1699)        
2024-08-20T22:05:15.2918166Z remote: Counting objects:  99% (1683/1699)        
2024-08-20T22:05:15.2918675Z remote: Counting objects: 100% (1699/1699)        
2024-08-20T22:05:15.2919219Z remote: Counting objects: 100% (1699/1699), done.        
2024-08-20T22:05:15.3131616Z remote: Compressing objects:   0% (1/826)        
2024-08-20T22:05:15.3439378Z remote: Compressing objects:   1% (9/826)        
2024-08-20T22:05:15.3752555Z remote: Compressing objects:   2% (17/826)        
2024-08-20T22:05:15.4279867Z remote: Compressing objects:   3% (25/826)        
2024-08-20T22:05:15.5131522Z remote: Compressing objects:   4% (34/826)        
2024-08-20T22:05:15.5546568Z remote: Compressing objects:   5% (42/826)        
2024-08-20T22:05:15.6222732Z remote: Compressing objects:   6% (50/826)        
2024-08-20T22:05:15.6647440Z remote: Compressing objects:   7% (58/826)        
2024-08-20T22:05:15.6905460Z remote: Compressing objects:   8% (67/826)        
2024-08-20T22:05:15.7122436Z remote: Compressing objects:   9% (75/826)        
2024-08-20T22:05:15.7371276Z remote: Compressing objects:  10% (83/826)        
2024-08-20T22:05:15.7580096Z remote: Compressing objects:  11% (91/826)        
2024-08-20T22:05:15.7773919Z remote: Compressing objects:  12% (100/826)        
2024-08-20T22:05:15.7922446Z remote: Compressing objects:  13% (108/826)        
2024-08-20T22:05:15.8038586Z remote: Compressing objects:  14% (116/826)        
2024-08-20T22:05:15.8127125Z remote: Compressing objects:  15% (124/826)        
2024-08-20T22:05:15.8214460Z remote: Compressing objects:  16% (133/826)        
2024-08-20T22:05:15.8274518Z remote: Compressing objects:  17% (141/826)        
2024-08-20T22:05:15.8316301Z remote: Compressing objects:  18% (149/826)        
2024-08-20T22:05:15.8364419Z remote: Compressing objects:  19% (157/826)        
2024-08-20T22:05:15.8395304Z remote: Compressing objects:  20% (166/826)        
2024-08-20T22:05:15.8417077Z remote: Compressing objects:  21% (174/826)        
2024-08-20T22:05:15.8421348Z remote: Compressing objects:  22% (182/826)        
2024-08-20T22:05:15.8426290Z remote: Compressing objects:  23% (190/826)        
2024-08-20T22:05:15.8434464Z remote: Compressing objects:  24% (199/826)        
2024-08-20T22:05:15.8439702Z remote: Compressing objects:  25% (207/826)        
2024-08-20T22:05:15.8450708Z remote: Compressing objects:  26% (215/826)        
2024-08-20T22:05:15.8459853Z remote: Compressing objects:  27% (224/826)        
2024-08-20T22:05:15.8486950Z remote: Compressing objects:  28% (232/826)        
2024-08-20T22:05:15.8492845Z remote: Compressing objects:  29% (240/826)        
2024-08-20T22:05:15.8506649Z remote: Compressing objects:  30% (248/826)        
2024-08-20T22:05:15.8515330Z remote: Compressing objects:  31% (257/826)        
2024-08-20T22:05:15.8521617Z remote: Compressing objects:  32% (265/826)        
2024-08-20T22:05:15.8530596Z remote: Compressing objects:  33% (273/826)        
2024-08-20T22:05:15.8538127Z remote: Compressing objects:  34% (281/826)        
2024-08-20T22:05:15.8544649Z remote: Compressing objects:  35% (290/826)        
2024-08-20T22:05:15.8549575Z remote: Compressing objects:  36% (298/826)        
2024-08-20T22:05:15.8559578Z remote: Compressing objects:  37% (306/826)        
2024-08-20T22:05:15.8569340Z remote: Compressing objects:  38% (314/826)        
2024-08-20T22:05:15.8573477Z remote: Compressing objects:  39% (323/826)        
2024-08-20T22:05:15.8579744Z remote: Compressing objects:  40% (331/826)        
2024-08-20T22:05:15.8584923Z remote: Compressing objects:  41% (339/826)        
2024-08-20T22:05:15.8594521Z remote: Compressing objects:  42% (347/826)        
2024-08-20T22:05:15.8598808Z remote: Compressing objects:  43% (356/826)        
2024-08-20T22:05:15.8605675Z remote: Compressing objects:  44% (364/826)        
2024-08-20T22:05:15.8610356Z remote: Compressing objects:  45% (372/826)        
2024-08-20T22:05:15.8615762Z remote: Compressing objects:  46% (380/826)        
2024-08-20T22:05:15.8618866Z remote: Compressing objects:  47% (389/826)        
2024-08-20T22:05:15.8621754Z remote: Compressing objects:  48% (397/826)        
2024-08-20T22:05:15.8625094Z remote: Compressing objects:  49% (405/826)        
2024-08-20T22:05:15.8628056Z remote: Compressing objects:  50% (413/826)        
2024-08-20T22:05:15.8631012Z remote: Compressing objects:  51% (422/826)        
2024-08-20T22:05:15.8632807Z remote: Compressing objects:  52% (430/826)        
2024-08-20T22:05:15.8635001Z remote: Compressing objects:  53% (438/826)        
2024-08-20T22:05:15.8636857Z remote: Compressing objects:  54% (447/826)        
2024-08-20T22:05:15.8637775Z remote: Compressing objects:  55% (455/826)        
2024-08-20T22:05:15.8639656Z remote: Compressing objects:  56% (463/826)        
2024-08-20T22:05:15.8640354Z remote: Compressing objects:  57% (471/826)        
2024-08-20T22:05:15.8641279Z remote: Compressing objects:  58% (480/826)        
2024-08-20T22:05:15.8642171Z remote: Compressing objects:  59% (488/826)        
2024-08-20T22:05:15.8642881Z remote: Compressing objects:  60% (496/826)        
2024-08-20T22:05:15.8644239Z remote: Compressing objects:  61% (504/826)        
2024-08-20T22:05:15.8644958Z remote: Compressing objects:  62% (513/826)        
2024-08-20T22:05:15.8645777Z remote: Compressing objects:  63% (521/826)        
2024-08-20T22:05:15.8649291Z remote: Compressing objects:  64% (529/826)        
2024-08-20T22:05:15.8657712Z remote: Compressing objects:  65% (537/826)        
2024-08-20T22:05:15.8663548Z remote: Compressing objects:  66% (546/826)        
2024-08-20T22:05:15.8669015Z remote: Compressing objects:  67% (554/826)        
2024-08-20T22:05:15.8672018Z remote: Compressing objects:  68% (562/826)        
2024-08-20T22:05:15.8677069Z remote: Compressing objects:  69% (570/826)        
2024-08-20T22:05:15.8681078Z remote: Compressing objects:  70% (579/826)        
2024-08-20T22:05:15.8684587Z remote: Compressing objects:  71% (587/826)        
2024-08-20T22:05:15.8687576Z remote: Compressing objects:  72% (595/826)        
2024-08-20T22:05:15.8691472Z remote: Compressing objects:  73% (603/826)        
2024-08-20T22:05:15.8694608Z remote: Compressing objects:  74% (612/826)        
2024-08-20T22:05:15.8697396Z remote: Compressing objects:  75% (620/826)        
2024-08-20T22:05:15.8700715Z remote: Compressing objects:  76% (628/826)        
2024-08-20T22:05:15.8703385Z remote: Compressing objects:  77% (637/826)        
2024-08-20T22:05:15.8704779Z remote: Compressing objects:  78% (645/826)        
2024-08-20T22:05:15.8707110Z remote: Compressing objects:  79% (653/826)        
2024-08-20T22:05:15.8709206Z remote: Compressing objects:  80% (661/826)        
2024-08-20T22:05:15.8710732Z remote: Compressing objects:  81% (670/826)        
2024-08-20T22:05:15.8713289Z remote: Compressing objects:  82% (678/826)        
2024-08-20T22:05:15.8715169Z remote: Compressing objects:  83% (686/826)        
2024-08-20T22:05:15.8716547Z remote: Compressing objects:  84% (694/826)        
2024-08-20T22:05:15.8718576Z remote: Compressing objects:  85% (703/826)        
2024-08-20T22:05:15.8720326Z remote: Compressing objects:  86% (711/826)        
2024-08-20T22:05:15.8721687Z remote: Compressing objects:  87% (719/826)        
2024-08-20T22:05:15.8724606Z remote: Compressing objects:  88% (727/826)        
2024-08-20T22:05:15.8726734Z remote: Compressing objects:  89% (736/826)        
2024-08-20T22:05:15.8728696Z remote: Compressing objects:  90% (744/826)        
2024-08-20T22:05:15.8729605Z remote: Compressing objects:  91% (752/826)        
2024-08-20T22:05:15.8730682Z remote: Compressing objects:  92% (760/826)        
2024-08-20T22:05:15.8731367Z remote: Compressing objects:  93% (769/826)        
2024-08-20T22:05:15.8732479Z remote: Compressing objects:  94% (777/826)        
2024-08-20T22:05:15.8734302Z remote: Compressing objects:  95% (785/826)        
2024-08-20T22:05:15.8736226Z remote: Compressing objects:  96% (793/826)        
2024-08-20T22:05:15.8736945Z remote: Compressing objects:  97% (802/826)        
2024-08-20T22:05:15.8737869Z remote: Compressing objects:  98% (810/826)        
2024-08-20T22:05:15.8739996Z remote: Compressing objects:  99% (818/826)        
2024-08-20T22:05:15.8740700Z remote: Compressing objects: 100% (826/826)        
2024-08-20T22:05:15.8741320Z remote: Compressing objects: 100% (826/826), done.        
2024-08-20T22:05:39.5094568Z remote: Total 1010208 (delta 1106), reused 1335 (delta 870), pack-reused 1008509 (from 1)        
2024-08-20T22:06:02.4664573Z [command]/usr/bin/git rev-parse --verify --quiet 40ec5f6ddd9787aca0449b24128343ff4c4a88b3^{object}
2024-08-20T22:06:02.4701632Z 40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:06:02.4707575Z ##[endgroup]
2024-08-20T22:06:02.4710924Z ##[group]Determining the checkout info
2024-08-20T22:06:02.4711615Z ##[endgroup]
2024-08-20T22:06:02.4712314Z ##[group]Checking out the ref
2024-08-20T22:06:02.4714271Z [command]/usr/bin/git checkout --quiet --force 40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:06:04.1424015Z ##[endgroup]
2024-08-20T22:06:04.1427234Z ##[group]Setting up auth for fetching submodules
2024-08-20T22:06:04.1430025Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2024-08-20T22:06:04.1486193Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2024-08-20T22:06:04.1525127Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2024-08-20T22:06:04.1563850Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2024-08-20T22:06:04.1600744Z ##[endgroup]
2024-08-20T22:06:04.1601368Z ##[group]Fetching submodules
2024-08-20T22:06:04.1604426Z [command]/usr/bin/git submodule sync --recursive
2024-08-20T22:06:04.1979968Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2024-08-20T22:06:04.2340849Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2024-08-20T22:06:04.2342782Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2024-08-20T22:06:04.2346018Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2024-08-20T22:06:04.2349146Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2024-08-20T22:06:04.2353108Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2024-08-20T22:06:04.2355412Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2024-08-20T22:06:04.2359782Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2024-08-20T22:06:04.2363480Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib'
2024-08-20T22:06:04.2367139Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2024-08-20T22:06:04.2371807Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2024-08-20T22:06:04.2375010Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2024-08-20T22:06:04.2379037Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2024-08-20T22:06:04.2382945Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2024-08-20T22:06:04.2386984Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2024-08-20T22:06:04.2390804Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2024-08-20T22:06:04.2396194Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2024-08-20T22:06:04.2403434Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2024-08-20T22:06:04.2407695Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2024-08-20T22:06:04.2411810Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2024-08-20T22:06:04.2416299Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2024-08-20T22:06:04.2420779Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2024-08-20T22:06:04.2425312Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc'
2024-08-20T22:06:04.2429763Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2024-08-20T22:06:04.2434391Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2024-08-20T22:06:04.2438882Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2024-08-20T22:06:04.2444039Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp'
2024-08-20T22:06:04.2448645Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2024-08-20T22:06:04.2453591Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2024-08-20T22:06:04.2458649Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2024-08-20T22:06:04.2463846Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2024-08-20T22:06:04.2469203Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2024-08-20T22:06:04.2475145Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2024-08-20T22:06:04.2479896Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2024-08-20T22:06:04.2488499Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2024-08-20T22:06:04.2520994Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2024-08-20T22:06:04.5701600Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2024-08-20T22:06:04.7566082Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2024-08-20T22:06:04.9361565Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2024-08-20T22:06:05.1928695Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2024-08-20T22:06:07.0858705Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2024-08-20T22:06:18.6131586Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2024-08-20T22:06:19.0282459Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'...
2024-08-20T22:06:19.4676438Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2024-08-20T22:06:20.0752840Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2024-08-20T22:06:21.3204811Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2024-08-20T22:06:23.4346479Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2024-08-20T22:06:29.4035273Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2024-08-20T22:06:30.8353770Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2024-08-20T22:06:32.7194263Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2024-08-20T22:06:34.0333489Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2024-08-20T22:06:34.4376173Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2024-08-20T22:06:34.8000872Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2024-08-20T22:06:36.0476385Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2024-08-20T22:06:36.3973974Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2024-08-20T22:06:36.6592980Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2024-08-20T22:06:38.2840984Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'...
2024-08-20T22:06:38.9895532Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2024-08-20T22:06:39.6628454Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2024-08-20T22:06:45.8221378Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2024-08-20T22:06:47.8251743Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'...
2024-08-20T22:06:52.1181788Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2024-08-20T22:06:52.3540410Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2024-08-20T22:07:01.6251512Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2024-08-20T22:07:01.8464718Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2024-08-20T22:07:02.0484117Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2024-08-20T22:07:03.0867454Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2024-08-20T22:07:03.3907804Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2024-08-20T22:07:04.0414084Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2024-08-20T22:07:04.4667749Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2024-08-20T22:07:04.4814152Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2024-08-20T22:07:04.4929970Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2024-08-20T22:07:04.5244785Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2024-08-20T22:07:04.5685773Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2024-08-20T22:07:05.7576681Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2'
2024-08-20T22:07:05.7868366Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2024-08-20T22:07:05.8395744Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006'
2024-08-20T22:07:05.9537828Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5'
2024-08-20T22:07:05.9948184Z Submodule path 'third_party/cudnn_frontend': checked out '23511ba176243f27b3b275da1fb3814ea805a171'
2024-08-20T22:07:06.6225022Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49'
2024-08-20T22:07:06.9121785Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2024-08-20T22:07:07.0062751Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e'
2024-08-20T22:07:07.0082704Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2024-08-20T22:07:07.0086280Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2024-08-20T22:07:07.0089660Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass'
2024-08-20T22:07:07.0093223Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2024-08-20T22:07:07.0096813Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2024-08-20T22:07:07.0128068Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2024-08-20T22:07:08.0012175Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2024-08-20T22:07:08.6372017Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'...
2024-08-20T22:07:10.8171049Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2024-08-20T22:07:11.8690119Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2024-08-20T22:07:12.2078410Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2024-08-20T22:07:12.3194485Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2024-08-20T22:07:12.8318247Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6'
2024-08-20T22:07:12.9016116Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2024-08-20T22:07:12.9165161Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1'
2024-08-20T22:07:13.0679792Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b'
2024-08-20T22:07:13.1146867Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592'
2024-08-20T22:07:13.1604992Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2024-08-20T22:07:13.1930732Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509'
2024-08-20T22:07:13.2465573Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2024-08-20T22:07:13.2625094Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b'
2024-08-20T22:07:13.2644302Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2024-08-20T22:07:13.2673979Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2024-08-20T22:07:26.6632656Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67'
2024-08-20T22:07:26.6850970Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2024-08-20T22:07:26.7848105Z Submodule path 'third_party/kineto': checked out 'd9753139d181b9ff42872465aac0e5d3018be415'
2024-08-20T22:07:26.7868683Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog'
2024-08-20T22:07:26.7872346Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2024-08-20T22:07:26.7875886Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2024-08-20T22:07:26.7912286Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'...
2024-08-20T22:07:27.3370233Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2024-08-20T22:07:28.5954161Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2024-08-20T22:07:29.7023715Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e'
2024-08-20T22:07:29.7043954Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-08-20T22:07:29.7047281Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-08-20T22:07:29.7050959Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-08-20T22:07:29.7054510Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-08-20T22:07:29.7058244Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-08-20T22:07:29.7062346Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-08-20T22:07:29.7066164Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-08-20T22:07:29.7069992Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-08-20T22:07:29.7108386Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'...
2024-08-20T22:07:30.5449221Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'...
2024-08-20T22:07:31.0027068Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'...
2024-08-20T22:07:32.2447299Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'...
2024-08-20T22:07:32.5291676Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'...
2024-08-20T22:07:33.0771331Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'...
2024-08-20T22:07:34.0791095Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'...
2024-08-20T22:07:40.8509430Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'...
2024-08-20T22:07:41.2585306Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9'
2024-08-20T22:07:41.2820210Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400'
2024-08-20T22:07:41.3252819Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2024-08-20T22:07:41.3419879Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067'
2024-08-20T22:07:41.3443000Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-08-20T22:07:41.3473806Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'...
2024-08-20T22:07:41.6447929Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4'
2024-08-20T22:07:41.6673983Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446'
2024-08-20T22:07:41.7155328Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850'
2024-08-20T22:07:41.8357407Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5'
2024-08-20T22:07:41.8557122Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150'
2024-08-20T22:07:41.9015840Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164'
2024-08-20T22:07:41.9677498Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2024-08-20T22:07:42.0141776Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78'
2024-08-20T22:07:42.0439389Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290'
2024-08-20T22:07:42.1705549Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2024-08-20T22:07:42.6824010Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557'
2024-08-20T22:07:42.6859486Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2024-08-20T22:07:42.6863181Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2024-08-20T22:07:42.6894623Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2024-08-20T22:07:43.1338076Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2024-08-20T22:07:44.0838026Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6'
2024-08-20T22:07:44.1246285Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242'
2024-08-20T22:07:44.2156478Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878'
2024-08-20T22:07:44.2178107Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-08-20T22:07:44.2181614Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest'
2024-08-20T22:07:44.2184862Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-08-20T22:07:44.2188497Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-08-20T22:07:44.2192290Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-08-20T22:07:44.2195919Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-08-20T22:07:44.2199664Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-08-20T22:07:44.2203267Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-08-20T22:07:44.2240883Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'...
2024-08-20T22:07:44.6974614Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'...
2024-08-20T22:07:45.7598571Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'...
2024-08-20T22:07:46.0631315Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'...
2024-08-20T22:07:52.1230025Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'...
2024-08-20T22:07:52.3803095Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'...
2024-08-20T22:07:52.6278154Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'...
2024-08-20T22:07:52.9539448Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'...
2024-08-20T22:07:59.7966569Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2'
2024-08-20T22:07:59.8437597Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1'
2024-08-20T22:07:59.8626530Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa'
2024-08-20T22:07:59.9857371Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d'
2024-08-20T22:08:00.0022538Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce'
2024-08-20T22:08:00.0206139Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5'
2024-08-20T22:08:00.0406132Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d'
2024-08-20T22:08:00.0425316Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-08-20T22:08:00.0428779Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-08-20T22:08:00.0460296Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'...
2024-08-20T22:08:01.9733948Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'...
2024-08-20T22:08:03.2878552Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4'
2024-08-20T22:08:03.3420309Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2024-08-20T22:08:04.0135309Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50'
2024-08-20T22:08:04.0282427Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe'
2024-08-20T22:08:04.3469734Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2024-08-20T22:08:04.3491941Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2024-08-20T22:08:04.3495225Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2024-08-20T22:08:04.3526425Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2024-08-20T22:08:04.7951518Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2024-08-20T22:08:05.8627159Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2024-08-20T22:08:05.9447304Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2024-08-20T22:08:05.9564444Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2024-08-20T22:08:05.9723018Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8'
2024-08-20T22:08:06.0173341Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c'
2024-08-20T22:08:06.0524853Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2024-08-20T22:08:06.1022139Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b'
2024-08-20T22:08:06.1363768Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2024-08-20T22:08:06.1384638Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2024-08-20T22:08:06.1387863Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2024-08-20T22:08:06.1391486Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2024-08-20T22:08:06.1394973Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2024-08-20T22:08:06.1427238Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2024-08-20T22:08:07.2101885Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2024-08-20T22:08:07.4709711Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2024-08-20T22:08:08.6354389Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2024-08-20T22:08:09.6130055Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2024-08-20T22:08:09.6322498Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2024-08-20T22:08:09.7108450Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2024-08-20T22:08:09.7448805Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2024-08-20T22:08:09.7467951Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-08-20T22:08:09.7500084Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2024-08-20T22:08:09.9594480Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2024-08-20T22:08:09.9640477Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2024-08-20T22:08:10.0010240Z Entering 'android/libs/fbjni'
2024-08-20T22:08:10.0062717Z Entering 'third_party/FP16'
2024-08-20T22:08:10.0115505Z Entering 'third_party/FXdiv'
2024-08-20T22:08:10.0166301Z Entering 'third_party/NNPACK'
2024-08-20T22:08:10.0225792Z Entering 'third_party/VulkanMemoryAllocator'
2024-08-20T22:08:10.0281653Z Entering 'third_party/XNNPACK'
2024-08-20T22:08:10.0353052Z Entering 'third_party/benchmark'
2024-08-20T22:08:10.0405383Z Entering 'third_party/cpp-httplib'
2024-08-20T22:08:10.0455678Z Entering 'third_party/cpuinfo'
2024-08-20T22:08:10.0507848Z Entering 'third_party/cudnn_frontend'
2024-08-20T22:08:10.0559112Z Entering 'third_party/cutlass'
2024-08-20T22:08:10.0621153Z Entering 'third_party/eigen'
2024-08-20T22:08:10.0680079Z Entering 'third_party/fbgemm'
2024-08-20T22:08:10.0730327Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-08-20T22:08:10.0782122Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-08-20T22:08:10.0832201Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-08-20T22:08:10.0890914Z Entering 'third_party/fbgemm/third_party/googletest'
2024-08-20T22:08:10.0939489Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-08-20T22:08:10.0992062Z Entering 'third_party/flatbuffers'
2024-08-20T22:08:10.1046831Z Entering 'third_party/fmt'
2024-08-20T22:08:10.1102111Z Entering 'third_party/gemmlowp/gemmlowp'
2024-08-20T22:08:10.1153570Z Entering 'third_party/gloo'
2024-08-20T22:08:10.1205341Z Entering 'third_party/googletest'
2024-08-20T22:08:10.1261141Z Entering 'third_party/ideep'
2024-08-20T22:08:10.1315100Z Entering 'third_party/ideep/mkl-dnn'
2024-08-20T22:08:10.1375080Z Entering 'third_party/ittapi'
2024-08-20T22:08:10.1426019Z Entering 'third_party/kineto'
2024-08-20T22:08:10.1475802Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-08-20T22:08:10.1524955Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-08-20T22:08:10.1577740Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-08-20T22:08:10.1628901Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-08-20T22:08:10.1680669Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-08-20T22:08:10.1730582Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-08-20T22:08:10.1785818Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-08-20T22:08:10.1836612Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-08-20T22:08:10.1888092Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-08-20T22:08:10.1941322Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-08-20T22:08:10.1997733Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-08-20T22:08:10.2047913Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-08-20T22:08:10.2101477Z Entering 'third_party/mimalloc'
2024-08-20T22:08:10.2153250Z Entering 'third_party/nccl/nccl'
2024-08-20T22:08:10.2209702Z Entering 'third_party/nlohmann'
2024-08-20T22:08:10.2262010Z Entering 'third_party/onnx'
2024-08-20T22:08:10.2328311Z Entering 'third_party/onnx/third_party/benchmark'
2024-08-20T22:08:10.2380022Z Entering 'third_party/onnx/third_party/pybind11'
2024-08-20T22:08:10.2435342Z Entering 'third_party/opentelemetry-cpp'
2024-08-20T22:08:10.2487228Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-08-20T22:08:10.2537731Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-08-20T22:08:10.2587857Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-08-20T22:08:10.2637133Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-08-20T22:08:10.2688450Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-08-20T22:08:10.2737981Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-08-20T22:08:10.2788614Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-08-20T22:08:10.2836546Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-08-20T22:08:10.2888483Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-08-20T22:08:10.2942023Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-08-20T22:08:10.3016854Z Entering 'third_party/pocketfft'
2024-08-20T22:08:10.3070830Z Entering 'third_party/protobuf'
2024-08-20T22:08:10.3124180Z Entering 'third_party/protobuf/third_party/benchmark'
2024-08-20T22:08:10.3174947Z Entering 'third_party/protobuf/third_party/googletest'
2024-08-20T22:08:10.3228640Z Entering 'third_party/psimd'
2024-08-20T22:08:10.3280434Z Entering 'third_party/pthreadpool'
2024-08-20T22:08:10.3331666Z Entering 'third_party/pybind11'
2024-08-20T22:08:10.3383744Z Entering 'third_party/python-peachpy'
2024-08-20T22:08:10.3434720Z Entering 'third_party/sleef'
2024-08-20T22:08:10.3485750Z Entering 'third_party/tensorpipe'
2024-08-20T22:08:10.3536557Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-08-20T22:08:10.3591380Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-08-20T22:08:10.3640627Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-08-20T22:08:10.3691760Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-08-20T22:08:10.3739761Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-08-20T22:08:10.3814943Z ##[endgroup]
2024-08-20T22:08:10.3818135Z ##[group]Persisting credentials for submodules
2024-08-20T22:08:10.3821549Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :"
2024-08-20T22:08:10.4188212Z Entering 'android/libs/fbjni'
2024-08-20T22:08:10.4255727Z Entering 'third_party/FP16'
2024-08-20T22:08:10.4321806Z Entering 'third_party/FXdiv'
2024-08-20T22:08:10.4388554Z Entering 'third_party/NNPACK'
2024-08-20T22:08:10.4457635Z Entering 'third_party/VulkanMemoryAllocator'
2024-08-20T22:08:10.4524325Z Entering 'third_party/XNNPACK'
2024-08-20T22:08:10.4606363Z Entering 'third_party/benchmark'
2024-08-20T22:08:10.4673945Z Entering 'third_party/cpp-httplib'
2024-08-20T22:08:10.4741998Z Entering 'third_party/cpuinfo'
2024-08-20T22:08:10.4811224Z Entering 'third_party/cudnn_frontend'
2024-08-20T22:08:10.4879388Z Entering 'third_party/cutlass'
2024-08-20T22:08:10.4953680Z Entering 'third_party/eigen'
2024-08-20T22:08:10.5023499Z Entering 'third_party/fbgemm'
2024-08-20T22:08:10.5089496Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-08-20T22:08:10.5155436Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-08-20T22:08:10.5227219Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-08-20T22:08:10.5299523Z Entering 'third_party/fbgemm/third_party/googletest'
2024-08-20T22:08:10.5364693Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-08-20T22:08:10.5434031Z Entering 'third_party/flatbuffers'
2024-08-20T22:08:10.5505446Z Entering 'third_party/fmt'
2024-08-20T22:08:10.5574995Z Entering 'third_party/gemmlowp/gemmlowp'
2024-08-20T22:08:10.5643299Z Entering 'third_party/gloo'
2024-08-20T22:08:10.5712271Z Entering 'third_party/googletest'
2024-08-20T22:08:10.5780808Z Entering 'third_party/ideep'
2024-08-20T22:08:10.5846263Z Entering 'third_party/ideep/mkl-dnn'
2024-08-20T22:08:10.5923870Z Entering 'third_party/ittapi'
2024-08-20T22:08:10.5992615Z Entering 'third_party/kineto'
2024-08-20T22:08:10.6059294Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-08-20T22:08:10.6124150Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-08-20T22:08:10.6193990Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-08-20T22:08:10.6261591Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-08-20T22:08:10.6329527Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-08-20T22:08:10.6395418Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-08-20T22:08:10.6466163Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-08-20T22:08:10.6535295Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-08-20T22:08:10.6602376Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-08-20T22:08:10.6671401Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-08-20T22:08:10.6743462Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-08-20T22:08:10.6811394Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-08-20T22:08:10.6881530Z Entering 'third_party/mimalloc'
2024-08-20T22:08:10.6949726Z Entering 'third_party/nccl/nccl'
2024-08-20T22:08:10.7017938Z Entering 'third_party/nlohmann'
2024-08-20T22:08:10.7086712Z Entering 'third_party/onnx'
2024-08-20T22:08:10.7168258Z Entering 'third_party/onnx/third_party/benchmark'
2024-08-20T22:08:10.7237270Z Entering 'third_party/onnx/third_party/pybind11'
2024-08-20T22:08:10.7308600Z Entering 'third_party/opentelemetry-cpp'
2024-08-20T22:08:10.7376002Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-08-20T22:08:10.7441700Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-08-20T22:08:10.7508072Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-08-20T22:08:10.7573903Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-08-20T22:08:10.7640312Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-08-20T22:08:10.7707848Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-08-20T22:08:10.7773132Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-08-20T22:08:10.7837076Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-08-20T22:08:10.7906736Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-08-20T22:08:10.7977111Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-08-20T22:08:10.8063144Z Entering 'third_party/pocketfft'
2024-08-20T22:08:10.8130523Z Entering 'third_party/protobuf'
2024-08-20T22:08:10.8200252Z Entering 'third_party/protobuf/third_party/benchmark'
2024-08-20T22:08:10.8266376Z Entering 'third_party/protobuf/third_party/googletest'
2024-08-20T22:08:10.8336046Z Entering 'third_party/psimd'
2024-08-20T22:08:10.8404520Z Entering 'third_party/pthreadpool'
2024-08-20T22:08:10.8471137Z Entering 'third_party/pybind11'
2024-08-20T22:08:10.8537788Z Entering 'third_party/python-peachpy'
2024-08-20T22:08:10.8605096Z Entering 'third_party/sleef'
2024-08-20T22:08:10.8673391Z Entering 'third_party/tensorpipe'
2024-08-20T22:08:10.8739900Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-08-20T22:08:10.8806309Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-08-20T22:08:10.8875765Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-08-20T22:08:10.8941106Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-08-20T22:08:10.9005522Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-08-20T22:08:10.9097711Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url"
2024-08-20T22:08:10.9469862Z Entering 'android/libs/fbjni'
2024-08-20T22:08:10.9531480Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2024-08-20T22:08:10.9553563Z Entering 'third_party/FP16'
2024-08-20T22:08:10.9616030Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2024-08-20T22:08:10.9637733Z Entering 'third_party/FXdiv'
2024-08-20T22:08:10.9701041Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2024-08-20T22:08:10.9722638Z Entering 'third_party/NNPACK'
2024-08-20T22:08:10.9785288Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2024-08-20T22:08:10.9807613Z Entering 'third_party/VulkanMemoryAllocator'
2024-08-20T22:08:10.9869626Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2024-08-20T22:08:10.9891557Z Entering 'third_party/XNNPACK'
2024-08-20T22:08:10.9953428Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2024-08-20T22:08:10.9990997Z Entering 'third_party/benchmark'
2024-08-20T22:08:11.0054344Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2024-08-20T22:08:11.0075576Z Entering 'third_party/cpp-httplib'
2024-08-20T22:08:11.0138104Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config	remote.origin.url
2024-08-20T22:08:11.0159650Z Entering 'third_party/cpuinfo'
2024-08-20T22:08:11.0222629Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2024-08-20T22:08:11.0245508Z Entering 'third_party/cudnn_frontend'
2024-08-20T22:08:11.0307609Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2024-08-20T22:08:11.0329167Z Entering 'third_party/cutlass'
2024-08-20T22:08:11.0394063Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2024-08-20T22:08:11.0422410Z Entering 'third_party/eigen'
2024-08-20T22:08:11.0486988Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2024-08-20T22:08:11.0511571Z Entering 'third_party/fbgemm'
2024-08-20T22:08:11.0575431Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2024-08-20T22:08:11.0595927Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-08-20T22:08:11.0656986Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2024-08-20T22:08:11.0680075Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-08-20T22:08:11.0741200Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2024-08-20T22:08:11.0762824Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-08-20T22:08:11.0832467Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config	remote.origin.url
2024-08-20T22:08:11.0860041Z Entering 'third_party/fbgemm/third_party/googletest'
2024-08-20T22:08:11.0921136Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2024-08-20T22:08:11.0946817Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-08-20T22:08:11.1008092Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2024-08-20T22:08:11.1032653Z Entering 'third_party/flatbuffers'
2024-08-20T22:08:11.1094447Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2024-08-20T22:08:11.1118407Z Entering 'third_party/fmt'
2024-08-20T22:08:11.1179500Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2024-08-20T22:08:11.1201246Z Entering 'third_party/gemmlowp/gemmlowp'
2024-08-20T22:08:11.1263019Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2024-08-20T22:08:11.1284741Z Entering 'third_party/gloo'
2024-08-20T22:08:11.1346841Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2024-08-20T22:08:11.1368438Z Entering 'third_party/googletest'
2024-08-20T22:08:11.1430400Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2024-08-20T22:08:11.1451875Z Entering 'third_party/ideep'
2024-08-20T22:08:11.1513761Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2024-08-20T22:08:11.1532396Z Entering 'third_party/ideep/mkl-dnn'
2024-08-20T22:08:11.1593228Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2024-08-20T22:08:11.1621579Z Entering 'third_party/ittapi'
2024-08-20T22:08:11.1685050Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2024-08-20T22:08:11.1706593Z Entering 'third_party/kineto'
2024-08-20T22:08:11.1770016Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2024-08-20T22:08:11.1789613Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-08-20T22:08:11.1851703Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config	remote.origin.url
2024-08-20T22:08:11.1870537Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-08-20T22:08:11.1932670Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config	remote.origin.url
2024-08-20T22:08:11.1959325Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-08-20T22:08:11.2023993Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config	remote.origin.url
2024-08-20T22:08:11.2045286Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-08-20T22:08:11.2107000Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config	remote.origin.url
2024-08-20T22:08:11.2127242Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-08-20T22:08:11.2194189Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config	remote.origin.url
2024-08-20T22:08:11.2212363Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-08-20T22:08:11.2284637Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config	remote.origin.url
2024-08-20T22:08:11.2307683Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-08-20T22:08:11.2369874Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config	remote.origin.url
2024-08-20T22:08:11.2390634Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-08-20T22:08:11.2452977Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config	remote.origin.url
2024-08-20T22:08:11.2473669Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-08-20T22:08:11.2535970Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config	remote.origin.url
2024-08-20T22:08:11.2557176Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-08-20T22:08:11.2624169Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config	remote.origin.url
2024-08-20T22:08:11.2647514Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-08-20T22:08:11.2707654Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2024-08-20T22:08:11.2727376Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-08-20T22:08:11.2789155Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2024-08-20T22:08:11.2812383Z Entering 'third_party/mimalloc'
2024-08-20T22:08:11.2874844Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config	remote.origin.url
2024-08-20T22:08:11.2896529Z Entering 'third_party/nccl/nccl'
2024-08-20T22:08:11.2957821Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2024-08-20T22:08:11.2979675Z Entering 'third_party/nlohmann'
2024-08-20T22:08:11.3041412Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2024-08-20T22:08:11.3062747Z Entering 'third_party/onnx'
2024-08-20T22:08:11.3124228Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2024-08-20T22:08:11.3159345Z Entering 'third_party/onnx/third_party/benchmark'
2024-08-20T22:08:11.3221105Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2024-08-20T22:08:11.3242232Z Entering 'third_party/onnx/third_party/pybind11'
2024-08-20T22:08:11.3303874Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2024-08-20T22:08:11.3329591Z Entering 'third_party/opentelemetry-cpp'
2024-08-20T22:08:11.3391860Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config	remote.origin.url
2024-08-20T22:08:11.3412732Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-08-20T22:08:11.3474212Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config	remote.origin.url
2024-08-20T22:08:11.3493952Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-08-20T22:08:11.3559557Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config	remote.origin.url
2024-08-20T22:08:11.3579906Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-08-20T22:08:11.3640410Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config	remote.origin.url
2024-08-20T22:08:11.3659908Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-08-20T22:08:11.3722037Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config	remote.origin.url
2024-08-20T22:08:11.3743198Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-08-20T22:08:11.3803789Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config	remote.origin.url
2024-08-20T22:08:11.3823009Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-08-20T22:08:11.3884388Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config	remote.origin.url
2024-08-20T22:08:11.3904008Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-08-20T22:08:11.3966903Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config	remote.origin.url
2024-08-20T22:08:11.3987341Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-08-20T22:08:11.4048905Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2024-08-20T22:08:11.4070500Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-08-20T22:08:11.4131913Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2024-08-20T22:08:11.4154708Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-08-20T22:08:11.4217415Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config	remote.origin.url
2024-08-20T22:08:11.4262056Z Entering 'third_party/pocketfft'
2024-08-20T22:08:11.4324917Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2024-08-20T22:08:11.4347759Z Entering 'third_party/protobuf'
2024-08-20T22:08:11.4410656Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2024-08-20T22:08:11.4432510Z Entering 'third_party/protobuf/third_party/benchmark'
2024-08-20T22:08:11.4494226Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2024-08-20T22:08:11.4513986Z Entering 'third_party/protobuf/third_party/googletest'
2024-08-20T22:08:11.4576226Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2024-08-20T22:08:11.4601113Z Entering 'third_party/psimd'
2024-08-20T22:08:11.4663272Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2024-08-20T22:08:11.4683503Z Entering 'third_party/pthreadpool'
2024-08-20T22:08:11.4744169Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2024-08-20T22:08:11.4765206Z Entering 'third_party/pybind11'
2024-08-20T22:08:11.4826276Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2024-08-20T22:08:11.4847867Z Entering 'third_party/python-peachpy'
2024-08-20T22:08:11.4911804Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2024-08-20T22:08:11.4933337Z Entering 'third_party/sleef'
2024-08-20T22:08:11.4997608Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2024-08-20T22:08:11.5019255Z Entering 'third_party/tensorpipe'
2024-08-20T22:08:11.5082408Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2024-08-20T22:08:11.5102520Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-08-20T22:08:11.5163184Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2024-08-20T22:08:11.5185538Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-08-20T22:08:11.5245116Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2024-08-20T22:08:11.5265484Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-08-20T22:08:11.5327429Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2024-08-20T22:08:11.5347393Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-08-20T22:08:11.5409358Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2024-08-20T22:08:11.5427187Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-08-20T22:08:11.5489679Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2024-08-20T22:08:11.6111351Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2024-08-20T22:08:11.6475331Z Entering 'android/libs/fbjni'
2024-08-20T22:08:11.6527897Z Entering 'third_party/FP16'
2024-08-20T22:08:11.6579973Z Entering 'third_party/FXdiv'
2024-08-20T22:08:11.6630975Z Entering 'third_party/NNPACK'
2024-08-20T22:08:11.6682897Z Entering 'third_party/VulkanMemoryAllocator'
2024-08-20T22:08:11.6734283Z Entering 'third_party/XNNPACK'
2024-08-20T22:08:11.6801905Z Entering 'third_party/benchmark'
2024-08-20T22:08:11.6853178Z Entering 'third_party/cpp-httplib'
2024-08-20T22:08:11.6906611Z Entering 'third_party/cpuinfo'
2024-08-20T22:08:11.6959041Z Entering 'third_party/cudnn_frontend'
2024-08-20T22:08:11.7011847Z Entering 'third_party/cutlass'
2024-08-20T22:08:11.7072289Z Entering 'third_party/eigen'
2024-08-20T22:08:11.7128632Z Entering 'third_party/fbgemm'
2024-08-20T22:08:11.7179861Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-08-20T22:08:11.7231474Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-08-20T22:08:11.7282350Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-08-20T22:08:11.7338569Z Entering 'third_party/fbgemm/third_party/googletest'
2024-08-20T22:08:11.7389729Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-08-20T22:08:11.7442797Z Entering 'third_party/flatbuffers'
2024-08-20T22:08:11.7499134Z Entering 'third_party/fmt'
2024-08-20T22:08:11.7551139Z Entering 'third_party/gemmlowp/gemmlowp'
2024-08-20T22:08:11.7604073Z Entering 'third_party/gloo'
2024-08-20T22:08:11.7660134Z Entering 'third_party/googletest'
2024-08-20T22:08:11.7712453Z Entering 'third_party/ideep'
2024-08-20T22:08:11.7762429Z Entering 'third_party/ideep/mkl-dnn'
2024-08-20T22:08:11.7822194Z Entering 'third_party/ittapi'
2024-08-20T22:08:11.7875130Z Entering 'third_party/kineto'
2024-08-20T22:08:11.7926692Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-08-20T22:08:11.7976290Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-08-20T22:08:11.8027924Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-08-20T22:08:11.8078517Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-08-20T22:08:11.8128625Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-08-20T22:08:11.8177401Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-08-20T22:08:11.8231839Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-08-20T22:08:11.8283134Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-08-20T22:08:11.8334098Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-08-20T22:08:11.8385783Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-08-20T22:08:11.8438445Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-08-20T22:08:11.8491061Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-08-20T22:08:11.8544062Z Entering 'third_party/mimalloc'
2024-08-20T22:08:11.8596320Z Entering 'third_party/nccl/nccl'
2024-08-20T22:08:11.8647831Z Entering 'third_party/nlohmann'
2024-08-20T22:08:11.8700389Z Entering 'third_party/onnx'
2024-08-20T22:08:11.8764478Z Entering 'third_party/onnx/third_party/benchmark'
2024-08-20T22:08:11.8820398Z Entering 'third_party/onnx/third_party/pybind11'
2024-08-20T22:08:11.8877083Z Entering 'third_party/opentelemetry-cpp'
2024-08-20T22:08:11.8931190Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-08-20T22:08:11.8986509Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-08-20T22:08:11.9036330Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-08-20T22:08:11.9086397Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-08-20T22:08:11.9137966Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-08-20T22:08:11.9186841Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-08-20T22:08:11.9239237Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-08-20T22:08:11.9287382Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-08-20T22:08:11.9340553Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-08-20T22:08:11.9395102Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-08-20T22:08:11.9465918Z Entering 'third_party/pocketfft'
2024-08-20T22:08:11.9517395Z Entering 'third_party/protobuf'
2024-08-20T22:08:11.9570358Z Entering 'third_party/protobuf/third_party/benchmark'
2024-08-20T22:08:11.9620631Z Entering 'third_party/protobuf/third_party/googletest'
2024-08-20T22:08:11.9676507Z Entering 'third_party/psimd'
2024-08-20T22:08:11.9727837Z Entering 'third_party/pthreadpool'
2024-08-20T22:08:11.9779208Z Entering 'third_party/pybind11'
2024-08-20T22:08:11.9830801Z Entering 'third_party/python-peachpy'
2024-08-20T22:08:11.9882279Z Entering 'third_party/sleef'
2024-08-20T22:08:11.9933230Z Entering 'third_party/tensorpipe'
2024-08-20T22:08:11.9985126Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-08-20T22:08:12.0034941Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-08-20T22:08:12.0090358Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-08-20T22:08:12.0140365Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-08-20T22:08:12.0190490Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-08-20T22:08:12.0268142Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2024-08-20T22:08:12.0634960Z Entering 'android/libs/fbjni'
2024-08-20T22:08:12.0685828Z Entering 'third_party/FP16'
2024-08-20T22:08:12.0736887Z Entering 'third_party/FXdiv'
2024-08-20T22:08:12.0788743Z Entering 'third_party/NNPACK'
2024-08-20T22:08:12.0841460Z Entering 'third_party/VulkanMemoryAllocator'
2024-08-20T22:08:12.0895375Z Entering 'third_party/XNNPACK'
2024-08-20T22:08:12.0962605Z Entering 'third_party/benchmark'
2024-08-20T22:08:12.1018098Z Entering 'third_party/cpp-httplib'
2024-08-20T22:08:12.1069511Z Entering 'third_party/cpuinfo'
2024-08-20T22:08:12.1124119Z Entering 'third_party/cudnn_frontend'
2024-08-20T22:08:12.1180886Z Entering 'third_party/cutlass'
2024-08-20T22:08:12.1245935Z Entering 'third_party/eigen'
2024-08-20T22:08:12.1300858Z Entering 'third_party/fbgemm'
2024-08-20T22:08:12.1352004Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-08-20T22:08:12.1403083Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-08-20T22:08:12.1457592Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-08-20T22:08:12.1514864Z Entering 'third_party/fbgemm/third_party/googletest'
2024-08-20T22:08:12.1568602Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-08-20T22:08:12.1626723Z Entering 'third_party/flatbuffers'
2024-08-20T22:08:12.1682069Z Entering 'third_party/fmt'
2024-08-20T22:08:12.1732846Z Entering 'third_party/gemmlowp/gemmlowp'
2024-08-20T22:08:12.1785351Z Entering 'third_party/gloo'
2024-08-20T22:08:12.1836474Z Entering 'third_party/googletest'
2024-08-20T22:08:12.1887716Z Entering 'third_party/ideep'
2024-08-20T22:08:12.1954044Z Entering 'third_party/ideep/mkl-dnn'
2024-08-20T22:08:12.2014001Z Entering 'third_party/ittapi'
2024-08-20T22:08:12.2064447Z Entering 'third_party/kineto'
2024-08-20T22:08:12.2115000Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-08-20T22:08:12.2168559Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-08-20T22:08:12.2224330Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-08-20T22:08:12.2274207Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-08-20T22:08:12.2324776Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-08-20T22:08:12.2373048Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-08-20T22:08:12.2427819Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-08-20T22:08:12.2478999Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-08-20T22:08:12.2530061Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-08-20T22:08:12.2580860Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-08-20T22:08:12.2633572Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-08-20T22:08:12.2684298Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-08-20T22:08:12.2737612Z Entering 'third_party/mimalloc'
2024-08-20T22:08:12.2789237Z Entering 'third_party/nccl/nccl'
2024-08-20T22:08:12.2842129Z Entering 'third_party/nlohmann'
2024-08-20T22:08:12.2895312Z Entering 'third_party/onnx'
2024-08-20T22:08:12.2959818Z Entering 'third_party/onnx/third_party/benchmark'
2024-08-20T22:08:12.3011467Z Entering 'third_party/onnx/third_party/pybind11'
2024-08-20T22:08:12.3067194Z Entering 'third_party/opentelemetry-cpp'
2024-08-20T22:08:12.3122324Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-08-20T22:08:12.3177591Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-08-20T22:08:12.3227476Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-08-20T22:08:12.3277190Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-08-20T22:08:12.3328405Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-08-20T22:08:12.3378540Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-08-20T22:08:12.3430306Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-08-20T22:08:12.3478265Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-08-20T22:08:12.3529703Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-08-20T22:08:12.3584356Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-08-20T22:08:12.3657067Z Entering 'third_party/pocketfft'
2024-08-20T22:08:12.3709594Z Entering 'third_party/protobuf'
2024-08-20T22:08:12.3764830Z Entering 'third_party/protobuf/third_party/benchmark'
2024-08-20T22:08:12.3814597Z Entering 'third_party/protobuf/third_party/googletest'
2024-08-20T22:08:12.3868450Z Entering 'third_party/psimd'
2024-08-20T22:08:12.3921678Z Entering 'third_party/pthreadpool'
2024-08-20T22:08:12.3975887Z Entering 'third_party/pybind11'
2024-08-20T22:08:12.4028167Z Entering 'third_party/python-peachpy'
2024-08-20T22:08:12.4080104Z Entering 'third_party/sleef'
2024-08-20T22:08:12.4132089Z Entering 'third_party/tensorpipe'
2024-08-20T22:08:12.4183646Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-08-20T22:08:12.4232927Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-08-20T22:08:12.4282309Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-08-20T22:08:12.4332218Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-08-20T22:08:12.4380939Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-08-20T22:08:12.4453305Z ##[endgroup]
2024-08-20T22:08:12.4502477Z [command]/usr/bin/git log -1 --format='%H'
2024-08-20T22:08:12.4537802Z '40ec5f6ddd9787aca0449b24128343ff4c4a88b3'
2024-08-20T22:08:12.4763827Z Prepare all required actions
2024-08-20T22:08:12.4764319Z Getting action download info
2024-08-20T22:08:12.6352401Z ##[group]Run ./.github/actions/setup-linux
2024-08-20T22:08:12.6352824Z env:
2024-08-20T22:08:12.6353095Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:08:12.6353442Z ##[endgroup]
2024-08-20T22:08:12.6413776Z ##[group]Run set -euo pipefail
2024-08-20T22:08:12.6414202Z set -euo pipefail
2024-08-20T22:08:12.6414595Z function get_ec2_metadata() {
2024-08-20T22:08:12.6415131Z   # Pulled from instance metadata endpoint for EC2
2024-08-20T22:08:12.6416024Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2024-08-20T22:08:12.6416797Z   category=$1
2024-08-20T22:08:12.6417332Z   # If it is GCP runner (runner name contains gcp), do not run this
2024-08-20T22:08:12.6417971Z   runner_name_str=i-0479ed8028e52c92f
2024-08-20T22:08:12.6418533Z   if [[ -f /.inarc ]]; then
2024-08-20T22:08:12.6419033Z     echo "ARC Runner, no info on ec2 metadata"
2024-08-20T22:08:12.6419607Z   elif [[ $runner_name_str == *"gcp"* ]]; then
2024-08-20T22:08:12.6420305Z     echo "Runner is from Google Cloud Platform, No info on ec2 metadata"
2024-08-20T22:08:12.6420910Z   else
2024-08-20T22:08:12.6421408Z     curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2024-08-20T22:08:12.6421982Z   fi
2024-08-20T22:08:12.6422271Z }
2024-08-20T22:08:12.6422627Z echo "ami-id: $(get_ec2_metadata ami-id)"
2024-08-20T22:08:12.6423215Z echo "instance-id: $(get_ec2_metadata instance-id)"
2024-08-20T22:08:12.6423866Z echo "instance-type: $(get_ec2_metadata instance-type)"
2024-08-20T22:08:12.6424439Z echo "system info $(uname -a)"
2024-08-20T22:08:12.6435791Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:08:12.6436293Z env:
2024-08-20T22:08:12.6436577Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:08:12.6436924Z ##[endgroup]
2024-08-20T22:08:12.6544401Z ami-id: ami-06c68f701d8090592
2024-08-20T22:08:12.6606214Z instance-id: i-0479ed8028e52c92f
2024-08-20T22:08:12.6661377Z instance-type: g5.4xlarge
2024-08-20T22:08:12.6675355Z system info Linux ip-10-0-77-54.ec2.internal 6.1.94-99.176.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
2024-08-20T22:08:12.6698765Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)"  >> $GITHUB_OUTPUT
2024-08-20T22:08:12.6699675Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)"  >> $GITHUB_OUTPUT
2024-08-20T22:08:12.6708779Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:08:12.6709265Z env:
2024-08-20T22:08:12.6709553Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:08:12.6709904Z ##[endgroup]
2024-08-20T22:08:12.6783915Z ##[group]Run if systemctl is-active --quiet docker; then
2024-08-20T22:08:12.6784520Z if systemctl is-active --quiet docker; then
2024-08-20T22:08:12.6785062Z     echo "Docker daemon is running...";
2024-08-20T22:08:12.6785517Z else
2024-08-20T22:08:12.6786003Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2024-08-20T22:08:12.6786760Z fi
2024-08-20T22:08:12.6794880Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:08:12.6795370Z env:
2024-08-20T22:08:12.6795651Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:08:12.6795995Z ##[endgroup]
2024-08-20T22:08:12.6885876Z Docker daemon is running...
2024-08-20T22:08:12.6946701Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2024-08-20T22:08:12.6947255Z with:
2024-08-20T22:08:12.6947522Z   shell: bash
2024-08-20T22:08:12.6947821Z   timeout_minutes: 5
2024-08-20T22:08:12.6948148Z   max_attempts: 3
2024-08-20T22:08:12.6948467Z   retry_wait_seconds: 30
2024-08-20T22:08:12.6951564Z   command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \
    --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"

# For LF Runners we need to make sure we also login to Meta's ECR docker registry too.
META_AWS_ACCOUNT_ID=308535385114
if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then
    aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \
        --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
fi

2024-08-20T22:08:12.6954529Z   polling_interval_seconds: 1
2024-08-20T22:08:12.6954901Z   warning_on_retry: true
2024-08-20T22:08:12.6955247Z   continue_on_error: false
2024-08-20T22:08:12.6955577Z env:
2024-08-20T22:08:12.6955856Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:08:12.6956214Z   AWS_RETRY_MODE: standard
2024-08-20T22:08:12.6956563Z   AWS_MAX_ATTEMPTS: 5
2024-08-20T22:08:12.6956909Z   AWS_DEFAULT_REGION: us-east-1
2024-08-20T22:08:12.6957284Z ##[endgroup]
2024-08-20T22:08:13.8528912Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-08-20T22:08:13.8529955Z Configure a credential helper to remove this warning. See
2024-08-20T22:08:13.8531021Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-08-20T22:08:13.8531713Z 
2024-08-20T22:08:13.8531891Z Login Succeeded
2024-08-20T22:08:14.7529801Z Command completed after 1 attempt(s).
2024-08-20T22:08:14.7590502Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2024-08-20T22:08:14.7591224Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2024-08-20T22:08:14.7591882Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2024-08-20T22:08:14.7600930Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:08:14.7601423Z env:
2024-08-20T22:08:14.7601709Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:08:14.7602063Z ##[endgroup]
2024-08-20T22:08:14.7697615Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2024-08-20T22:08:14.7698454Z # ignore expansion of "docker ps -q" since it could be empty
2024-08-20T22:08:14.7699045Z # shellcheck disable=SC2046
2024-08-20T22:08:14.7699508Z docker stop $(docker ps -q) || true
2024-08-20T22:08:14.7699993Z # Prune all of the docker images
2024-08-20T22:08:14.7700447Z docker system prune -af
2024-08-20T22:08:14.7709322Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:08:14.7709821Z env:
2024-08-20T22:08:14.7710101Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:08:14.7710447Z ##[endgroup]
2024-08-20T22:08:14.8027613Z "docker stop" requires at least 1 argument.
2024-08-20T22:08:14.8028401Z See 'docker stop --help'.
2024-08-20T22:08:14.8028644Z 
2024-08-20T22:08:14.8028860Z Usage:  docker stop [OPTIONS] CONTAINER [CONTAINER...]
2024-08-20T22:08:14.8029309Z 
2024-08-20T22:08:14.8029526Z Stop one or more running containers
2024-08-20T22:08:14.8215757Z Total reclaimed space: 0B
2024-08-20T22:08:14.8269820Z ##[group]Run set +e
2024-08-20T22:08:14.8270266Z set +e
2024-08-20T22:08:14.8270564Z set -x
2024-08-20T22:08:14.8270853Z 
2024-08-20T22:08:14.8271169Z PT_DOMAIN=download.pytorch.org
2024-08-20T22:08:14.8272131Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400,
2024-08-20T22:08:14.8273188Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last
2024-08-20T22:08:14.8273933Z # one is returned at random
2024-08-20T22:08:14.8274468Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1)
2024-08-20T22:08:14.8274966Z 
2024-08-20T22:08:14.8275279Z if [ -z "${RESOLVED_IP}" ]; then
2024-08-20T22:08:14.8275895Z   echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..."
2024-08-20T22:08:14.8276817Z   RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1)
2024-08-20T22:08:14.8277366Z 
2024-08-20T22:08:14.8277691Z   if [ -z "${RESOLVED_IP}" ]; then
2024-08-20T22:08:14.8278234Z     echo "Couldn't resolve ${PT_DOMAIN}, exiting..."
2024-08-20T22:08:14.8278736Z     exit 1
2024-08-20T22:08:14.8279051Z   fi
2024-08-20T22:08:14.8279338Z fi
2024-08-20T22:08:14.8279615Z 
2024-08-20T22:08:14.8279972Z if grep -r "${PT_DOMAIN}" /etc/hosts; then
2024-08-20T22:08:14.8280482Z   # Clean up any old records first
2024-08-20T22:08:14.8280987Z   sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts
2024-08-20T22:08:14.8281433Z fi
2024-08-20T22:08:14.8281699Z 
2024-08-20T22:08:14.8282126Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts
2024-08-20T22:08:14.8282673Z cat /etc/hosts
2024-08-20T22:08:14.8291541Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:08:14.8292029Z env:
2024-08-20T22:08:14.8292319Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:08:14.8292667Z ##[endgroup]
2024-08-20T22:08:14.8324961Z + PT_DOMAIN=download.pytorch.org
2024-08-20T22:08:14.8331070Z ++ dig -4 +short download.pytorch.org
2024-08-20T22:08:14.8331615Z ++ tail -n1
2024-08-20T22:08:14.8739856Z + RESOLVED_IP=18.160.10.22
2024-08-20T22:08:14.8740517Z + '[' -z 18.160.10.22 ']'
2024-08-20T22:08:14.8741079Z + grep -r download.pytorch.org /etc/hosts
2024-08-20T22:08:14.8758942Z + echo '18.160.10.22 download.pytorch.org'
2024-08-20T22:08:14.8759585Z + sudo tee -a /etc/hosts
2024-08-20T22:08:15.1440080Z 18.160.10.22 download.pytorch.org
2024-08-20T22:08:15.1462041Z + cat /etc/hosts
2024-08-20T22:08:15.1473458Z 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
2024-08-20T22:08:15.1485553Z ::1         localhost6 localhost6.localdomain6
2024-08-20T22:08:15.1486409Z 18.160.10.22 download.pytorch.org
2024-08-20T22:08:15.1671268Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main
2024-08-20T22:08:15.1671864Z with:
2024-08-20T22:08:15.1672948Z   docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:08:15.1674170Z   docker-build-dir: .ci/docker
2024-08-20T22:08:15.1674555Z   working-directory: .
2024-08-20T22:08:15.1675013Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-08-20T22:08:15.1675526Z   force-push: false
2024-08-20T22:08:15.1675821Z env:
2024-08-20T22:08:15.1676096Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:08:15.1676427Z ##[endgroup]
2024-08-20T22:08:15.1694953Z ##[group]Run set -ex
2024-08-20T22:08:15.1695323Z set -ex
2024-08-20T22:08:15.1695612Z 
2024-08-20T22:08:15.1696174Z # If the docker build directory or the build script doesn't exist, the action will
2024-08-20T22:08:15.1697155Z # gracefully return the docker image name as it is.  Pulling docker image in Linux
2024-08-20T22:08:15.1697946Z # job could then download the pre-built image as usual
2024-08-20T22:08:15.1698800Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then
2024-08-20T22:08:15.1699464Z   echo "skip=true" >> "${GITHUB_OUTPUT}"
2024-08-20T22:08:15.1700255Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2024-08-20T22:08:15.1700802Z 
2024-08-20T22:08:15.1701301Z   echo "There is no Docker build script in ${REPO_NAME} repo, skipping..."
2024-08-20T22:08:15.1701923Z   exit 0
2024-08-20T22:08:15.1702224Z else
2024-08-20T22:08:15.1702585Z   echo "skip=false" >> "${GITHUB_OUTPUT}"
2024-08-20T22:08:15.1703032Z fi
2024-08-20T22:08:15.1703307Z 
2024-08-20T22:08:15.1703767Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then
2024-08-20T22:08:15.1704601Z   # The docker image name already includes the ECR prefix and tag, so we can just
2024-08-20T22:08:15.1705366Z   # use it as it is, but first let's extract the tag
2024-08-20T22:08:15.1706058Z   DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}')
2024-08-20T22:08:15.1706776Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2024-08-20T22:08:15.1707474Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2024-08-20T22:08:15.1708026Z else
2024-08-20T22:08:15.1708456Z   DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}")
2024-08-20T22:08:15.1709106Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2024-08-20T22:08:15.1709979Z   echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2024-08-20T22:08:15.1710707Z fi
2024-08-20T22:08:15.1722665Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:08:15.1723162Z env:
2024-08-20T22:08:15.1723444Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:08:15.1723796Z   REPO_NAME: pytorch
2024-08-20T22:08:15.1724914Z   DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:08:15.1726117Z   DOCKER_BUILD_DIR: .ci/docker
2024-08-20T22:08:15.1726616Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-08-20T22:08:15.1727125Z ##[endgroup]
2024-08-20T22:08:15.1758615Z + [[ ! -d .ci/docker ]]
2024-08-20T22:08:15.1759352Z + [[ ! -f .ci/docker/build.sh ]]
2024-08-20T22:08:15.1759862Z + echo skip=false
2024-08-20T22:08:15.1761973Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]]
2024-08-20T22:08:15.1765925Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:08:15.1767300Z ++ awk -F '[:,]' '{print $2}'
2024-08-20T22:08:15.1792209Z + DOCKER_TAG=f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:08:15.1793282Z + echo docker-tag=f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:08:15.1795333Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:08:15.1828154Z ##[group]Run set +e
2024-08-20T22:08:15.1828631Z set +e
2024-08-20T22:08:15.1829099Z set -x
2024-08-20T22:08:15.1829478Z 
2024-08-20T22:08:15.1829810Z login() {
2024-08-20T22:08:15.1830652Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2024-08-20T22:08:15.1831459Z }
2024-08-20T22:08:15.1831899Z 
2024-08-20T22:08:15.1832235Z retry () {
2024-08-20T22:08:15.1832715Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2024-08-20T22:08:15.1833318Z }
2024-08-20T22:08:15.1833651Z 
2024-08-20T22:08:15.1834054Z retry login "${DOCKER_REGISTRY}"
2024-08-20T22:08:15.1834782Z 
2024-08-20T22:08:15.1835341Z # Check if image already exists, if it does then skip building it
2024-08-20T22:08:15.1836120Z if docker manifest inspect "${DOCKER_IMAGE}"; then
2024-08-20T22:08:15.1836765Z   exit 0
2024-08-20T22:08:15.1837155Z fi
2024-08-20T22:08:15.1837543Z 
2024-08-20T22:08:15.1838147Z # NB: This part requires a full checkout. Otherwise, the merge base will
2024-08-20T22:08:15.1839059Z # be empty.  The default action would be to continue rebuild the image
2024-08-20T22:08:15.1839906Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then
2024-08-20T22:08:15.1840789Z   # if we're on the base branch then use the parent commit
2024-08-20T22:08:15.1841526Z   MERGE_BASE=$(git rev-parse HEAD~)
2024-08-20T22:08:15.1842122Z else
2024-08-20T22:08:15.1842745Z   # otherwise we're on a PR, so use the most recent base commit
2024-08-20T22:08:15.1843549Z   MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION")
2024-08-20T22:08:15.1844124Z fi
2024-08-20T22:08:15.1844575Z 
2024-08-20T22:08:15.1845002Z if [[ -z "${MERGE_BASE}" ]]; then
2024-08-20T22:08:15.1845595Z   echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2024-08-20T22:08:15.1846172Z 
2024-08-20T22:08:15.1846929Z   echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..."
2024-08-20T22:08:15.1847763Z   exit 0
2024-08-20T22:08:15.1848289Z fi
2024-08-20T22:08:15.1848624Z 
2024-08-20T22:08:15.1849151Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then
2024-08-20T22:08:15.1850274Z   echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit"
2024-08-20T22:08:15.1851155Z   exit 1
2024-08-20T22:08:15.1851510Z fi
2024-08-20T22:08:15.1851945Z 
2024-08-20T22:08:15.1852642Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}")
2024-08-20T22:08:15.1853856Z # If no image exists but the hash is the same as the previous hash then we should error out here
2024-08-20T22:08:15.1854823Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then
2024-08-20T22:08:15.1855862Z   echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch"
2024-08-20T22:08:15.1857176Z   echo "         Will re-build docker image to store in local cache, TTS may be longer"
2024-08-20T22:08:15.1858068Z fi
2024-08-20T22:08:15.1858544Z 
2024-08-20T22:08:15.1859007Z echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2024-08-20T22:08:15.1867443Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:08:15.1868101Z env:
2024-08-20T22:08:15.1868469Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:08:15.1869378Z   DOCKER_BUILD_DIR: .ci/docker
2024-08-20T22:08:15.1870038Z   BASE_REVISION: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:08:15.1871423Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:08:15.1872840Z   DOCKER_TAG: f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:08:15.1873565Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-08-20T22:08:15.1874191Z ##[endgroup]
2024-08-20T22:08:15.1904480Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-08-20T22:08:15.1905221Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-08-20T22:08:15.1906220Z + aws ecr get-login-password --region us-east-1
2024-08-20T22:08:15.1908594Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-08-20T22:08:15.7339160Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-08-20T22:08:15.7340768Z Configure a credential helper to remove this warning. See
2024-08-20T22:08:15.7341763Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-08-20T22:08:15.7342342Z 
2024-08-20T22:08:15.7342490Z Login Succeeded
2024-08-20T22:08:15.7361839Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:08:15.9800177Z {
2024-08-20T22:08:15.9800801Z 	"schemaVersion": 2,
2024-08-20T22:08:15.9802220Z 	"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
2024-08-20T22:08:15.9802902Z 	"config": {
2024-08-20T22:08:15.9803818Z 		"mediaType": "application/vnd.docker.container.image.v1+json",
2024-08-20T22:08:15.9804464Z 		"size": 48711,
2024-08-20T22:08:15.9805062Z 		"digest": "sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb"
2024-08-20T22:08:15.9805876Z 	},
2024-08-20T22:08:15.9806431Z 	"layers": [
2024-08-20T22:08:15.9807036Z 		{
2024-08-20T22:08:15.9807648Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9808790Z 			"size": 28580681,
2024-08-20T22:08:15.9809970Z 			"digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a"
2024-08-20T22:08:15.9810939Z 		},
2024-08-20T22:08:15.9811581Z 		{
2024-08-20T22:08:15.9812249Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9813113Z 			"size": 7943451,
2024-08-20T22:08:15.9814038Z 			"digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d"
2024-08-20T22:08:15.9814966Z 		},
2024-08-20T22:08:15.9815365Z 		{
2024-08-20T22:08:15.9816131Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9816770Z 			"size": 55728572,
2024-08-20T22:08:15.9817458Z 			"digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450"
2024-08-20T22:08:15.9818135Z 		},
2024-08-20T22:08:15.9818577Z 		{
2024-08-20T22:08:15.9819186Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9819794Z 			"size": 186,
2024-08-20T22:08:15.9820425Z 			"digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195"
2024-08-20T22:08:15.9821213Z 		},
2024-08-20T22:08:15.9821551Z 		{
2024-08-20T22:08:15.9822036Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9822746Z 			"size": 6886,
2024-08-20T22:08:15.9823373Z 			"digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80"
2024-08-20T22:08:15.9824073Z 		},
2024-08-20T22:08:15.9824793Z 		{
2024-08-20T22:08:15.9825317Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9825992Z 			"size": 1329001756,
2024-08-20T22:08:15.9826714Z 			"digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108"
2024-08-20T22:08:15.9827698Z 		},
2024-08-20T22:08:15.9828112Z 		{
2024-08-20T22:08:15.9828631Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9829231Z 			"size": 62501,
2024-08-20T22:08:15.9829939Z 			"digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198"
2024-08-20T22:08:15.9830636Z 		},
2024-08-20T22:08:15.9830992Z 		{
2024-08-20T22:08:15.9831522Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9832161Z 			"size": 1684,
2024-08-20T22:08:15.9832798Z 			"digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a"
2024-08-20T22:08:15.9833538Z 		},
2024-08-20T22:08:15.9833834Z 		{
2024-08-20T22:08:15.9834375Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9835070Z 			"size": 1523,
2024-08-20T22:08:15.9835673Z 			"digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1"
2024-08-20T22:08:15.9836407Z 		},
2024-08-20T22:08:15.9836793Z 		{
2024-08-20T22:08:15.9837327Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9838088Z 			"size": 2528295403,
2024-08-20T22:08:15.9838795Z 			"digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5"
2024-08-20T22:08:15.9839516Z 		},
2024-08-20T22:08:15.9839850Z 		{
2024-08-20T22:08:15.9840401Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9841034Z 			"size": 86016,
2024-08-20T22:08:15.9841671Z 			"digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b"
2024-08-20T22:08:15.9842454Z 		},
2024-08-20T22:08:15.9842751Z 		{
2024-08-20T22:08:15.9843270Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9843973Z 			"size": 1822,
2024-08-20T22:08:15.9844566Z 			"digest": "sha256:ac23c5fcc1ef310e9cd0d138b180281188e20a3e28c51cd14751622e3c7d92ea"
2024-08-20T22:08:15.9845268Z 		},
2024-08-20T22:08:15.9845675Z 		{
2024-08-20T22:08:15.9846188Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9846813Z 			"size": 246761563,
2024-08-20T22:08:15.9847515Z 			"digest": "sha256:ca426ffc9d2bd051ad1ff5c4f02fd5d9be161e70953cc37882c815080c708f4d"
2024-08-20T22:08:15.9848224Z 		},
2024-08-20T22:08:15.9848632Z 		{
2024-08-20T22:08:15.9849109Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9849738Z 			"size": 544,
2024-08-20T22:08:15.9850416Z 			"digest": "sha256:4373aa743caf6660c671295b837048c8350a2165f2336dc3b7b6c96462579e50"
2024-08-20T22:08:15.9851064Z 		},
2024-08-20T22:08:15.9851408Z 		{
2024-08-20T22:08:15.9851992Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9852626Z 			"size": 1258,
2024-08-20T22:08:15.9853225Z 			"digest": "sha256:75b2abb16c685854c017d0cf75c626526c956734c656d9e78fdd4520317e9b3c"
2024-08-20T22:08:15.9853954Z 		},
2024-08-20T22:08:15.9854303Z 		{
2024-08-20T22:08:15.9854876Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9855519Z 			"size": 484,
2024-08-20T22:08:15.9856137Z 			"digest": "sha256:ef1e3ef1e85f0124270b8506721219150e230184181f8207d64463bf5c67e656"
2024-08-20T22:08:15.9856841Z 		},
2024-08-20T22:08:15.9857244Z 		{
2024-08-20T22:08:15.9857726Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9858501Z 			"size": 91711766,
2024-08-20T22:08:15.9859194Z 			"digest": "sha256:0bf61114d37b2d09065aa6852b1e78a32580a1ff011dd5d23f03b0f5c4e710af"
2024-08-20T22:08:15.9859873Z 		},
2024-08-20T22:08:15.9860212Z 		{
2024-08-20T22:08:15.9860779Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9861437Z 			"size": 3390,
2024-08-20T22:08:15.9862119Z 			"digest": "sha256:2df849f72ba4258df10c9c45ace21a66d93b73196e78e96f12c13098822f60e2"
2024-08-20T22:08:15.9862896Z 		},
2024-08-20T22:08:15.9863229Z 		{
2024-08-20T22:08:15.9863738Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9864424Z 			"size": 1909,
2024-08-20T22:08:15.9865039Z 			"digest": "sha256:34c47d63c5de27418739aa048b4dd781c3006d65691e9b903c95304d0509d89c"
2024-08-20T22:08:15.9865727Z 		},
2024-08-20T22:08:15.9866096Z 		{
2024-08-20T22:08:15.9866605Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9867236Z 			"size": 701,
2024-08-20T22:08:15.9867940Z 			"digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2"
2024-08-20T22:08:15.9868594Z 		},
2024-08-20T22:08:15.9869394Z 		{
2024-08-20T22:08:15.9869999Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9870636Z 			"size": 2864254879,
2024-08-20T22:08:15.9871287Z 			"digest": "sha256:6efb96fc87ea1bca54876d85331e5585e28f325a1d60ee1aef7811eabb74f933"
2024-08-20T22:08:15.9872041Z 		},
2024-08-20T22:08:15.9872370Z 		{
2024-08-20T22:08:15.9872952Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9873552Z 			"size": 380,
2024-08-20T22:08:15.9874169Z 			"digest": "sha256:b119c4fc6c58d9e72744201ff4850b34184a94eb3566031b7d36e87ecce950a6"
2024-08-20T22:08:15.9875085Z 		},
2024-08-20T22:08:15.9875384Z 		{
2024-08-20T22:08:15.9875917Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9876599Z 			"size": 12900,
2024-08-20T22:08:15.9877239Z 			"digest": "sha256:33b639226eeefa41d7bb26a8ecc20d271ceef45571dabfd80aaee3634926381b"
2024-08-20T22:08:15.9877930Z 		},
2024-08-20T22:08:15.9878356Z 		{
2024-08-20T22:08:15.9878860Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9879490Z 			"size": 803,
2024-08-20T22:08:15.9880155Z 			"digest": "sha256:d3547bd43146930aa830d54562c98aa678e355cf5f84527e9717eb72fffe8ca3"
2024-08-20T22:08:15.9880865Z 		},
2024-08-20T22:08:15.9881162Z 		{
2024-08-20T22:08:15.9881721Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9882371Z 			"size": 106,
2024-08-20T22:08:15.9882951Z 			"digest": "sha256:5645520256fa6d7f2270608eaed53849a131d03fbf4c5d4472c9cfde1a439e47"
2024-08-20T22:08:15.9883697Z 		},
2024-08-20T22:08:15.9884052Z 		{
2024-08-20T22:08:15.9884565Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9885242Z 			"size": 503,
2024-08-20T22:08:15.9885858Z 			"digest": "sha256:829ed7d91e3818d0e11834c71551117c7aeec0a51d798e93d641faa0c02582cb"
2024-08-20T22:08:15.9886540Z 		},
2024-08-20T22:08:15.9886947Z 		{
2024-08-20T22:08:15.9887420Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9888054Z 			"size": 121477267,
2024-08-20T22:08:15.9888762Z 			"digest": "sha256:385d0421d67c533d628985c453a186028a61f6ad4209ae519ec3d4ecea661d3e"
2024-08-20T22:08:15.9889460Z 		},
2024-08-20T22:08:15.9889759Z 		{
2024-08-20T22:08:15.9890365Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9890997Z 			"size": 109,
2024-08-20T22:08:15.9891608Z 			"digest": "sha256:7cbfce999d76851e07dd71056a6d852629d789ccfd46be44cc405b451fb8bb58"
2024-08-20T22:08:15.9892370Z 		},
2024-08-20T22:08:15.9892705Z 		{
2024-08-20T22:08:15.9893295Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9893965Z 			"size": 490,
2024-08-20T22:08:15.9894632Z 			"digest": "sha256:6658e558ebe496f0d628947534b0ea1e44808f84f2dfd8a9a704e8aa5329d791"
2024-08-20T22:08:15.9895403Z 		},
2024-08-20T22:08:15.9895747Z 		{
2024-08-20T22:08:15.9896226Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9896933Z 			"size": 297,
2024-08-20T22:08:15.9897554Z 			"digest": "sha256:33444a5a7e01a3010fd1cc16b32a66892a08600d7f700747d23faf3c36624f3a"
2024-08-20T22:08:15.9898297Z 		},
2024-08-20T22:08:15.9898832Z 		{
2024-08-20T22:08:15.9899352Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9900012Z 			"size": 103,
2024-08-20T22:08:15.9900732Z 			"digest": "sha256:7c47122f4e09b47e891b4ae9e499a35a4eb46faacae87d0e946615adda6e0706"
2024-08-20T22:08:15.9901455Z 		},
2024-08-20T22:08:15.9901790Z 		{
2024-08-20T22:08:15.9902354Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9902979Z 			"size": 1473,
2024-08-20T22:08:15.9903611Z 			"digest": "sha256:1811ff4d3556ac255604f346d879528aff2aadd7652c2d39baf4fe6eaa60aec3"
2024-08-20T22:08:15.9904360Z 		},
2024-08-20T22:08:15.9904723Z 		{
2024-08-20T22:08:15.9905200Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9905881Z 			"size": 594729516,
2024-08-20T22:08:15.9906542Z 			"digest": "sha256:7754ee068b33f760d9c320c26a1a3935247ace0c8c463a976105c1586a43fb1c"
2024-08-20T22:08:15.9907191Z 		},
2024-08-20T22:08:15.9907679Z 		{
2024-08-20T22:08:15.9908199Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9908898Z 			"size": 164,
2024-08-20T22:08:15.9909560Z 			"digest": "sha256:0479fbac3361713c075bf788e778117917dcacdd1ff71ed79a789dd2016a1714"
2024-08-20T22:08:15.9910255Z 		},
2024-08-20T22:08:15.9910610Z 		{
2024-08-20T22:08:15.9911185Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9911906Z 			"size": 564,
2024-08-20T22:08:15.9912555Z 			"digest": "sha256:f1d6874dee9c34d154278d2b6935d04447f07fa89d25d4a09398234040d8a7e8"
2024-08-20T22:08:15.9913322Z 		},
2024-08-20T22:08:15.9913692Z 		{
2024-08-20T22:08:15.9914251Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9914976Z 			"size": 35874203,
2024-08-20T22:08:15.9915648Z 			"digest": "sha256:afc429e87b593a64f3e0a1be51e09e966ea1aa71e6e8a0bec4edc9f2f7bc1c05"
2024-08-20T22:08:15.9916319Z 		},
2024-08-20T22:08:15.9916707Z 		{
2024-08-20T22:08:15.9917249Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9917940Z 			"size": 104,
2024-08-20T22:08:15.9918522Z 			"digest": "sha256:019620471c2592cd22437fe8ed1a94c784114cf7445c5c2ab36dd0bae727fc66"
2024-08-20T22:08:15.9919344Z + exit 0
2024-08-20T22:08:15.9919709Z 		},
2024-08-20T22:08:15.9920068Z 		{
2024-08-20T22:08:15.9920670Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9921365Z 			"size": 425,
2024-08-20T22:08:15.9921997Z 			"digest": "sha256:27aeb1b2cd5e28d8f0e54fc1090baf88abc29ebf5fce088b493fcae99c9550d1"
2024-08-20T22:08:15.9922707Z 		},
2024-08-20T22:08:15.9923103Z 		{
2024-08-20T22:08:15.9923611Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9924267Z 			"size": 20262183,
2024-08-20T22:08:15.9925007Z 			"digest": "sha256:829c0e19f4b7797e9120a86172fabfd710f9b3971ffd714a616931a3fadc7192"
2024-08-20T22:08:15.9925731Z 		},
2024-08-20T22:08:15.9926038Z 		{
2024-08-20T22:08:15.9926639Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9927359Z 			"size": 438,
2024-08-20T22:08:15.9927983Z 			"digest": "sha256:560cd6122a3868e7e82e75eb103d182edf6512114d61ccdfae099af00cb16b4c"
2024-08-20T22:08:15.9928713Z 		},
2024-08-20T22:08:15.9929050Z 		{
2024-08-20T22:08:15.9929561Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9930232Z 			"size": 701,
2024-08-20T22:08:15.9930864Z 			"digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2"
2024-08-20T22:08:15.9931567Z 		},
2024-08-20T22:08:15.9931983Z 		{
2024-08-20T22:08:15.9932469Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9933128Z 			"size": 141,
2024-08-20T22:08:15.9933811Z 			"digest": "sha256:5a4eb19d72198a01f3babd4711c40fcf7c6e5cb9d5cfedc507b0061408d47097"
2024-08-20T22:08:15.9934582Z 		},
2024-08-20T22:08:15.9934910Z 		{
2024-08-20T22:08:15.9935489Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9936247Z 			"size": 135,
2024-08-20T22:08:15.9936933Z 			"digest": "sha256:da6a838a66c72b5de9a5590c233779880ea901c1264940a1c618825d2501ffd1"
2024-08-20T22:08:15.9937589Z 		},
2024-08-20T22:08:15.9937942Z 		{
2024-08-20T22:08:15.9938563Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9939165Z 			"size": 4838068375,
2024-08-20T22:08:15.9939834Z 			"digest": "sha256:cbb79a7053537bed2d80b77198ea552b985563b11d3f8616593c6ab575a3a090"
2024-08-20T22:08:15.9940579Z 		},
2024-08-20T22:08:15.9940989Z 		{
2024-08-20T22:08:15.9941486Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9942166Z 			"size": 194,
2024-08-20T22:08:15.9942808Z 			"digest": "sha256:4d4fa6802ab3e6b36d8d4adb650046e9370c4a16a19876970559ef8a312ca855"
2024-08-20T22:08:15.9943503Z 		},
2024-08-20T22:08:15.9943849Z 		{
2024-08-20T22:08:15.9944379Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9945015Z 			"size": 564,
2024-08-20T22:08:15.9945659Z 			"digest": "sha256:0f085d4fa279366fcd52d303b7ad8abddc5962a8872174b8109f3f5d5ca5f2cf"
2024-08-20T22:08:15.9946377Z 		},
2024-08-20T22:08:15.9946709Z 		{
2024-08-20T22:08:15.9947347Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9947968Z 			"size": 43162921,
2024-08-20T22:08:15.9948601Z 			"digest": "sha256:4a0412373305517489d891a4749e6d2b80b4aeb7df3b8adca0ed0abb2f3e5300"
2024-08-20T22:08:15.9949468Z 		},
2024-08-20T22:08:15.9949803Z 		{
2024-08-20T22:08:15.9950282Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9950994Z 			"size": 106,
2024-08-20T22:08:15.9951603Z 			"digest": "sha256:302c8c1779444e54919f83e91c6d5e1fdea653fc899be802b3a2a34d34d99d5d"
2024-08-20T22:08:15.9952309Z 		},
2024-08-20T22:08:15.9952678Z 		{
2024-08-20T22:08:15.9953185Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9953904Z 			"size": 1380,
2024-08-20T22:08:15.9954585Z 			"digest": "sha256:02970347dff622ee45629bc28788716d00528464f71f88ec7b74ae5f44a486e2"
2024-08-20T22:08:15.9955265Z 		},
2024-08-20T22:08:15.9955618Z 		{
2024-08-20T22:08:15.9956178Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9956770Z 			"size": 701,
2024-08-20T22:08:15.9957414Z 			"digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2"
2024-08-20T22:08:15.9958167Z 		},
2024-08-20T22:08:15.9958520Z 		{
2024-08-20T22:08:15.9959000Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9959684Z 			"size": 139,
2024-08-20T22:08:15.9960316Z 			"digest": "sha256:916c5e8580164926a3aec87e88a5d1c2d02756e49fdeed4198e1e6304efa2fd3"
2024-08-20T22:08:15.9961125Z 		},
2024-08-20T22:08:15.9961426Z 		{
2024-08-20T22:08:15.9961955Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9962638Z 			"size": 118,
2024-08-20T22:08:15.9963224Z 			"digest": "sha256:c42b3602a1fb4df5111224650f15adffb025979ae9c614807379ea84e189533b"
2024-08-20T22:08:15.9963937Z 		},
2024-08-20T22:08:15.9964315Z 		{
2024-08-20T22:08:15.9964849Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9965452Z 			"size": 4324672728,
2024-08-20T22:08:15.9966164Z 			"digest": "sha256:11fad972ef6cb9d2dc236504b0ee5b6e5ea996e4576ece0ed3262fbb9ee2de60"
2024-08-20T22:08:15.9966888Z 		},
2024-08-20T22:08:15.9967296Z 		{
2024-08-20T22:08:15.9967840Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9968498Z 			"size": 174,
2024-08-20T22:08:15.9969439Z 			"digest": "sha256:f957c16244819132b478272aaf24231699c4fcbd495c229a574f94daab8d6b86"
2024-08-20T22:08:15.9970183Z 		},
2024-08-20T22:08:15.9970549Z 		{
2024-08-20T22:08:15.9971071Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9971797Z 			"size": 907,
2024-08-20T22:08:15.9972396Z 			"digest": "sha256:768c31dca016b80bbdc973c8ee818034bdbd17b7ea9bdadbf0ec45ea51005720"
2024-08-20T22:08:15.9973258Z 		},
2024-08-20T22:08:15.9973682Z 		{
2024-08-20T22:08:15.9974196Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9974870Z 			"size": 701,
2024-08-20T22:08:15.9975564Z 			"digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2"
2024-08-20T22:08:15.9976259Z 		},
2024-08-20T22:08:15.9976556Z 		{
2024-08-20T22:08:15.9977150Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9977783Z 			"size": 134,
2024-08-20T22:08:15.9978492Z 			"digest": "sha256:8c32d0ecebbe2054c96a3a5c1f3d6c8b87c2e826686b369e5242cd55d0f01c2d"
2024-08-20T22:08:15.9979215Z 		},
2024-08-20T22:08:15.9979552Z 		{
2024-08-20T22:08:15.9980084Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9980770Z 			"size": 32,
2024-08-20T22:08:15.9981436Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-08-20T22:08:15.9982165Z 		},
2024-08-20T22:08:15.9982557Z 		{
2024-08-20T22:08:15.9983102Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9983703Z 			"size": 158,
2024-08-20T22:08:15.9984377Z 			"digest": "sha256:78ee77eddef23f20afc78e2924f4467041cadb754634ee3df431c68211cf676d"
2024-08-20T22:08:15.9985105Z 		},
2024-08-20T22:08:15.9985403Z 		{
2024-08-20T22:08:15.9985966Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9986753Z 			"size": 1841,
2024-08-20T22:08:15.9987435Z 			"digest": "sha256:37513e9b43d7123e7ddee09100f62a72b57af7b3833eaecd515e247e2f63e13f"
2024-08-20T22:08:15.9988122Z 		},
2024-08-20T22:08:15.9988532Z 		{
2024-08-20T22:08:15.9989113Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9989772Z 			"size": 7529776,
2024-08-20T22:08:15.9990380Z 			"digest": "sha256:a7a5392e2bf6b73d9be292cc22a99ad4637636c15733beb7afbe6403b5ba21a8"
2024-08-20T22:08:15.9991137Z 		},
2024-08-20T22:08:15.9991496Z 		{
2024-08-20T22:08:15.9991980Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9992674Z 			"size": 163,
2024-08-20T22:08:15.9993322Z 			"digest": "sha256:b10de02255c021b2366b85bf12ddf593aac89f88964201c455b71a145bc5ca2e"
2024-08-20T22:08:15.9994016Z 		},
2024-08-20T22:08:15.9994402Z 		{
2024-08-20T22:08:15.9994991Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9995629Z 			"size": 7943,
2024-08-20T22:08:15.9996341Z 			"digest": "sha256:e56eeff2db93ffe5002a2bb764795b7333321c6802cc28bef0801a619fe6a07e"
2024-08-20T22:08:15.9997005Z 		},
2024-08-20T22:08:15.9997342Z 		{
2024-08-20T22:08:15.9997928Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:15.9998564Z 			"size": 8068,
2024-08-20T22:08:15.9999160Z 			"digest": "sha256:70ac3e38d0fd1f0f46ad710c3fc485fd01406606054e8cf84a63af1570bc9f3a"
2024-08-20T22:08:15.9999934Z 		},
2024-08-20T22:08:16.0000284Z 		{
2024-08-20T22:08:16.0000797Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:16.0001489Z 			"size": 302,
2024-08-20T22:08:16.0002182Z 			"digest": "sha256:eb09cbf3e4168749cc8605e3188ed35070f6037d13930e443c38d8941734488f"
2024-08-20T22:08:16.0002902Z 		},
2024-08-20T22:08:16.0003256Z 		{
2024-08-20T22:08:16.0003783Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:16.0004445Z 			"size": 7629638,
2024-08-20T22:08:16.0005123Z 			"digest": "sha256:ee1273153675e6543b58f5be50321abee5e91f7351701d5d501a6837eaf676ef"
2024-08-20T22:08:16.0005776Z 		},
2024-08-20T22:08:16.0006131Z 		{
2024-08-20T22:08:16.0006707Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:16.0007324Z 			"size": 108,
2024-08-20T22:08:16.0007959Z 			"digest": "sha256:e22979f9e0eedfa16f98eb4fa26f30a50c237f5ffd2f6e5b898f2b908058496d"
2024-08-20T22:08:16.0008790Z 		},
2024-08-20T22:08:16.0009144Z 		{
2024-08-20T22:08:16.0009623Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:16.0010438Z 			"size": 54145661,
2024-08-20T22:08:16.0011098Z 			"digest": "sha256:6c1f0aabce7c84f9b0b9ab3b96325778dba1c1465b7463c5b4c6be1cf52b74db"
2024-08-20T22:08:16.0011863Z 		},
2024-08-20T22:08:16.0012188Z 		{
2024-08-20T22:08:16.0012706Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:16.0013391Z 			"size": 473,
2024-08-20T22:08:16.0014048Z 			"digest": "sha256:7ebe49c3a23e71b289ce012b48dad35791a2f2aa7fcee2326a64d763b0d14df8"
2024-08-20T22:08:16.0014774Z 		},
2024-08-20T22:08:16.0015233Z 		{
2024-08-20T22:08:16.0015771Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:16.0016373Z 			"size": 1374858766,
2024-08-20T22:08:16.0017112Z 			"digest": "sha256:5d115b29b5d53e5418762bc41ae975dd245bcffc3d058b8902c58146d505cb37"
2024-08-20T22:08:16.0017808Z 		},
2024-08-20T22:08:16.0018142Z 		{
2024-08-20T22:08:16.0018756Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:16.0019392Z 			"size": 106,
2024-08-20T22:08:16.0020039Z 			"digest": "sha256:7f2d7a04fbe65bbe1e895138e0cfeeef53ad926c890bab8e7e132ad1e655e48f"
2024-08-20T22:08:16.0020825Z 		},
2024-08-20T22:08:16.0021125Z 		{
2024-08-20T22:08:16.0021642Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:16.0022427Z 			"size": 559,
2024-08-20T22:08:16.0023008Z 			"digest": "sha256:d79fd7d526989e554455eda7d459f2c88444030f61710101e52ec8c7a49b7b79"
2024-08-20T22:08:16.0023828Z 		},
2024-08-20T22:08:16.0024231Z 		{
2024-08-20T22:08:16.0024745Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:16.0025365Z 			"size": 46248488,
2024-08-20T22:08:16.0026054Z 			"digest": "sha256:9752d6fbbcab4e7403f26777c5b65911320de6389c81b833a47ba45c4f3354b4"
2024-08-20T22:08:16.0026773Z 		},
2024-08-20T22:08:16.0027104Z 		{
2024-08-20T22:08:16.0027641Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:16.0028297Z 			"size": 111,
2024-08-20T22:08:16.0028998Z 			"digest": "sha256:83808b2eff4d71ed46db7c29f66fe918f328405c25b314225dfad01b4868d1f4"
2024-08-20T22:08:16.0029743Z 		},
2024-08-20T22:08:16.0030133Z 		{
2024-08-20T22:08:16.0030693Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:16.0031353Z 			"size": 32,
2024-08-20T22:08:16.0031945Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-08-20T22:08:16.0032702Z 		},
2024-08-20T22:08:16.0033064Z 		{
2024-08-20T22:08:16.0033579Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:16.0034244Z 			"size": 32,
2024-08-20T22:08:16.0034888Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-08-20T22:08:16.0035663Z 		},
2024-08-20T22:08:16.0036071Z 		{
2024-08-20T22:08:16.0036548Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:16.0037184Z 			"size": 32,
2024-08-20T22:08:16.0037884Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-08-20T22:08:16.0038583Z 		},
2024-08-20T22:08:16.0038880Z 		{
2024-08-20T22:08:16.0039471Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:08:16.0040113Z 			"size": 32,
2024-08-20T22:08:16.0040694Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-08-20T22:08:16.0041469Z 		}
2024-08-20T22:08:16.0229121Z 	]
2024-08-20T22:08:16.0229423Z }
2024-08-20T22:08:16.0339238Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/}
2024-08-20T22:08:16.0339723Z tag=${ECR_DOCKER_IMAGE##*/}
2024-08-20T22:08:16.0340267Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}"
2024-08-20T22:08:16.0350607Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:08:16.0351100Z env:
2024-08-20T22:08:16.0351376Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:08:16.0352526Z   ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:08:16.0353694Z ##[endgroup]
2024-08-20T22:08:16.0386148Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:08:16.0437161Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2024-08-20T22:08:16.0437734Z with:
2024-08-20T22:08:16.0438808Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:08:16.0440101Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-08-20T22:08:16.0440612Z env:
2024-08-20T22:08:16.0440884Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:08:16.0441227Z ##[endgroup]
2024-08-20T22:08:16.0458294Z ##[group]Run set -x
2024-08-20T22:08:16.0458644Z set -x
2024-08-20T22:08:16.0458941Z set +e
2024-08-20T22:08:16.0459229Z 
2024-08-20T22:08:16.0459502Z login() {
2024-08-20T22:08:16.0460154Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2024-08-20T22:08:16.0460860Z }
2024-08-20T22:08:16.0461139Z 
2024-08-20T22:08:16.0461440Z retry () {
2024-08-20T22:08:16.0461836Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2024-08-20T22:08:16.0462448Z }
2024-08-20T22:08:16.0462724Z 
2024-08-20T22:08:16.0463050Z retry login "${DOCKER_REGISTRY}"
2024-08-20T22:08:16.0463458Z 
2024-08-20T22:08:16.0463732Z set -e
2024-08-20T22:08:16.0464205Z # ignore output since only exit code is used for conditional
2024-08-20T22:08:16.0464954Z # only pull docker image if it's not available locally
2024-08-20T22:08:16.0465711Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2024-08-20T22:08:16.0466401Z   retry docker pull "${DOCKER_IMAGE}"
2024-08-20T22:08:16.0466840Z fi
2024-08-20T22:08:16.0475264Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:08:16.0475751Z env:
2024-08-20T22:08:16.0476026Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:08:16.0477165Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:08:16.0478480Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-08-20T22:08:16.0478988Z ##[endgroup]
2024-08-20T22:08:16.0507666Z + set +e
2024-08-20T22:08:16.0508326Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-08-20T22:08:16.0508955Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-08-20T22:08:16.0512011Z + aws ecr get-login-password --region us-east-1
2024-08-20T22:08:16.0513112Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-08-20T22:08:16.5810057Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-08-20T22:08:16.5811670Z Configure a credential helper to remove this warning. See
2024-08-20T22:08:16.5813645Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-08-20T22:08:16.5814607Z 
2024-08-20T22:08:16.5814739Z Login Succeeded
2024-08-20T22:08:16.5835876Z + set -e
2024-08-20T22:08:16.5837241Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:08:16.5996559Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:08:16.5998694Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:08:16.8661765Z f6d216893d65c7b8ae43df4daaf247db808378e9: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks
2024-08-20T22:08:16.8665558Z 7a2c55901189: Pulling fs layer
2024-08-20T22:08:16.8666127Z 224fe954d725: Pulling fs layer
2024-08-20T22:08:16.8666944Z 75722010b82e: Pulling fs layer
2024-08-20T22:08:16.8667445Z d527cbbb87e3: Pulling fs layer
2024-08-20T22:08:16.8667959Z b57676e46aee: Pulling fs layer
2024-08-20T22:08:16.8668490Z a8c1e85b5e14: Pulling fs layer
2024-08-20T22:08:16.8669227Z a41a8d1c11c8: Pulling fs layer
2024-08-20T22:08:16.8669717Z 0c1227890755: Pulling fs layer
2024-08-20T22:08:16.8670210Z d8d1234baab3: Pulling fs layer
2024-08-20T22:08:16.8670730Z 7ed32bc8e469: Pulling fs layer
2024-08-20T22:08:16.8671241Z ec1e7978c1fe: Pulling fs layer
2024-08-20T22:08:16.8671766Z ac23c5fcc1ef: Pulling fs layer
2024-08-20T22:08:16.8672291Z ca426ffc9d2b: Pulling fs layer
2024-08-20T22:08:16.8672685Z 4373aa743caf: Pulling fs layer
2024-08-20T22:08:16.8673294Z 75b2abb16c68: Pulling fs layer
2024-08-20T22:08:16.8673808Z ef1e3ef1e85f: Pulling fs layer
2024-08-20T22:08:16.8674320Z 0bf61114d37b: Pulling fs layer
2024-08-20T22:08:16.8674831Z 2df849f72ba4: Pulling fs layer
2024-08-20T22:08:16.8675364Z 34c47d63c5de: Pulling fs layer
2024-08-20T22:08:16.8675891Z 1fef4bd19792: Pulling fs layer
2024-08-20T22:08:16.8676441Z 6efb96fc87ea: Pulling fs layer
2024-08-20T22:08:16.8676947Z b119c4fc6c58: Pulling fs layer
2024-08-20T22:08:16.8677787Z b57676e46aee: Waiting
2024-08-20T22:08:16.8678225Z a8c1e85b5e14: Waiting
2024-08-20T22:08:16.8678657Z 33b639226eee: Pulling fs layer
2024-08-20T22:08:16.8679165Z d3547bd43146: Pulling fs layer
2024-08-20T22:08:16.8679627Z a41a8d1c11c8: Waiting
2024-08-20T22:08:16.8680060Z 5645520256fa: Pulling fs layer
2024-08-20T22:08:16.8680534Z 829ed7d91e38: Pulling fs layer
2024-08-20T22:08:16.8681126Z 4373aa743caf: Waiting
2024-08-20T22:08:16.8681661Z 385d0421d67c: Pulling fs layer
2024-08-20T22:08:16.8682118Z 7cbfce999d76: Pulling fs layer
2024-08-20T22:08:16.8682488Z 6658e558ebe4: Pulling fs layer
2024-08-20T22:08:16.8682840Z d527cbbb87e3: Waiting
2024-08-20T22:08:16.8683174Z 33444a5a7e01: Pulling fs layer
2024-08-20T22:08:16.8683530Z 7ed32bc8e469: Waiting
2024-08-20T22:08:16.8683905Z 0c1227890755: Waiting
2024-08-20T22:08:16.8684224Z ec1e7978c1fe: Waiting
2024-08-20T22:08:16.8684630Z ac23c5fcc1ef: Waiting
2024-08-20T22:08:16.8685020Z ca426ffc9d2b: Waiting
2024-08-20T22:08:16.8685339Z d8d1234baab3: Waiting
2024-08-20T22:08:16.8685652Z 6efb96fc87ea: Waiting
2024-08-20T22:08:16.8685972Z ef1e3ef1e85f: Waiting
2024-08-20T22:08:16.8686288Z b119c4fc6c58: Waiting
2024-08-20T22:08:16.8686595Z 2df849f72ba4: Waiting
2024-08-20T22:08:16.8686932Z 7c47122f4e09: Pulling fs layer
2024-08-20T22:08:16.8687305Z 1811ff4d3556: Pulling fs layer
2024-08-20T22:08:16.8687673Z 7754ee068b33: Pulling fs layer
2024-08-20T22:08:16.8688046Z 0479fbac3361: Pulling fs layer
2024-08-20T22:08:16.8688418Z f1d6874dee9c: Pulling fs layer
2024-08-20T22:08:16.8688793Z afc429e87b59: Pulling fs layer
2024-08-20T22:08:16.8689156Z 019620471c25: Pulling fs layer
2024-08-20T22:08:16.8689527Z 27aeb1b2cd5e: Pulling fs layer
2024-08-20T22:08:16.8689892Z 7754ee068b33: Waiting
2024-08-20T22:08:16.8690208Z f1d6874dee9c: Waiting
2024-08-20T22:08:16.8690538Z 829c0e19f4b7: Pulling fs layer
2024-08-20T22:08:16.8690918Z 560cd6122a38: Pulling fs layer
2024-08-20T22:08:16.8691286Z 5a4eb19d7219: Pulling fs layer
2024-08-20T22:08:16.8691654Z da6a838a66c7: Pulling fs layer
2024-08-20T22:08:16.8692026Z 6658e558ebe4: Waiting
2024-08-20T22:08:16.8692339Z 0479fbac3361: Waiting
2024-08-20T22:08:16.8692659Z 1811ff4d3556: Waiting
2024-08-20T22:08:16.8692964Z 33b639226eee: Waiting
2024-08-20T22:08:16.8693276Z 33444a5a7e01: Waiting
2024-08-20T22:08:16.8693594Z 27aeb1b2cd5e: Waiting
2024-08-20T22:08:16.8693918Z 385d0421d67c: Waiting
2024-08-20T22:08:16.8694243Z cbb79a705353: Pulling fs layer
2024-08-20T22:08:16.8694609Z 7c47122f4e09: Waiting
2024-08-20T22:08:16.8694921Z 560cd6122a38: Waiting
2024-08-20T22:08:16.8695245Z 4d4fa6802ab3: Pulling fs layer
2024-08-20T22:08:16.8695609Z 829ed7d91e38: Waiting
2024-08-20T22:08:16.8695939Z 0f085d4fa279: Pulling fs layer
2024-08-20T22:08:16.8696297Z 019620471c25: Waiting
2024-08-20T22:08:16.8696616Z 4a0412373305: Pulling fs layer
2024-08-20T22:08:16.8697170Z da6a838a66c7: Waiting
2024-08-20T22:08:16.8697508Z 302c8c177944: Pulling fs layer
2024-08-20T22:08:16.8697857Z 0f085d4fa279: Waiting
2024-08-20T22:08:16.8698269Z 02970347dff6: Pulling fs layer
2024-08-20T22:08:16.8698642Z 916c5e858016: Pulling fs layer
2024-08-20T22:08:16.8698997Z 5a4eb19d7219: Waiting
2024-08-20T22:08:16.8699295Z 4a0412373305: Waiting
2024-08-20T22:08:16.8699616Z 02970347dff6: Waiting
2024-08-20T22:08:16.8699939Z c42b3602a1fb: Pulling fs layer
2024-08-20T22:08:16.8700312Z 11fad972ef6c: Pulling fs layer
2024-08-20T22:08:16.8700661Z 302c8c177944: Waiting
2024-08-20T22:08:16.8700983Z f957c1624481: Pulling fs layer
2024-08-20T22:08:16.8701333Z c42b3602a1fb: Waiting
2024-08-20T22:08:16.8701653Z 768c31dca016: Pulling fs layer
2024-08-20T22:08:16.8702001Z 916c5e858016: Waiting
2024-08-20T22:08:16.8702311Z 768c31dca016: Waiting
2024-08-20T22:08:16.8702636Z 8c32d0ecebbe: Pulling fs layer
2024-08-20T22:08:16.8702993Z 11fad972ef6c: Waiting
2024-08-20T22:08:16.8703320Z 4f4fb700ef54: Pulling fs layer
2024-08-20T22:08:16.8703694Z 78ee77eddef2: Pulling fs layer
2024-08-20T22:08:16.8704047Z 5645520256fa: Waiting
2024-08-20T22:08:16.8704351Z afc429e87b59: Waiting
2024-08-20T22:08:16.8704748Z 829c0e19f4b7: Waiting
2024-08-20T22:08:16.8705066Z 37513e9b43d7: Pulling fs layer
2024-08-20T22:08:16.8705431Z a7a5392e2bf6: Pulling fs layer
2024-08-20T22:08:16.8705794Z b10de02255c0: Pulling fs layer
2024-08-20T22:08:16.8706165Z e56eeff2db93: Pulling fs layer
2024-08-20T22:08:16.8706519Z b10de02255c0: Waiting
2024-08-20T22:08:16.8706848Z 70ac3e38d0fd: Pulling fs layer
2024-08-20T22:08:16.8707206Z 4f4fb700ef54: Waiting
2024-08-20T22:08:16.8707533Z eb09cbf3e416: Pulling fs layer
2024-08-20T22:08:16.8707896Z e56eeff2db93: Waiting
2024-08-20T22:08:16.8708218Z ee1273153675: Pulling fs layer
2024-08-20T22:08:16.8708585Z e22979f9e0ee: Pulling fs layer
2024-08-20T22:08:16.8708955Z 6c1f0aabce7c: Pulling fs layer
2024-08-20T22:08:16.8709311Z eb09cbf3e416: Waiting
2024-08-20T22:08:16.8709616Z a7a5392e2bf6: Waiting
2024-08-20T22:08:16.8709930Z ee1273153675: Waiting
2024-08-20T22:08:16.8710246Z e22979f9e0ee: Waiting
2024-08-20T22:08:16.8710565Z 7ebe49c3a23e: Pulling fs layer
2024-08-20T22:08:16.8710943Z 5d115b29b5d5: Pulling fs layer
2024-08-20T22:08:16.8711293Z 37513e9b43d7: Waiting
2024-08-20T22:08:16.8711617Z 7f2d7a04fbe6: Pulling fs layer
2024-08-20T22:08:16.8712041Z 70ac3e38d0fd: Waiting
2024-08-20T22:08:16.8712372Z d79fd7d52698: Pulling fs layer
2024-08-20T22:08:16.8712784Z 9752d6fbbcab: Pulling fs layer
2024-08-20T22:08:16.8713152Z 83808b2eff4d: Pulling fs layer
2024-08-20T22:08:16.8713500Z 7f2d7a04fbe6: Waiting
2024-08-20T22:08:16.8713812Z 9752d6fbbcab: Waiting
2024-08-20T22:08:16.8714126Z d79fd7d52698: Waiting
2024-08-20T22:08:17.0083457Z 224fe954d725: Verifying Checksum
2024-08-20T22:08:17.0083967Z 224fe954d725: Download complete
2024-08-20T22:08:17.0687053Z d527cbbb87e3: Verifying Checksum
2024-08-20T22:08:17.0687498Z d527cbbb87e3: Download complete
2024-08-20T22:08:17.1509310Z b57676e46aee: Verifying Checksum
2024-08-20T22:08:17.1509887Z b57676e46aee: Download complete
2024-08-20T22:08:17.2182964Z 7a2c55901189: Verifying Checksum
2024-08-20T22:08:17.2183432Z 7a2c55901189: Download complete
2024-08-20T22:08:17.3137060Z a41a8d1c11c8: Download complete
2024-08-20T22:08:17.3720651Z 0c1227890755: Verifying Checksum
2024-08-20T22:08:17.4503847Z 0c1227890755: Download complete
2024-08-20T22:08:17.4504435Z d8d1234baab3: Verifying Checksum
2024-08-20T22:08:17.4504840Z d8d1234baab3: Download complete
2024-08-20T22:08:17.4719368Z 75722010b82e: Verifying Checksum
2024-08-20T22:08:17.4719904Z 75722010b82e: Download complete
2024-08-20T22:08:17.5569841Z ec1e7978c1fe: Download complete
2024-08-20T22:08:17.6425943Z ac23c5fcc1ef: Verifying Checksum
2024-08-20T22:08:17.6426534Z ac23c5fcc1ef: Download complete
2024-08-20T22:08:18.3796598Z 7a2c55901189: Pull complete
2024-08-20T22:08:18.6903622Z 224fe954d725: Pull complete
2024-08-20T22:08:19.5989808Z 75722010b82e: Pull complete
2024-08-20T22:08:19.6105552Z d527cbbb87e3: Pull complete
2024-08-20T22:08:19.6251519Z b57676e46aee: Pull complete
2024-08-20T22:08:20.1593713Z ca426ffc9d2b: Verifying Checksum
2024-08-20T22:08:20.1594721Z ca426ffc9d2b: Download complete
2024-08-20T22:08:20.2246098Z 4373aa743caf: Verifying Checksum
2024-08-20T22:08:20.2246540Z 4373aa743caf: Download complete
2024-08-20T22:08:20.2989868Z 75b2abb16c68: Download complete
2024-08-20T22:08:20.3749465Z ef1e3ef1e85f: Verifying Checksum
2024-08-20T22:08:20.3749979Z ef1e3ef1e85f: Download complete
2024-08-20T22:08:21.3448169Z 0bf61114d37b: Verifying Checksum
2024-08-20T22:08:21.3448633Z 0bf61114d37b: Download complete
2024-08-20T22:08:21.4167176Z 2df849f72ba4: Verifying Checksum
2024-08-20T22:08:21.4167620Z 2df849f72ba4: Download complete
2024-08-20T22:08:21.4834374Z 34c47d63c5de: Verifying Checksum
2024-08-20T22:08:21.4834997Z 34c47d63c5de: Download complete
2024-08-20T22:08:21.5575715Z 1fef4bd19792: Download complete
2024-08-20T22:08:30.4944585Z a8c1e85b5e14: Verifying Checksum
2024-08-20T22:08:30.4945899Z a8c1e85b5e14: Download complete
2024-08-20T22:08:30.5616924Z b119c4fc6c58: Verifying Checksum
2024-08-20T22:08:30.5617509Z b119c4fc6c58: Download complete
2024-08-20T22:08:30.6432321Z 33b639226eee: Verifying Checksum
2024-08-20T22:08:30.6433335Z 33b639226eee: Download complete
2024-08-20T22:08:30.7145673Z d3547bd43146: Download complete
2024-08-20T22:08:30.7861306Z 5645520256fa: Verifying Checksum
2024-08-20T22:08:30.7861902Z 5645520256fa: Download complete
2024-08-20T22:08:30.8525038Z 829ed7d91e38: Verifying Checksum
2024-08-20T22:08:30.8525506Z 829ed7d91e38: Download complete
2024-08-20T22:08:32.2281443Z 385d0421d67c: Download complete
2024-08-20T22:08:32.2942003Z 7cbfce999d76: Download complete
2024-08-20T22:08:32.3700544Z 6658e558ebe4: Verifying Checksum
2024-08-20T22:08:32.3701166Z 6658e558ebe4: Download complete
2024-08-20T22:08:32.4395391Z 33444a5a7e01: Download complete
2024-08-20T22:08:32.5320918Z 7c47122f4e09: Verifying Checksum
2024-08-20T22:08:32.5321514Z 7c47122f4e09: Download complete
2024-08-20T22:08:32.5982974Z 1811ff4d3556: Verifying Checksum
2024-08-20T22:08:32.5983598Z 1811ff4d3556: Download complete
2024-08-20T22:08:38.5968669Z 7754ee068b33: Verifying Checksum
2024-08-20T22:08:38.5969632Z 7754ee068b33: Download complete
2024-08-20T22:08:38.6673844Z 0479fbac3361: Verifying Checksum
2024-08-20T22:08:38.6674808Z 0479fbac3361: Download complete
2024-08-20T22:08:38.7392146Z f1d6874dee9c: Download complete
2024-08-20T22:08:39.1510417Z afc429e87b59: Verifying Checksum
2024-08-20T22:08:39.1511560Z afc429e87b59: Download complete
2024-08-20T22:08:39.2311301Z 019620471c25: Verifying Checksum
2024-08-20T22:08:39.2311983Z 019620471c25: Download complete
2024-08-20T22:08:39.3135903Z 27aeb1b2cd5e: Verifying Checksum
2024-08-20T22:08:39.3136376Z 27aeb1b2cd5e: Download complete
2024-08-20T22:08:39.5616435Z 829c0e19f4b7: Verifying Checksum
2024-08-20T22:08:39.5617108Z 829c0e19f4b7: Download complete
2024-08-20T22:08:39.6273446Z 560cd6122a38: Download complete
2024-08-20T22:08:39.6901293Z 5a4eb19d7219: Verifying Checksum
2024-08-20T22:08:39.6901877Z 5a4eb19d7219: Download complete
2024-08-20T22:08:39.7593356Z da6a838a66c7: Verifying Checksum
2024-08-20T22:08:39.7593941Z da6a838a66c7: Download complete
2024-08-20T22:08:42.7952075Z 7ed32bc8e469: Verifying Checksum
2024-08-20T22:08:42.7952548Z 7ed32bc8e469: Download complete
2024-08-20T22:08:42.8624626Z 4d4fa6802ab3: Download complete
2024-08-20T22:08:42.9451149Z 0f085d4fa279: Verifying Checksum
2024-08-20T22:08:42.9451666Z 0f085d4fa279: Download complete
2024-08-20T22:08:43.4480596Z 4a0412373305: Verifying Checksum
2024-08-20T22:08:43.4481052Z 4a0412373305: Download complete
2024-08-20T22:08:43.5471875Z a8c1e85b5e14: Pull complete
2024-08-20T22:08:43.5541198Z 302c8c177944: Download complete
2024-08-20T22:08:43.6275359Z 02970347dff6: Verifying Checksum
2024-08-20T22:08:43.6275910Z 02970347dff6: Download complete
2024-08-20T22:08:43.7030549Z 916c5e858016: Download complete
2024-08-20T22:08:43.7042964Z a41a8d1c11c8: Pull complete
2024-08-20T22:08:43.7874560Z c42b3602a1fb: Verifying Checksum
2024-08-20T22:08:43.7875447Z c42b3602a1fb: Download complete
2024-08-20T22:08:43.8670270Z 0c1227890755: Pull complete
2024-08-20T22:08:44.1291407Z d8d1234baab3: Pull complete
2024-08-20T22:08:50.2497043Z 6efb96fc87ea: Verifying Checksum
2024-08-20T22:08:50.2497558Z 6efb96fc87ea: Download complete
2024-08-20T22:08:50.3181070Z f957c1624481: Download complete
2024-08-20T22:08:50.3959696Z 768c31dca016: Download complete
2024-08-20T22:08:50.4765855Z 8c32d0ecebbe: Verifying Checksum
2024-08-20T22:08:50.4766330Z 8c32d0ecebbe: Download complete
2024-08-20T22:08:50.4854334Z 4f4fb700ef54: Verifying Checksum
2024-08-20T22:08:50.4854765Z 4f4fb700ef54: Download complete
2024-08-20T22:08:50.5615899Z 78ee77eddef2: Verifying Checksum
2024-08-20T22:08:50.5616499Z 78ee77eddef2: Download complete
2024-08-20T22:08:50.6380304Z 37513e9b43d7: Download complete
2024-08-20T22:08:50.7802917Z a7a5392e2bf6: Download complete
2024-08-20T22:08:50.8494827Z b10de02255c0: Verifying Checksum
2024-08-20T22:08:50.8495400Z b10de02255c0: Download complete
2024-08-20T22:08:50.9310949Z e56eeff2db93: Download complete
2024-08-20T22:08:51.0410773Z 70ac3e38d0fd: Verifying Checksum
2024-08-20T22:08:51.0411410Z 70ac3e38d0fd: Download complete
2024-08-20T22:08:51.1184970Z eb09cbf3e416: Download complete
2024-08-20T22:08:51.2424178Z ee1273153675: Verifying Checksum
2024-08-20T22:08:51.2425417Z ee1273153675: Download complete
2024-08-20T22:08:51.3212447Z e22979f9e0ee: Verifying Checksum
2024-08-20T22:08:51.3213035Z e22979f9e0ee: Download complete
2024-08-20T22:08:51.9194627Z 6c1f0aabce7c: Verifying Checksum
2024-08-20T22:08:51.9195281Z 6c1f0aabce7c: Download complete
2024-08-20T22:08:51.9839337Z 7ebe49c3a23e: Verifying Checksum
2024-08-20T22:08:51.9840002Z 7ebe49c3a23e: Download complete
2024-08-20T22:09:17.7587901Z 5d115b29b5d5: Verifying Checksum
2024-08-20T22:09:17.7588366Z 5d115b29b5d5: Download complete
2024-08-20T22:09:17.8339284Z 7f2d7a04fbe6: Verifying Checksum
2024-08-20T22:09:17.8339780Z 7f2d7a04fbe6: Download complete
2024-08-20T22:09:17.9167819Z d79fd7d52698: Download complete
2024-08-20T22:09:19.2284162Z 9752d6fbbcab: Verifying Checksum
2024-08-20T22:09:19.2284860Z 9752d6fbbcab: Download complete
2024-08-20T22:09:19.3007843Z 83808b2eff4d: Verifying Checksum
2024-08-20T22:09:19.3008275Z 83808b2eff4d: Download complete
2024-08-20T22:09:58.0189523Z 11fad972ef6c: Verifying Checksum
2024-08-20T22:09:58.0189985Z 11fad972ef6c: Download complete
2024-08-20T22:09:59.8220847Z cbb79a705353: Verifying Checksum
2024-08-20T22:09:59.8221298Z cbb79a705353: Download complete
2024-08-20T22:10:21.8626567Z 7ed32bc8e469: Pull complete
2024-08-20T22:10:22.0941437Z ec1e7978c1fe: Pull complete
2024-08-20T22:10:22.3266982Z ac23c5fcc1ef: Pull complete
2024-08-20T22:10:34.9841032Z ca426ffc9d2b: Pull complete
2024-08-20T22:10:35.0093618Z 4373aa743caf: Pull complete
2024-08-20T22:10:35.0352347Z 75b2abb16c68: Pull complete
2024-08-20T22:10:35.0573374Z ef1e3ef1e85f: Pull complete
2024-08-20T22:10:37.4954049Z 0bf61114d37b: Pull complete
2024-08-20T22:10:37.6553208Z 2df849f72ba4: Pull complete
2024-08-20T22:10:37.8064957Z 34c47d63c5de: Pull complete
2024-08-20T22:10:37.9342400Z 1fef4bd19792: Pull complete
2024-08-20T22:11:37.7712509Z 6efb96fc87ea: Pull complete
2024-08-20T22:11:37.8706045Z b119c4fc6c58: Pull complete
2024-08-20T22:11:37.9522951Z 33b639226eee: Pull complete
2024-08-20T22:11:38.0151633Z d3547bd43146: Pull complete
2024-08-20T22:11:38.1069447Z 5645520256fa: Pull complete
2024-08-20T22:11:38.2066581Z 829ed7d91e38: Pull complete
2024-08-20T22:11:42.0162924Z 385d0421d67c: Pull complete
2024-08-20T22:11:42.6975537Z 7cbfce999d76: Pull complete
2024-08-20T22:11:43.8834095Z 6658e558ebe4: Pull complete
2024-08-20T22:11:44.1075502Z 33444a5a7e01: Pull complete
2024-08-20T22:11:44.2330564Z 7c47122f4e09: Pull complete
2024-08-20T22:11:44.3483325Z 1811ff4d3556: Pull complete
2024-08-20T22:11:54.9626700Z 7754ee068b33: Pull complete
2024-08-20T22:11:55.1488931Z 0479fbac3361: Pull complete
2024-08-20T22:11:55.3193118Z f1d6874dee9c: Pull complete
2024-08-20T22:11:56.2361763Z afc429e87b59: Pull complete
2024-08-20T22:11:56.4574870Z 019620471c25: Pull complete
2024-08-20T22:11:56.6736537Z 27aeb1b2cd5e: Pull complete
2024-08-20T22:11:57.1812495Z 829c0e19f4b7: Pull complete
2024-08-20T22:11:57.4084923Z 560cd6122a38: Pull complete
2024-08-20T22:11:57.6560902Z 5a4eb19d7219: Pull complete
2024-08-20T22:11:57.7907747Z da6a838a66c7: Pull complete
2024-08-20T22:12:37.3437539Z cbb79a705353: Pull complete
2024-08-20T22:12:37.5712242Z 4d4fa6802ab3: Pull complete
2024-08-20T22:12:37.7490800Z 0f085d4fa279: Pull complete
2024-08-20T22:12:40.3625989Z 4a0412373305: Pull complete
2024-08-20T22:12:40.5950643Z 302c8c177944: Pull complete
2024-08-20T22:12:40.8283045Z 02970347dff6: Pull complete
2024-08-20T22:12:41.2669770Z 916c5e858016: Pull complete
2024-08-20T22:12:41.4921122Z c42b3602a1fb: Pull complete
2024-08-20T22:14:03.5354175Z 11fad972ef6c: Pull complete
2024-08-20T22:14:03.6849215Z f957c1624481: Pull complete
2024-08-20T22:14:03.8612104Z 768c31dca016: Pull complete
2024-08-20T22:14:04.2910524Z 8c32d0ecebbe: Pull complete
2024-08-20T22:14:04.5103718Z 4f4fb700ef54: Pull complete
2024-08-20T22:14:04.7422195Z 78ee77eddef2: Pull complete
2024-08-20T22:14:04.9791152Z 37513e9b43d7: Pull complete
2024-08-20T22:14:05.3517770Z a7a5392e2bf6: Pull complete
2024-08-20T22:14:05.5851030Z b10de02255c0: Pull complete
2024-08-20T22:14:05.8209276Z e56eeff2db93: Pull complete
2024-08-20T22:14:06.0478203Z 70ac3e38d0fd: Pull complete
2024-08-20T22:14:06.2706788Z eb09cbf3e416: Pull complete
2024-08-20T22:14:07.6842905Z ee1273153675: Pull complete
2024-08-20T22:14:07.9195318Z e22979f9e0ee: Pull complete
2024-08-20T22:14:10.0460076Z 6c1f0aabce7c: Pull complete
2024-08-20T22:14:10.2773995Z 7ebe49c3a23e: Pull complete
2024-08-20T22:14:23.5648631Z 5d115b29b5d5: Pull complete
2024-08-20T22:14:23.7636925Z 7f2d7a04fbe6: Pull complete
2024-08-20T22:14:23.9703927Z d79fd7d52698: Pull complete
2024-08-20T22:14:24.8277320Z 9752d6fbbcab: Pull complete
2024-08-20T22:14:25.0823776Z 83808b2eff4d: Pull complete
2024-08-20T22:14:25.9531992Z Digest: sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402
2024-08-20T22:14:25.9900283Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:14:26.0088995Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:14:26.0176736Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT"
2024-08-20T22:14:26.0177656Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT"
2024-08-20T22:14:26.0189161Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:14:26.0189663Z env:
2024-08-20T22:14:26.0189938Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:14:26.0190291Z ##[endgroup]
2024-08-20T22:14:26.0377711Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main
2024-08-20T22:14:26.0378252Z with:
2024-08-20T22:14:26.0378542Z   driver-version: 550.54.15
2024-08-20T22:14:26.0378953Z env:
2024-08-20T22:14:26.0379242Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:14:26.0379585Z ##[endgroup]
2024-08-20T22:14:26.0517625Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2024-08-20T22:14:26.0518182Z with:
2024-08-20T22:14:26.0518460Z   timeout_minutes: 10
2024-08-20T22:14:26.0518784Z   max_attempts: 3
2024-08-20T22:14:26.0553620Z   command: # Is it disgusting to have a full shell script here in this github action? Sure
# But is it the best way to make it so that this action relies on nothing else? Absolutely
set -eou pipefail

DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID)
DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run"

install_nvidia_docker2_amzn2() {
    (
        set -x
        # Needed for yum-config-manager
        sudo yum install -y yum-utils
        if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then
          YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo"
        else
          # Amazon Linux 2
          YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo"
        fi

        sudo yum-config-manager --add-repo "${YUM_REPO_URL}"
        sudo yum install -y nvidia-docker2
        sudo systemctl restart docker
    )
}

install_nvidia_docker2_ubuntu20() {
    (
        set -x
        # Install nvidia-driver package if not installed
        status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)"
        if [ ! $? = 0 ] || [ ! "$status" = installed ]; then
          sudo apt-get install -y nvidia-docker2
          sudo systemctl restart docker
        fi
    )
}

pre_install_nvidia_driver_amzn2() {
    (
        # Purge any nvidia driver installed from RHEL repo
        sudo yum remove -y nvidia-driver-latest-dkms
    )
}

install_nvidia_driver_common() {
    (
        # Try to gather more information about the runner and its existing NVIDIA driver if any
        echo "Before installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        HAS_NVIDIA_DRIVER=0
        # Check if NVIDIA driver has already been installed
        if [ -x "$(command -v nvidia-smi)" ]; then
            set +e
            # The driver exists, check its version next. Also check only the first GPU if there are more than one of them
            # so that the same driver version is not print over multiple lines
            INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
            NVIDIA_SMI_STATUS=$?

            if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing"
            elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing"
            else
                HAS_NVIDIA_DRIVER=1
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation"
            fi
            set -e
        fi

        if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then
            # CAUTION: this may need to be updated in future
            if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then
                  sudo yum groupinstall -y "Development Tools"
                  # ensure our kernel install is the same as our underlying kernel,
                  # groupinstall "Development Tools" has a habit of mismatching kernel headers
                  sudo yum install -y "kernel-devel-uname-r == $(uname -r)"
                  sudo modprobe backlight
            fi
            sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN"

            set +e
            sudo /bin/bash /tmp/nvidia_driver -s --no-drm
            NVIDIA_INSTALLATION_STATUS=$?

            RESET_GPU=0
            if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then
                sudo cat /var/log/nvidia-installer.log
                # Fail to install NVIDIA driver, try to reset the GPU
                RESET_GPU=1
            elif [ -x "$(command -v nvidia-smi)" ]; then
                # Check again if nvidia-smi works even if the driver installation completes successfully
                INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
                NVIDIA_SMI_STATUS=$?

                if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                    RESET_GPU=1
                fi
            fi

            if [ "$RESET_GPU" -eq 1 ]; then
                NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1)
                # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this
                # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388
                for PCI_ID in $NVIDIA_DEVICES; do
                    DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable)

                    echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)"
                    # This requires sudo permission of course
                    echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset
                    sleep 1
                done
            fi

            sudo rm -fv /tmp/nvidia_driver
            set -e
        fi
    )
}

post_install_nvidia_driver_common() {
    (
        sudo modprobe nvidia || true
        echo "After installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        (
            set +e

            nvidia-smi
            # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in
            # the case where the driver has already crashed as it still can get the driver version
            # and some basic information like the bus ID.  However, the rest of the information
            # would be missing (ERR!), for example:
            #
            # +-----------------------------------------------------------------------------+
            # | NVIDIA-SMI 525.89.02    Driver Version: 525.89.02    CUDA Version: 12.0     |
            # |-------------------------------+----------------------+----------------------+
            # | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
            # | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
            # |                               |                      |               MIG M. |
            # |===============================+======================+======================|
            # |   0  ERR!                Off  | 00000000:00:1E.0 Off |                 ERR! |
            # |ERR!  ERR! ERR!    ERR! / ERR! |   4184MiB / 23028MiB |    ERR!      Default |
            # |                               |                      |                 ERR! |
            # +-------------------------------+----------------------+----------------------+
            #
            # +-----------------------------------------------------------------------------+
            # | Processes:                                                                  |
            # |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
            # |        ID   ID                                                   Usage      |
            # |=============================================================================|
            # +-----------------------------------------------------------------------------+
            #
            # This should be reported as a failure instead as it will guarantee to fail when
            # Docker tries to run with --gpus all
            #
            # So, the correct check here is to query one of the missing piece of info like
            # GPU name, so that the command can fail accordingly
            nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0
            NVIDIA_SMI_STATUS=$?

            # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285
            if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then
                echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}"
            else
                echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}"
                exit ${NVIDIA_SMI_STATUS}
            fi
            set -e
        )
    )
}

install_nvidia_driver_amzn2() {
    (
        set -x
        pre_install_nvidia_driver_amzn2
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

install_nvidia_driver_ubuntu20() {
    (
        set -x
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

echo "== Installing nvidia driver ${DRIVER_FN} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_driver_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_driver_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac

# Install container toolkit based on distribution
echo "== Installing nvidia container toolkit for ${DISTRIBUTION} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_docker2_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_docker2_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac
echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}"

# Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with
# more than one GPUs. This just needs to be run once. The command fails
# on subsequent runs and complains that the mode is already on, but that's
# ok
sudo nvidia-persistenced || true
# This should show persistence mode ON
nvidia-smi

2024-08-20T22:14:26.0588541Z   retry_wait_seconds: 10
2024-08-20T22:14:26.0588908Z   polling_interval_seconds: 1
2024-08-20T22:14:26.0589271Z   warning_on_retry: true
2024-08-20T22:14:26.0589943Z   continue_on_error: false
2024-08-20T22:14:26.0590278Z env:
2024-08-20T22:14:26.0590704Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:14:26.0591045Z   DRIVER_VERSION: 550.54.15
2024-08-20T22:14:26.0591388Z ##[endgroup]
2024-08-20T22:14:26.1414291Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run ==
2024-08-20T22:14:26.1415035Z + pre_install_nvidia_driver_amzn2
2024-08-20T22:14:26.1417593Z + sudo yum remove -y nvidia-driver-latest-dkms
2024-08-20T22:14:26.6255434Z No match for argument: nvidia-driver-latest-dkms
2024-08-20T22:14:26.6256429Z No packages marked for removal.
2024-08-20T22:14:26.6317950Z Dependencies resolved.
2024-08-20T22:14:26.6327799Z Nothing to do.
2024-08-20T22:14:26.6328232Z Complete!
2024-08-20T22:14:26.6967929Z + install_nvidia_driver_common
2024-08-20T22:14:26.6968707Z + echo 'Before installing NVIDIA driver'
2024-08-20T22:14:26.6969458Z + lspci
2024-08-20T22:14:26.6969845Z Before installing NVIDIA driver
2024-08-20T22:14:26.7145294Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma]
2024-08-20T22:14:26.7146052Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2024-08-20T22:14:26.7146977Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
2024-08-20T22:14:26.7147762Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111
2024-08-20T22:14:26.7148576Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller
2024-08-20T22:14:26.7149387Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2024-08-20T22:14:26.7150108Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1)
2024-08-20T22:14:26.7151400Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller
2024-08-20T22:14:26.7151985Z + lsmod
2024-08-20T22:14:26.7191796Z Module                  Size  Used by
2024-08-20T22:14:26.7192221Z xt_conntrack           16384  1
2024-08-20T22:14:26.7192599Z nft_chain_nat          16384  3
2024-08-20T22:14:26.7192973Z xt_MASQUERADE          20480  1
2024-08-20T22:14:26.7193425Z nf_nat                 57344  2 nft_chain_nat,xt_MASQUERADE
2024-08-20T22:14:26.7193906Z nf_conntrack_netlink    57344  0
2024-08-20T22:14:26.7194482Z nf_conntrack          184320  4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE
2024-08-20T22:14:26.7195113Z nf_defrag_ipv6         24576  1 nf_conntrack
2024-08-20T22:14:26.7195572Z nf_defrag_ipv4         16384  1 nf_conntrack
2024-08-20T22:14:26.7199553Z xfrm_user              57344  1
2024-08-20T22:14:26.7199978Z xfrm_algo              16384  1 xfrm_user
2024-08-20T22:14:26.7200507Z xt_addrtype            16384  2
2024-08-20T22:14:26.7200980Z nft_compat             20480  4
2024-08-20T22:14:26.7201491Z nf_tables             307200  57 nft_compat,nft_chain_nat
2024-08-20T22:14:26.7202117Z nfnetlink              20480  4 nft_compat,nf_conntrack_netlink,nf_tables
2024-08-20T22:14:26.7202657Z br_netfilter           36864  0
2024-08-20T22:14:26.7203058Z bridge                307200  1 br_netfilter
2024-08-20T22:14:26.7203500Z stp                    16384  1 bridge
2024-08-20T22:14:26.7203931Z llc                    16384  2 bridge,stp
2024-08-20T22:14:26.7204337Z overlay               167936  0
2024-08-20T22:14:26.7204697Z tls                   114688  0
2024-08-20T22:14:26.7205059Z nls_ascii              16384  1
2024-08-20T22:14:26.7205438Z nls_cp437              20480  1
2024-08-20T22:14:26.7205790Z vfat                   24576  1
2024-08-20T22:14:26.7206157Z fat                    86016  1 vfat
2024-08-20T22:14:26.7206543Z sunrpc                692224  1
2024-08-20T22:14:26.7206902Z ena                   167936  0
2024-08-20T22:14:26.7207263Z ghash_clmulni_intel    16384  0
2024-08-20T22:14:26.7207642Z ptp                    36864  1 ena
2024-08-20T22:14:26.7208031Z pps_core               24576  1 ptp
2024-08-20T22:14:26.7208423Z aesni_intel           393216  0
2024-08-20T22:14:26.7208778Z i8042                  45056  0
2024-08-20T22:14:26.7209142Z serio                  28672  3 i8042
2024-08-20T22:14:26.7209559Z crypto_simd            16384  1 aesni_intel
2024-08-20T22:14:26.7210285Z cryptd                 28672  2 crypto_simd,ghash_clmulni_intel
2024-08-20T22:14:26.7210773Z button                 24576  0
2024-08-20T22:14:26.7211138Z sch_fq_codel           20480  17
2024-08-20T22:14:26.7211504Z dm_mod                188416  0
2024-08-20T22:14:26.7211854Z fuse                  163840  1
2024-08-20T22:14:26.7212225Z dax                    45056  1 dm_mod
2024-08-20T22:14:26.7212614Z loop                   36864  0
2024-08-20T22:14:26.7212976Z configfs               57344  1
2024-08-20T22:14:26.7213334Z dmi_sysfs              20480  0
2024-08-20T22:14:26.7213772Z crc32_pclmul           16384  0
2024-08-20T22:14:26.7214256Z crc32c_intel           24576  0
2024-08-20T22:14:26.7214737Z efivarfs               24576  1
2024-08-20T22:14:26.7215210Z + modinfo nvidia
2024-08-20T22:14:26.7216147Z filename:       /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko
2024-08-20T22:14:26.7217144Z alias:          char-major-195-*
2024-08-20T22:14:26.7217647Z version:        550.54.15
2024-08-20T22:14:26.7218112Z supported:      external
2024-08-20T22:14:26.7218456Z license:        NVIDIA
2024-08-20T22:14:26.7218952Z firmware:       nvidia/550.54.15/gsp_tu10x.bin
2024-08-20T22:14:26.7219427Z firmware:       nvidia/550.54.15/gsp_ga10x.bin
2024-08-20T22:14:26.7219886Z srcversion:     833721318DA517F0C2FEC97
2024-08-20T22:14:26.7220346Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2024-08-20T22:14:26.7220840Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2024-08-20T22:14:26.7221439Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2024-08-20T22:14:26.7221966Z depends:        i2c-core,drm
2024-08-20T22:14:26.7222322Z retpoline:      Y
2024-08-20T22:14:26.7222656Z name:           nvidia
2024-08-20T22:14:26.7223403Z vermagic:       6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 
2024-08-20T22:14:26.7224114Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2024-08-20T22:14:26.7224967Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2024-08-20T22:14:26.7225581Z parm:           NVreg_ResmanDebugLevel:int
2024-08-20T22:14:26.7226019Z parm:           NVreg_RmLogonRC:int
2024-08-20T22:14:26.7226446Z parm:           NVreg_ModifyDeviceFiles:int
2024-08-20T22:14:26.7226902Z parm:           NVreg_DeviceFileUID:int
2024-08-20T22:14:26.7227333Z parm:           NVreg_DeviceFileGID:int
2024-08-20T22:14:26.7227761Z parm:           NVreg_DeviceFileMode:int
2024-08-20T22:14:26.7228377Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2024-08-20T22:14:26.7228944Z parm:           NVreg_UsePageAttributeTable:int
2024-08-20T22:14:26.7229420Z parm:           NVreg_EnablePCIeGen3:int
2024-08-20T22:14:26.7229842Z parm:           NVreg_EnableMSI:int
2024-08-20T22:14:26.7230262Z parm:           NVreg_TCEBypassMode:int
2024-08-20T22:14:26.7230718Z parm:           NVreg_EnableStreamMemOPs:int
2024-08-20T22:14:26.7231242Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2024-08-20T22:14:26.7231811Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2024-08-20T22:14:26.7232368Z parm:           NVreg_EnableS0ixPowerManagement:int
2024-08-20T22:14:26.7232973Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2024-08-20T22:14:26.7233565Z parm:           NVreg_DynamicPowerManagement:int
2024-08-20T22:14:26.7234167Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2024-08-20T22:14:26.7234752Z parm:           NVreg_EnableGpuFirmware:int
2024-08-20T22:14:26.7235240Z parm:           NVreg_EnableGpuFirmwareLogs:int
2024-08-20T22:14:26.7235765Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2024-08-20T22:14:26.7236299Z parm:           NVreg_EnableUserNUMAManagement:int
2024-08-20T22:14:26.7236789Z parm:           NVreg_MemoryPoolSize:int
2024-08-20T22:14:26.7237252Z parm:           NVreg_KMallocHeapMaxSize:int
2024-08-20T22:14:26.7237736Z parm:           NVreg_VMallocHeapMaxSize:int
2024-08-20T22:14:26.7238198Z parm:           NVreg_IgnoreMMIOCheck:int
2024-08-20T22:14:26.7238641Z parm:           NVreg_NvLinkDisable:int
2024-08-20T22:14:26.7239249Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2024-08-20T22:14:26.7239769Z parm:           NVreg_RegisterPCIDriver:int
2024-08-20T22:14:26.7240244Z parm:           NVreg_EnableResizableBar:int
2024-08-20T22:14:26.7240730Z parm:           NVreg_EnableDbgBreakpoint:int
2024-08-20T22:14:26.7241226Z parm:           NVreg_EnableNonblockingOpen:int
2024-08-20T22:14:26.7241701Z parm:           NVreg_RegistryDwords:charp
2024-08-20T22:14:26.7242207Z parm:           NVreg_RegistryDwordsPerDevice:charp
2024-08-20T22:14:26.7242684Z parm:           NVreg_RmMsg:charp
2024-08-20T22:14:26.7243095Z parm:           NVreg_GpuBlacklist:charp
2024-08-20T22:14:26.7243556Z parm:           NVreg_TemporaryFilePath:charp
2024-08-20T22:14:26.7244027Z parm:           NVreg_ExcludedGpus:charp
2024-08-20T22:14:26.7244479Z parm:           NVreg_DmaRemapPeerMmio:int
2024-08-20T22:14:26.7244955Z parm:           NVreg_RmNvlinkBandwidth:charp
2024-08-20T22:14:26.7245421Z parm:           NVreg_ImexChannelCount:int
2024-08-20T22:14:26.7245880Z parm:           rm_firmware_active:charp
2024-08-20T22:14:26.7246292Z + HAS_NVIDIA_DRIVER=0
2024-08-20T22:14:26.7246695Z ++ command -v nvidia-smi
2024-08-20T22:14:26.7247140Z + '[' -x /usr/bin/nvidia-smi ']'
2024-08-20T22:14:26.7247504Z + set +e
2024-08-20T22:14:26.7248013Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0
2024-08-20T22:14:28.9700699Z + INSTALLED_DRIVER_VERSION=550.54.15
2024-08-20T22:14:28.9701345Z + NVIDIA_SMI_STATUS=0
2024-08-20T22:14:28.9701920Z + '[' 0 -ne 0 ']'
2024-08-20T22:14:28.9702289Z + '[' 550.54.15 '!=' 550.54.15 ']'
2024-08-20T22:14:28.9702664Z + HAS_NVIDIA_DRIVER=1
2024-08-20T22:14:28.9703730Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation'
2024-08-20T22:14:28.9704451Z + set -e
2024-08-20T22:14:28.9704765Z + '[' 1 -eq 0 ']'
2024-08-20T22:14:28.9705336Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation
2024-08-20T22:14:28.9706020Z + post_install_nvidia_driver_common
2024-08-20T22:14:28.9707308Z + sudo modprobe nvidia
2024-08-20T22:14:29.1071387Z + echo 'After installing NVIDIA driver'
2024-08-20T22:14:29.1071851Z + lspci
2024-08-20T22:14:29.1072144Z After installing NVIDIA driver
2024-08-20T22:14:29.1177727Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma]
2024-08-20T22:14:29.1178514Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2024-08-20T22:14:29.1179706Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
2024-08-20T22:14:29.1180556Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111
2024-08-20T22:14:29.1181387Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller
2024-08-20T22:14:29.1182167Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2024-08-20T22:14:29.1182899Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1)
2024-08-20T22:14:29.1183706Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller
2024-08-20T22:14:29.1184305Z + lsmod
2024-08-20T22:14:29.1209266Z Module                  Size  Used by
2024-08-20T22:14:29.1209826Z nvidia_uvm           4706304  0
2024-08-20T22:14:29.1210349Z nvidia              54071296  1 nvidia_uvm
2024-08-20T22:14:29.1210849Z drm                   602112  1 nvidia
2024-08-20T22:14:29.1211293Z drm_panel_orientation_quirks    28672  1 drm
2024-08-20T22:14:29.1211808Z backlight              24576  1 drm
2024-08-20T22:14:29.1212229Z i2c_core              106496  2 nvidia,drm
2024-08-20T22:14:29.1212653Z xt_conntrack           16384  1
2024-08-20T22:14:29.1213026Z nft_chain_nat          16384  3
2024-08-20T22:14:29.1213407Z xt_MASQUERADE          20480  1
2024-08-20T22:14:29.1213888Z nf_nat                 57344  2 nft_chain_nat,xt_MASQUERADE
2024-08-20T22:14:29.1214377Z nf_conntrack_netlink    57344  0
2024-08-20T22:14:29.1214955Z nf_conntrack          184320  4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE
2024-08-20T22:14:29.1215839Z nf_defrag_ipv6         24576  1 nf_conntrack
2024-08-20T22:14:29.1216299Z nf_defrag_ipv4         16384  1 nf_conntrack
2024-08-20T22:14:29.1216730Z xfrm_user              57344  1
2024-08-20T22:14:29.1217118Z xfrm_algo              16384  1 xfrm_user
2024-08-20T22:14:29.1217543Z xt_addrtype            16384  2
2024-08-20T22:14:29.1217972Z nft_compat             20480  4
2024-08-20T22:14:29.1218428Z nf_tables             307200  57 nft_compat,nft_chain_nat
2024-08-20T22:14:29.1219156Z nfnetlink              20480  4 nft_compat,nf_conntrack_netlink,nf_tables
2024-08-20T22:14:29.1219689Z br_netfilter           36864  0
2024-08-20T22:14:29.1220093Z bridge                307200  1 br_netfilter
2024-08-20T22:14:29.1220538Z stp                    16384  1 bridge
2024-08-20T22:14:29.1220953Z llc                    16384  2 bridge,stp
2024-08-20T22:14:29.1221370Z overlay               167936  0
2024-08-20T22:14:29.1221736Z tls                   114688  0
2024-08-20T22:14:29.1222101Z nls_ascii              16384  1
2024-08-20T22:14:29.1222465Z nls_cp437              20480  1
2024-08-20T22:14:29.1222824Z vfat                   24576  1
2024-08-20T22:14:29.1223186Z fat                    86016  1 vfat
2024-08-20T22:14:29.1223566Z sunrpc                692224  1
2024-08-20T22:14:29.1223932Z ena                   167936  0
2024-08-20T22:14:29.1224296Z ghash_clmulni_intel    16384  0
2024-08-20T22:14:29.1224678Z ptp                    36864  1 ena
2024-08-20T22:14:29.1225061Z pps_core               24576  1 ptp
2024-08-20T22:14:29.1225451Z aesni_intel           393216  0
2024-08-20T22:14:29.1225848Z i8042                  45056  0
2024-08-20T22:14:29.1226339Z serio                  28672  3 i8042
2024-08-20T22:14:29.1227128Z crypto_simd            16384  1 aesni_intel
2024-08-20T22:14:29.1227881Z cryptd                 28672  2 crypto_simd,ghash_clmulni_intel
2024-08-20T22:14:29.1228533Z button                 24576  0
2024-08-20T22:14:29.1229011Z sch_fq_codel           20480  17
2024-08-20T22:14:29.1229502Z dm_mod                188416  0
2024-08-20T22:14:29.1229985Z fuse                  163840  1
2024-08-20T22:14:29.1230410Z dax                    45056  1 dm_mod
2024-08-20T22:14:29.1230857Z loop                   36864  0
2024-08-20T22:14:29.1231336Z configfs               57344  1
2024-08-20T22:14:29.1231802Z dmi_sysfs              20480  0
2024-08-20T22:14:29.1232233Z crc32_pclmul           16384  0
2024-08-20T22:14:29.1232591Z crc32c_intel           24576  0
2024-08-20T22:14:29.1232957Z efivarfs               24576  1
2024-08-20T22:14:29.1233309Z + modinfo nvidia
2024-08-20T22:14:29.1234001Z filename:       /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko
2024-08-20T22:14:29.1234698Z alias:          char-major-195-*
2024-08-20T22:14:29.1235075Z version:        550.54.15
2024-08-20T22:14:29.1235421Z supported:      external
2024-08-20T22:14:29.1235758Z license:        NVIDIA
2024-08-20T22:14:29.1236132Z firmware:       nvidia/550.54.15/gsp_tu10x.bin
2024-08-20T22:14:29.1236612Z firmware:       nvidia/550.54.15/gsp_ga10x.bin
2024-08-20T22:14:29.1237066Z srcversion:     833721318DA517F0C2FEC97
2024-08-20T22:14:29.1237515Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2024-08-20T22:14:29.1238006Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2024-08-20T22:14:29.1238494Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2024-08-20T22:14:29.1238981Z depends:        i2c-core,drm
2024-08-20T22:14:29.1239334Z retpoline:      Y
2024-08-20T22:14:29.1239640Z name:           nvidia
2024-08-20T22:14:29.1240249Z vermagic:       6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 
2024-08-20T22:14:29.1240942Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2024-08-20T22:14:29.1241579Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2024-08-20T22:14:29.1242165Z parm:           NVreg_ResmanDebugLevel:int
2024-08-20T22:14:29.1242609Z parm:           NVreg_RmLogonRC:int
2024-08-20T22:14:29.1243039Z parm:           NVreg_ModifyDeviceFiles:int
2024-08-20T22:14:29.1243653Z parm:           NVreg_DeviceFileUID:int
2024-08-20T22:14:29.1244091Z parm:           NVreg_DeviceFileGID:int
2024-08-20T22:14:29.1244529Z parm:           NVreg_DeviceFileMode:int
2024-08-20T22:14:29.1245051Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2024-08-20T22:14:29.1245601Z parm:           NVreg_UsePageAttributeTable:int
2024-08-20T22:14:29.1246078Z parm:           NVreg_EnablePCIeGen3:int
2024-08-20T22:14:29.1246509Z parm:           NVreg_EnableMSI:int
2024-08-20T22:14:29.1246916Z parm:           NVreg_TCEBypassMode:int
2024-08-20T22:14:29.1247372Z parm:           NVreg_EnableStreamMemOPs:int
2024-08-20T22:14:29.1247900Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2024-08-20T22:14:29.1248484Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2024-08-20T22:14:29.1249031Z parm:           NVreg_EnableS0ixPowerManagement:int
2024-08-20T22:14:29.1249633Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2024-08-20T22:14:29.1250229Z parm:           NVreg_DynamicPowerManagement:int
2024-08-20T22:14:29.1250842Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2024-08-20T22:14:29.1251422Z parm:           NVreg_EnableGpuFirmware:int
2024-08-20T22:14:29.1251910Z parm:           NVreg_EnableGpuFirmwareLogs:int
2024-08-20T22:14:29.1252438Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2024-08-20T22:14:29.1252978Z parm:           NVreg_EnableUserNUMAManagement:int
2024-08-20T22:14:29.1253471Z parm:           NVreg_MemoryPoolSize:int
2024-08-20T22:14:29.1253931Z parm:           NVreg_KMallocHeapMaxSize:int
2024-08-20T22:14:29.1254409Z parm:           NVreg_VMallocHeapMaxSize:int
2024-08-20T22:14:29.1254880Z parm:           NVreg_IgnoreMMIOCheck:int
2024-08-20T22:14:29.1255417Z parm:           NVreg_NvLinkDisable:int
2024-08-20T22:14:29.1255927Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2024-08-20T22:14:29.1256446Z parm:           NVreg_RegisterPCIDriver:int
2024-08-20T22:14:29.1256925Z parm:           NVreg_EnableResizableBar:int
2024-08-20T22:14:29.1257400Z parm:           NVreg_EnableDbgBreakpoint:int
2024-08-20T22:14:29.1257892Z parm:           NVreg_EnableNonblockingOpen:int
2024-08-20T22:14:29.1258374Z parm:           NVreg_RegistryDwords:charp
2024-08-20T22:14:29.1258960Z parm:           NVreg_RegistryDwordsPerDevice:charp
2024-08-20T22:14:29.1259434Z parm:           NVreg_RmMsg:charp
2024-08-20T22:14:29.1259842Z parm:           NVreg_GpuBlacklist:charp
2024-08-20T22:14:29.1260303Z parm:           NVreg_TemporaryFilePath:charp
2024-08-20T22:14:29.1260762Z parm:           NVreg_ExcludedGpus:charp
2024-08-20T22:14:29.1261215Z parm:           NVreg_DmaRemapPeerMmio:int
2024-08-20T22:14:29.1261695Z parm:           NVreg_RmNvlinkBandwidth:charp
2024-08-20T22:14:29.1262168Z parm:           NVreg_ImexChannelCount:int
2024-08-20T22:14:29.1262612Z parm:           rm_firmware_active:charp
2024-08-20T22:14:29.1263009Z + set +e
2024-08-20T22:14:29.1263330Z + nvidia-smi
2024-08-20T22:14:30.6969613Z Tue Aug 20 22:14:30 2024       
2024-08-20T22:14:30.6970462Z +-----------------------------------------------------------------------------------------+
2024-08-20T22:14:30.6971306Z | NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
2024-08-20T22:14:30.6972105Z |-----------------------------------------+------------------------+----------------------+
2024-08-20T22:14:30.6972910Z | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
2024-08-20T22:14:30.6973807Z | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
2024-08-20T22:14:30.6974512Z |                                         |                        |               MIG M. |
2024-08-20T22:14:30.6975081Z |=========================================+========================+======================|
2024-08-20T22:14:30.7070427Z |   0  NVIDIA A10G                    Off |   00000000:00:1E.0 Off |                    0 |
2024-08-20T22:14:30.7072287Z |  0%   29C    P0             58W /  300W |       0MiB /  23028MiB |      5%      Default |
2024-08-20T22:14:30.7073544Z |                                         |                        |                  N/A |
2024-08-20T22:14:30.7074953Z +-----------------------------------------+------------------------+----------------------+
2024-08-20T22:14:30.7076112Z                                                                                          
2024-08-20T22:14:30.7077406Z +-----------------------------------------------------------------------------------------+
2024-08-20T22:14:30.7078126Z | Processes:                                                                              |
2024-08-20T22:14:30.7078843Z |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
2024-08-20T22:14:30.7079540Z |        ID   ID                                                               Usage      |
2024-08-20T22:14:30.7080099Z |=========================================================================================|
2024-08-20T22:14:30.7080761Z |  No running processes found                                                             |
2024-08-20T22:14:30.7081526Z +-----------------------------------------------------------------------------------------+
2024-08-20T22:14:31.3318864Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0
2024-08-20T22:14:32.8989420Z NVIDIA A10G
2024-08-20T22:14:33.3519456Z + NVIDIA_SMI_STATUS=0
2024-08-20T22:14:33.3520058Z + '[' 0 -eq 0 ']'
2024-08-20T22:14:33.3520785Z + echo 'INFO: Ignoring allowed status 0'
2024-08-20T22:14:33.3521223Z + set -e
2024-08-20T22:14:33.3521527Z INFO: Ignoring allowed status 0
2024-08-20T22:14:33.3527835Z == Installing nvidia container toolkit for amzn2023 ==
2024-08-20T22:14:33.3531845Z + sudo yum install -y yum-utils
2024-08-20T22:14:36.3578004Z Last metadata expiration check: 0:09:56 ago on Tue Aug 20 22:04:40 2024.
2024-08-20T22:14:36.3906001Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed.
2024-08-20T22:14:36.4204375Z Dependencies resolved.
2024-08-20T22:14:36.4328806Z Nothing to do.
2024-08-20T22:14:36.4329474Z Complete!
2024-08-20T22:14:36.4986060Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]]
2024-08-20T22:14:36.4987203Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
2024-08-20T22:14:36.4988681Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
2024-08-20T22:14:36.7853686Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
2024-08-20T22:14:36.8582585Z + sudo yum install -y nvidia-docker2
2024-08-20T22:14:37.3398531Z nvidia-container-toolkit                         10 kB/s | 833  B     00:00    
2024-08-20T22:14:37.3618955Z Package nvidia-docker2-2.14.0-1.noarch is already installed.
2024-08-20T22:14:37.3917855Z Dependencies resolved.
2024-08-20T22:14:37.4041599Z Nothing to do.
2024-08-20T22:14:37.4041907Z Complete!
2024-08-20T22:14:37.4873655Z + sudo systemctl restart docker
2024-08-20T22:15:32.5517649Z Tue Aug 20 22:15:32 2024       
2024-08-20T22:15:32.5518501Z +-----------------------------------------------------------------------------------------+
2024-08-20T22:15:32.5519341Z | NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
2024-08-20T22:15:32.5520142Z |-----------------------------------------+------------------------+----------------------+
2024-08-20T22:15:32.5520941Z | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
2024-08-20T22:15:32.5521854Z | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
2024-08-20T22:15:32.5522573Z |                                         |                        |               MIG M. |
2024-08-20T22:15:32.5523107Z |=========================================+========================+======================|
2024-08-20T22:15:32.5640158Z |   0  NVIDIA A10G                    On  |   00000000:00:1E.0 Off |                    0 |
2024-08-20T22:15:32.5641535Z |  0%   30C    P0             58W /  300W |       0MiB /  23028MiB |      5%      Default |
2024-08-20T22:15:32.5642184Z |                                         |                        |                  N/A |
2024-08-20T22:15:32.5643032Z +-----------------------------------------+------------------------+----------------------+
2024-08-20T22:15:32.5643615Z                                                                                          
2024-08-20T22:15:32.5644478Z +-----------------------------------------------------------------------------------------+
2024-08-20T22:15:32.5645301Z | Processes:                                                                              |
2024-08-20T22:15:32.5646007Z |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
2024-08-20T22:15:32.5646708Z |        ID   ID                                                               Usage      |
2024-08-20T22:15:32.5647276Z |=========================================================================================|
2024-08-20T22:15:32.5647925Z |  No running processes found                                                             |
2024-08-20T22:15:32.5648693Z +-----------------------------------------------------------------------------------------+
2024-08-20T22:15:33.1816858Z Command completed after 1 attempt(s).
2024-08-20T22:15:33.1896439Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84
2024-08-20T22:15:33.1897205Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84
2024-08-20T22:15:33.1897887Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2024-08-20T22:15:33.1898525Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}"
2024-08-20T22:15:33.1912810Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:15:33.1913311Z env:
2024-08-20T22:15:33.1913594Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:33.1914055Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:33.1914538Z ##[endgroup]
2024-08-20T22:15:33.4813530Z Defaulting to user installation because normal site-packages is not writeable
2024-08-20T22:15:33.9012027Z Collecting psutil==5.9.1
2024-08-20T22:15:33.9393948Z   Downloading psutil-5.9.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB)
2024-08-20T22:15:33.9893555Z Collecting nvidia-ml-py==11.525.84
2024-08-20T22:15:33.9935838Z   Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB)
2024-08-20T22:15:34.0795922Z Installing collected packages: psutil, nvidia-ml-py
2024-08-20T22:15:34.2351828Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1
2024-08-20T22:15:34.4299542Z Prepare all required actions
2024-08-20T22:15:34.4300092Z Getting action download info
2024-08-20T22:15:34.5357561Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6)
2024-08-20T22:15:34.7414394Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a)
2024-08-20T22:15:34.8799089Z ##[group]Run ./.github/actions/download-build-artifacts
2024-08-20T22:15:34.8799572Z with:
2024-08-20T22:15:34.8799899Z   name: linux-focal-cuda12.1-py3.10-gcc9-sm86
2024-08-20T22:15:34.8800351Z   s3-bucket: gha-artifacts
2024-08-20T22:15:34.8800696Z env:
2024-08-20T22:15:34.8800965Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:34.8801415Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:34.8801929Z ##[endgroup]
2024-08-20T22:15:34.8835011Z ##[group]Run seemethere/download-artifact-s3@v4
2024-08-20T22:15:34.8835450Z with:
2024-08-20T22:15:34.8835815Z   name: linux-focal-cuda12.1-py3.10-gcc9-sm86
2024-08-20T22:15:34.8836271Z   s3-bucket: gha-artifacts
2024-08-20T22:15:34.8836625Z   region: us-east-1
2024-08-20T22:15:34.8836924Z env:
2024-08-20T22:15:34.8837190Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:34.8837631Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:34.8838274Z ##[endgroup]
2024-08-20T22:15:35.3683710Z (node:52498) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2024-08-20T22:15:35.3684401Z 
2024-08-20T22:15:35.3684662Z Please migrate your code to use AWS SDK for JavaScript (v3).
2024-08-20T22:15:35.3685381Z For more information, check the migration guide at https://a.co/7PzMCcy
2024-08-20T22:15:35.3686247Z (Use `node --trace-warnings ...` to show where the warning was created)
2024-08-20T22:15:35.4439057Z Found 1 objects with prefix pytorch/pytorch/10479310721/linux-focal-cuda12.1-py3.10-gcc9-sm86/
2024-08-20T22:15:35.4441184Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2024-08-20T22:15:42.9713274Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2024-08-20T22:15:42.9719643Z Artifact download has finished successfully
2024-08-20T22:15:43.0096700Z ##[group]Run unzip -o artifacts.zip
2024-08-20T22:15:43.0097152Z unzip -o artifacts.zip
2024-08-20T22:15:43.0106176Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:15:43.0106665Z env:
2024-08-20T22:15:43.0106941Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:43.0107392Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:43.0107875Z ##[endgroup]
2024-08-20T22:15:43.0188386Z Archive:  artifacts.zip
2024-08-20T22:15:43.0189128Z    creating: dist/
2024-08-20T22:15:45.1465128Z   inflating: dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl  
2024-08-20T22:15:45.1465789Z    creating: build/custom_test_artifacts/
2024-08-20T22:15:45.1466380Z    creating: build/custom_test_artifacts/custom-op-build/
2024-08-20T22:15:45.1467108Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2024-08-20T22:15:45.1467949Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/
2024-08-20T22:15:45.1474769Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml  
2024-08-20T22:15:45.1475726Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/
2024-08-20T22:15:45.1476674Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2024-08-20T22:15:45.1477671Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/
2024-08-20T22:15:45.1478648Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-08-20T22:15:45.1479747Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2024-08-20T22:15:45.1481434Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2024-08-20T22:15:45.1482448Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-08-20T22:15:45.1483436Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-08-20T22:15:45.1484829Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2024-08-20T22:15:45.1486525Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2024-08-20T22:15:45.1488767Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2024-08-20T22:15:45.1489913Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2024-08-20T22:15:45.1491219Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2024-08-20T22:15:45.1492640Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2024-08-20T22:15:45.1493677Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2024-08-20T22:15:45.1494737Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2024-08-20T22:15:45.1534906Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2024-08-20T22:15:45.1574686Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2024-08-20T22:15:45.1576127Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2024-08-20T22:15:45.1621624Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2024-08-20T22:15:45.1623026Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2024-08-20T22:15:45.1624460Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2024-08-20T22:15:45.1625919Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2024-08-20T22:15:45.1627322Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2024-08-20T22:15:45.1628694Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2024-08-20T22:15:45.1630070Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2024-08-20T22:15:45.1631449Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2024-08-20T22:15:45.1632785Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2024-08-20T22:15:45.1634059Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2024-08-20T22:15:45.1635294Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2024-08-20T22:15:45.1636501Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2024-08-20T22:15:45.1637727Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2024-08-20T22:15:45.1638914Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2024-08-20T22:15:45.1640120Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2024-08-20T22:15:45.1696848Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2024-08-20T22:15:45.1757704Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2024-08-20T22:15:45.1758898Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2024-08-20T22:15:45.1759897Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/
2024-08-20T22:15:45.1760760Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2024-08-20T22:15:45.1761675Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2024-08-20T22:15:45.1762601Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2024-08-20T22:15:45.1763625Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2024-08-20T22:15:45.1764786Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2024-08-20T22:15:45.1765898Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2024-08-20T22:15:45.1766941Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2024-08-20T22:15:45.1768194Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2024-08-20T22:15:45.1769450Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2024-08-20T22:15:45.1770550Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2024-08-20T22:15:45.1771625Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2024-08-20T22:15:45.1772695Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2024-08-20T22:15:45.1789945Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2024-08-20T22:15:45.1934970Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2024-08-20T22:15:45.1936032Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2024-08-20T22:15:45.1937152Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2024-08-20T22:15:45.1938375Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2024-08-20T22:15:45.1939616Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2024-08-20T22:15:45.1940710Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2024-08-20T22:15:45.1941838Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2024-08-20T22:15:45.1942984Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2024-08-20T22:15:45.1944128Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2024-08-20T22:15:45.1945254Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2024-08-20T22:15:45.1946382Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2024-08-20T22:15:45.1963280Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2024-08-20T22:15:45.2046676Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2024-08-20T22:15:45.2047874Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2024-08-20T22:15:45.2048949Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2024-08-20T22:15:45.2050153Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2024-08-20T22:15:45.2051088Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2024-08-20T22:15:45.2051987Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2024-08-20T22:15:45.2052900Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2024-08-20T22:15:45.2054983Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2024-08-20T22:15:45.2055975Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2024-08-20T22:15:45.2056798Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2024-08-20T22:15:45.2177862Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2024-08-20T22:15:45.2240721Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2024-08-20T22:15:45.2241479Z    creating: build/custom_test_artifacts/jit-hook-build/
2024-08-20T22:15:45.2242187Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2024-08-20T22:15:45.2243014Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/
2024-08-20T22:15:45.2249619Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml  
2024-08-20T22:15:45.2250749Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/
2024-08-20T22:15:45.2251682Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2024-08-20T22:15:45.2252673Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/
2024-08-20T22:15:45.2253632Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-08-20T22:15:45.2254723Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2024-08-20T22:15:45.2256078Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2024-08-20T22:15:45.2257393Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-08-20T22:15:45.2258440Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-08-20T22:15:45.2260319Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2024-08-20T22:15:45.2262007Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2024-08-20T22:15:45.2264133Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2024-08-20T22:15:45.2265274Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2024-08-20T22:15:45.2266653Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2024-08-20T22:15:45.2267981Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2024-08-20T22:15:45.2269171Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2024-08-20T22:15:45.2270182Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2024-08-20T22:15:45.2310043Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2024-08-20T22:15:45.2349717Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2024-08-20T22:15:45.2351168Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2024-08-20T22:15:45.2397757Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2024-08-20T22:15:45.2399167Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2024-08-20T22:15:45.2400818Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2024-08-20T22:15:45.2402301Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2024-08-20T22:15:45.2403727Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2024-08-20T22:15:45.2405109Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2024-08-20T22:15:45.2406493Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2024-08-20T22:15:45.2407886Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2024-08-20T22:15:45.2409238Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2024-08-20T22:15:45.2410527Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2024-08-20T22:15:45.2411760Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2024-08-20T22:15:45.2413113Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2024-08-20T22:15:45.2414391Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2024-08-20T22:15:45.2415576Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2024-08-20T22:15:45.2416784Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2024-08-20T22:15:45.2473122Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2024-08-20T22:15:45.2533526Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2024-08-20T22:15:45.2535505Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2024-08-20T22:15:45.2537481Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/
2024-08-20T22:15:45.2539289Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2024-08-20T22:15:45.2541084Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2024-08-20T22:15:45.2542961Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2024-08-20T22:15:45.2544636Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2024-08-20T22:15:45.2545832Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2024-08-20T22:15:45.2546992Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2024-08-20T22:15:45.2548086Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2024-08-20T22:15:45.2549198Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2024-08-20T22:15:45.2550313Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2024-08-20T22:15:45.2551424Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2024-08-20T22:15:45.2552530Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2024-08-20T22:15:45.2553622Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2024-08-20T22:15:45.2566192Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2024-08-20T22:15:45.2631321Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2024-08-20T22:15:45.2632510Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2024-08-20T22:15:45.2633586Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2024-08-20T22:15:45.2634618Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2024-08-20T22:15:45.2635516Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2024-08-20T22:15:45.2636662Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2024-08-20T22:15:45.2637600Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2024-08-20T22:15:45.2639866Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2024-08-20T22:15:45.2640770Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2024-08-20T22:15:45.2641599Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2024-08-20T22:15:45.2692061Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2024-08-20T22:15:45.2692831Z    creating: build/custom_test_artifacts/custom-backend-build/
2024-08-20T22:15:45.2693794Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2024-08-20T22:15:45.2694681Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/
2024-08-20T22:15:45.2700916Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml  
2024-08-20T22:15:45.2701904Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/
2024-08-20T22:15:45.2702915Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2024-08-20T22:15:45.2703970Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/
2024-08-20T22:15:45.2705012Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-08-20T22:15:45.2706175Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2024-08-20T22:15:45.2707468Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2024-08-20T22:15:45.2708544Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-08-20T22:15:45.2709599Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-08-20T22:15:45.2711049Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2024-08-20T22:15:45.2712562Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2024-08-20T22:15:45.2714395Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2024-08-20T22:15:45.2715594Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2024-08-20T22:15:45.2717155Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2024-08-20T22:15:45.2718498Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2024-08-20T22:15:45.2719595Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2024-08-20T22:15:45.2720655Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2024-08-20T22:15:45.2760366Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2024-08-20T22:15:45.2800293Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2024-08-20T22:15:45.2801999Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2024-08-20T22:15:45.2847074Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2024-08-20T22:15:45.2848540Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2024-08-20T22:15:45.2850016Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2024-08-20T22:15:45.2851541Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2024-08-20T22:15:45.2852998Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2024-08-20T22:15:45.2854656Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2024-08-20T22:15:45.2856112Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2024-08-20T22:15:45.2857557Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2024-08-20T22:15:45.2859206Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2024-08-20T22:15:45.2860548Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2024-08-20T22:15:45.2861833Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2024-08-20T22:15:45.2863109Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2024-08-20T22:15:45.2864445Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2024-08-20T22:15:45.2865688Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2024-08-20T22:15:45.2866962Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2024-08-20T22:15:45.2923088Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2024-08-20T22:15:45.2983644Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2024-08-20T22:15:45.2984884Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2024-08-20T22:15:45.2985931Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/
2024-08-20T22:15:45.2986862Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2024-08-20T22:15:45.2987954Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2024-08-20T22:15:45.2988972Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2024-08-20T22:15:45.2990110Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2024-08-20T22:15:45.2991480Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2024-08-20T22:15:45.2992700Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2024-08-20T22:15:45.2993839Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2024-08-20T22:15:45.2995075Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2024-08-20T22:15:45.2996415Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2024-08-20T22:15:45.2997623Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2024-08-20T22:15:45.2998806Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2024-08-20T22:15:45.2999979Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2024-08-20T22:15:45.3001230Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2024-08-20T22:15:45.3125223Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2024-08-20T22:15:45.3127549Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2024-08-20T22:15:45.3129910Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2024-08-20T22:15:45.3132507Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2024-08-20T22:15:45.3134848Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2024-08-20T22:15:45.3136206Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2024-08-20T22:15:45.3137430Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2024-08-20T22:15:45.3138750Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2024-08-20T22:15:45.3139991Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2024-08-20T22:15:45.3141223Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2024-08-20T22:15:45.3142438Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2024-08-20T22:15:45.3154607Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2024-08-20T22:15:45.3210752Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2024-08-20T22:15:45.3212036Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2024-08-20T22:15:45.3213174Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2024-08-20T22:15:45.3214199Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2024-08-20T22:15:45.3215234Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2024-08-20T22:15:45.3216201Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2024-08-20T22:15:45.3217163Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2024-08-20T22:15:45.3219714Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2024-08-20T22:15:45.3220678Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2024-08-20T22:15:45.3221646Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2024-08-20T22:15:45.3326329Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2024-08-20T22:15:45.3368687Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2024-08-20T22:15:45.3369657Z    creating: build/lib/
2024-08-20T22:15:45.3458177Z   inflating: build/lib/libprotobuf-lite.a  
2024-08-20T22:15:45.3913195Z   inflating: build/lib/libprotobuf.a  
2024-08-20T22:15:45.3922953Z   inflating: build/lib/libpthreadpool.a  
2024-08-20T22:15:45.3931336Z   inflating: build/lib/libcpuinfo.a  
2024-08-20T22:15:45.3939544Z   inflating: build/lib/libcpuinfo_internals.a  
2024-08-20T22:15:45.3940171Z   inflating: build/lib/libclog.a     
2024-08-20T22:15:45.3958700Z   inflating: build/lib/libnnpack.a   
2024-08-20T22:15:45.3961998Z   inflating: build/lib/libnnpack_reference_layers.a  
2024-08-20T22:15:45.4027418Z   inflating: build/lib/libgtest.a    
2024-08-20T22:15:45.4102285Z   inflating: build/lib/libbenchmark.a  
2024-08-20T22:15:45.4165891Z   inflating: build/lib/libasmjit.a   
2024-08-20T22:15:45.4173886Z   inflating: build/lib/libittnotify.a  
2024-08-20T22:15:45.4201768Z   inflating: build/lib/libtensorpipe_uv.a  
2024-08-20T22:15:45.4328670Z   inflating: build/lib/libgloo.a     
2024-08-20T22:15:45.4349790Z   inflating: build/lib/libfmt.a      
2024-08-20T22:15:45.4444732Z   inflating: build/lib/libc10.so     
2024-08-20T22:15:45.4446413Z   inflating: build/lib/libcaffe2_nvrtc.so  
2024-08-20T22:15:45.4447614Z   inflating: build/lib/libtorch_global_deps.so  
2024-08-20T22:15:45.4951316Z   inflating: build/lib/libprotoc.a   
2024-08-20T22:15:45.4971030Z   inflating: build/lib/libpytorch_qnnpack.a  
2024-08-20T22:15:45.4989193Z   inflating: build/lib/libgmock.a    
2024-08-20T22:15:45.4989716Z   inflating: build/lib/libgtest_main.a  
2024-08-20T22:15:45.4990791Z   inflating: build/lib/libbenchmark_main.a  
2024-08-20T22:15:45.5380849Z   inflating: build/lib/libgloo_cuda.a  
2024-08-20T22:15:46.5542486Z   inflating: build/lib/libdnnl.a     
2024-08-20T22:15:46.6115739Z   inflating: build/lib/libtensorpipe.a  
2024-08-20T22:15:46.6173710Z   inflating: build/lib/libc10_cuda.so  
2024-08-20T22:15:46.6174541Z   inflating: build/lib/libgmock_main.a  
2024-08-20T22:15:46.7431022Z   inflating: build/lib/libfbgemm.a   
2024-08-20T22:15:46.7687378Z   inflating: build/lib/libtensorpipe_cuda.a  
2024-08-20T22:15:46.8192983Z   inflating: build/lib/libkineto.a   
2024-08-20T22:15:46.8376122Z   inflating: build/lib/libXNNPACK.a  
2024-08-20T22:15:46.8418724Z   inflating: build/lib/libonnx_proto.a  
2024-08-20T22:15:46.9112413Z   inflating: build/lib/libonnx.a     
2024-08-20T22:15:49.3539431Z   inflating: build/lib/libtorch_cpu.so  
2024-08-20T22:15:49.3544491Z   inflating: build/lib/libunbox_lib.a  
2024-08-20T22:15:49.3548822Z   inflating: build/lib/libshm.so     
2024-08-20T22:15:51.4147059Z   inflating: build/lib/libtorch_cuda.so  
2024-08-20T22:15:51.4148856Z   inflating: build/lib/libtorch.so   
2024-08-20T22:15:51.4151368Z   inflating: build/lib/libc10d_cuda_test.so  
2024-08-20T22:15:52.2310467Z   inflating: build/lib/libtorch_cuda_linalg.so  
2024-08-20T22:15:52.4303639Z   inflating: build/lib/libtorch_python.so  
2024-08-20T22:15:52.4376205Z   inflating: build/lib/libtorchbind_test.so  
2024-08-20T22:15:52.4397534Z   inflating: build/lib/libjitbackend_test.so  
2024-08-20T22:15:52.4422535Z   inflating: build/lib/libbackend_with_compiler.so  
2024-08-20T22:15:52.4448510Z   inflating: build/lib/libaoti_custom_ops.so  
2024-08-20T22:15:52.4483301Z   inflating: build/lib/libnnapi_backend.so  
2024-08-20T22:15:52.4483780Z    creating: build/bin/
2024-08-20T22:15:52.4534690Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2024-08-20T22:15:52.4585982Z   inflating: build/bin/c10_DeviceGuard_test  
2024-08-20T22:15:52.4637550Z   inflating: build/bin/c10_Device_test  
2024-08-20T22:15:52.4697037Z   inflating: build/bin/c10_DispatchKeySet_test  
2024-08-20T22:15:52.4750629Z   inflating: build/bin/c10_Scalar_test  
2024-08-20T22:15:52.4800290Z   inflating: build/bin/c10_StreamGuard_test  
2024-08-20T22:15:52.4851090Z   inflating: build/bin/c10_SymInt_test  
2024-08-20T22:15:52.4906058Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2024-08-20T22:15:52.4961666Z   inflating: build/bin/c10_InlineStreamGuard_test  
2024-08-20T22:15:52.5018483Z   inflating: build/bin/c10_SizesAndStrides_test  
2024-08-20T22:15:52.5089724Z   inflating: build/bin/c10_cow_test  
2024-08-20T22:15:52.5143160Z   inflating: build/bin/c10_Bitset_test  
2024-08-20T22:15:52.5191972Z   inflating: build/bin/c10_ConstexprCrc_test  
2024-08-20T22:15:52.5242175Z   inflating: build/bin/c10_DeadlockDetection_test  
2024-08-20T22:15:52.5292945Z   inflating: build/bin/c10_Half_test  
2024-08-20T22:15:52.5349383Z   inflating: build/bin/c10_LeftRight_test  
2024-08-20T22:15:52.5404828Z   inflating: build/bin/c10_Metaprogramming_test  
2024-08-20T22:15:52.5455468Z   inflating: build/bin/c10_Synchronized_test  
2024-08-20T22:15:52.5511133Z   inflating: build/bin/c10_ThreadLocal_test  
2024-08-20T22:15:52.5563309Z   inflating: build/bin/c10_TypeIndex_test  
2024-08-20T22:15:52.5614284Z   inflating: build/bin/c10_TypeList_test  
2024-08-20T22:15:52.5663779Z   inflating: build/bin/c10_TypeTraits_test  
2024-08-20T22:15:52.5715367Z   inflating: build/bin/c10_accumulate_test  
2024-08-20T22:15:52.5771220Z   inflating: build/bin/c10_bfloat16_test  
2024-08-20T22:15:52.5821836Z   inflating: build/bin/c10_bit_cast_test  
2024-08-20T22:15:52.5878572Z   inflating: build/bin/c10_complex_math_test  
2024-08-20T22:15:52.5935032Z   inflating: build/bin/c10_complex_test  
2024-08-20T22:15:52.5987502Z   inflating: build/bin/c10_exception_test  
2024-08-20T22:15:52.6038422Z   inflating: build/bin/c10_flags_test  
2024-08-20T22:15:52.6088467Z   inflating: build/bin/c10_generic_math_test  
2024-08-20T22:15:52.6253738Z   inflating: build/bin/c10_intrusive_ptr_test  
2024-08-20T22:15:52.6304926Z   inflating: build/bin/c10_irange_test  
2024-08-20T22:15:52.6358688Z   inflating: build/bin/c10_lazy_test  
2024-08-20T22:15:52.6416239Z   inflating: build/bin/c10_logging_test  
2024-08-20T22:15:52.6492264Z   inflating: build/bin/c10_optional_test  
2024-08-20T22:15:52.6554277Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2024-08-20T22:15:52.6608717Z   inflating: build/bin/c10_registry_test  
2024-08-20T22:15:52.6760950Z   inflating: build/bin/c10_small_vector_test  
2024-08-20T22:15:52.6812879Z   inflating: build/bin/c10_ssize_test  
2024-08-20T22:15:52.6865079Z   inflating: build/bin/c10_string_util_test  
2024-08-20T22:15:52.6915687Z   inflating: build/bin/c10_tempfile_test  
2024-08-20T22:15:52.6975380Z   inflating: build/bin/c10_string_view_test  
2024-08-20T22:15:52.7023456Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2024-08-20T22:15:52.7079851Z   inflating: build/bin/c10_typeid_test  
2024-08-20T22:15:52.7527891Z   inflating: build/bin/protoc-3.13.0.0  
2024-08-20T22:15:52.7974178Z   inflating: build/bin/protoc        
2024-08-20T22:15:52.8027424Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test  
2024-08-20T22:15:52.8080675Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream  
2024-08-20T22:15:52.8133667Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device  
2024-08-20T22:15:52.8186085Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes  
2024-08-20T22:15:52.8239626Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads  
2024-08-20T22:15:52.8292034Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks  
2024-08-20T22:15:52.8341482Z   inflating: build/bin/c10_cuda_CUDATest  
2024-08-20T22:15:52.8394476Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block  
2024-08-20T22:15:52.8725254Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2024-08-20T22:15:52.9071719Z   inflating: build/bin/vec_test_all_types_AVX512  
2024-08-20T22:15:52.9431854Z   inflating: build/bin/vec_test_all_types_AVX2  
2024-08-20T22:15:52.9484948Z   inflating: build/bin/BackoffTest   
2024-08-20T22:15:52.9538718Z   inflating: build/bin/FileStoreTest  
2024-08-20T22:15:52.9594105Z   inflating: build/bin/TCPStoreTest  
2024-08-20T22:15:52.9647813Z   inflating: build/bin/HashStoreTest  
2024-08-20T22:15:52.9661535Z   inflating: build/bin/ProcessGroupMPITest  
2024-08-20T22:15:52.9715859Z   inflating: build/bin/test_edge_op_registration  
2024-08-20T22:15:52.9720282Z   inflating: build/bin/torch_shm_manager  
2024-08-20T22:15:52.9723299Z   inflating: build/bin/example_allreduce  
2024-08-20T22:15:52.9778947Z   inflating: build/bin/test_dist_autograd  
2024-08-20T22:15:52.9848062Z   inflating: build/bin/test_cpp_rpc  
2024-08-20T22:15:52.9851457Z   inflating: build/bin/parallel_benchmark  
2024-08-20T22:15:52.9919476Z   inflating: build/bin/test_mobile_nnc  
2024-08-20T22:15:52.9928540Z   inflating: build/bin/aot_model_compiler_test  
2024-08-20T22:15:53.0275678Z   inflating: build/bin/test_lazy     
2024-08-20T22:15:53.1459570Z   inflating: build/bin/test_api      
2024-08-20T22:15:53.1533783Z   inflating: build/bin/Dict_test     
2024-08-20T22:15:53.1586220Z   inflating: build/bin/Dimname_test  
2024-08-20T22:15:53.1651330Z   inflating: build/bin/MaybeOwned_test  
2024-08-20T22:15:53.1709169Z   inflating: build/bin/NamedTensor_test  
2024-08-20T22:15:53.1768538Z   inflating: build/bin/apply_utils_test  
2024-08-20T22:15:53.1827914Z   inflating: build/bin/atest         
2024-08-20T22:15:53.1891640Z   inflating: build/bin/basic         
2024-08-20T22:15:53.1946838Z   inflating: build/bin/broadcast_test  
2024-08-20T22:15:53.1998627Z   inflating: build/bin/cpu_allocator_test  
2024-08-20T22:15:53.2056478Z   inflating: build/bin/cpu_generator_test  
2024-08-20T22:15:53.2110778Z   inflating: build/bin/cpu_profiling_allocator_test  
2024-08-20T22:15:53.2203824Z   inflating: build/bin/cpu_rng_test  
2024-08-20T22:15:53.2254561Z   inflating: build/bin/dispatch_key_set_test  
2024-08-20T22:15:53.2304810Z   inflating: build/bin/dlconvertor_test  
2024-08-20T22:15:53.2363031Z   inflating: build/bin/extension_backend_test  
2024-08-20T22:15:53.2418708Z   inflating: build/bin/half_test     
2024-08-20T22:15:53.2514850Z   inflating: build/bin/ivalue_test   
2024-08-20T22:15:53.2565106Z   inflating: build/bin/lazy_tensor_test  
2024-08-20T22:15:53.2619693Z   inflating: build/bin/math_kernel_test  
2024-08-20T22:15:53.2674189Z   inflating: build/bin/memory_format_test  
2024-08-20T22:15:53.2727310Z   inflating: build/bin/memory_overlapping_test  
2024-08-20T22:15:53.2780698Z   inflating: build/bin/mobile_memory_cleanup  
2024-08-20T22:15:53.2837250Z   inflating: build/bin/native_test   
2024-08-20T22:15:53.2887599Z   inflating: build/bin/operator_name_test  
2024-08-20T22:15:53.2939562Z   inflating: build/bin/operators_test  
2024-08-20T22:15:53.2991416Z   inflating: build/bin/packedtensoraccessor_test  
2024-08-20T22:15:53.3058735Z   inflating: build/bin/pow_test      
2024-08-20T22:15:53.3116132Z   inflating: build/bin/quantized_test  
2024-08-20T22:15:53.3166721Z   inflating: build/bin/reduce_ops_test  
2024-08-20T22:15:53.3218056Z   inflating: build/bin/reportMemoryUsage_test  
2024-08-20T22:15:53.3275236Z   inflating: build/bin/scalar_tensor_test  
2024-08-20T22:15:53.3334620Z   inflating: build/bin/scalar_test   
2024-08-20T22:15:53.3386271Z   inflating: build/bin/StorageUtils_test  
2024-08-20T22:15:53.3439077Z   inflating: build/bin/stride_properties_test  
2024-08-20T22:15:53.3518002Z   inflating: build/bin/tensor_iterator_test  
2024-08-20T22:15:53.3573123Z   inflating: build/bin/test_parallel  
2024-08-20T22:15:53.3576040Z   inflating: build/bin/thread_init_test  
2024-08-20T22:15:53.3631775Z   inflating: build/bin/type_ptr_test  
2024-08-20T22:15:53.3692345Z   inflating: build/bin/type_test     
2024-08-20T22:15:53.3744508Z   inflating: build/bin/undefined_tensor_test  
2024-08-20T22:15:53.3746284Z   inflating: build/bin/verify_api_visibility  
2024-08-20T22:15:53.3815833Z   inflating: build/bin/legacy_vmap_test  
2024-08-20T22:15:53.3867592Z   inflating: build/bin/weakref_test  
2024-08-20T22:15:53.3919966Z   inflating: build/bin/wrapdim_test  
2024-08-20T22:15:53.3972102Z   inflating: build/bin/xla_tensor_test  
2024-08-20T22:15:53.4031872Z   inflating: build/bin/IListRef_test  
2024-08-20T22:15:53.4138472Z   inflating: build/bin/List_test     
2024-08-20T22:15:53.4205864Z   inflating: build/bin/KernelFunction_test  
2024-08-20T22:15:53.4327826Z   inflating: build/bin/kernel_function_legacy_test  
2024-08-20T22:15:53.4425054Z   inflating: build/bin/kernel_function_test  
2024-08-20T22:15:53.4554475Z   inflating: build/bin/kernel_lambda_legacy_test  
2024-08-20T22:15:53.4658515Z   inflating: build/bin/kernel_lambda_test  
2024-08-20T22:15:53.4719834Z   inflating: build/bin/kernel_stackbased_test  
2024-08-20T22:15:53.4816408Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2024-08-20T22:15:53.4867341Z   inflating: build/bin/CppSignature_test  
2024-08-20T22:15:53.4916735Z   inflating: build/bin/op_allowlist_test  
2024-08-20T22:15:53.4973277Z   inflating: build/bin/backend_fallback_test  
2024-08-20T22:15:53.5276334Z   inflating: build/bin/op_registration_test  
2024-08-20T22:15:53.5340681Z   inflating: build/bin/inline_container_test  
2024-08-20T22:15:53.5394093Z   inflating: build/bin/cuda_apply_test  
2024-08-20T22:15:53.5446462Z   inflating: build/bin/cuda_allocator_test  
2024-08-20T22:15:53.5506080Z   inflating: build/bin/cuda_atomic_ops_test  
2024-08-20T22:15:53.5561648Z   inflating: build/bin/cuda_caching_host_allocator_test  
2024-08-20T22:15:53.5631273Z   inflating: build/bin/cuda_complex_math_test  
2024-08-20T22:15:53.5691141Z   inflating: build/bin/cuda_complex_test  
2024-08-20T22:15:53.5740477Z   inflating: build/bin/cuda_device_test  
2024-08-20T22:15:53.5798770Z   inflating: build/bin/cuda_cub_test  
2024-08-20T22:15:53.5849413Z   inflating: build/bin/cuda_dlconvertor_test  
2024-08-20T22:15:53.5914369Z   inflating: build/bin/cuda_distributions_test  
2024-08-20T22:15:53.5971956Z   inflating: build/bin/cuda_generator_test  
2024-08-20T22:15:53.6021269Z   inflating: build/bin/cuda_half_test  
2024-08-20T22:15:53.6072914Z   inflating: build/bin/cuda_integer_divider_test  
2024-08-20T22:15:53.6122190Z   inflating: build/bin/cuda_optional_test  
2024-08-20T22:15:53.6174690Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2024-08-20T22:15:53.6227058Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2024-08-20T22:15:53.6277338Z   inflating: build/bin/cuda_allocatorTraceTracker_test  
2024-08-20T22:15:53.6338200Z   inflating: build/bin/cuda_stream_test  
2024-08-20T22:15:53.6388097Z   inflating: build/bin/cuda_cudnn_test  
2024-08-20T22:15:53.6440844Z   inflating: build/bin/cuda_vectorized_test  
2024-08-20T22:15:53.6454920Z   inflating: build/bin/tutorial_tensorexpr  
2024-08-20T22:15:53.6520935Z   inflating: build/bin/ProcessGroupGlooTest  
2024-08-20T22:15:53.6578738Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2024-08-20T22:15:53.6642380Z   inflating: build/bin/ProcessGroupNCCLTest  
2024-08-20T22:15:53.6705545Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2024-08-20T22:15:53.7534484Z   inflating: build/bin/test_tensorexpr  
2024-08-20T22:15:53.8116789Z   inflating: build/bin/test_jit      
2024-08-20T22:15:53.8117658Z    creating: .additional_ci_files/
2024-08-20T22:15:53.8177950Z   inflating: .additional_ci_files/test-times.json  
2024-08-20T22:15:53.8418640Z   inflating: .additional_ci_files/test-class-times.json  
2024-08-20T22:15:53.8462223Z ##[group]Run rm artifacts.zip
2024-08-20T22:15:53.8462814Z rm artifacts.zip
2024-08-20T22:15:53.8474659Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:15:53.8475405Z env:
2024-08-20T22:15:53.8475814Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:53.8476473Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:53.8477174Z ##[endgroup]
2024-08-20T22:15:53.9573850Z ##[group]Run df -H
2024-08-20T22:15:53.9574185Z df -H
2024-08-20T22:15:53.9583117Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:15:53.9583611Z env:
2024-08-20T22:15:53.9583891Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:53.9584347Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:53.9584828Z ##[endgroup]
2024-08-20T22:15:53.9636724Z Filesystem        Size  Used Avail Use% Mounted on
2024-08-20T22:15:53.9637399Z devtmpfs          4.2M     0  4.2M   0% /dev
2024-08-20T22:15:53.9637872Z tmpfs              34G     0   34G   0% /dev/shm
2024-08-20T22:15:53.9638340Z tmpfs              14G  553k   14G   1% /run
2024-08-20T22:15:53.9638791Z /dev/nvme0n1p1    161G   53G  109G  33% /
2024-08-20T22:15:53.9639469Z tmpfs              34G  4.1k   34G   1% /tmp
2024-08-20T22:15:53.9639957Z /dev/nvme0n1p128   11M  1.4M  9.2M  13% /boot/efi
2024-08-20T22:15:53.9640848Z tmpfs             6.7G     0  6.7G   0% /run/user/0
2024-08-20T22:15:53.9681428Z Prepare all required actions
2024-08-20T22:15:53.9681884Z Getting action download info
2024-08-20T22:15:54.1414063Z ##[group]Run ./.github/actions/download-td-artifacts
2024-08-20T22:15:54.1414528Z with:
2024-08-20T22:15:54.1414776Z env:
2024-08-20T22:15:54.1415049Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:54.1415498Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:54.1415984Z ##[endgroup]
2024-08-20T22:15:54.1448631Z ##[group]Run seemethere/download-artifact-s3@v4
2024-08-20T22:15:54.1449071Z with:
2024-08-20T22:15:54.1449333Z   name: td_results
2024-08-20T22:15:54.1449655Z   s3-bucket: gha-artifacts
2024-08-20T22:15:54.1450001Z   region: us-east-1
2024-08-20T22:15:54.1450291Z env:
2024-08-20T22:15:54.1450568Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:54.1451026Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:54.1451505Z ##[endgroup]
2024-08-20T22:15:54.6192331Z (node:52518) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2024-08-20T22:15:54.6193493Z 
2024-08-20T22:15:54.6193819Z Please migrate your code to use AWS SDK for JavaScript (v3).
2024-08-20T22:15:54.6194587Z For more information, check the migration guide at https://a.co/7PzMCcy
2024-08-20T22:15:54.6195442Z (Use `node --trace-warnings ...` to show where the warning was created)
2024-08-20T22:15:54.6893592Z Found 0 objects with prefix pytorch/pytorch/10479310721/td_results/
2024-08-20T22:15:54.6900483Z Artifact download has finished successfully
2024-08-20T22:15:54.7219893Z ##[group]Run mkdir -p .additional_ci_files
2024-08-20T22:15:54.7220370Z mkdir -p .additional_ci_files
2024-08-20T22:15:54.7220925Z mv td_results.json .additional_ci_files/td_results.json
2024-08-20T22:15:54.7229851Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:15:54.7230345Z env:
2024-08-20T22:15:54.7230619Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:54.7231081Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:54.7231570Z ##[endgroup]
2024-08-20T22:15:54.7287113Z mv: cannot stat 'td_results.json': No such file or directory
2024-08-20T22:15:54.7299811Z ##[error]Process completed with exit code 1.
2024-08-20T22:15:54.7334005Z ##[group]Run .github/scripts/parse_ref.py
2024-08-20T22:15:54.7334488Z .github/scripts/parse_ref.py
2024-08-20T22:15:54.7343354Z shell: /usr/bin/bash -e {0}
2024-08-20T22:15:54.7343707Z env:
2024-08-20T22:15:54.7343992Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:54.7344444Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:54.7344922Z ##[endgroup]
2024-08-20T22:15:54.7627171Z Prepare all required actions
2024-08-20T22:15:54.7665722Z ##[group]Run ./.github/actions/get-workflow-job-id
2024-08-20T22:15:54.7666178Z with:
2024-08-20T22:15:54.7666746Z   github-token: ***
2024-08-20T22:15:54.7667076Z env:
2024-08-20T22:15:54.7667350Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:54.7667811Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:54.7668320Z ##[endgroup]
2024-08-20T22:15:54.7709327Z ##[group]Run set -eux
2024-08-20T22:15:54.7709660Z set -eux
2024-08-20T22:15:54.7710239Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2024-08-20T22:15:54.7719371Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:15:54.7719865Z env:
2024-08-20T22:15:54.7720145Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:54.7720595Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:54.7721232Z   GITHUB_TOKEN: ***
2024-08-20T22:15:54.7721536Z ##[endgroup]
2024-08-20T22:15:54.7749283Z + python3 .github/scripts/get_workflow_job_id.py 10479310721 i-0479ed8028e52c92f
2024-08-20T22:15:55.4262479Z setting job-id=29026300273
2024-08-20T22:15:55.4264623Z setting job-name=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu)
2024-08-20T22:15:55.4482762Z Prepare all required actions
2024-08-20T22:15:55.4483197Z Getting action download info
2024-08-20T22:15:55.5710772Z ##[group]Run ./.github/actions/filter-test-configs
2024-08-20T22:15:55.5711218Z with:
2024-08-20T22:15:55.5711663Z   github-token: ***
2024-08-20T22:15:55.5718125Z   test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}
2024-08-20T22:15:55.5725664Z   job-name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu)
2024-08-20T22:15:55.5726585Z env:
2024-08-20T22:15:55.5726862Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:55.5727314Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:55.5727790Z ##[endgroup]
2024-08-20T22:15:55.5769657Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2024-08-20T22:15:55.5770202Z with:
2024-08-20T22:15:55.5770468Z   shell: bash
2024-08-20T22:15:55.5770768Z   timeout_minutes: 10
2024-08-20T22:15:55.5771086Z   max_attempts: 5
2024-08-20T22:15:55.5771404Z   retry_wait_seconds: 30
2024-08-20T22:15:55.5772512Z   command: set -eux
# PyYAML 6.0 doesn't work with MacOS x86 anymore
# This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2
python3 -m pip install requests==2.27.1 pyyaml==6.0.1

2024-08-20T22:15:55.5773664Z   polling_interval_seconds: 1
2024-08-20T22:15:55.5774035Z   warning_on_retry: true
2024-08-20T22:15:55.5774379Z   continue_on_error: false
2024-08-20T22:15:55.5774724Z env:
2024-08-20T22:15:55.5774993Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:55.5775436Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:55.5776067Z   GITHUB_TOKEN: ***
2024-08-20T22:15:55.5776379Z ##[endgroup]
2024-08-20T22:15:55.6535185Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1
2024-08-20T22:15:55.8920169Z Defaulting to user installation because normal site-packages is not writeable
2024-08-20T22:15:56.0388373Z Collecting requests==2.27.1
2024-08-20T22:15:56.0790150Z   Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB)
2024-08-20T22:15:56.2731510Z Collecting pyyaml==6.0.1
2024-08-20T22:15:56.2795212Z   Downloading PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (738 kB)
2024-08-20T22:15:56.3377670Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10)
2024-08-20T22:15:56.6368215Z Collecting charset-normalizer~=2.0.0
2024-08-20T22:15:56.6432466Z   Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
2024-08-20T22:15:56.6919030Z Collecting certifi>=2017.4.17
2024-08-20T22:15:56.6980345Z   Downloading certifi-2024.7.4-py3-none-any.whl (162 kB)
2024-08-20T22:15:56.7072594Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10)
2024-08-20T22:15:56.7818747Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml
2024-08-20T22:15:56.8980958Z Successfully installed certifi-2024.7.4 charset-normalizer-2.0.12 pyyaml-6.0.1 requests-2.27.1
2024-08-20T22:15:57.6325396Z Command completed after 1 attempt(s).
2024-08-20T22:15:57.6386023Z ##[group]Run set -x
2024-08-20T22:15:57.6386353Z set -x
2024-08-20T22:15:57.6386645Z 
2024-08-20T22:15:57.6387183Z # Use relative path here as this could be checked out anywhere, not necessarily
2024-08-20T22:15:57.6387843Z # in runner workspace
2024-08-20T22:15:57.6388379Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py"
2024-08-20T22:15:57.6398576Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:15:57.6399256Z env:
2024-08-20T22:15:57.6399532Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:57.6399978Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:57.6400463Z ##[endgroup]
2024-08-20T22:15:57.6432794Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py
2024-08-20T22:15:57.6673610Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}"
2024-08-20T22:15:57.6674158Z echo "Workflow: ${GITHUB_WORKFLOW}"
2024-08-20T22:15:57.6674625Z echo "Job name: ${JOB_NAME}"
2024-08-20T22:15:57.6675026Z 
2024-08-20T22:15:57.6675568Z # Use relative path here as this could be checked out anywhere, not necessarily
2024-08-20T22:15:57.6676248Z # in runner workspace
2024-08-20T22:15:57.6676843Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \
2024-08-20T22:15:57.6677493Z   --workflow "${GITHUB_WORKFLOW}" \
2024-08-20T22:15:57.6677967Z   --job-name "${JOB_NAME}" \
2024-08-20T22:15:57.6684615Z   --test-matrix "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \
2024-08-20T22:15:57.6691608Z   --selected-test-configs "" \
2024-08-20T22:15:57.6692082Z   --pr-number "${PR_NUMBER}" \
2024-08-20T22:15:57.6692678Z   --tag "${TAG}" \
2024-08-20T22:15:57.6693087Z   --event-name "${EVENT_NAME}" \
2024-08-20T22:15:57.6693533Z   --schedule "${SCHEDULE}" \
2024-08-20T22:15:57.6693957Z   --branch "${HEAD_BRANCH}"
2024-08-20T22:15:57.6703167Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:15:57.6703668Z env:
2024-08-20T22:15:57.6703938Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:57.6704389Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:57.6705063Z   GITHUB_TOKEN: ***
2024-08-20T22:15:57.6705915Z   JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu)
2024-08-20T22:15:57.6706828Z   PR_NUMBER: 
2024-08-20T22:15:57.6707131Z   TAG: ciflow/inductor/133712
2024-08-20T22:15:57.6707492Z   EVENT_NAME: push
2024-08-20T22:15:57.6707797Z   SCHEDULE: 
2024-08-20T22:15:57.6708075Z   HEAD_BRANCH: 
2024-08-20T22:15:57.6708372Z ##[endgroup]
2024-08-20T22:15:57.6736616Z Workflow: inductor-periodic
2024-08-20T22:15:57.6737650Z Job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu)
2024-08-20T22:15:57.9457112Z INFO:root:Found no test-config label on the PR, so all test configs are included
2024-08-20T22:15:58.1459843Z ##[group]Run echo "Filtered matrix:"
2024-08-20T22:15:58.1460285Z echo "Filtered matrix:"
2024-08-20T22:15:58.1466877Z echo "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}"
2024-08-20T22:15:58.1474219Z 
2024-08-20T22:15:58.1474501Z echo
2024-08-20T22:15:58.1474883Z echo "Is the current job unstable? False"
2024-08-20T22:15:58.1475335Z 
2024-08-20T22:15:58.1475610Z echo
2024-08-20T22:15:58.1475971Z echo "Is keep-going label set? False"
2024-08-20T22:15:58.1476406Z 
2024-08-20T22:15:58.1476677Z echo
2024-08-20T22:15:58.1476996Z echo "Renabled issues? "
2024-08-20T22:15:58.1485960Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:15:58.1486454Z env:
2024-08-20T22:15:58.1486735Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:58.1487192Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:58.1487677Z ##[endgroup]
2024-08-20T22:15:58.1517383Z Filtered matrix:
2024-08-20T22:15:58.1525278Z {include: [{config: dynamo_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}]}
2024-08-20T22:15:58.1532002Z 
2024-08-20T22:15:58.1532159Z Is the current job unstable? False
2024-08-20T22:15:58.1532441Z 
2024-08-20T22:15:58.1532741Z Is keep-going label set? False
2024-08-20T22:15:58.1532991Z 
2024-08-20T22:15:58.1533117Z Renabled issues? 
2024-08-20T22:15:58.1576419Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2024-08-20T22:15:58.1577190Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2024-08-20T22:15:58.1585477Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:15:58.1585966Z env:
2024-08-20T22:15:58.1586238Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:58.1586683Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:58.1587203Z   JOB_TIMEOUT: 240
2024-08-20T22:15:58.1587506Z ##[endgroup]
2024-08-20T22:15:58.1668625Z ##[group]Run set -x
2024-08-20T22:15:58.1669396Z set -x
2024-08-20T22:15:58.1669693Z 
2024-08-20T22:15:58.1670043Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2024-08-20T22:15:58.1670577Z   TEST_COMMAND=.ci/pytorch/multigpu-test.sh
2024-08-20T22:15:58.1671136Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2024-08-20T22:15:58.1671644Z   TEST_COMMAND=.ci/onnx/test.sh
2024-08-20T22:15:58.1672056Z else
2024-08-20T22:15:58.1672396Z   TEST_COMMAND=.ci/pytorch/test.sh
2024-08-20T22:15:58.1672818Z fi
2024-08-20T22:15:58.1673095Z 
2024-08-20T22:15:58.1673559Z # detached container should get cleaned up by teardown_ec2_linux
2024-08-20T22:15:58.1674308Z # TODO: Stop building test binaries as part of the build phase
2024-08-20T22:15:58.1674997Z # Used for GPU_FLAG since that doesn't play nice
2024-08-20T22:15:58.1675574Z # shellcheck disable=SC2086,SC2090
2024-08-20T22:15:58.1676666Z container_name=$(docker run \
2024-08-20T22:15:58.1677212Z   ${GPU_FLAG:-} \
2024-08-20T22:15:58.1677771Z   -e BUILD_ENVIRONMENT \
2024-08-20T22:15:58.1678215Z   -e PR_NUMBER \
2024-08-20T22:15:58.1678674Z   -e GITHUB_ACTIONS \
2024-08-20T22:15:58.1679086Z   -e GITHUB_REPOSITORY \
2024-08-20T22:15:58.1679490Z   -e GITHUB_WORKFLOW \
2024-08-20T22:15:58.1679871Z   -e GITHUB_JOB \
2024-08-20T22:15:58.1680232Z   -e GITHUB_RUN_ID \
2024-08-20T22:15:58.1680608Z   -e GITHUB_RUN_NUMBER \
2024-08-20T22:15:58.1681007Z   -e GITHUB_RUN_ATTEMPT \
2024-08-20T22:15:58.1681399Z   -e JOB_ID \
2024-08-20T22:15:58.1681737Z   -e JOB_NAME \
2024-08-20T22:15:58.1682082Z   -e BASE_SHA \
2024-08-20T22:15:58.1682433Z   -e BRANCH \
2024-08-20T22:15:58.1682763Z   -e SHA1 \
2024-08-20T22:15:58.1683109Z   -e AWS_DEFAULT_REGION \
2024-08-20T22:15:58.1683511Z   -e IN_WHEEL_TEST \
2024-08-20T22:15:58.1683891Z   -e SHARD_NUMBER \
2024-08-20T22:15:58.1684259Z   -e TEST_CONFIG \
2024-08-20T22:15:58.1684624Z   -e NUM_TEST_SHARDS \
2024-08-20T22:15:58.1685018Z   -e REENABLED_ISSUES \
2024-08-20T22:15:58.1685430Z   -e CONTINUE_THROUGH_ERROR \
2024-08-20T22:15:58.1685853Z   -e VERBOSE_TEST_LOGS \
2024-08-20T22:15:58.1686252Z   -e TEST_SHOWLOCALS \
2024-08-20T22:15:58.1686646Z   -e NO_TEST_TIMEOUT \
2024-08-20T22:15:58.1687020Z   -e NO_TD \
2024-08-20T22:15:58.1687367Z   -e TD_DISTRIBUTED \
2024-08-20T22:15:58.1687740Z   -e PR_LABELS \
2024-08-20T22:15:58.1688147Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2024-08-20T22:15:58.1688600Z   -e SCCACHE_BUCKET \
2024-08-20T22:15:58.1688992Z   -e SCCACHE_S3_KEY_PREFIX \
2024-08-20T22:15:58.1689394Z   -e XLA_CUDA \
2024-08-20T22:15:58.1689997Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2024-08-20T22:15:58.1690508Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2024-08-20T22:15:58.1691024Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2024-08-20T22:15:58.1691530Z   -e SKIP_SCCACHE_INITIALIZATION=1 \
2024-08-20T22:15:58.1692000Z   -e HUGGING_FACE_HUB_TOKEN \
2024-08-20T22:15:58.1692453Z   -e SCRIBE_GRAPHQL_ACCESS_TOKEN \
2024-08-20T22:15:58.1692897Z   -e DASHBOARD_TAG \
2024-08-20T22:15:58.1693365Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2024-08-20T22:15:58.1693924Z   --security-opt seccomp=unconfined \
2024-08-20T22:15:58.1694388Z   --cap-add=SYS_PTRACE \
2024-08-20T22:15:58.1694778Z   --ipc=host \
2024-08-20T22:15:58.1695129Z   --shm-size="${SHM_SIZE}" \
2024-08-20T22:15:58.1695526Z   --tty \
2024-08-20T22:15:58.1695845Z   --detach \
2024-08-20T22:15:58.1696353Z   --name="${container_name}" \
2024-08-20T22:15:58.1696784Z   --user jenkins \
2024-08-20T22:15:58.1697273Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2024-08-20T22:15:58.1697826Z   -w /var/lib/jenkins/workspace \
2024-08-20T22:15:58.1698250Z   "${DOCKER_IMAGE}"
2024-08-20T22:15:58.1698665Z )
2024-08-20T22:15:58.1699075Z # Propagate download.pytorch.org IP to container
2024-08-20T22:15:58.1699995Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts"
2024-08-20T22:15:58.1700958Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}"
2024-08-20T22:15:58.1701867Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"
2024-08-20T22:15:58.1710893Z shell: /usr/bin/bash -e {0}
2024-08-20T22:15:58.1711241Z env:
2024-08-20T22:15:58.1711515Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:58.1711968Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:58.1712563Z   BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86
2024-08-20T22:15:58.1713066Z   PR_NUMBER: 
2024-08-20T22:15:58.1713388Z   GITHUB_REPOSITORY: pytorch/pytorch
2024-08-20T22:15:58.1713813Z   GITHUB_WORKFLOW: inductor-periodic
2024-08-20T22:15:58.1714212Z   GITHUB_JOB: test
2024-08-20T22:15:58.1714526Z   GITHUB_RUN_ID: 10479310721
2024-08-20T22:15:58.1714882Z   GITHUB_RUN_NUMBER: 54565
2024-08-20T22:15:58.1715231Z   GITHUB_RUN_ATTEMPT: 1
2024-08-20T22:15:58.1715563Z   JOB_ID: 29026300273
2024-08-20T22:15:58.1716434Z   JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu)
2024-08-20T22:15:58.1717352Z   BRANCH: 
2024-08-20T22:15:58.1717694Z   SHA1: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:15:58.1718253Z   BASE_SHA: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:15:58.1718777Z   TEST_CONFIG: dynamic_aot_eager_huggingface
2024-08-20T22:15:58.1719210Z   SHARD_NUMBER: 1
2024-08-20T22:15:58.1719519Z   NUM_TEST_SHARDS: 1
2024-08-20T22:15:58.1719844Z   REENABLED_ISSUES: 
2024-08-20T22:15:58.1720183Z   CONTINUE_THROUGH_ERROR: False
2024-08-20T22:15:58.1720562Z   VERBOSE_TEST_LOGS: False
2024-08-20T22:15:58.1720921Z   TEST_SHOWLOCALS: False
2024-08-20T22:15:58.1721268Z   NO_TEST_TIMEOUT: False
2024-08-20T22:15:58.1721599Z   NO_TD: False
2024-08-20T22:15:58.1721893Z   TD_DISTRIBUTED: False
2024-08-20T22:15:58.1722309Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2024-08-20T22:15:58.1722830Z   SCCACHE_S3_KEY_PREFIX: inductor-periodic
2024-08-20T22:15:58.1723255Z   SHM_SIZE: 2g
2024-08-20T22:15:58.1724333Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:15:58.1725498Z   XLA_CUDA: 
2024-08-20T22:15:58.1725974Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2024-08-20T22:15:58.1726699Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2024-08-20T22:15:58.1727135Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2024-08-20T22:15:58.1727544Z   DASHBOARD_TAG: 
2024-08-20T22:15:58.1728111Z   HUGGING_FACE_HUB_TOKEN: ***
2024-08-20T22:15:58.1728488Z   SCRIBE_GRAPHQL_ACCESS_TOKEN: 
2024-08-20T22:15:58.1728856Z ##[endgroup]
2024-08-20T22:15:58.1758777Z + [[ dynamic_aot_eager_huggingface == \m\u\l\t\i\g\p\u ]]
2024-08-20T22:15:58.1759618Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]]
2024-08-20T22:15:58.1760126Z + TEST_COMMAND=.ci/pytorch/test.sh
2024-08-20T22:15:58.1767638Z +++ nproc --ignore=2
2024-08-20T22:15:58.1809383Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10479310721 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:16:09.6042021Z + container_name=0069e58a274b850ea9dc89e5a84233030f0d0aca44320cdb9963abb2a7d1ef4f
2024-08-20T22:16:09.6044560Z + grep download.pytorch.org /etc/hosts
2024-08-20T22:16:09.6046376Z + docker exec -i 0069e58a274b850ea9dc89e5a84233030f0d0aca44320cdb9963abb2a7d1ef4f sudo bash -c '/bin/cat >> /etc/hosts'
2024-08-20T22:16:09.7238194Z + echo DOCKER_CONTAINER_ID=0069e58a274b850ea9dc89e5a84233030f0d0aca44320cdb9963abb2a7d1ef4f
2024-08-20T22:16:09.7243106Z ++ echo dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl
2024-08-20T22:16:09.7246231Z + docker exec -t 0069e58a274b850ea9dc89e5a84233030f0d0aca44320cdb9963abb2a7d1ef4f sh -c 'pip install dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh'
2024-08-20T22:16:10.1214789Z Processing ./dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+git40ec5f6)
2024-08-20T22:16:10.4449579Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.13.1)
2024-08-20T22:16:10.4452430Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (4.12.2)
2024-08-20T22:16:10.4454610Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.8.8)
2024-08-20T22:16:10.4457140Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.1.4)
2024-08-20T22:16:10.4460844Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2024.6.1)
2024-08-20T22:16:10.4466230Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.13.1)
2024-08-20T22:16:10.4483019Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.3.0)
2024-08-20T22:16:10.4496733Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.3.0)
2024-08-20T22:16:10.4514355Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.22.4)
2024-08-20T22:16:10.4901851Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.1.5)
2024-08-20T22:16:11.0675115Z Installing collected packages: torch
2024-08-20T22:16:20.8218770Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
2024-08-20T22:16:20.8220656Z timm 1.0.9.dev0 requires torchvision, which is not installed.
2024-08-20T22:16:20.8221459Z Successfully installed torch-2.5.0a0+git40ec5f6
2024-08-20T22:16:20.9038169Z + export TERM=vt100
2024-08-20T22:16:20.9038565Z + TERM=vt100
2024-08-20T22:16:20.9038886Z ++ dirname .ci/pytorch/test.sh
2024-08-20T22:16:20.9048665Z + source .ci/pytorch/common.sh
2024-08-20T22:16:20.9053329Z +++ dirname .ci/pytorch/common.sh
2024-08-20T22:16:20.9063992Z ++ source .ci/pytorch/common_utils.sh
2024-08-20T22:16:20.9065935Z +++ declare -f -t trap_add
2024-08-20T22:16:20.9072342Z ++ set -ex
2024-08-20T22:16:20.9072973Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]]
2024-08-20T22:16:20.9073519Z ++ BUILD_TEST_LIBTORCH=0
2024-08-20T22:16:20.9074097Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]]
2024-08-20T22:16:20.9077328Z ++ stat -c %u /var/lib/jenkins/workspace
2024-08-20T22:16:20.9098271Z + WORKSPACE_ORIGINAL_OWNER_ID=1000
2024-08-20T22:16:20.9105294Z + trap_add cleanup_workspace EXIT
2024-08-20T22:16:20.9105967Z + trap_add_cmd=cleanup_workspace
2024-08-20T22:16:20.9106454Z + shift
2024-08-20T22:16:20.9106822Z + for trap_add_name in "$@"
2024-08-20T22:16:20.9109511Z +++ trap -p EXIT
2024-08-20T22:16:20.9110031Z ++ eval 'extract_trap_cmd '
2024-08-20T22:16:20.9110523Z +++ extract_trap_cmd
2024-08-20T22:16:20.9110949Z +++ printf '%s\n' ''
2024-08-20T22:16:20.9111348Z ++ printf '%s\n' cleanup_workspace
2024-08-20T22:16:20.9113874Z + trap -- '
2024-08-20T22:16:20.9114291Z cleanup_workspace' EXIT
2024-08-20T22:16:20.9114791Z + sudo chown -R jenkins /var/lib/jenkins/workspace
2024-08-20T22:16:21.6154347Z + git config --global --add safe.directory /var/lib/jenkins/workspace
2024-08-20T22:16:21.6177838Z + echo 'Environment variables:'
2024-08-20T22:16:21.6178374Z Environment variables:
2024-08-20T22:16:21.6178890Z + env
2024-08-20T22:16:21.6189343Z INSTALLED_DB=yes
2024-08-20T22:16:21.6189839Z NV_LIBCUBLAS_VERSION=12.1.3.1-1
2024-08-20T22:16:21.6190313Z NVIDIA_VISIBLE_DEVICES=all
2024-08-20T22:16:21.6190864Z NV_NVML_DEV_VERSION=12.1.105-1
2024-08-20T22:16:21.6191590Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-08-20T22:16:21.6192164Z CONTINUE_THROUGH_ERROR=False
2024-08-20T22:16:21.6192707Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1
2024-08-20T22:16:21.6193311Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1
2024-08-20T22:16:21.6194075Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86
2024-08-20T22:16:21.6194673Z HOSTNAME=0069e58a274b
2024-08-20T22:16:21.6195519Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_7f0c3573-2fe6-440d-a041-3befcdd7c05d
2024-08-20T22:16:21.6196593Z GITHUB_ACTION=__self
2024-08-20T22:16:21.6197025Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2024-08-20T22:16:21.6200915Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526
2024-08-20T22:16:21.6205100Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1
2024-08-20T22:16:21.6205655Z NV_NVTX_VERSION=12.1.105-1
2024-08-20T22:16:21.6206007Z GITHUB_RUN_NUMBER=54565
2024-08-20T22:16:21.6206382Z TEST_CONFIG=dynamic_aot_eager_huggingface
2024-08-20T22:16:21.6206821Z GITHUB_REPOSITORY_OWNER_ID=21003710
2024-08-20T22:16:21.6207292Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2024-08-20T22:16:21.6207750Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1
2024-08-20T22:16:21.6208354Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1
2024-08-20T22:16:21.6208769Z SCRIBE_GRAPHQL_ACCESS_TOKEN=
2024-08-20T22:16:21.6209162Z NV_LIBNPP_VERSION=12.1.0.40-1
2024-08-20T22:16:21.6209611Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot]
2024-08-20T22:16:21.6210099Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2024-08-20T22:16:21.6210558Z GITHUB_REF_TYPE=tag
2024-08-20T22:16:21.6210874Z TORCH_CUDA_ARCH_LIST=Maxwell
2024-08-20T22:16:21.6211252Z NCCL_VERSION=2.17.1-1
2024-08-20T22:16:21.6211640Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:16:21.6212079Z XLA_CUDA=
2024-08-20T22:16:21.6212566Z HUGGING_FACE_HUB_TOKEN=***
2024-08-20T22:16:21.6213195Z ***
2024-08-20T22:16:21.6213484Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2024-08-20T22:16:21.6213883Z GITHUB_REPOSITORY_ID=65600975
2024-08-20T22:16:21.6214241Z GITHUB_ACTIONS=true
2024-08-20T22:16:21.6214575Z NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:16:21.6215077Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1
2024-08-20T22:16:21.6215610Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1
2024-08-20T22:16:21.6216097Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:16:21.6216617Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2024-08-20T22:16:21.6217100Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:16:21.6218031Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/133712
2024-08-20T22:16:21.6218889Z UCC_HOME=/usr
2024-08-20T22:16:21.6219232Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1
2024-08-20T22:16:21.6219633Z VERBOSE_TEST_LOGS=False
2024-08-20T22:16:21.6219970Z NVIDIA_PRODUCT_NAME=CUDA
2024-08-20T22:16:21.6220434Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1
2024-08-20T22:16:21.6220931Z GITHUB_REF=refs/tags/ciflow/inductor/133712
2024-08-20T22:16:21.6221403Z NV_CUDA_CUDART_VERSION=12.1.105-1
2024-08-20T22:16:21.6221777Z SHARD_NUMBER=1
2024-08-20T22:16:21.6222090Z GITHUB_REF_PROTECTED=false
2024-08-20T22:16:21.6222439Z HOME=/var/lib/jenkins
2024-08-20T22:16:21.6222796Z GITHUB_API_URL=https://api.github.com
2024-08-20T22:16:21.6223246Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2024-08-20T22:16:21.6223724Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb
2024-08-20T22:16:21.6224278Z SCCACHE_S3_KEY_PREFIX=inductor-periodic
2024-08-20T22:16:21.6224684Z CUDA_VERSION=12.1.1
2024-08-20T22:16:21.6225113Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1
2024-08-20T22:16:21.6225550Z NUM_TEST_SHARDS=1
2024-08-20T22:16:21.6225856Z UCX_HOME=/usr
2024-08-20T22:16:21.6226371Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1
2024-08-20T22:16:21.6227453Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_7f0c3573-2fe6-440d-a041-3befcdd7c05d
2024-08-20T22:16:21.6228926Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu)
2024-08-20T22:16:21.6230351Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_7f0c3573-2fe6-440d-a041-3befcdd7c05d
2024-08-20T22:16:21.6231458Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2024-08-20T22:16:21.6232213Z GITHUB_EVENT_NAME=push
2024-08-20T22:16:21.6232542Z DASHBOARD_TAG=
2024-08-20T22:16:21.6232847Z GITHUB_RUN_ID=10479310721
2024-08-20T22:16:21.6233305Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1
2024-08-20T22:16:21.6233844Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1
2024-08-20T22:16:21.6234845Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_7f0c3573-2fe6-440d-a041-3befcdd7c05d
2024-08-20T22:16:21.6235753Z GITHUB_ACTOR=pytorch-bot[bot]
2024-08-20T22:16:21.6236155Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1
2024-08-20T22:16:21.6236533Z PR_NUMBER=
2024-08-20T22:16:21.6236815Z GITHUB_RUN_ATTEMPT=1
2024-08-20T22:16:21.6237148Z ANACONDA_PYTHON_VERSION=3.10
2024-08-20T22:16:21.6237595Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2024-08-20T22:16:21.6238053Z TERM=vt100
2024-08-20T22:16:21.6238390Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1
2024-08-20T22:16:21.6238900Z INSTALLED_VISION=yes
2024-08-20T22:16:21.6239216Z BRANCH=
2024-08-20T22:16:21.6239502Z OPENSSL_ROOT_DIR=/opt/openssl
2024-08-20T22:16:21.6239902Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2024-08-20T22:16:21.6240323Z CUDA_PATH=/usr/local/cuda
2024-08-20T22:16:21.6241091Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2024-08-20T22:16:21.6241837Z GITHUB_SERVER_URL=https://github.com
2024-08-20T22:16:21.6242326Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b
2024-08-20T22:16:21.6242787Z REENABLED_ISSUES=
2024-08-20T22:16:21.6243086Z SHLVL=1
2024-08-20T22:16:21.6243350Z MAX_JOBS=14
2024-08-20T22:16:21.6243660Z NV_CUDA_LIB_VERSION=12.1.1-1
2024-08-20T22:16:21.6244013Z NVARCH=x86_64
2024-08-20T22:16:21.6244311Z GITHUB_ACTOR_ID=54816060
2024-08-20T22:16:21.6244768Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:16:21.6245297Z GITHUB_REF_NAME=ciflow/inductor/133712
2024-08-20T22:16:21.6245797Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1
2024-08-20T22:16:21.6246472Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2024-08-20T22:16:21.6247037Z GITHUB_JOB=test
2024-08-20T22:16:21.6247439Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1
2024-08-20T22:16:21.6247995Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2024-08-20T22:16:21.6248506Z NO_TEST_TIMEOUT=False
2024-08-20T22:16:21.6248827Z TD_DISTRIBUTED=False
2024-08-20T22:16:21.6249212Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1
2024-08-20T22:16:21.6249653Z GITHUB_REPOSITORY=pytorch/pytorch
2024-08-20T22:16:21.6250075Z NV_NVPROF_VERSION=12.1.105-1
2024-08-20T22:16:21.6250435Z GITHUB_RETENTION_DAYS=90
2024-08-20T22:16:21.6250782Z OPENSSL_DIR=/opt/openssl
2024-08-20T22:16:21.6251131Z GITHUB_ACTION_REPOSITORY=
2024-08-20T22:16:21.6252167Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-08-20T22:16:21.6253257Z GITHUB_BASE_REF=
2024-08-20T22:16:21.6253591Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2024-08-20T22:16:21.6253961Z CI=true
2024-08-20T22:16:21.6254277Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1
2024-08-20T22:16:21.6254683Z GITHUB_REPOSITORY_OWNER=pytorch
2024-08-20T22:16:21.6255053Z JOB_ID=29026300273
2024-08-20T22:16:21.6255365Z INSTALLED_PROTOBUF=yes
2024-08-20T22:16:21.6255689Z GITHUB_HEAD_REF=
2024-08-20T22:16:21.6255994Z GITHUB_ACTION_REF=
2024-08-20T22:16:21.6256432Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2024-08-20T22:16:21.6256903Z TEST_SHOWLOCALS=False
2024-08-20T22:16:21.6257293Z GITHUB_WORKFLOW=inductor-periodic
2024-08-20T22:16:21.6257699Z DEBIAN_FRONTEND=noninteractive
2024-08-20T22:16:21.6258699Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_7f0c3573-2fe6-440d-a041-3befcdd7c05d
2024-08-20T22:16:21.6259510Z NO_TD=False
2024-08-20T22:16:21.6259807Z SKIP_SCCACHE_INITIALIZATION=1
2024-08-20T22:16:21.6260170Z _=/usr/bin/env
2024-08-20T22:16:21.6260659Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2024-08-20T22:16:21.6427102Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch
2024-08-20T22:16:21.6428299Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2024-08-20T22:16:21.6429321Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib
2024-08-20T22:16:21.6430355Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test
2024-08-20T22:16:21.6431054Z + BUILD_DIR=build
2024-08-20T22:16:21.6431376Z + BUILD_RENAMED_DIR=build_renamed
2024-08-20T22:16:21.6431776Z + BUILD_BIN_DIR=build/bin
2024-08-20T22:16:21.6432114Z + SHARD_NUMBER=1
2024-08-20T22:16:21.6432415Z + NUM_TEST_SHARDS=1
2024-08-20T22:16:21.6432724Z + export VALGRIND=ON
2024-08-20T22:16:21.6433039Z + VALGRIND=ON
2024-08-20T22:16:21.6433574Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]]
2024-08-20T22:16:21.6434180Z + [[ 0 == \1 ]]
2024-08-20T22:16:21.6434476Z + [[ False == \1 ]]
2024-08-20T22:16:21.6435215Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]]
2024-08-20T22:16:21.6435764Z ++ realpath build/custom_test_artifacts
2024-08-20T22:16:21.6446770Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2024-08-20T22:16:21.6447650Z + [[ -n '' ]]
2024-08-20T22:16:21.6448016Z + echo 'Environment variables'
2024-08-20T22:16:21.6448393Z Environment variables
2024-08-20T22:16:21.6448700Z + env
2024-08-20T22:16:21.6455757Z INSTALLED_DB=yes
2024-08-20T22:16:21.6456297Z NV_LIBCUBLAS_VERSION=12.1.3.1-1
2024-08-20T22:16:21.6456811Z NVIDIA_VISIBLE_DEVICES=all
2024-08-20T22:16:21.6457318Z NV_NVML_DEV_VERSION=12.1.105-1
2024-08-20T22:16:21.6458096Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-08-20T22:16:21.6458930Z CONTINUE_THROUGH_ERROR=False
2024-08-20T22:16:21.6459623Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1
2024-08-20T22:16:21.6460288Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1
2024-08-20T22:16:21.6461042Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86
2024-08-20T22:16:21.6461542Z HOSTNAME=0069e58a274b
2024-08-20T22:16:21.6462381Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_7f0c3573-2fe6-440d-a041-3befcdd7c05d
2024-08-20T22:16:21.6463407Z GITHUB_ACTION=__self
2024-08-20T22:16:21.6463790Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2024-08-20T22:16:21.6467966Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526
2024-08-20T22:16:21.6472485Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1
2024-08-20T22:16:21.6473005Z NV_NVTX_VERSION=12.1.105-1
2024-08-20T22:16:21.6473352Z GITHUB_RUN_NUMBER=54565
2024-08-20T22:16:21.6473721Z TEST_CONFIG=dynamic_aot_eager_huggingface
2024-08-20T22:16:21.6474149Z GITHUB_REPOSITORY_OWNER_ID=21003710
2024-08-20T22:16:21.6474620Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2024-08-20T22:16:21.6475076Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1
2024-08-20T22:16:21.6475503Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1
2024-08-20T22:16:21.6475901Z SCRIBE_GRAPHQL_ACCESS_TOKEN=
2024-08-20T22:16:21.6476294Z NV_LIBNPP_VERSION=12.1.0.40-1
2024-08-20T22:16:21.6476736Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot]
2024-08-20T22:16:21.6477218Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2024-08-20T22:16:21.6477674Z GITHUB_REF_TYPE=tag
2024-08-20T22:16:21.6477992Z TORCH_CUDA_ARCH_LIST=Maxwell
2024-08-20T22:16:21.6478579Z NCCL_VERSION=2.17.1-1
2024-08-20T22:16:21.6478961Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:16:21.6479401Z XLA_CUDA=
2024-08-20T22:16:21.6479852Z HUGGING_FACE_HUB_TOKEN=***
2024-08-20T22:16:21.6480230Z ***
2024-08-20T22:16:21.6480500Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2024-08-20T22:16:21.6480889Z GITHUB_REPOSITORY_ID=65600975
2024-08-20T22:16:21.6481246Z GITHUB_ACTIONS=true
2024-08-20T22:16:21.6481572Z NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:16:21.6482061Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1
2024-08-20T22:16:21.6482591Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1
2024-08-20T22:16:21.6483056Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:16:21.6483568Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2024-08-20T22:16:21.6484040Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:16:21.6484960Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/133712
2024-08-20T22:16:21.6485886Z UCC_HOME=/usr
2024-08-20T22:16:21.6486236Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1
2024-08-20T22:16:21.6486623Z VERBOSE_TEST_LOGS=False
2024-08-20T22:16:21.6486963Z NVIDIA_PRODUCT_NAME=CUDA
2024-08-20T22:16:21.6487417Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1
2024-08-20T22:16:21.6487899Z GITHUB_REF=refs/tags/ciflow/inductor/133712
2024-08-20T22:16:21.6488372Z NV_CUDA_CUDART_VERSION=12.1.105-1
2024-08-20T22:16:21.6488742Z SHARD_NUMBER=1
2024-08-20T22:16:21.6489045Z GITHUB_REF_PROTECTED=false
2024-08-20T22:16:21.6489383Z HOME=/var/lib/jenkins
2024-08-20T22:16:21.6489737Z GITHUB_API_URL=https://api.github.com
2024-08-20T22:16:21.6490166Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2024-08-20T22:16:21.6490640Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb
2024-08-20T22:16:21.6491177Z SCCACHE_S3_KEY_PREFIX=inductor-periodic
2024-08-20T22:16:21.6491582Z CUDA_VERSION=12.1.1
2024-08-20T22:16:21.6492000Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1
2024-08-20T22:16:21.6492424Z NUM_TEST_SHARDS=1
2024-08-20T22:16:21.6492732Z UCX_HOME=/usr
2024-08-20T22:16:21.6493248Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1
2024-08-20T22:16:21.6494319Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_7f0c3573-2fe6-440d-a041-3befcdd7c05d
2024-08-20T22:16:21.6495757Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu)
2024-08-20T22:16:21.6497170Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_7f0c3573-2fe6-440d-a041-3befcdd7c05d
2024-08-20T22:16:21.6498272Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2024-08-20T22:16:21.6499108Z GITHUB_EVENT_NAME=push
2024-08-20T22:16:21.6499437Z DASHBOARD_TAG=
2024-08-20T22:16:21.6499728Z GITHUB_RUN_ID=10479310721
2024-08-20T22:16:21.6500189Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1
2024-08-20T22:16:21.6500722Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1
2024-08-20T22:16:21.6501701Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_7f0c3573-2fe6-440d-a041-3befcdd7c05d
2024-08-20T22:16:21.6502651Z GITHUB_ACTOR=pytorch-bot[bot]
2024-08-20T22:16:21.6503049Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1
2024-08-20T22:16:21.6503420Z PR_NUMBER=
2024-08-20T22:16:21.6503690Z GITHUB_RUN_ATTEMPT=1
2024-08-20T22:16:21.6504050Z VALGRIND=ON
2024-08-20T22:16:21.6504388Z ANACONDA_PYTHON_VERSION=3.10
2024-08-20T22:16:21.6504826Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2024-08-20T22:16:21.6505270Z TERM=vt100
2024-08-20T22:16:21.6505608Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1
2024-08-20T22:16:21.6506016Z INSTALLED_VISION=yes
2024-08-20T22:16:21.6506316Z BRANCH=
2024-08-20T22:16:21.6506595Z OPENSSL_ROOT_DIR=/opt/openssl
2024-08-20T22:16:21.6506998Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2024-08-20T22:16:21.6507415Z CUDA_PATH=/usr/local/cuda
2024-08-20T22:16:21.6508160Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2024-08-20T22:16:21.6509025Z GITHUB_SERVER_URL=https://github.com
2024-08-20T22:16:21.6509585Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b
2024-08-20T22:16:21.6510116Z REENABLED_ISSUES=
2024-08-20T22:16:21.6510405Z SHLVL=1
2024-08-20T22:16:21.6510659Z MAX_JOBS=14
2024-08-20T22:16:21.6510980Z NV_CUDA_LIB_VERSION=12.1.1-1
2024-08-20T22:16:21.6511317Z NVARCH=x86_64
2024-08-20T22:16:21.6511607Z GITHUB_ACTOR_ID=54816060
2024-08-20T22:16:21.6512056Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:16:21.6512590Z GITHUB_REF_NAME=ciflow/inductor/133712
2024-08-20T22:16:21.6513076Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1
2024-08-20T22:16:21.6513732Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2024-08-20T22:16:21.6514295Z GITHUB_JOB=test
2024-08-20T22:16:21.6514700Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1
2024-08-20T22:16:21.6515241Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2024-08-20T22:16:21.6515861Z NO_TEST_TIMEOUT=False
2024-08-20T22:16:21.6516200Z TD_DISTRIBUTED=False
2024-08-20T22:16:21.6516576Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1
2024-08-20T22:16:21.6517018Z GITHUB_REPOSITORY=pytorch/pytorch
2024-08-20T22:16:21.6517433Z NV_NVPROF_VERSION=12.1.105-1
2024-08-20T22:16:21.6517793Z GITHUB_RETENTION_DAYS=90
2024-08-20T22:16:21.6518133Z OPENSSL_DIR=/opt/openssl
2024-08-20T22:16:21.6518471Z GITHUB_ACTION_REPOSITORY=
2024-08-20T22:16:21.6519505Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-08-20T22:16:21.6520561Z GITHUB_BASE_REF=
2024-08-20T22:16:21.6520884Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2024-08-20T22:16:21.6521241Z CI=true
2024-08-20T22:16:21.6521552Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1
2024-08-20T22:16:21.6521959Z GITHUB_REPOSITORY_OWNER=pytorch
2024-08-20T22:16:21.6522319Z JOB_ID=29026300273
2024-08-20T22:16:21.6522626Z INSTALLED_PROTOBUF=yes
2024-08-20T22:16:21.6522953Z GITHUB_HEAD_REF=
2024-08-20T22:16:21.6523254Z GITHUB_ACTION_REF=
2024-08-20T22:16:21.6523685Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2024-08-20T22:16:21.6524140Z TEST_SHOWLOCALS=False
2024-08-20T22:16:21.6524527Z GITHUB_WORKFLOW=inductor-periodic
2024-08-20T22:16:21.6524924Z DEBIAN_FRONTEND=noninteractive
2024-08-20T22:16:21.6525810Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_7f0c3573-2fe6-440d-a041-3befcdd7c05d
2024-08-20T22:16:21.6526604Z NO_TD=False
2024-08-20T22:16:21.6526905Z SKIP_SCCACHE_INITIALIZATION=1
2024-08-20T22:16:21.6527260Z _=/usr/bin/env
2024-08-20T22:16:21.6527593Z + echo 'Testing pytorch'
2024-08-20T22:16:21.6527929Z Testing pytorch
2024-08-20T22:16:21.6528250Z + export LANG=C.UTF-8
2024-08-20T22:16:21.6528576Z + LANG=C.UTF-8
2024-08-20T22:16:21.6528862Z + PR_NUMBER=
2024-08-20T22:16:21.6529232Z + [[ dynamic_aot_eager_huggingface == \d\e\f\a\u\l\t ]]
2024-08-20T22:16:21.6529815Z + [[ dynamic_aot_eager_huggingface == \d\i\s\t\r\i\b\u\t\e\d ]]
2024-08-20T22:16:21.6530376Z + [[ dynamic_aot_eager_huggingface == \s\l\o\w ]]
2024-08-20T22:16:21.6531039Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]]
2024-08-20T22:16:21.6531725Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]]
2024-08-20T22:16:21.6532284Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2024-08-20T22:16:21.6532779Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2024-08-20T22:16:21.6533251Z + [[ dynamic_aot_eager_huggingface == *crossref* ]]
2024-08-20T22:16:21.6533855Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]]
2024-08-20T22:16:21.6534487Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]]
2024-08-20T22:16:21.6535126Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]]
2024-08-20T22:16:21.6535685Z + pip_install --user ninja==1.10.2
2024-08-20T22:16:21.6536216Z + pip install --progress-bar off --user ninja==1.10.2
2024-08-20T22:16:22.9709691Z Collecting ninja==1.10.2
2024-08-20T22:16:22.9898202Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB)
2024-08-20T22:16:23.0453522Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2024-08-20T22:16:23.6557346Z Installing collected packages: ninja
2024-08-20T22:16:23.6636386Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2024-08-20T22:16:23.6637677Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2024-08-20T22:16:23.7061378Z Successfully installed ninja-1.10.2
2024-08-20T22:16:23.7864582Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-08-20T22:16:23.7867077Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-08-20T22:16:23.7868697Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]]
2024-08-20T22:16:23.7869465Z + install_tlparse
2024-08-20T22:16:23.7869870Z + pip_install --user tlparse==0.3.25
2024-08-20T22:16:23.7870426Z + pip install --progress-bar off --user tlparse==0.3.25
2024-08-20T22:16:24.2264839Z Collecting tlparse==0.3.25
2024-08-20T22:16:24.2434321Z   Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB)
2024-08-20T22:16:24.3052987Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB)
2024-08-20T22:16:24.9718732Z Installing collected packages: tlparse
2024-08-20T22:16:25.0446410Z Successfully installed tlparse-0.3.25
2024-08-20T22:16:25.1223789Z ++ python -m site --user-base
2024-08-20T22:16:25.1465306Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-08-20T22:16:25.1467316Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]]
2024-08-20T22:16:25.1468054Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]]
2024-08-20T22:16:25.1469127Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]]
2024-08-20T22:16:25.1470175Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass'
2024-08-20T22:16:25.1471331Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass
2024-08-20T22:16:25.1472028Z + cd test
2024-08-20T22:16:25.1472558Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)'
2024-08-20T22:16:26.7596661Z + [[ dynamic_aot_eager_huggingface == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2024-08-20T22:16:26.7597367Z + [[ dynamic_aot_eager_huggingface == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2024-08-20T22:16:26.7600726Z + DYNAMO_BENCHMARK_FLAGS=()
2024-08-20T22:16:26.7601375Z + [[ dynamic_aot_eager_huggingface == *pr_time_benchmarks* ]]
2024-08-20T22:16:26.7602196Z + [[ dynamic_aot_eager_huggingface == *dynamo_eager* ]]
2024-08-20T22:16:26.7602789Z + [[ dynamic_aot_eager_huggingface == *aot_eager* ]]
2024-08-20T22:16:26.7603550Z + DYNAMO_BENCHMARK_FLAGS+=(--backend aot_eager)
2024-08-20T22:16:26.7604055Z + [[ dynamic_aot_eager_huggingface == *dynamic* ]]
2024-08-20T22:16:26.7604710Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only)
2024-08-20T22:16:26.7605293Z + [[ dynamic_aot_eager_huggingface == *cpu* ]]
2024-08-20T22:16:26.7605818Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda)
2024-08-20T22:16:26.7634815Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]]
2024-08-20T22:16:26.7635500Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]]
2024-08-20T22:16:26.7637730Z + cd test
2024-08-20T22:16:26.7638254Z + python -c 'import torch; print(torch.__config__.show())'
2024-08-20T22:16:28.2327788Z PyTorch built with:
2024-08-20T22:16:28.2328494Z   - GCC 9.4
2024-08-20T22:16:28.2328850Z   - C++ Version: 201703
2024-08-20T22:16:28.2330224Z   - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2024-08-20T22:16:28.2331301Z   - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67)
2024-08-20T22:16:28.2331965Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2024-08-20T22:16:28.2332471Z   - LAPACK is enabled (usually provided by MKL)
2024-08-20T22:16:28.2332951Z   - NNPACK is enabled
2024-08-20T22:16:28.2333332Z   - CPU capability usage: AVX2
2024-08-20T22:16:28.2333745Z   - CUDA Runtime 12.1
2024-08-20T22:16:28.2334259Z   - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86
2024-08-20T22:16:28.2334871Z   - CuDNN 90.1  (built against CUDA 12.4)
2024-08-20T22:16:28.2335314Z   - Magma 2.6.1
2024-08-20T22:16:28.2342406Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 
2024-08-20T22:16:28.2348749Z 
2024-08-20T22:16:28.5516512Z + cd test
2024-08-20T22:16:28.5517278Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2024-08-20T22:16:29.8892306Z ATen/Parallel:
2024-08-20T22:16:29.8892717Z 	at::get_num_threads() : 8
2024-08-20T22:16:29.8893113Z 	at::get_num_interop_threads() : 16
2024-08-20T22:16:29.8893590Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2024-08-20T22:16:29.8894058Z 	omp_get_max_threads() : 8
2024-08-20T22:16:29.8895148Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2024-08-20T22:16:29.8895940Z 	mkl_get_max_threads() : 8
2024-08-20T22:16:29.8896558Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67)
2024-08-20T22:16:29.8897221Z std::thread::hardware_concurrency() : 16
2024-08-20T22:16:29.8897701Z Environment variables:
2024-08-20T22:16:29.8898044Z 	OMP_NUM_THREADS : [not set]
2024-08-20T22:16:29.8898515Z 	MKL_NUM_THREADS : [not set]
2024-08-20T22:16:29.8898922Z ATen parallel backend: OpenMP
2024-08-20T22:16:29.8899185Z 
2024-08-20T22:16:30.1683092Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]]
2024-08-20T22:16:30.1683886Z + [[ dynamic_aot_eager_huggingface == *backward* ]]
2024-08-20T22:16:30.1684575Z + [[ dynamic_aot_eager_huggingface == *xla* ]]
2024-08-20T22:16:30.1685311Z + [[ dynamic_aot_eager_huggingface == *executorch* ]]
2024-08-20T22:16:30.1686085Z + [[ dynamic_aot_eager_huggingface == \j\i\t\_\l\e\g\a\c\y ]]
2024-08-20T22:16:30.1686976Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]]
2024-08-20T22:16:30.1687730Z + [[ dynamic_aot_eager_huggingface == distributed ]]
2024-08-20T22:16:30.1688483Z + [[ dynamic_aot_eager_huggingface == *inductor_distributed* ]]
2024-08-20T22:16:30.1689375Z + [[ dynamic_aot_eager_huggingface == *inductor-halide* ]]
2024-08-20T22:16:30.1690234Z + [[ dynamic_aot_eager_huggingface == *inductor-micro-benchmark* ]]
2024-08-20T22:16:30.1690846Z + [[ dynamic_aot_eager_huggingface == *huggingface* ]]
2024-08-20T22:16:30.1691677Z + install_torchvision
2024-08-20T22:16:30.1692006Z + local orig_preload
2024-08-20T22:16:30.1692319Z + local commit
2024-08-20T22:16:30.1692626Z ++ get_pinned_commit vision
2024-08-20T22:16:30.1693016Z ++ cat .github/ci_commit_pins/vision.txt
2024-08-20T22:16:30.1708520Z + commit=d23a6e1664d20707c11781299611436e1f0c104f
2024-08-20T22:16:30.1708973Z + orig_preload=
2024-08-20T22:16:30.1709335Z + '[' -n '' ']'
2024-08-20T22:16:30.1710444Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-08-20T22:16:30.1711974Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-08-20T22:16:30.5058532Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-08-20T22:16:30.5062296Z   Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-x1ut7h5o
2024-08-20T22:16:30.5093914Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-x1ut7h5o
2024-08-20T22:16:32.0049788Z   Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f'
2024-08-20T22:16:32.0077121Z   Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f
2024-08-20T22:16:33.3367905Z   Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f
2024-08-20T22:16:33.6264835Z   Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f
2024-08-20T22:16:36.1478075Z   Preparing metadata (setup.py) ... [?25l- \ done
2024-08-20T22:16:36.1509787Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4)
2024-08-20T22:16:36.1513942Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+git40ec5f6)
2024-08-20T22:16:36.1518265Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0)
2024-08-20T22:16:36.1589581Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1)
2024-08-20T22:16:36.1593939Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2)
2024-08-20T22:16:36.1597207Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8)
2024-08-20T22:16:36.1600464Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4)
2024-08-20T22:16:36.1603694Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.6.1)
2024-08-20T22:16:36.1608811Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.1)
2024-08-20T22:16:36.1623113Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0)
2024-08-20T22:16:36.2099302Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5)
2024-08-20T22:16:36.2161753Z Building wheels for collected packages: torchvision
2024-08-20T22:17:52.7491004Z   Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done
2024-08-20T22:17:52.7523951Z [?25h  Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2116138 sha256=c50da785e60006273b68c3e6618e773dc830188bf561faafc8e5a18e0b91a717
2024-08-20T22:17:52.7525632Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96
2024-08-20T22:17:52.7559635Z Successfully built torchvision
2024-08-20T22:17:53.2423228Z Installing collected packages: torchvision
2024-08-20T22:17:53.6556128Z Successfully installed torchvision-0.19.0a0+d23a6e1
2024-08-20T22:17:53.7937707Z + '[' -n '' ']'
2024-08-20T22:17:53.7938030Z + id=0
2024-08-20T22:17:53.7944670Z + test_dynamo_benchmark huggingface 0
2024-08-20T22:17:53.7945151Z ++ pwd
2024-08-20T22:17:53.7945821Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2024-08-20T22:17:53.7946356Z + local suite=huggingface
2024-08-20T22:17:53.7946697Z + shift
2024-08-20T22:17:53.7950217Z + local shard_id=0
2024-08-20T22:17:53.7951174Z + shift
2024-08-20T22:17:53.7951570Z + [[ dynamic_aot_eager_huggingface == *perf_compare* ]]
2024-08-20T22:17:53.7952118Z + [[ dynamic_aot_eager_huggingface == *perf* ]]
2024-08-20T22:17:53.7952623Z + [[ dynamic_aot_eager_huggingface == *cpu* ]]
2024-08-20T22:17:53.7953137Z + [[ dynamic_aot_eager_huggingface == *aot_inductor* ]]
2024-08-20T22:17:53.7953939Z + test_single_dynamo_benchmark inference huggingface 0 --inference --bfloat16
2024-08-20T22:17:53.7954513Z ++ pwd
2024-08-20T22:17:53.7954982Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2024-08-20T22:17:53.7955653Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports
2024-08-20T22:17:53.7977376Z + local name=inference
2024-08-20T22:17:53.7977839Z + shift
2024-08-20T22:17:53.7978152Z + local suite=huggingface
2024-08-20T22:17:53.7978583Z + shift
2024-08-20T22:17:53.7978848Z + local shard_id=0
2024-08-20T22:17:53.7979146Z + shift
2024-08-20T22:17:53.7979420Z + partition_flags=()
2024-08-20T22:17:53.7982189Z + local partition_flags
2024-08-20T22:17:53.7982782Z + [[ -n 1 ]]
2024-08-20T22:17:53.7983108Z + [[ -n 0 ]]
2024-08-20T22:17:53.7983708Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id")
2024-08-20T22:17:53.7984423Z + [[ dynamic_aot_eager_huggingface == *perf_compare* ]]
2024-08-20T22:17:53.7984958Z + [[ dynamic_aot_eager_huggingface == *perf* ]]
2024-08-20T22:17:53.7985475Z + [[ dynamic_aot_eager_huggingface == *aot_inductor* ]]
2024-08-20T22:17:53.7986000Z + [[ dynamic_aot_eager_huggingface == *_avx2* ]]
2024-08-20T22:17:53.7986502Z + [[ dynamic_aot_eager_huggingface == *_avx512* ]]
2024-08-20T22:17:53.7988392Z + python benchmarks/dynamo/huggingface.py --ci --accuracy --timing --explain --backend aot_eager --dynamic-shapes --dynamic-batch-only --device cuda --inference --bfloat16 --total-partitions 1 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv
2024-08-20T22:17:56.8484004Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:17:56.8485735Z   warnings.warn(
2024-08-20T22:17:56.8892304Z 
2024-08-20T22:17:56.8892702Z config.json:   0% 0.00/694 [00:00<?, ?B/s]
2024-08-20T22:17:56.8893213Z config.json: 100% 694/694 [00:00<00:00, 10.4MB/s]
2024-08-20T22:17:56.9395731Z 
2024-08-20T22:17:56.9396146Z config.json:   0% 0.00/1.21k [00:00<?, ?B/s]
2024-08-20T22:17:56.9396675Z config.json: 100% 1.21k/1.21k [00:00<00:00, 23.9MB/s]
2024-08-20T22:17:56.9781671Z 
2024-08-20T22:17:56.9782019Z config.json:   0% 0.00/762 [00:00<?, ?B/s]
2024-08-20T22:17:56.9782502Z config.json: 100% 762/762 [00:00<00:00, 12.6MB/s]
2024-08-20T22:17:57.0154780Z 
2024-08-20T22:17:57.0155046Z config.json:   0% 0.00/626 [00:00<?, ?B/s]
2024-08-20T22:17:57.0602584Z config.json: 100% 626/626 [00:00<00:00, 12.8MB/s]
2024-08-20T22:17:57.0603748Z 
2024-08-20T22:17:57.0603919Z config.json:   0% 0.00/674 [00:00<?, ?B/s]
2024-08-20T22:17:57.0604386Z config.json: 100% 674/674 [00:00<00:00, 13.9MB/s]
2024-08-20T22:17:57.1054683Z 
2024-08-20T22:17:57.1055509Z config.json:   0% 0.00/508 [00:00<?, ?B/s]
2024-08-20T22:17:57.1056043Z config.json: 100% 508/508 [00:00<00:00, 5.25MB/s]
2024-08-20T22:18:00.1909809Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:18:00.1911449Z   warnings.warn(
2024-08-20T22:18:00.3796758Z 
2024-08-20T22:18:03.1823649Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:18:03.1824116Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:18:03.1830856Z cuda eval  AlbertForMaskedLM                  
2024-08-20T22:18:20.3687508Z pass
2024-08-20T22:18:20.3780846Z TIMING: backend_compile:3.16305 entire_frame_compile:5.44462 total_wall_time:5.44462
2024-08-20T22:18:20.3781981Z STATS: call_* op count: 878 | FakeTensorMode.__torch_dispatch__:18635 | ProxyTorchDispatchMode.__torch_dispatch__:7624
2024-08-20T22:18:20.3782939Z Dynamo produced 1 graphs covering 878 ops with 0 graph breaks (0 unique)
2024-08-20T22:18:24.3124762Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:18:24.3127967Z   warnings.warn(
2024-08-20T22:18:24.4870648Z 
2024-08-20T22:18:27.2626392Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:18:27.2626850Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:18:27.2633673Z cuda eval  AlbertForQuestionAnswering         
2024-08-20T22:18:44.4066681Z pass
2024-08-20T22:18:44.4118036Z TIMING: backend_compile:3.18215 entire_frame_compile:5.44062 total_wall_time:5.44062
2024-08-20T22:18:44.4119074Z STATS: call_* op count: 878 | FakeTensorMode.__torch_dispatch__:18625 | ProxyTorchDispatchMode.__torch_dispatch__:7575
2024-08-20T22:18:44.4120037Z Dynamo produced 1 graphs covering 878 ops with 0 graph breaks (0 unique)
2024-08-20T22:18:48.2698112Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:18:48.2699796Z   warnings.warn(
2024-08-20T22:18:48.4576470Z 
2024-08-20T22:18:51.2080562Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:18:51.2081024Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:18:51.2152195Z cuda eval  AllenaiLongformerBase              
2024-08-20T22:18:54.5996811Z W0820 22:18:54.599000 1845 torch/_dynamo/variables/tensor.py:776] [0/0] Graph break from `Tensor.item()`, consider setting:
2024-08-20T22:18:54.5998078Z W0820 22:18:54.599000 1845 torch/_dynamo/variables/tensor.py:776] [0/0]     torch._dynamo.config.capture_scalar_outputs = True
2024-08-20T22:18:54.5999066Z W0820 22:18:54.599000 1845 torch/_dynamo/variables/tensor.py:776] [0/0] or:
2024-08-20T22:18:54.6000023Z W0820 22:18:54.599000 1845 torch/_dynamo/variables/tensor.py:776] [0/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2024-08-20T22:18:54.6001213Z W0820 22:18:54.599000 1845 torch/_dynamo/variables/tensor.py:776] [0/0] to include these operations in the captured graph.
2024-08-20T22:18:54.6002176Z W0820 22:18:54.599000 1845 torch/_dynamo/variables/tensor.py:776] [0/0] 
2024-08-20T22:18:54.6003052Z W0820 22:18:54.599000 1845 torch/_dynamo/variables/tensor.py:776] [0/0] Graph break: from user code at:
2024-08-20T22:18:54.6004630Z W0820 22:18:54.599000 1845 torch/_dynamo/variables/tensor.py:776] [0/0]   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2545, in run_n_iterations
2024-08-20T22:18:54.6006300Z W0820 22:18:54.599000 1845 torch/_dynamo/variables/tensor.py:776] [0/0]     self.model_iter_fn(mod, inputs, collect_outputs=False)
2024-08-20T22:18:54.6008053Z W0820 22:18:54.599000 1845 torch/_dynamo/variables/tensor.py:776] [0/0]   File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 518, in forward_pass
2024-08-20T22:18:54.6009368Z W0820 22:18:54.599000 1845 torch/_dynamo/variables/tensor.py:776] [0/0]     return mod(**inputs)
2024-08-20T22:18:54.6011251Z W0820 22:18:54.599000 1845 torch/_dynamo/variables/tensor.py:776] [0/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1835, in forward
2024-08-20T22:18:54.6012814Z W0820 22:18:54.599000 1845 torch/_dynamo/variables/tensor.py:776] [0/0]     outputs = self.longformer(
2024-08-20T22:18:54.6014714Z W0820 22:18:54.599000 1845 torch/_dynamo/variables/tensor.py:776] [0/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1738, in forward
2024-08-20T22:18:54.6016294Z W0820 22:18:54.599000 1845 torch/_dynamo/variables/tensor.py:776] [0/0]     encoder_outputs = self.encoder(
2024-08-20T22:18:54.6018018Z W0820 22:18:54.599000 1845 torch/_dynamo/variables/tensor.py:776] [0/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1291, in forward
2024-08-20T22:18:54.6019838Z W0820 22:18:54.599000 1845 torch/_dynamo/variables/tensor.py:776] [0/0]     is_global_attn = is_index_global_attn.flatten().any().item()
2024-08-20T22:18:54.6020834Z W0820 22:18:54.599000 1845 torch/_dynamo/variables/tensor.py:776] [0/0] 
2024-08-20T22:18:54.6021556Z W0820 22:18:54.599000 1845 torch/_dynamo/variables/tensor.py:776] [0/0] 
2024-08-20T22:19:18.1779929Z pass
2024-08-20T22:19:18.1879426Z TIMING: entire_frame_compile:24.81598 backend_compile:11.18296 total_wall_time:24.81598
2024-08-20T22:19:18.1882162Z STATS: call_* op count: 1739 | FakeTensorMode.__torch_dispatch__:44674 | ProxyTorchDispatchMode.__torch_dispatch__:18975 | FakeTensor.__torch_dispatch__:32
2024-08-20T22:19:18.1883529Z Dynamo produced 4 graphs covering 1739 ops with 4 graph breaks (1 unique)
2024-08-20T22:19:22.4677803Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:19:22.4679497Z   warnings.warn(
2024-08-20T22:19:22.6493842Z 
2024-08-20T22:19:26.6082642Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:19:26.6083108Z loading model: 0it [00:03, ?it/s]
2024-08-20T22:19:26.6162014Z cuda eval  BartForCausalLM                    
2024-08-20T22:19:36.0547694Z pass
2024-08-20T22:19:36.0786031Z TIMING: backend_compile:3.17995 entire_frame_compile:6.70478 total_wall_time:6.70478
2024-08-20T22:19:36.0787296Z STATS: call_* op count: 656 | FakeTensorMode.__torch_dispatch__:16523 | FakeTensor.__torch_dispatch__:216 | ProxyTorchDispatchMode.__torch_dispatch__:6722
2024-08-20T22:19:36.0789826Z Dynamo produced 1 graphs covering 656 ops with 0 graph breaks (0 unique)
2024-08-20T22:19:39.9371157Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:19:39.9374487Z   warnings.warn(
2024-08-20T22:19:40.1390378Z 
2024-08-20T22:19:47.0772296Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:19:47.0772764Z loading model: 0it [00:06, ?it/s]
2024-08-20T22:19:47.0899270Z cuda eval  BartForConditionalGeneration       
2024-08-20T22:20:10.6789153Z pass
2024-08-20T22:20:10.7013810Z TIMING: backend_compile:9.01703 entire_frame_compile:17.89216 total_wall_time:17.89216
2024-08-20T22:20:10.7015917Z STATS: call_* op count: 1776 | FakeTensorMode.__torch_dispatch__:44078 | FakeTensor.__torch_dispatch__:649 | ProxyTorchDispatchMode.__torch_dispatch__:18031
2024-08-20T22:20:10.7017397Z Dynamo produced 1 graphs covering 1776 ops with 0 graph breaks (0 unique)
2024-08-20T22:20:14.8469524Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:20:14.8471145Z   warnings.warn(
2024-08-20T22:20:15.0528264Z 
2024-08-20T22:20:17.0953769Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:20:17.0954242Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:20:17.1008451Z cuda eval  BertForMaskedLM                    
2024-08-20T22:20:26.2343777Z pass
2024-08-20T22:20:26.2412770Z TIMING: backend_compile:3.84516 entire_frame_compile:8.22184 total_wall_time:8.22184
2024-08-20T22:20:26.2414352Z STATS: call_* op count: 740 | FakeTensorMode.__torch_dispatch__:18861 | ProxyTorchDispatchMode.__torch_dispatch__:7686
2024-08-20T22:20:26.2415446Z Dynamo produced 1 graphs covering 740 ops with 0 graph breaks (0 unique)
2024-08-20T22:20:30.1448421Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:20:30.1450093Z   warnings.warn(
2024-08-20T22:20:30.3999143Z 
2024-08-20T22:20:32.1900073Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:20:32.1900550Z loading model: 0it [00:01, ?it/s]
2024-08-20T22:20:32.1951611Z cuda eval  BertForQuestionAnswering           
2024-08-20T22:20:41.1254384Z pass
2024-08-20T22:20:41.1262496Z TIMING: backend_compile:3.80489 entire_frame_compile:8.13662 total_wall_time:8.13662
2024-08-20T22:20:41.1263606Z STATS: call_* op count: 754 | FakeTensorMode.__torch_dispatch__:19037 | ProxyTorchDispatchMode.__torch_dispatch__:7707
2024-08-20T22:20:41.1264743Z Dynamo produced 1 graphs covering 754 ops with 0 graph breaks (0 unique)
2024-08-20T22:20:45.0331555Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:20:45.0333190Z   warnings.warn(
2024-08-20T22:20:45.2395534Z 
2024-08-20T22:21:14.5069372Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:21:14.5069894Z loading model: 0it [00:29, ?it/s]
2024-08-20T22:21:14.5224812Z cuda eval  BlenderbotForCausalLM              
2024-08-20T22:21:14.5227582Z pass_due_to_skip
2024-08-20T22:21:14.9550012Z TIMING:
2024-08-20T22:21:14.9550316Z STATS: call_* op count: 0
2024-08-20T22:21:14.9554855Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-08-20T22:21:18.5799085Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:21:18.5800733Z   warnings.warn(
2024-08-20T22:21:18.7857297Z 
2024-08-20T22:21:20.2367954Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:21:20.2368759Z loading model: 0it [00:01, ?it/s]
2024-08-20T22:21:20.2419756Z cuda eval  BlenderbotSmallForCausalLM         
2024-08-20T22:21:20.2621650Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForCausalLM. Setting accuracy check to cosine
2024-08-20T22:21:25.3984985Z pass
2024-08-20T22:21:25.4028845Z TIMING: backend_compile:2.46963 entire_frame_compile:4.86383 total_wall_time:4.86383
2024-08-20T22:21:25.4030276Z STATS: call_* op count: 652 | FakeTensorMode.__torch_dispatch__:12709 | ProxyTorchDispatchMode.__torch_dispatch__:5230
2024-08-20T22:21:25.4031234Z Dynamo produced 1 graphs covering 652 ops with 0 graph breaks (0 unique)
2024-08-20T22:21:29.2240651Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:21:29.2242292Z   warnings.warn(
2024-08-20T22:21:29.4088617Z 
2024-08-20T22:21:31.2331337Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:21:31.2331828Z loading model: 0it [00:01, ?it/s]
2024-08-20T22:21:31.2422393Z cuda eval  BlenderbotSmallForConditionalGeneration 
2024-08-20T22:21:31.2894670Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForConditionalGeneration. Setting accuracy check to cosine
2024-08-20T22:21:44.9230316Z pass
2024-08-20T22:21:44.9289521Z TIMING: backend_compile:6.78993 entire_frame_compile:13.22396 total_wall_time:13.22396
2024-08-20T22:21:44.9292116Z STATS: call_* op count: 1676 | FakeTensorMode.__torch_dispatch__:32419 | ProxyTorchDispatchMode.__torch_dispatch__:13481
2024-08-20T22:21:44.9294063Z Dynamo produced 1 graphs covering 1676 ops with 0 graph breaks (0 unique)
2024-08-20T22:21:48.9133681Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:21:48.9135298Z   warnings.warn(
2024-08-20T22:21:49.1073876Z 
2024-08-20T22:21:51.1944842Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:21:51.1945314Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:21:51.1997398Z cuda eval  CamemBert                          
2024-08-20T22:22:00.3272849Z pass
2024-08-20T22:22:00.3344965Z TIMING: backend_compile:3.85732 entire_frame_compile:8.2152 total_wall_time:8.2152
2024-08-20T22:22:00.3346089Z STATS: call_* op count: 756 | FakeTensorMode.__torch_dispatch__:19036 | ProxyTorchDispatchMode.__torch_dispatch__:7762
2024-08-20T22:22:00.3347087Z Dynamo produced 1 graphs covering 756 ops with 0 graph breaks (0 unique)
2024-08-20T22:22:04.1633827Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:22:04.1635468Z   warnings.warn(
2024-08-20T22:22:04.3670459Z 
2024-08-20T22:22:06.8082473Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:22:06.8082940Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:22:06.8125187Z cuda eval  DebertaForMaskedLM                 
2024-08-20T22:22:16.7417045Z pass
2024-08-20T22:22:16.7519055Z TIMING: backend_compile:4.70823 entire_frame_compile:8.93291 total_wall_time:8.93291
2024-08-20T22:22:16.7520091Z STATS: call_* op count: 1580 | FakeTensorMode.__torch_dispatch__:25990 | ProxyTorchDispatchMode.__torch_dispatch__:10635
2024-08-20T22:22:16.7521083Z Dynamo produced 1 graphs covering 1580 ops with 0 graph breaks (0 unique)
2024-08-20T22:22:20.6713934Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:22:20.6715677Z   warnings.warn(
2024-08-20T22:22:20.9678912Z 
2024-08-20T22:22:22.9986470Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:22:22.9986988Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:22:23.0027534Z cuda eval  DebertaForQuestionAnswering        
2024-08-20T22:22:32.6475116Z pass
2024-08-20T22:22:32.6483640Z TIMING: backend_compile:4.62965 entire_frame_compile:8.83901 total_wall_time:8.83901
2024-08-20T22:22:32.6484705Z STATS: call_* op count: 1594 | FakeTensorMode.__torch_dispatch__:26166 | ProxyTorchDispatchMode.__torch_dispatch__:10656
2024-08-20T22:22:32.6485705Z Dynamo produced 1 graphs covering 1594 ops with 0 graph breaks (0 unique)
2024-08-20T22:22:36.5876789Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:22:36.5878488Z   warnings.warn(
2024-08-20T22:22:36.8121508Z 
2024-08-20T22:22:49.4899061Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:22:49.4899524Z loading model: 0it [00:12, ?it/s]
2024-08-20T22:22:49.4995373Z cuda eval  DebertaV2ForMaskedLM               
2024-08-20T22:22:49.4998636Z pass_due_to_skip
2024-08-20T22:22:49.6216969Z TIMING:
2024-08-20T22:22:49.6217272Z STATS: call_* op count: 0
2024-08-20T22:22:49.6221894Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-08-20T22:22:53.2810392Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:22:53.2812133Z   warnings.warn(
2024-08-20T22:22:53.4937016Z 
2024-08-20T22:23:04.0965599Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:23:04.0966068Z loading model: 0it [00:10, ?it/s]
2024-08-20T22:23:04.1059800Z cuda eval  DebertaV2ForQuestionAnswering      
2024-08-20T22:23:24.8608112Z pass
2024-08-20T22:23:24.9009441Z TIMING: backend_compile:8.09293 entire_frame_compile:16.54281 total_wall_time:16.54281
2024-08-20T22:23:24.9010501Z STATS: call_* op count: 2172 | FakeTensorMode.__torch_dispatch__:38451 | ProxyTorchDispatchMode.__torch_dispatch__:16022
2024-08-20T22:23:24.9011486Z Dynamo produced 1 graphs covering 2172 ops with 0 graph breaks (0 unique)
2024-08-20T22:23:28.9633455Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:23:28.9635112Z   warnings.warn(
2024-08-20T22:23:29.2171195Z 
2024-08-20T22:23:30.6485550Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:23:30.6486025Z loading model: 0it [00:01, ?it/s]
2024-08-20T22:23:30.6514159Z cuda eval  DistilBertForMaskedLM              
2024-08-20T22:23:35.0507703Z pass
2024-08-20T22:23:35.0523734Z TIMING: backend_compile:2.03203 entire_frame_compile:3.98062 total_wall_time:3.98062
2024-08-20T22:23:35.0524769Z STATS: call_* op count: 412 | FakeTensorMode.__torch_dispatch__:10175 | ProxyTorchDispatchMode.__torch_dispatch__:4101
2024-08-20T22:23:35.0525731Z Dynamo produced 1 graphs covering 412 ops with 0 graph breaks (0 unique)
2024-08-20T22:23:38.8480126Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:23:38.8481790Z   warnings.warn(
2024-08-20T22:23:39.1864950Z 
2024-08-20T22:23:40.3843583Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:23:40.3844513Z loading model: 0it [00:01, ?it/s]
2024-08-20T22:23:40.3870468Z cuda eval  DistilBertForQuestionAnswering     
2024-08-20T22:23:44.7911674Z pass
2024-08-20T22:23:44.7913714Z TIMING: backend_compile:2.07486 entire_frame_compile:4.01371 total_wall_time:4.01371
2024-08-20T22:23:44.7914737Z STATS: call_* op count: 428 | FakeTensorMode.__torch_dispatch__:10373 | ProxyTorchDispatchMode.__torch_dispatch__:4132
2024-08-20T22:23:44.7915701Z Dynamo produced 1 graphs covering 428 ops with 0 graph breaks (0 unique)
2024-08-20T22:23:48.6189062Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:23:48.6191113Z   warnings.warn(
2024-08-20T22:23:48.8588116Z 
2024-08-20T22:23:50.8943427Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:23:50.8943918Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:23:50.8964280Z cuda eval  DistillGPT2                        
2024-08-20T22:23:55.5165363Z pass
2024-08-20T22:23:55.5302541Z TIMING: backend_compile:1.98056 entire_frame_compile:3.93835 total_wall_time:3.93835
2024-08-20T22:23:55.5303830Z STATS: call_* op count: 648 | FakeTensorMode.__torch_dispatch__:10118 | ProxyTorchDispatchMode.__torch_dispatch__:4034
2024-08-20T22:23:55.5306791Z Dynamo produced 1 graphs covering 648 ops with 0 graph breaks (0 unique)
2024-08-20T22:23:59.3388954Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:23:59.3391054Z   warnings.warn(
2024-08-20T22:23:59.5495095Z 
2024-08-20T22:23:59.5503875Z loading model: 0it [00:00, ?it/s]If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.`
2024-08-20T22:24:00.3023173Z 
2024-08-20T22:24:00.3023925Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:24:00.3075972Z cuda eval  ElectraForCausalLM                 
2024-08-20T22:24:08.9258603Z pass
2024-08-20T22:24:08.9355724Z TIMING: backend_compile:3.82572 entire_frame_compile:8.20334 total_wall_time:8.20334
2024-08-20T22:24:08.9356800Z STATS: call_* op count: 750 | FakeTensorMode.__torch_dispatch__:19145 | ProxyTorchDispatchMode.__torch_dispatch__:7770
2024-08-20T22:24:08.9357768Z Dynamo produced 1 graphs covering 750 ops with 0 graph breaks (0 unique)
2024-08-20T22:24:12.8259734Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:24:12.8261615Z   warnings.warn(
2024-08-20T22:24:13.0197124Z 
2024-08-20T22:24:13.7341430Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:24:13.7342009Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:24:13.7396167Z cuda eval  ElectraForQuestionAnswering        
2024-08-20T22:24:22.3846810Z pass
2024-08-20T22:24:22.3859050Z TIMING: backend_compile:3.88264 entire_frame_compile:8.25883 total_wall_time:8.25883
2024-08-20T22:24:22.3860228Z STATS: call_* op count: 756 | FakeTensorMode.__torch_dispatch__:19175 | ProxyTorchDispatchMode.__torch_dispatch__:7751
2024-08-20T22:24:22.3861482Z Dynamo produced 1 graphs covering 756 ops with 0 graph breaks (0 unique)
2024-08-20T22:24:26.2969292Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:24:26.2971050Z   warnings.warn(
2024-08-20T22:24:26.5146375Z 
2024-08-20T22:24:28.8178514Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:24:28.8178980Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:24:28.8219955Z cuda eval  GPT2ForSequenceClassification      
2024-08-20T22:24:40.6541992Z pass
2024-08-20T22:24:40.6658951Z TIMING: entire_frame_compile:10.36848 backend_compile:2.39271 total_wall_time:10.36848
2024-08-20T22:24:40.6660283Z STATS: call_* op count: 633 | FakeTensorMode.__torch_dispatch__:14740 | ProxyTorchDispatchMode.__torch_dispatch__:4253 | FakeTensor.__torch_dispatch__:2
2024-08-20T22:24:40.6661381Z Dynamo produced 2 graphs covering 633 ops with 2 graph breaks (1 unique)
2024-08-20T22:24:44.5788824Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:24:44.5790865Z   warnings.warn(
2024-08-20T22:24:44.8067626Z 
2024-08-20T22:24:46.3209651Z loading model: 0it [00:00, ?it/s]WARNING:common:Model GoogleFnet supports float32 only
2024-08-20T22:24:46.5614596Z 
2024-08-20T22:24:46.5617923Z loading model: 0it [00:01, ?it/s]
2024-08-20T22:24:46.5618906Z WARNING:common:Model GoogleFnet supports float32 only
2024-08-20T22:24:46.5628372Z cuda eval  GoogleFnet                         
2024-08-20T22:24:47.0920570Z WARNING:common:Model GoogleFnet supports float32 only
2024-08-20T22:24:51.5818188Z pass
2024-08-20T22:24:51.5935286Z TIMING: backend_compile:1.96147 entire_frame_compile:4.23458 total_wall_time:4.23458
2024-08-20T22:24:51.5936440Z STATS: call_* op count: 464 | FakeTensorMode.__torch_dispatch__:9795 | ProxyTorchDispatchMode.__torch_dispatch__:4158
2024-08-20T22:24:51.5937388Z Dynamo produced 1 graphs covering 464 ops with 0 graph breaks (0 unique)
2024-08-20T22:24:55.4014574Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:24:55.4016351Z   warnings.warn(
2024-08-20T22:24:55.6018379Z 
2024-08-20T22:24:57.6865314Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:24:57.6865782Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:24:57.6918799Z cuda eval  LayoutLMForMaskedLM                
2024-08-20T22:25:07.1998898Z pass
2024-08-20T22:25:07.2069178Z TIMING: backend_compile:4.02571 entire_frame_compile:8.5802 total_wall_time:8.5802
2024-08-20T22:25:07.2070439Z STATS: call_* op count: 792 | FakeTensorMode.__torch_dispatch__:19883 | ProxyTorchDispatchMode.__torch_dispatch__:8082
2024-08-20T22:25:07.2071408Z Dynamo produced 1 graphs covering 792 ops with 0 graph breaks (0 unique)
2024-08-20T22:25:11.1161357Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:25:11.1163036Z   warnings.warn(
2024-08-20T22:25:11.3231595Z 
2024-08-20T22:25:13.1653921Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:25:13.1654513Z loading model: 0it [00:01, ?it/s]
2024-08-20T22:25:13.1708366Z cuda eval  LayoutLMForSequenceClassification  
2024-08-20T22:25:26.0094535Z pass
2024-08-20T22:25:26.0104911Z TIMING: entire_frame_compile:11.98956 backend_compile:2.54707 total_wall_time:11.98956
2024-08-20T22:25:26.0105967Z STATS: call_* op count: 394 | FakeTensorMode.__torch_dispatch__:15574 | ProxyTorchDispatchMode.__torch_dispatch__:4518
2024-08-20T22:25:26.0106922Z Dynamo produced 2 graphs covering 394 ops with 2 graph breaks (1 unique)
2024-08-20T22:25:29.8967453Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:25:29.8969587Z   warnings.warn(
2024-08-20T22:25:30.1175639Z 
2024-08-20T22:25:40.7422172Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:25:40.7422748Z loading model: 0it [00:10, ?it/s]
2024-08-20T22:25:40.7542661Z cuda eval  M2M100ForConditionalGeneration     
2024-08-20T22:25:40.8282949Z WARNING:common:fp64 golden ref were not generated for M2M100ForConditionalGeneration. Setting accuracy check to cosine
2024-08-20T22:26:00.9294755Z pass
2024-08-20T22:26:00.9419504Z TIMING: backend_compile:10.0387 entire_frame_compile:19.37174 total_wall_time:19.37174
2024-08-20T22:26:00.9420559Z STATS: call_* op count: 2580 | FakeTensorMode.__torch_dispatch__:49003 | ProxyTorchDispatchMode.__torch_dispatch__:20441
2024-08-20T22:26:00.9421564Z Dynamo produced 1 graphs covering 2580 ops with 0 graph breaks (0 unique)
2024-08-20T22:26:05.1352139Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:26:05.1354225Z   warnings.warn(
2024-08-20T22:26:05.3712889Z 
2024-08-20T22:26:09.3684061Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:26:09.3684535Z loading model: 0it [00:03, ?it/s]
2024-08-20T22:26:09.3763026Z cuda eval  MBartForCausalLM                   
2024-08-20T22:26:09.4055254Z WARNING:common:fp64 golden ref were not generated for MBartForCausalLM. Setting accuracy check to cosine
2024-08-20T22:26:17.3863659Z pass
2024-08-20T22:26:17.4061250Z TIMING: backend_compile:3.84378 entire_frame_compile:7.58768 total_wall_time:7.58768
2024-08-20T22:26:17.4062320Z STATS: call_* op count: 964 | FakeTensorMode.__torch_dispatch__:18709 | ProxyTorchDispatchMode.__torch_dispatch__:7748
2024-08-20T22:26:17.4063703Z Dynamo produced 1 graphs covering 964 ops with 0 graph breaks (0 unique)
2024-08-20T22:26:21.2735925Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:26:21.2738919Z   warnings.warn(
2024-08-20T22:26:21.4578805Z 
2024-08-20T22:26:28.4280313Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:26:28.4281046Z loading model: 0it [00:06, ?it/s]
2024-08-20T22:26:28.4407590Z cuda eval  MBartForConditionalGeneration      
2024-08-20T22:26:28.5331476Z WARNING:common:fp64 golden ref were not generated for MBartForConditionalGeneration. Setting accuracy check to cosine
2024-08-20T22:26:49.5576253Z pass
2024-08-20T22:26:49.5837490Z TIMING: backend_compile:10.13792 entire_frame_compile:19.51092 total_wall_time:19.51092
2024-08-20T22:26:49.5841031Z STATS: call_* op count: 2526 | FakeTensorMode.__torch_dispatch__:48703 | FakeTensor.__torch_dispatch__:2 | ProxyTorchDispatchMode.__torch_dispatch__:20317
2024-08-20T22:26:49.5842174Z Dynamo produced 1 graphs covering 2526 ops with 0 graph breaks (0 unique)
2024-08-20T22:26:53.7551880Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:26:53.7553590Z   warnings.warn(
2024-08-20T22:26:53.9215141Z 
2024-08-20T22:26:57.9616934Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:26:57.9617407Z loading model: 0it [00:04, ?it/s]
2024-08-20T22:26:57.9673941Z cuda eval  MT5ForConditionalGeneration        
2024-08-20T22:27:13.7076217Z pass
2024-08-20T22:27:13.7310966Z TIMING: backend_compile:7.44214 entire_frame_compile:14.9133 total_wall_time:14.9133
2024-08-20T22:27:13.7312029Z STATS: call_* op count: 2432 | FakeTensorMode.__torch_dispatch__:43207 | ProxyTorchDispatchMode.__torch_dispatch__:18164
2024-08-20T22:27:13.7313008Z Dynamo produced 1 graphs covering 2432 ops with 0 graph breaks (0 unique)
2024-08-20T22:27:17.7460992Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:27:17.7462633Z   warnings.warn(
2024-08-20T22:27:17.9609964Z 
2024-08-20T22:27:17.9647242Z loading model: 0it [00:00, ?it/s]If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.`
2024-08-20T22:27:22.6779514Z 
2024-08-20T22:27:22.6780080Z loading model: 0it [00:04, ?it/s]
2024-08-20T22:27:22.6883145Z cuda eval  MegatronBertForCausalLM            
2024-08-20T22:27:41.1489042Z pass
2024-08-20T22:27:41.1694676Z TIMING: backend_compile:7.63276 entire_frame_compile:16.1291 total_wall_time:16.1291
2024-08-20T22:27:41.1696409Z STATS: call_* op count: 1442 | FakeTensorMode.__torch_dispatch__:36583 | ProxyTorchDispatchMode.__torch_dispatch__:15012
2024-08-20T22:27:41.1700374Z Dynamo produced 1 graphs covering 1442 ops with 0 graph breaks (0 unique)
2024-08-20T22:27:45.1846436Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:27:45.1848197Z   warnings.warn(
2024-08-20T22:27:45.3944332Z 
2024-08-20T22:27:49.8330436Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:27:49.8330914Z loading model: 0it [00:04, ?it/s]
2024-08-20T22:27:49.8434103Z cuda eval  MegatronBertForQuestionAnswering   
2024-08-20T22:28:08.1181931Z pass
2024-08-20T22:28:08.1335906Z TIMING: backend_compile:7.61408 entire_frame_compile:16.07396 total_wall_time:16.07396
2024-08-20T22:28:08.1337527Z STATS: call_* op count: 1448 | FakeTensorMode.__torch_dispatch__:36645 | ProxyTorchDispatchMode.__torch_dispatch__:14993
2024-08-20T22:28:08.1338668Z Dynamo produced 1 graphs covering 1448 ops with 0 graph breaks (0 unique)
2024-08-20T22:28:12.2065154Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:28:12.2066918Z   warnings.warn(
2024-08-20T22:28:12.4337139Z 
2024-08-20T22:28:13.7088006Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:28:13.7088541Z loading model: 0it [00:01, ?it/s]
2024-08-20T22:28:13.7390034Z cuda eval  MobileBertForMaskedLM              
2024-08-20T22:28:50.4460267Z pass
2024-08-20T22:28:50.4487210Z TIMING: backend_compile:15.86823 entire_frame_compile:35.50298 total_wall_time:35.50298
2024-08-20T22:28:50.4488461Z STATS: call_* op count: 2894 | FakeTensorMode.__torch_dispatch__:73179 | ProxyTorchDispatchMode.__torch_dispatch__:30329
2024-08-20T22:28:50.4489515Z Dynamo produced 1 graphs covering 2894 ops with 0 graph breaks (0 unique)
2024-08-20T22:28:55.0084471Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:28:55.0086112Z   warnings.warn(
2024-08-20T22:28:55.2547592Z 
2024-08-20T22:28:56.3429706Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:28:56.3430216Z loading model: 0it [00:01, ?it/s]
2024-08-20T22:28:56.3713209Z cuda eval  MobileBertForQuestionAnswering     
2024-08-20T22:29:32.8419145Z pass
2024-08-20T22:29:32.8419776Z TIMING: backend_compile:15.75659 entire_frame_compile:35.30067 total_wall_time:35.30067
2024-08-20T22:29:32.8420839Z STATS: call_* op count: 2902 | FakeTensorMode.__torch_dispatch__:73299 | ProxyTorchDispatchMode.__torch_dispatch__:30323
2024-08-20T22:29:32.8421834Z Dynamo produced 1 graphs covering 2902 ops with 0 graph breaks (0 unique)
2024-08-20T22:29:37.4055423Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:29:37.4057058Z   warnings.warn(
2024-08-20T22:29:37.6784563Z 
2024-08-20T22:29:40.0727358Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:29:40.0727812Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:29:40.0781168Z cuda eval  OPTForCausalLM                     
2024-08-20T22:29:40.0985918Z WARNING:common:fp64 golden ref were not generated for OPTForCausalLM. Setting accuracy check to cosine
2024-08-20T22:29:48.2489898Z pass
2024-08-20T22:29:48.2690648Z TIMING: backend_compile:3.82458 entire_frame_compile:7.61504 total_wall_time:7.61504
2024-08-20T22:29:48.2691713Z STATS: call_* op count: 1068 | FakeTensorMode.__torch_dispatch__:18759 | ProxyTorchDispatchMode.__torch_dispatch__:7792
2024-08-20T22:29:48.2693066Z Dynamo produced 1 graphs covering 1068 ops with 0 graph breaks (0 unique)
2024-08-20T22:29:52.1575724Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:29:52.1577359Z   warnings.warn(
2024-08-20T22:29:52.3429802Z 
2024-08-20T22:29:54.2322192Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:29:54.2322662Z loading model: 0it [00:01, ?it/s]
2024-08-20T22:29:54.2362401Z cuda eval  PLBartForCausalLM                  
2024-08-20T22:29:54.3444140Z WARNING:common:fp64 golden ref were not generated for PLBartForCausalLM. Setting accuracy check to cosine
2024-08-20T22:29:58.4345887Z pass
2024-08-20T22:29:58.4481389Z TIMING: backend_compile:1.97675 entire_frame_compile:3.83617 total_wall_time:3.83617
2024-08-20T22:29:58.4482427Z STATS: call_* op count: 506 | FakeTensorMode.__torch_dispatch__:9825 | ProxyTorchDispatchMode.__torch_dispatch__:4016
2024-08-20T22:29:58.4483375Z Dynamo produced 1 graphs covering 506 ops with 0 graph breaks (0 unique)
2024-08-20T22:30:02.2736959Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:30:02.2738662Z   warnings.warn(
2024-08-20T22:30:02.5183424Z 
2024-08-20T22:30:05.9331795Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:30:05.9332670Z loading model: 0it [00:03, ?it/s]
2024-08-20T22:30:05.9396535Z cuda eval  PLBartForConditionalGeneration     
2024-08-20T22:30:05.9881191Z WARNING:common:fp64 golden ref were not generated for PLBartForConditionalGeneration. Setting accuracy check to cosine
2024-08-20T22:30:16.3724817Z pass
2024-08-20T22:30:16.3879324Z TIMING: backend_compile:4.91512 entire_frame_compile:9.76911 total_wall_time:9.76911
2024-08-20T22:30:16.3880498Z STATS: call_* op count: 1312 | FakeTensorMode.__torch_dispatch__:25161 | FakeTensor.__torch_dispatch__:2 | ProxyTorchDispatchMode.__torch_dispatch__:10451
2024-08-20T22:30:16.3881613Z Dynamo produced 1 graphs covering 1312 ops with 0 graph breaks (0 unique)
2024-08-20T22:30:20.3024516Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:30:20.3026195Z   warnings.warn(
2024-08-20T22:30:20.5910311Z 
2024-08-20T22:30:26.7848414Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:30:26.7848890Z loading model: 0it [00:06, ?it/s]
2024-08-20T22:30:26.7925095Z cuda eval  PegasusForCausalLM                 
2024-08-20T22:30:26.8225669Z WARNING:common:fp64 golden ref were not generated for PegasusForCausalLM. Setting accuracy check to cosine
2024-08-20T22:30:34.5877234Z pass
2024-08-20T22:30:34.5953316Z TIMING: backend_compile:3.70596 entire_frame_compile:7.39304 total_wall_time:7.39304
2024-08-20T22:30:34.5954356Z STATS: call_* op count: 956 | FakeTensorMode.__torch_dispatch__:18579 | ProxyTorchDispatchMode.__torch_dispatch__:7693
2024-08-20T22:30:34.5955331Z Dynamo produced 1 graphs covering 956 ops with 0 graph breaks (0 unique)
2024-08-20T22:30:38.4007947Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:30:38.4009576Z   warnings.warn(
2024-08-20T22:30:38.5966519Z 
2024-08-20T22:30:48.8490887Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:30:48.8491845Z loading model: 0it [00:10, ?it/s]
2024-08-20T22:30:48.8615115Z cuda eval  PegasusForConditionalGeneration    
2024-08-20T22:30:48.9312580Z WARNING:common:fp64 golden ref were not generated for PegasusForConditionalGeneration. Setting accuracy check to cosine
2024-08-20T22:31:08.8847526Z pass
2024-08-20T22:31:08.8999650Z TIMING: backend_compile:9.95054 entire_frame_compile:19.24491 total_wall_time:19.24491
2024-08-20T22:31:08.9001948Z STATS: call_* op count: 2484 | FakeTensorMode.__torch_dispatch__:48007 | ProxyTorchDispatchMode.__torch_dispatch__:20007
2024-08-20T22:31:08.9002963Z Dynamo produced 1 graphs covering 2484 ops with 0 graph breaks (0 unique)
2024-08-20T22:31:13.0336038Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:31:13.0340690Z   warnings.warn(
2024-08-20T22:31:13.2409138Z 
2024-08-20T22:31:13.2437918Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`
2024-08-20T22:31:15.4301814Z We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked.
2024-08-20T22:31:15.4303722Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded.
2024-08-20T22:31:15.6334765Z 
2024-08-20T22:31:15.6335122Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:31:15.6388348Z cuda eval  RobertaForCausalLM                 
2024-08-20T22:31:24.7773127Z pass
2024-08-20T22:31:24.7875778Z TIMING: backend_compile:3.81201 entire_frame_compile:8.15226 total_wall_time:8.15226
2024-08-20T22:31:24.7876822Z STATS: call_* op count: 764 | FakeTensorMode.__torch_dispatch__:19129 | ProxyTorchDispatchMode.__torch_dispatch__:7794
2024-08-20T22:31:24.7877808Z Dynamo produced 1 graphs covering 764 ops with 0 graph breaks (0 unique)
2024-08-20T22:31:28.6758623Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:31:28.6760351Z   warnings.warn(
2024-08-20T22:31:28.9183627Z 
2024-08-20T22:31:30.6675554Z loading model: 0it [00:00, ?it/s]We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked.
2024-08-20T22:31:30.6677612Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded.
2024-08-20T22:31:30.8882284Z 
2024-08-20T22:31:30.8882604Z loading model: 0it [00:01, ?it/s]
2024-08-20T22:31:30.8935949Z cuda eval  RobertaForQuestionAnswering        
2024-08-20T22:31:39.9430409Z pass
2024-08-20T22:31:39.9432999Z TIMING: backend_compile:3.88313 entire_frame_compile:8.25402 total_wall_time:8.25402
2024-08-20T22:31:39.9434040Z STATS: call_* op count: 768 | FakeTensorMode.__torch_dispatch__:19187 | ProxyTorchDispatchMode.__torch_dispatch__:7775
2024-08-20T22:31:39.9435010Z Dynamo produced 1 graphs covering 768 ops with 0 graph breaks (0 unique)
2024-08-20T22:31:43.8204965Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:31:43.8207634Z   warnings.warn(
2024-08-20T22:31:44.0239096Z 
2024-08-20T22:31:44.6139583Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:31:44.6140277Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:31:44.6177920Z cuda eval  Speech2Text2ForCausalLM            
2024-08-20T22:31:44.7252093Z WARNING:common:fp64 golden ref were not generated for Speech2Text2ForCausalLM. Setting accuracy check to cosine
2024-08-20T22:31:48.7498509Z pass
2024-08-20T22:31:48.7511794Z TIMING: backend_compile:1.95629 entire_frame_compile:3.81142 total_wall_time:3.81142
2024-08-20T22:31:48.7512951Z STATS: call_* op count: 518 | FakeTensorMode.__torch_dispatch__:9935 | ProxyTorchDispatchMode.__torch_dispatch__:4065
2024-08-20T22:31:48.7513921Z Dynamo produced 1 graphs covering 518 ops with 0 graph breaks (0 unique)
2024-08-20T22:31:52.5724536Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:31:52.5728239Z   warnings.warn(
2024-08-20T22:31:52.7583217Z 
2024-08-20T22:31:54.2738121Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:31:54.2738664Z loading model: 0it [00:01, ?it/s]
2024-08-20T22:31:54.2777228Z cuda eval  T5ForConditionalGeneration         
2024-08-20T22:32:06.3741663Z pass
2024-08-20T22:32:06.3998235Z TIMING: backend_compile:5.28888 entire_frame_compile:10.76287 total_wall_time:10.76287
2024-08-20T22:32:06.3999306Z STATS: call_* op count: 1662 | FakeTensorMode.__torch_dispatch__:29937 | ProxyTorchDispatchMode.__torch_dispatch__:12507
2024-08-20T22:32:06.4000287Z Dynamo produced 1 graphs covering 1662 ops with 0 graph breaks (0 unique)
2024-08-20T22:32:10.3448855Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:32:10.3452082Z   warnings.warn(
2024-08-20T22:32:10.6171721Z 
2024-08-20T22:32:11.9989910Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:32:11.9990373Z loading model: 0it [00:01, ?it/s]
2024-08-20T22:32:12.0027152Z cuda eval  T5Small                            
2024-08-20T22:32:23.9439597Z pass
2024-08-20T22:32:23.9696873Z TIMING: backend_compile:5.16019 entire_frame_compile:10.61712 total_wall_time:10.61712
2024-08-20T22:32:23.9697926Z STATS: call_* op count: 1662 | FakeTensorMode.__torch_dispatch__:29937 | ProxyTorchDispatchMode.__torch_dispatch__:12507
2024-08-20T22:32:23.9698971Z Dynamo produced 1 graphs covering 1662 ops with 0 graph breaks (0 unique)
2024-08-20T22:32:27.9095968Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:32:27.9097732Z   warnings.warn(
2024-08-20T22:32:28.0879931Z 
2024-08-20T22:32:32.0604819Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:32:32.0605449Z loading model: 0it [00:03, ?it/s]
2024-08-20T22:32:32.0683708Z cuda eval  TrOCRForCausalLM                   
2024-08-20T22:32:32.0993096Z WARNING:common:fp64 golden ref were not generated for TrOCRForCausalLM. Setting accuracy check to cosine
2024-08-20T22:32:39.9120224Z pass
2024-08-20T22:32:39.9249611Z TIMING: backend_compile:3.72397 entire_frame_compile:7.42561 total_wall_time:7.42561
2024-08-20T22:32:39.9250896Z STATS: call_* op count: 958 | FakeTensorMode.__torch_dispatch__:18625 | ProxyTorchDispatchMode.__torch_dispatch__:7712
2024-08-20T22:32:39.9251857Z Dynamo produced 1 graphs covering 958 ops with 0 graph breaks (0 unique)
2024-08-20T22:32:43.7576382Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:32:43.7578050Z   warnings.warn(
2024-08-20T22:32:43.9544513Z 
2024-08-20T22:32:53.7637557Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:32:53.7638030Z loading model: 0it [00:09, ?it/s]
2024-08-20T22:32:53.7728333Z cuda eval  XGLMForCausalLM                    
2024-08-20T22:32:53.8224348Z WARNING:common:fp64 golden ref were not generated for XGLMForCausalLM. Setting accuracy check to cosine
2024-08-20T22:33:09.2494941Z pass
2024-08-20T22:33:09.2572678Z TIMING: backend_compile:7.72376 entire_frame_compile:14.88201 total_wall_time:14.88201
2024-08-20T22:33:09.2573921Z STATS: call_* op count: 1982 | FakeTensorMode.__torch_dispatch__:37626 | FakeTensor.__torch_dispatch__:1 | ProxyTorchDispatchMode.__torch_dispatch__:15735
2024-08-20T22:33:09.2575038Z Dynamo produced 1 graphs covering 1982 ops with 0 graph breaks (0 unique)
2024-08-20T22:33:13.2588010Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:33:13.2589731Z   warnings.warn(
2024-08-20T22:33:13.4650811Z 
2024-08-20T22:33:17.9095242Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:33:17.9095726Z loading model: 0it [00:04, ?it/s]
2024-08-20T22:33:17.9189380Z cuda eval  XLNetLMHeadModel                   
2024-08-20T22:33:45.8931076Z pass
2024-08-20T22:33:45.9227869Z TIMING: backend_compile:14.72339 entire_frame_compile:25.02618 total_wall_time:25.02618
2024-08-20T22:33:45.9228929Z STATS: call_* op count: 1632 | FakeTensorMode.__torch_dispatch__:77980 | ProxyTorchDispatchMode.__torch_dispatch__:31646
2024-08-20T22:33:45.9229908Z Dynamo produced 1 graphs covering 1632 ops with 0 graph breaks (0 unique)
2024-08-20T22:33:50.2906005Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:33:50.2907733Z   warnings.warn(
2024-08-20T22:33:50.4658367Z 
2024-08-20T22:33:52.6245588Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:33:52.6246154Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:33:52.6320551Z cuda eval  YituTechConvBert                   
2024-08-20T22:34:06.1890869Z pass
2024-08-20T22:34:06.1975991Z TIMING: backend_compile:6.0383 entire_frame_compile:12.55874 total_wall_time:12.55874
2024-08-20T22:34:06.1977273Z STATS: call_* op count: 1268 | FakeTensorMode.__torch_dispatch__:28862 | ProxyTorchDispatchMode.__torch_dispatch__:11736
2024-08-20T22:34:06.1978247Z Dynamo produced 1 graphs covering 1268 ops with 0 graph breaks (0 unique)
2024-08-20T22:34:07.2610081Z accuracy            pass_rate=95.65%
2024-08-20T22:34:07.2614538Z calls_captured      gmean=0.00x mean=1214.261x
2024-08-20T22:34:07.2618446Z unique_graphs       gmean=0.00x mean=1.065x
2024-08-20T22:34:07.2622724Z graph_breaks        gmean=0.00x mean=0.174x
2024-08-20T22:34:07.2626614Z unique_graph_breaks gmean=0.00x mean=0.065x
2024-08-20T22:34:07.2630745Z autograd_captures   gmean=0.00x mean=0.000x
2024-08-20T22:34:07.2634613Z autograd_compiles   gmean=0.00x mean=0.000x
2024-08-20T22:34:07.2638491Z cudagraph_skips     gmean=0.00x mean=0.000x
2024-08-20T22:34:07.9033988Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_aot_eager_huggingface_inference.csv
2024-08-20T22:34:08.1651721Z AlbertForMaskedLM                   PASS
2024-08-20T22:34:08.1656072Z AlbertForQuestionAnswering          PASS
2024-08-20T22:34:08.1660293Z AllenaiLongformerBase               PASS
2024-08-20T22:34:08.1664647Z BartForCausalLM                     PASS
2024-08-20T22:34:08.1669492Z BartForConditionalGeneration        PASS
2024-08-20T22:34:08.1674520Z BertForMaskedLM                     PASS
2024-08-20T22:34:08.1679281Z BertForQuestionAnswering            PASS
2024-08-20T22:34:08.1683710Z BlenderbotForCausalLM               XFAIL
2024-08-20T22:34:08.1688409Z BlenderbotSmallForCausalLM          PASS
2024-08-20T22:34:08.1692961Z BlenderbotSmallForConditionalGeneration  PASS
2024-08-20T22:34:08.1697594Z CamemBert                           PASS
2024-08-20T22:34:08.1702088Z DebertaForMaskedLM                  PASS
2024-08-20T22:34:08.1706571Z DebertaForQuestionAnswering         PASS
2024-08-20T22:34:08.1711044Z DebertaV2ForMaskedLM                XFAIL
2024-08-20T22:34:08.1715561Z DebertaV2ForQuestionAnswering       PASS
2024-08-20T22:34:08.1720123Z DistilBertForMaskedLM               PASS
2024-08-20T22:34:08.1724645Z DistilBertForQuestionAnswering      PASS
2024-08-20T22:34:08.1729194Z DistillGPT2                         PASS
2024-08-20T22:34:08.1733642Z ElectraForCausalLM                  PASS
2024-08-20T22:34:08.1738147Z ElectraForQuestionAnswering         PASS
2024-08-20T22:34:08.1742806Z GPT2ForSequenceClassification       PASS
2024-08-20T22:34:08.1747433Z GoogleFnet                          PASS
2024-08-20T22:34:08.1751656Z LayoutLMForMaskedLM                 PASS
2024-08-20T22:34:08.1756281Z LayoutLMForSequenceClassification   PASS
2024-08-20T22:34:08.1760683Z M2M100ForConditionalGeneration      PASS
2024-08-20T22:34:08.1765270Z MBartForCausalLM                    PASS
2024-08-20T22:34:08.1770208Z MBartForConditionalGeneration       PASS
2024-08-20T22:34:08.1774872Z MT5ForConditionalGeneration         PASS
2024-08-20T22:34:08.1779282Z MegatronBertForCausalLM             PASS
2024-08-20T22:34:08.1784028Z MegatronBertForQuestionAnswering    PASS
2024-08-20T22:34:08.1788452Z MobileBertForMaskedLM               PASS
2024-08-20T22:34:08.1792925Z MobileBertForQuestionAnswering      PASS
2024-08-20T22:34:08.1797345Z OPTForCausalLM                      PASS
2024-08-20T22:34:08.1801933Z PLBartForCausalLM                   PASS
2024-08-20T22:34:08.1806466Z PLBartForConditionalGeneration      PASS
2024-08-20T22:34:08.1810933Z PegasusForCausalLM                  PASS
2024-08-20T22:34:08.1815438Z PegasusForConditionalGeneration     PASS
2024-08-20T22:34:08.1820124Z RobertaForCausalLM                  PASS
2024-08-20T22:34:08.1824741Z RobertaForQuestionAnswering         PASS
2024-08-20T22:34:08.1829184Z Speech2Text2ForCausalLM             PASS
2024-08-20T22:34:08.1833626Z T5ForConditionalGeneration          PASS
2024-08-20T22:34:08.1838161Z T5Small                             PASS
2024-08-20T22:34:08.1842615Z TrOCRForCausalLM                    PASS
2024-08-20T22:34:08.1847300Z XGLMForCausalLM                     PASS
2024-08-20T22:34:08.1851627Z XLNetLMHeadModel                    PASS
2024-08-20T22:34:08.1856293Z YituTechConvBert                    PASS
2024-08-20T22:34:08.2277826Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_aot_eager_huggingface_inference.csv
2024-08-20T22:34:08.4895115Z AlbertForMaskedLM                   PASS
2024-08-20T22:34:08.4899305Z AlbertForQuestionAnswering          PASS
2024-08-20T22:34:08.4904771Z AllenaiLongformerBase               PASS
2024-08-20T22:34:08.4910274Z BartForCausalLM                     PASS
2024-08-20T22:34:08.4915434Z BartForConditionalGeneration        PASS
2024-08-20T22:34:08.4920526Z BertForMaskedLM                     PASS
2024-08-20T22:34:08.4925695Z BertForQuestionAnswering            PASS
2024-08-20T22:34:08.4930744Z BlenderbotForCausalLM               PASS
2024-08-20T22:34:08.4936001Z BlenderbotSmallForCausalLM          PASS
2024-08-20T22:34:08.4941148Z BlenderbotSmallForConditionalGeneration  PASS
2024-08-20T22:34:08.4946130Z CamemBert                           PASS
2024-08-20T22:34:08.4950928Z DebertaForMaskedLM                  PASS
2024-08-20T22:34:08.4955900Z DebertaForQuestionAnswering         PASS
2024-08-20T22:34:08.4960852Z DebertaV2ForMaskedLM                PASS
2024-08-20T22:34:08.4965967Z DebertaV2ForQuestionAnswering       PASS
2024-08-20T22:34:08.4971275Z DistilBertForMaskedLM               PASS
2024-08-20T22:34:08.4977198Z DistilBertForQuestionAnswering      PASS
2024-08-20T22:34:08.4982126Z DistillGPT2                         PASS
2024-08-20T22:34:08.4987257Z ElectraForCausalLM                  PASS
2024-08-20T22:34:08.4992189Z ElectraForQuestionAnswering         PASS
2024-08-20T22:34:08.4997284Z GPT2ForSequenceClassification       PASS
2024-08-20T22:34:08.5002107Z GoogleFnet                          PASS
2024-08-20T22:34:08.5007218Z LayoutLMForMaskedLM                 PASS
2024-08-20T22:34:08.5012144Z LayoutLMForSequenceClassification   PASS
2024-08-20T22:34:08.5017223Z M2M100ForConditionalGeneration      PASS
2024-08-20T22:34:08.5022374Z MBartForCausalLM                    PASS
2024-08-20T22:34:08.5027424Z MBartForConditionalGeneration       PASS
2024-08-20T22:34:08.5032212Z MT5ForConditionalGeneration         PASS
2024-08-20T22:34:08.5037296Z MegatronBertForCausalLM             PASS
2024-08-20T22:34:08.5042129Z MegatronBertForQuestionAnswering    PASS
2024-08-20T22:34:08.5047412Z MobileBertForMaskedLM               PASS
2024-08-20T22:34:08.5052119Z MobileBertForQuestionAnswering      PASS
2024-08-20T22:34:08.5057175Z OPTForCausalLM                      PASS
2024-08-20T22:34:08.5062313Z PLBartForCausalLM                   PASS
2024-08-20T22:34:08.5067283Z PLBartForConditionalGeneration      PASS
2024-08-20T22:34:08.5072732Z PegasusForCausalLM                  PASS
2024-08-20T22:34:08.5077855Z PegasusForConditionalGeneration     PASS
2024-08-20T22:34:08.5082626Z RobertaForCausalLM                  PASS
2024-08-20T22:34:08.5087696Z RobertaForQuestionAnswering         PASS
2024-08-20T22:34:08.5092526Z Speech2Text2ForCausalLM             PASS
2024-08-20T22:34:08.5097593Z T5ForConditionalGeneration          PASS
2024-08-20T22:34:08.5102782Z T5Small                             PASS
2024-08-20T22:34:08.5107737Z TrOCRForCausalLM                    PASS
2024-08-20T22:34:08.5112635Z XGLMForCausalLM                     PASS
2024-08-20T22:34:08.5117662Z XLNetLMHeadModel                    PASS
2024-08-20T22:34:08.5122697Z YituTechConvBert                    PASS
2024-08-20T22:34:08.5537910Z + test_single_dynamo_benchmark training huggingface 0 --training --amp
2024-08-20T22:34:08.5542917Z ++ pwd
2024-08-20T22:34:08.5543512Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2024-08-20T22:34:08.5544203Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports
2024-08-20T22:34:08.5567868Z + local name=training
2024-08-20T22:34:08.5568258Z + shift
2024-08-20T22:34:08.5568626Z + local suite=huggingface
2024-08-20T22:34:08.5569283Z + shift
2024-08-20T22:34:08.5569607Z + local shard_id=0
2024-08-20T22:34:08.5569968Z + shift
2024-08-20T22:34:08.5570242Z + partition_flags=()
2024-08-20T22:34:08.5570573Z + local partition_flags
2024-08-20T22:34:08.5575041Z + [[ -n 1 ]]
2024-08-20T22:34:08.5575453Z + [[ -n 0 ]]
2024-08-20T22:34:08.5576073Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id")
2024-08-20T22:34:08.5576849Z + [[ dynamic_aot_eager_huggingface == *perf_compare* ]]
2024-08-20T22:34:08.5577442Z + [[ dynamic_aot_eager_huggingface == *perf* ]]
2024-08-20T22:34:08.5578159Z + [[ dynamic_aot_eager_huggingface == *aot_inductor* ]]
2024-08-20T22:34:08.5578876Z + [[ dynamic_aot_eager_huggingface == *_avx2* ]]
2024-08-20T22:34:08.5579557Z + [[ dynamic_aot_eager_huggingface == *_avx512* ]]
2024-08-20T22:34:08.5581569Z + python benchmarks/dynamo/huggingface.py --ci --accuracy --timing --explain --backend aot_eager --dynamic-shapes --dynamic-batch-only --device cuda --training --amp --total-partitions 1 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/training_huggingface.csv
2024-08-20T22:34:11.4351491Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:34:11.4353133Z   warnings.warn(
2024-08-20T22:34:14.6540569Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:34:14.6542420Z   warnings.warn(
2024-08-20T22:34:14.8799936Z 
2024-08-20T22:34:17.9111035Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:34:17.9111508Z loading model: 0it [00:03, ?it/s]
2024-08-20T22:34:17.9111960Z cuda train AlbertForMaskedLM                  
2024-08-20T22:35:02.6776526Z pass
2024-08-20T22:35:02.7592785Z TIMING: entire_frame_compile:8.88618 backend_compile:6.35961 total_wall_time:8.88618
2024-08-20T22:35:02.7593980Z STATS: call_* op count: 441 | FakeTensorMode.__torch_dispatch__:26344 | ProxyTorchDispatchMode.__torch_dispatch__:13657 | FakeTensor.__torch_dispatch__:12
2024-08-20T22:35:02.7595098Z Dynamo produced 1 graphs covering 441 ops with 4 graph breaks (4 unique)
2024-08-20T22:35:06.7587747Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:35:06.7589396Z   warnings.warn(
2024-08-20T22:35:06.9444519Z 
2024-08-20T22:35:09.9090680Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:35:09.9091187Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:35:09.9091651Z cuda train AlbertForQuestionAnswering         
2024-08-20T22:35:53.9024711Z W0820 22:35:53.901000 6076 torch/_logging/_internal.py:1058] [4/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T22:35:56.4449435Z pass
2024-08-20T22:35:56.4972987Z TIMING: entire_frame_compile:10.56245 backend_compile:7.16954 total_wall_time:10.56245
2024-08-20T22:35:56.4974224Z STATS: call_* op count: 533 | FakeTensorMode.__torch_dispatch__:28532 | ProxyTorchDispatchMode.__torch_dispatch__:14646 | FakeTensor.__torch_dispatch__:518
2024-08-20T22:35:56.4975368Z Dynamo produced 2 graphs covering 533 ops with 5 graph breaks (4 unique)
2024-08-20T22:36:00.5951390Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:36:00.5954514Z   warnings.warn(
2024-08-20T22:36:00.7969062Z 
2024-08-20T22:36:03.7473076Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:36:03.7473539Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:36:03.7477580Z cuda train AllenaiLongformerBase              
2024-08-20T22:36:11.4243183Z W0820 22:36:11.423000 6172 torch/_dynamo/variables/tensor.py:776] [3/0] Graph break from `Tensor.item()`, consider setting:
2024-08-20T22:36:11.4244503Z W0820 22:36:11.423000 6172 torch/_dynamo/variables/tensor.py:776] [3/0]     torch._dynamo.config.capture_scalar_outputs = True
2024-08-20T22:36:11.4245700Z W0820 22:36:11.423000 6172 torch/_dynamo/variables/tensor.py:776] [3/0] or:
2024-08-20T22:36:11.4246972Z W0820 22:36:11.423000 6172 torch/_dynamo/variables/tensor.py:776] [3/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2024-08-20T22:36:11.4248536Z W0820 22:36:11.423000 6172 torch/_dynamo/variables/tensor.py:776] [3/0] to include these operations in the captured graph.
2024-08-20T22:36:11.4249702Z W0820 22:36:11.423000 6172 torch/_dynamo/variables/tensor.py:776] [3/0] 
2024-08-20T22:36:11.4250857Z W0820 22:36:11.423000 6172 torch/_dynamo/variables/tensor.py:776] [3/0] Graph break: from user code at:
2024-08-20T22:36:11.4252779Z W0820 22:36:11.423000 6172 torch/_dynamo/variables/tensor.py:776] [3/0]   File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in torch_dynamo_resume_in_forward_and_backward_pass_at_522
2024-08-20T22:36:11.4254364Z W0820 22:36:11.423000 6172 torch/_dynamo/variables/tensor.py:776] [3/0]     pred = mod(**cloned_inputs)
2024-08-20T22:36:11.4256764Z W0820 22:36:11.423000 6172 torch/_dynamo/variables/tensor.py:776] [3/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1835, in forward
2024-08-20T22:36:11.4258427Z W0820 22:36:11.423000 6172 torch/_dynamo/variables/tensor.py:776] [3/0]     outputs = self.longformer(
2024-08-20T22:36:11.4260179Z W0820 22:36:11.423000 6172 torch/_dynamo/variables/tensor.py:776] [3/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1738, in forward
2024-08-20T22:36:11.4261762Z W0820 22:36:11.423000 6172 torch/_dynamo/variables/tensor.py:776] [3/0]     encoder_outputs = self.encoder(
2024-08-20T22:36:11.4263502Z W0820 22:36:11.423000 6172 torch/_dynamo/variables/tensor.py:776] [3/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/longformer/modeling_longformer.py", line 1291, in forward
2024-08-20T22:36:11.4265367Z W0820 22:36:11.423000 6172 torch/_dynamo/variables/tensor.py:776] [3/0]     is_global_attn = is_index_global_attn.flatten().any().item()
2024-08-20T22:36:11.4266377Z W0820 22:36:11.423000 6172 torch/_dynamo/variables/tensor.py:776] [3/0] 
2024-08-20T22:36:11.4267093Z W0820 22:36:11.423000 6172 torch/_dynamo/variables/tensor.py:776] [3/0] 
2024-08-20T22:36:44.1285438Z W0820 22:36:44.127000 6172 torch/_logging/_internal.py:1058] [16/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T22:37:02.5401530Z pass
2024-08-20T22:37:02.6048034Z TIMING: entire_frame_compile:51.23129 backend_compile:30.77608 total_wall_time:51.23129
2024-08-20T22:37:02.6050504Z STATS: call_* op count: 2366 | FakeTensorMode.__torch_dispatch__:91163 | ProxyTorchDispatchMode.__torch_dispatch__:48592 | FakeTensor.__torch_dispatch__:3764
2024-08-20T22:37:02.6051664Z Dynamo produced 7 graphs covering 2366 ops with 9 graph breaks (5 unique)
2024-08-20T22:37:07.5452394Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:37:07.5454054Z   warnings.warn(
2024-08-20T22:37:07.7727028Z 
2024-08-20T22:37:11.7736871Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:37:11.7737468Z loading model: 0it [00:04, ?it/s]
2024-08-20T22:37:11.7738011Z cuda train BartForCausalLM                    
2024-08-20T22:37:21.8814246Z pass
2024-08-20T22:37:22.0307866Z TIMING: entire_frame_compile:2.35565 backend_compile:1.43123 total_wall_time:2.35565
2024-08-20T22:37:22.0309280Z STATS: call_* op count: 39 | FakeTensorMode.__torch_dispatch__:2599 | FakeTensor.__torch_dispatch__:65 | ProxyTorchDispatchMode.__torch_dispatch__:1122
2024-08-20T22:37:22.0310390Z Dynamo produced 5 graphs covering 39 ops with 6 graph breaks (5 unique)
2024-08-20T22:37:25.9416387Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:37:25.9418140Z   warnings.warn(
2024-08-20T22:37:26.1220297Z 
2024-08-20T22:37:33.2392496Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:37:33.2393162Z loading model: 0it [00:07, ?it/s]
2024-08-20T22:37:33.2393884Z cuda train BartForConditionalGeneration       
2024-08-20T22:37:53.3021727Z pass
2024-08-20T22:37:53.5479668Z TIMING: entire_frame_compile:3.71326 backend_compile:2.21338 total_wall_time:3.71326
2024-08-20T22:37:53.5480840Z STATS: call_* op count: 93 | FakeTensorMode.__torch_dispatch__:5785 | FakeTensor.__torch_dispatch__:150 | ProxyTorchDispatchMode.__torch_dispatch__:2721
2024-08-20T22:37:53.5483446Z Dynamo produced 7 graphs covering 93 ops with 8 graph breaks (5 unique)
2024-08-20T22:37:57.4653019Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:37:57.4655174Z   warnings.warn(
2024-08-20T22:37:57.7021432Z 
2024-08-20T22:37:59.8520625Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:37:59.8521238Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:37:59.8521749Z cuda train BertForMaskedLM                    
2024-08-20T22:38:14.8314995Z W0820 22:38:14.830000 6459 torch/_logging/_internal.py:1058] [4/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T22:38:32.6481434Z pass
2024-08-20T22:38:32.7709901Z TIMING: entire_frame_compile:29.67956 backend_compile:15.98696 total_wall_time:29.67956
2024-08-20T22:38:32.7711454Z STATS: call_* op count: 995 | FakeTensorMode.__torch_dispatch__:46009 | ProxyTorchDispatchMode.__torch_dispatch__:21485 | FakeTensor.__torch_dispatch__:3686
2024-08-20T22:38:32.7712851Z Dynamo produced 2 graphs covering 995 ops with 5 graph breaks (4 unique)
2024-08-20T22:38:37.2282363Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:38:37.2284063Z   warnings.warn(
2024-08-20T22:38:37.4632940Z 
2024-08-20T22:38:39.3137954Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:38:39.3138929Z loading model: 0it [00:01, ?it/s]
2024-08-20T22:38:39.3140989Z cuda train BertForQuestionAnswering           
2024-08-20T22:38:53.9515024Z W0820 22:38:53.950000 6555 torch/_logging/_internal.py:1058] [4/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T22:39:11.4898060Z pass
2024-08-20T22:39:11.6088077Z TIMING: entire_frame_compile:29.40598 backend_compile:15.88889 total_wall_time:29.40598
2024-08-20T22:39:11.6089659Z STATS: call_* op count: 993 | FakeTensorMode.__torch_dispatch__:45714 | ProxyTorchDispatchMode.__torch_dispatch__:21302 | FakeTensor.__torch_dispatch__:3638
2024-08-20T22:39:11.6090909Z Dynamo produced 2 graphs covering 993 ops with 5 graph breaks (4 unique)
2024-08-20T22:39:16.0673943Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:39:16.0676356Z   warnings.warn(
2024-08-20T22:39:16.2670929Z 
2024-08-20T22:39:45.4806086Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:39:45.4808422Z loading model: 0it [00:29, ?it/s]
2024-08-20T22:39:45.4809146Z cuda train BlenderbotForCausalLM              
2024-08-20T22:39:45.4813879Z Traceback (most recent call last):
2024-08-20T22:39:45.4814857Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model
2024-08-20T22:39:45.4815666Z     self.model_iter_fn(model, example_inputs)
2024-08-20T22:39:45.4817374Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in forward_and_backward_pass
2024-08-20T22:39:45.4818193Z     pred = mod(**cloned_inputs)
2024-08-20T22:39:45.4819876Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T22:39:45.4820808Z     return self._call_impl(*args, **kwargs)
2024-08-20T22:39:45.4821756Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T22:39:45.4822593Z     return forward_call(*args, **kwargs)
2024-08-20T22:39:45.4823675Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 1531, in forward
2024-08-20T22:39:45.4824639Z     outputs = self.model.decoder(
2024-08-20T22:39:45.4825958Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T22:39:45.4826878Z     return self._call_impl(*args, **kwargs)
2024-08-20T22:39:45.4827819Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T22:39:45.4828650Z     return forward_call(*args, **kwargs)
2024-08-20T22:39:45.4829716Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 997, in forward
2024-08-20T22:39:45.4830670Z     layer_outputs = decoder_layer(
2024-08-20T22:39:45.4831851Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T22:39:45.4832897Z     return self._call_impl(*args, **kwargs)
2024-08-20T22:39:45.4833833Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T22:39:45.4834859Z     return forward_call(*args, **kwargs)
2024-08-20T22:39:45.4835919Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 397, in forward
2024-08-20T22:39:45.4837014Z     hidden_states, self_attn_weights, present_key_value = self.self_attn(
2024-08-20T22:39:45.4838153Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T22:39:45.4839040Z     return self._call_impl(*args, **kwargs)
2024-08-20T22:39:45.4839973Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T22:39:45.4840794Z     return forward_call(*args, **kwargs)
2024-08-20T22:39:45.4841855Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/blenderbot/modeling_blenderbot.py", line 152, in forward
2024-08-20T22:39:45.4842885Z     query_states = self.q_proj(hidden_states) * self.scaling
2024-08-20T22:39:45.4843959Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T22:39:45.4844895Z     return self._call_impl(*args, **kwargs)
2024-08-20T22:39:45.4845826Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T22:39:45.4846656Z     return forward_call(*args, **kwargs)
2024-08-20T22:39:45.4847550Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 125, in forward
2024-08-20T22:39:45.4848404Z     return F.linear(input, self.weight, self.bias)
2024-08-20T22:39:45.4851724Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 14.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 6.44 MiB is free. Process 60284 has 21.96 GiB memory in use. Of the allocated memory 21.61 GiB is allocated by PyTorch, and 27.77 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-08-20T22:39:45.4854513Z 
2024-08-20T22:39:45.4854817Z The above exception was the direct cause of the following exception:
2024-08-20T22:39:45.4855260Z 
2024-08-20T22:39:45.4855413Z Traceback (most recent call last):
2024-08-20T22:39:45.4856055Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run
2024-08-20T22:39:45.4856711Z     ) = runner.load_model(
2024-08-20T22:39:45.4857376Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 454, in load_model
2024-08-20T22:39:45.4858140Z     self.validate_model(model, example_inputs)
2024-08-20T22:39:45.4859012Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model
2024-08-20T22:39:45.4859765Z     raise RuntimeError("Eager run failed") from e
2024-08-20T22:39:45.4860364Z RuntimeError: Eager run failed
2024-08-20T22:39:45.4860617Z 
2024-08-20T22:39:45.4860740Z eager_fail_to_run
2024-08-20T22:39:49.1446924Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:39:49.1448646Z   warnings.warn(
2024-08-20T22:39:49.3629185Z 
2024-08-20T22:39:50.8699683Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:39:50.8700162Z loading model: 0it [00:01, ?it/s]
2024-08-20T22:39:50.8700623Z cuda train BlenderbotSmallForCausalLM         
2024-08-20T22:39:50.8921146Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForCausalLM. Setting accuracy check to cosine
2024-08-20T22:39:54.0668743Z pass
2024-08-20T22:39:54.0949137Z TIMING: entire_frame_compile:2.46703 backend_compile:1.48879 total_wall_time:2.46703
2024-08-20T22:39:54.0950744Z STATS: call_* op count: 58 | FakeTensorMode.__torch_dispatch__:2991 | FakeTensor.__torch_dispatch__:58 | ProxyTorchDispatchMode.__torch_dispatch__:1286
2024-08-20T22:39:54.0951859Z Dynamo produced 6 graphs covering 58 ops with 6 graph breaks (5 unique)
2024-08-20T22:39:58.0154313Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:39:58.0155964Z   warnings.warn(
2024-08-20T22:39:58.2309845Z 
2024-08-20T22:40:00.1898879Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:40:00.1899345Z loading model: 0it [00:01, ?it/s]
2024-08-20T22:40:00.1899838Z cuda train BlenderbotSmallForConditionalGeneration 
2024-08-20T22:40:00.2380497Z WARNING:common:fp64 golden ref were not generated for BlenderbotSmallForConditionalGeneration. Setting accuracy check to cosine
2024-08-20T22:40:05.4870888Z pass
2024-08-20T22:40:05.5386344Z TIMING: entire_frame_compile:3.89575 backend_compile:2.18488 total_wall_time:3.89575
2024-08-20T22:40:05.5387739Z STATS: call_* op count: 121 | FakeTensorMode.__torch_dispatch__:6219 | ProxyTorchDispatchMode.__torch_dispatch__:2976 | FakeTensor.__torch_dispatch__:124
2024-08-20T22:40:05.5388858Z Dynamo produced 7 graphs covering 121 ops with 8 graph breaks (5 unique)
2024-08-20T22:40:09.5207100Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:40:09.5210108Z   warnings.warn(
2024-08-20T22:40:09.7059037Z 
2024-08-20T22:40:11.8639887Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:40:11.8640368Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:40:11.8640846Z cuda train CamemBert                          
2024-08-20T22:40:26.9174284Z W0820 22:40:26.916000 6884 torch/_logging/_internal.py:1058] [4/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T22:40:44.6707443Z pass
2024-08-20T22:40:44.7885750Z TIMING: entire_frame_compile:29.66903 backend_compile:16.00025 total_wall_time:29.66903
2024-08-20T22:40:44.7887145Z STATS: call_* op count: 1003 | FakeTensorMode.__torch_dispatch__:46106 | ProxyTorchDispatchMode.__torch_dispatch__:21521 | FakeTensor.__torch_dispatch__:3686
2024-08-20T22:40:44.7892132Z Dynamo produced 2 graphs covering 1003 ops with 5 graph breaks (4 unique)
2024-08-20T22:40:49.2493334Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:40:49.2495067Z   warnings.warn(
2024-08-20T22:40:49.4366258Z 
2024-08-20T22:40:51.9513143Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:40:51.9514068Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:40:51.9514517Z cuda train DebertaForMaskedLM                 
2024-08-20T22:41:08.9373185Z W0820 22:41:08.936000 6980 torch/_logging/_internal.py:1058] [4/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T22:41:23.5696893Z pass
2024-08-20T22:41:23.6813240Z TIMING: entire_frame_compile:28.24409 backend_compile:15.23034 total_wall_time:28.24409
2024-08-20T22:41:23.6816284Z STATS: call_* op count: 1318 | FakeTensorMode.__torch_dispatch__:48482 | ProxyTorchDispatchMode.__torch_dispatch__:23799 | FakeTensor.__torch_dispatch__:3020
2024-08-20T22:41:23.6817675Z Dynamo produced 2 graphs covering 1318 ops with 5 graph breaks (4 unique)
2024-08-20T22:41:28.1660858Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:41:28.1662573Z   warnings.warn(
2024-08-20T22:41:28.4080956Z 
2024-08-20T22:41:30.5216791Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:41:30.5217847Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:41:30.5218629Z cuda train DebertaForQuestionAnswering        
2024-08-20T22:41:46.8376412Z W0820 22:41:46.836000 7076 torch/_logging/_internal.py:1058] [4/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T22:42:01.2607295Z pass
2024-08-20T22:42:01.3836573Z TIMING: entire_frame_compile:27.93578 backend_compile:15.38767 total_wall_time:27.93578
2024-08-20T22:42:01.3837865Z STATS: call_* op count: 1316 | FakeTensorMode.__torch_dispatch__:48187 | ProxyTorchDispatchMode.__torch_dispatch__:23616 | FakeTensor.__torch_dispatch__:2976
2024-08-20T22:42:01.3842996Z Dynamo produced 2 graphs covering 1316 ops with 5 graph breaks (4 unique)
2024-08-20T22:42:05.8414913Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:42:05.8416564Z   warnings.warn(
2024-08-20T22:42:06.0548908Z 
2024-08-20T22:42:18.9101087Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:42:18.9101640Z loading model: 0it [00:12, ?it/s]
2024-08-20T22:42:18.9102089Z cuda train DebertaV2ForMaskedLM               
2024-08-20T22:42:18.9102875Z pass_due_to_skip
2024-08-20T22:42:19.1351451Z TIMING:
2024-08-20T22:42:19.1351847Z STATS: call_* op count: 0
2024-08-20T22:42:19.1352456Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-08-20T22:42:22.7917011Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:42:22.7918649Z   warnings.warn(
2024-08-20T22:42:22.9990752Z 
2024-08-20T22:42:33.7249783Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:42:33.7250262Z loading model: 0it [00:10, ?it/s]
2024-08-20T22:42:33.7250719Z cuda train DebertaV2ForQuestionAnswering      
2024-08-20T22:42:44.3721802Z ERROR:common:
2024-08-20T22:42:44.3722272Z Traceback (most recent call last):
2024-08-20T22:42:44.3723058Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2732, in check_accuracy
2024-08-20T22:42:44.3723880Z     correct_result = self.run_n_iterations(
2024-08-20T22:42:44.3726715Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2545, in run_n_iterations
2024-08-20T22:42:44.3727580Z     self.model_iter_fn(mod, inputs, collect_outputs=False)
2024-08-20T22:42:44.3728496Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 524, in forward_and_backward_pass
2024-08-20T22:42:44.3729382Z     pred = mod(**cloned_inputs)
2024-08-20T22:42:44.3730875Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T22:42:44.3731767Z     return self._call_impl(*args, **kwargs)
2024-08-20T22:42:44.3732706Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T22:42:44.3733541Z     return forward_call(*args, **kwargs)
2024-08-20T22:42:44.3734780Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1486, in forward
2024-08-20T22:42:44.3735908Z     outputs = self.deberta(
2024-08-20T22:42:44.3736838Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T22:42:44.3737713Z     return self._call_impl(*args, **kwargs)
2024-08-20T22:42:44.3739290Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T22:42:44.3740216Z     return forward_call(*args, **kwargs)
2024-08-20T22:42:44.3741280Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1070, in forward
2024-08-20T22:42:44.3742220Z     encoder_outputs = self.encoder(
2024-08-20T22:42:44.3743169Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T22:42:44.3744055Z     return self._call_impl(*args, **kwargs)
2024-08-20T22:42:44.3745041Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T22:42:44.3745864Z     return forward_call(*args, **kwargs)
2024-08-20T22:42:44.3746911Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 514, in forward
2024-08-20T22:42:44.3747837Z     output_states = layer_module(
2024-08-20T22:42:44.3748792Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T22:42:44.3749671Z     return self._call_impl(*args, **kwargs)
2024-08-20T22:42:44.3750600Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T22:42:44.3751419Z     return forward_call(*args, **kwargs)
2024-08-20T22:42:44.3752458Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 362, in forward
2024-08-20T22:42:44.3753394Z     attention_output = self.attention(
2024-08-20T22:42:44.3754368Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T22:42:44.3755258Z     return self._call_impl(*args, **kwargs)
2024-08-20T22:42:44.3756182Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T22:42:44.3757014Z     return forward_call(*args, **kwargs)
2024-08-20T22:42:44.3758057Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 293, in forward
2024-08-20T22:42:44.3758978Z     self_output = self.self(
2024-08-20T22:42:44.3759889Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T22:42:44.3760768Z     return self._call_impl(*args, **kwargs)
2024-08-20T22:42:44.3761698Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T22:42:44.3762527Z     return forward_call(*args, **kwargs)
2024-08-20T22:42:44.3763564Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 733, in forward
2024-08-20T22:42:44.3764768Z     attention_probs = XSoftmax.apply(attention_scores, attention_mask, -1)
2024-08-20T22:42:44.3765822Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/function.py", line 575, in apply
2024-08-20T22:42:44.3766816Z     return super().apply(*args, **kwargs)  # type: ignore[misc]
2024-08-20T22:42:44.3767953Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 113, in forward
2024-08-20T22:42:44.3769233Z     output = torch.softmax(output, self.dim)
2024-08-20T22:42:44.3772623Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 24.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 10.44 MiB is free. Process 60973 has 21.96 GiB memory in use. Of the allocated memory 21.29 GiB is allocated by PyTorch, and 318.25 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-08-20T22:42:44.4053822Z eager_1st_run_OOM
2024-08-20T22:42:44.4054306Z TIMING:
2024-08-20T22:42:44.4054719Z STATS: call_* op count: 0
2024-08-20T22:42:44.4055498Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-08-20T22:42:48.1053460Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:42:48.1055094Z   warnings.warn(
2024-08-20T22:42:48.3470325Z 
2024-08-20T22:42:49.8242762Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:42:49.8243789Z loading model: 0it [00:01, ?it/s]
2024-08-20T22:42:49.8244681Z cuda train DistilBertForMaskedLM              
2024-08-20T22:42:57.0675852Z W0820 22:42:57.066000 7256 torch/_logging/_internal.py:1058] [4/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T22:43:06.3385574Z pass
2024-08-20T22:43:06.4020280Z TIMING: entire_frame_compile:15.35277 backend_compile:8.82825 total_wall_time:15.35277
2024-08-20T22:43:06.4021664Z STATS: call_* op count: 540 | FakeTensorMode.__torch_dispatch__:24058 | ProxyTorchDispatchMode.__torch_dispatch__:11147 | FakeTensor.__torch_dispatch__:1936
2024-08-20T22:43:06.4022784Z Dynamo produced 2 graphs covering 540 ops with 5 graph breaks (4 unique)
2024-08-20T22:43:10.5481752Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:43:10.5483445Z   warnings.warn(
2024-08-20T22:43:10.7211421Z 
2024-08-20T22:43:11.9540067Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:43:11.9540543Z loading model: 0it [00:01, ?it/s]
2024-08-20T22:43:11.9540995Z cuda train DistilBertForQuestionAnswering     
2024-08-20T22:43:19.0924123Z W0820 22:43:19.091000 7352 torch/_logging/_internal.py:1058] [4/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T22:43:28.1378525Z pass
2024-08-20T22:43:28.2011258Z TIMING: entire_frame_compile:15.12792 backend_compile:8.4797 total_wall_time:15.12792
2024-08-20T22:43:28.2012668Z STATS: call_* op count: 539 | FakeTensorMode.__torch_dispatch__:23806 | ProxyTorchDispatchMode.__torch_dispatch__:10978 | FakeTensor.__torch_dispatch__:1892
2024-08-20T22:43:28.2013779Z Dynamo produced 2 graphs covering 539 ops with 5 graph breaks (4 unique)
2024-08-20T22:43:32.3239593Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:43:32.3241306Z   warnings.warn(
2024-08-20T22:43:32.5363578Z 
2024-08-20T22:43:34.6031911Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:43:34.6032799Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:43:34.6033269Z cuda train DistillGPT2                        
2024-08-20T22:43:42.5104522Z W0820 22:43:42.509000 7448 torch/_logging/_internal.py:1058] [4/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T22:43:49.2712536Z pass
2024-08-20T22:43:49.3327188Z TIMING: entire_frame_compile:12.65933 backend_compile:7.27446 total_wall_time:12.65933
2024-08-20T22:43:49.3328578Z STATS: call_* op count: 571 | FakeTensorMode.__torch_dispatch__:20519 | ProxyTorchDispatchMode.__torch_dispatch__:9304 | FakeTensor.__torch_dispatch__:1442
2024-08-20T22:43:49.3330052Z Dynamo produced 2 graphs covering 571 ops with 5 graph breaks (4 unique)
2024-08-20T22:43:53.4547706Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:43:53.4549431Z   warnings.warn(
2024-08-20T22:43:53.8698875Z 
2024-08-20T22:43:53.8707549Z loading model: 0it [00:00, ?it/s]If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.`
2024-08-20T22:43:54.6816084Z 
2024-08-20T22:43:54.6816623Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:43:54.6817127Z cuda train ElectraForCausalLM                 
2024-08-20T22:44:08.3214274Z pass
2024-08-20T22:44:08.3428519Z TIMING: entire_frame_compile:12.24132 backend_compile:6.74777 total_wall_time:12.24132
2024-08-20T22:44:08.3429780Z STATS: call_* op count: 377 | FakeTensorMode.__torch_dispatch__:27303 | ProxyTorchDispatchMode.__torch_dispatch__:12488 | FakeTensor.__torch_dispatch__:12
2024-08-20T22:44:08.3430907Z Dynamo produced 1 graphs covering 377 ops with 4 graph breaks (4 unique)
2024-08-20T22:44:12.3886872Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:44:12.3888521Z   warnings.warn(
2024-08-20T22:44:12.5935571Z 
2024-08-20T22:44:13.3970335Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:44:13.3970809Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:44:13.3971260Z cuda train ElectraForQuestionAnswering        
2024-08-20T22:44:26.6707447Z W0820 22:44:26.669000 7639 torch/_logging/_internal.py:1058] [4/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T22:44:44.3546229Z pass
2024-08-20T22:44:44.3738719Z TIMING: entire_frame_compile:29.61629 backend_compile:15.9558 total_wall_time:29.61629
2024-08-20T22:44:44.3741656Z STATS: call_* op count: 1000 | FakeTensorMode.__torch_dispatch__:46099 | ProxyTorchDispatchMode.__torch_dispatch__:21467 | FakeTensor.__torch_dispatch__:3686
2024-08-20T22:44:44.3742985Z Dynamo produced 2 graphs covering 1000 ops with 5 graph breaks (4 unique)
2024-08-20T22:44:48.8004350Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:44:48.8006051Z   warnings.warn(
2024-08-20T22:44:49.0100572Z 
2024-08-20T22:44:51.4360069Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:44:51.4360982Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:44:51.4361902Z cuda train GPT2ForSequenceClassification      
2024-08-20T22:45:08.8521266Z W0820 22:45:08.851000 7735 torch/_logging/_internal.py:1058] [7/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T22:45:22.2118323Z pass
2024-08-20T22:45:22.3315021Z TIMING: entire_frame_compile:26.2902 backend_compile:13.23444 total_wall_time:26.2902
2024-08-20T22:45:22.3317729Z STATS: call_* op count: 1101 | FakeTensorMode.__torch_dispatch__:42781 | ProxyTorchDispatchMode.__torch_dispatch__:18351 | FakeTensor.__torch_dispatch__:2810
2024-08-20T22:45:22.3320117Z Dynamo produced 5 graphs covering 1101 ops with 7 graph breaks (5 unique)
2024-08-20T22:45:26.7267286Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:45:26.7271004Z   warnings.warn(
2024-08-20T22:45:26.9228740Z 
2024-08-20T22:45:28.4205849Z loading model: 0it [00:00, ?it/s]WARNING:common:Model GoogleFnet supports float32 only
2024-08-20T22:45:28.6307969Z 
2024-08-20T22:45:28.6308954Z loading model: 0it [00:01, ?it/s]
2024-08-20T22:45:28.6309975Z WARNING:common:Model GoogleFnet supports float32 only
2024-08-20T22:45:28.6317055Z cuda train GoogleFnet                         
2024-08-20T22:45:29.9710565Z WARNING:common:Model GoogleFnet supports float32 only
2024-08-20T22:45:36.6449650Z W0820 22:45:36.644000 7831 torch/_logging/_internal.py:1058] [4/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T22:45:45.9370230Z pass
2024-08-20T22:45:46.0003918Z TIMING: entire_frame_compile:15.30393 backend_compile:8.27446 total_wall_time:15.30393
2024-08-20T22:45:46.0005115Z STATS: call_* op count: 573 | FakeTensorMode.__torch_dispatch__:21478 | ProxyTorchDispatchMode.__torch_dispatch__:10146 | FakeTensor.__torch_dispatch__:1994
2024-08-20T22:45:46.0006234Z Dynamo produced 2 graphs covering 573 ops with 5 graph breaks (4 unique)
2024-08-20T22:45:50.0550809Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:45:50.0552509Z   warnings.warn(
2024-08-20T22:45:50.2354812Z 
2024-08-20T22:45:52.3802414Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:45:52.3803536Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:45:52.3804574Z cuda train LayoutLMForMaskedLM                
2024-08-20T22:46:07.7199751Z W0820 22:46:07.719000 7926 torch/_logging/_internal.py:1058] [4/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T22:46:25.9357668Z pass
2024-08-20T22:46:26.0630097Z TIMING: entire_frame_compile:30.4127 backend_compile:16.34077 total_wall_time:30.4127
2024-08-20T22:46:26.0631447Z STATS: call_* op count: 1033 | FakeTensorMode.__torch_dispatch__:47242 | ProxyTorchDispatchMode.__torch_dispatch__:21936 | FakeTensor.__torch_dispatch__:3762
2024-08-20T22:46:26.0632568Z Dynamo produced 2 graphs covering 1033 ops with 5 graph breaks (4 unique)
2024-08-20T22:46:30.5363453Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:46:30.5365106Z   warnings.warn(
2024-08-20T22:46:30.7727397Z 
2024-08-20T22:46:32.7404050Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:46:32.7404873Z loading model: 0it [00:01, ?it/s]
2024-08-20T22:46:32.7405343Z cuda train LayoutLMForSequenceClassification  
2024-08-20T22:46:49.8251182Z W0820 22:46:49.824000 8022 torch/_logging/_internal.py:1058] [7/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T22:47:08.1729788Z pass
2024-08-20T22:47:08.2967152Z TIMING: entire_frame_compile:32.62072 backend_compile:16.40994 total_wall_time:32.62072
2024-08-20T22:47:08.2968508Z STATS: call_* op count: 1030 | FakeTensorMode.__torch_dispatch__:50859 | ProxyTorchDispatchMode.__torch_dispatch__:22181 | FakeTensor.__torch_dispatch__:3776
2024-08-20T22:47:08.2969925Z Dynamo produced 5 graphs covering 1030 ops with 7 graph breaks (5 unique)
2024-08-20T22:47:12.7929058Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:47:12.7932270Z   warnings.warn(
2024-08-20T22:47:13.0091400Z 
2024-08-20T22:47:23.8004073Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:47:23.8004957Z loading model: 0it [00:10, ?it/s]
2024-08-20T22:47:23.8005845Z cuda train M2M100ForConditionalGeneration     
2024-08-20T22:47:23.8770706Z WARNING:common:fp64 golden ref were not generated for M2M100ForConditionalGeneration. Setting accuracy check to cosine
2024-08-20T22:47:54.8592335Z pass
2024-08-20T22:47:55.0793152Z TIMING: entire_frame_compile:28.77665 backend_compile:16.26375 total_wall_time:28.77665
2024-08-20T22:47:55.0795051Z STATS: call_* op count: 1296 | FakeTensorMode.__torch_dispatch__:67674 | ProxyTorchDispatchMode.__torch_dispatch__:32498 | FakeTensor.__torch_dispatch__:114
2024-08-20T22:47:55.0796193Z Dynamo produced 1 graphs covering 1296 ops with 4 graph breaks (4 unique)
2024-08-20T22:47:59.6607894Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:47:59.6609533Z   warnings.warn(
2024-08-20T22:47:59.9092865Z 
2024-08-20T22:48:03.9311543Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:48:03.9312053Z loading model: 0it [00:04, ?it/s]
2024-08-20T22:48:03.9312492Z cuda train MBartForCausalLM                   
2024-08-20T22:48:03.9611247Z WARNING:common:fp64 golden ref were not generated for MBartForCausalLM. Setting accuracy check to cosine
2024-08-20T22:48:07.6876688Z pass
2024-08-20T22:48:07.7735694Z TIMING: entire_frame_compile:2.50549 backend_compile:1.63751 total_wall_time:2.50549
2024-08-20T22:48:07.7738082Z STATS: call_* op count: 60 | FakeTensorMode.__torch_dispatch__:3031 | FakeTensor.__torch_dispatch__:58 | ProxyTorchDispatchMode.__torch_dispatch__:1295
2024-08-20T22:48:07.7740326Z Dynamo produced 6 graphs covering 60 ops with 6 graph breaks (5 unique)
2024-08-20T22:48:11.6779116Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:48:11.6780744Z   warnings.warn(
2024-08-20T22:48:11.9005332Z 
2024-08-20T22:48:19.0086212Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:48:19.0086883Z loading model: 0it [00:07, ?it/s]
2024-08-20T22:48:19.0087576Z cuda train MBartForConditionalGeneration      
2024-08-20T22:48:19.1073805Z WARNING:common:fp64 golden ref were not generated for MBartForConditionalGeneration. Setting accuracy check to cosine
2024-08-20T22:48:26.5232928Z pass
2024-08-20T22:48:26.7499817Z TIMING: entire_frame_compile:4.00057 backend_compile:2.38653 total_wall_time:4.00057
2024-08-20T22:48:26.7501039Z STATS: call_* op count: 136 | FakeTensorMode.__torch_dispatch__:6556 | FakeTensor.__torch_dispatch__:126 | ProxyTorchDispatchMode.__torch_dispatch__:3088
2024-08-20T22:48:26.7502146Z Dynamo produced 8 graphs covering 136 ops with 8 graph breaks (5 unique)
2024-08-20T22:48:30.7177760Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:48:30.7181088Z   warnings.warn(
2024-08-20T22:48:30.9065138Z 
2024-08-20T22:48:35.0778633Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:48:35.0779089Z loading model: 0it [00:04, ?it/s]
2024-08-20T22:48:35.0779546Z cuda train MT5ForConditionalGeneration        
2024-08-20T22:48:59.2776794Z W0820 22:48:59.276000 8406 torch/_logging/_internal.py:1058] [4/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T22:49:18.3519446Z pass
2024-08-20T22:49:18.6647925Z TIMING: entire_frame_compile:40.01438 backend_compile:22.4197 total_wall_time:40.01438
2024-08-20T22:49:18.6652372Z STATS: call_* op count: 1763 | FakeTensorMode.__torch_dispatch__:73137 | ProxyTorchDispatchMode.__torch_dispatch__:36442 | FakeTensor.__torch_dispatch__:3534
2024-08-20T22:49:18.6653605Z Dynamo produced 2 graphs covering 1763 ops with 5 graph breaks (4 unique)
2024-08-20T22:49:23.4394255Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:49:23.4395886Z   warnings.warn(
2024-08-20T22:49:23.6673386Z 
2024-08-20T22:49:23.6711811Z loading model: 0it [00:00, ?it/s]If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.`
2024-08-20T22:49:28.4839198Z 
2024-08-20T22:49:28.4839662Z loading model: 0it [00:04, ?it/s]
2024-08-20T22:49:28.4842462Z cuda train MegatronBertForCausalLM            
2024-08-20T22:49:59.0939706Z W0820 22:49:59.093000 8502 torch/_logging/_internal.py:1058] [4/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T22:50:34.5140991Z pass
2024-08-20T22:50:34.8717857Z TIMING: entire_frame_compile:57.9727 backend_compile:30.96762 total_wall_time:57.9727
2024-08-20T22:50:34.8719070Z STATS: call_* op count: 1922 | FakeTensorMode.__torch_dispatch__:89495 | ProxyTorchDispatchMode.__torch_dispatch__:42045 | FakeTensor.__torch_dispatch__:7142
2024-08-20T22:50:34.8720201Z Dynamo produced 2 graphs covering 1922 ops with 5 graph breaks (4 unique)
2024-08-20T22:50:39.9648767Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:50:39.9651073Z   warnings.warn(
2024-08-20T22:50:40.1597941Z 
2024-08-20T22:50:44.6531977Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:50:44.6532432Z loading model: 0it [00:04, ?it/s]
2024-08-20T22:51:14.8444598Z cuda train MegatronBertForQuestionAnswering   
2024-08-20T22:51:14.8446706Z W0820 22:51:14.843000 8598 torch/_logging/_internal.py:1058] [4/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T22:51:50.0112489Z pass
2024-08-20T22:51:50.3594516Z TIMING: entire_frame_compile:57.76891 backend_compile:30.91816 total_wall_time:57.76891
2024-08-20T22:51:50.3595735Z STATS: call_* op count: 1916 | FakeTensorMode.__torch_dispatch__:89092 | ProxyTorchDispatchMode.__torch_dispatch__:41827 | FakeTensor.__torch_dispatch__:7094
2024-08-20T22:51:50.3596877Z Dynamo produced 2 graphs covering 1916 ops with 5 graph breaks (4 unique)
2024-08-20T22:51:55.5063687Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:51:55.5065402Z   warnings.warn(
2024-08-20T22:51:55.7035559Z 
2024-08-20T22:51:57.1685705Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:51:57.1686179Z loading model: 0it [00:01, ?it/s]
2024-08-20T22:51:57.1686634Z cuda train MobileBertForMaskedLM              
2024-08-20T22:52:55.0554075Z pass
2024-08-20T22:52:55.1065418Z TIMING: entire_frame_compile:54.05515 backend_compile:27.10251 total_wall_time:54.05515
2024-08-20T22:52:55.1066654Z STATS: call_* op count: 1449 | FakeTensorMode.__torch_dispatch__:109311 | ProxyTorchDispatchMode.__torch_dispatch__:55808 | FakeTensor.__torch_dispatch__:14
2024-08-20T22:52:55.1067803Z Dynamo produced 1 graphs covering 1449 ops with 3 graph breaks (3 unique)
2024-08-20T22:53:00.1188467Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:53:00.1190171Z   warnings.warn(
2024-08-20T22:53:00.3220385Z 
2024-08-20T22:53:01.6217573Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:53:59.5538101Z loading model: 0it [00:01, ?it/s]
2024-08-20T22:53:59.5538713Z cuda train MobileBertForQuestionAnswering     
2024-08-20T22:53:59.5539171Z pass
2024-08-20T22:53:59.5902246Z TIMING: entire_frame_compile:54.24237 backend_compile:27.16552 total_wall_time:54.24237
2024-08-20T22:53:59.5903510Z STATS: call_* op count: 1453 | FakeTensorMode.__torch_dispatch__:109224 | ProxyTorchDispatchMode.__torch_dispatch__:55709 | FakeTensor.__torch_dispatch__:18
2024-08-20T22:53:59.5905104Z Dynamo produced 1 graphs covering 1453 ops with 3 graph breaks (3 unique)
2024-08-20T22:54:04.6066004Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:54:04.6067609Z   warnings.warn(
2024-08-20T22:54:04.8398105Z 
2024-08-20T22:54:07.4504833Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:54:07.4505566Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:54:07.4506243Z cuda train OPTForCausalLM                     
2024-08-20T22:54:07.4700539Z WARNING:common:fp64 golden ref were not generated for OPTForCausalLM. Setting accuracy check to cosine
2024-08-20T22:54:12.0743797Z pass
2024-08-20T22:54:12.1216914Z TIMING: entire_frame_compile:2.593 backend_compile:1.72692 total_wall_time:2.593
2024-08-20T22:54:12.1218078Z STATS: call_* op count: 82 | FakeTensorMode.__torch_dispatch__:3327 | FakeTensor.__torch_dispatch__:56 | ProxyTorchDispatchMode.__torch_dispatch__:1405
2024-08-20T22:54:12.1219220Z Dynamo produced 6 graphs covering 82 ops with 6 graph breaks (5 unique)
2024-08-20T22:54:16.0638160Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:54:16.0639793Z   warnings.warn(
2024-08-20T22:54:16.2557191Z 
2024-08-20T22:54:18.2659679Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:54:18.2660165Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:54:18.2660623Z cuda train PLBartForCausalLM                  
2024-08-20T22:54:18.2828390Z WARNING:common:fp64 golden ref were not generated for PLBartForCausalLM. Setting accuracy check to cosine
2024-08-20T22:54:21.3532431Z pass
2024-08-20T22:54:21.3975858Z TIMING: entire_frame_compile:2.38418 backend_compile:1.52822 total_wall_time:2.38418
2024-08-20T22:54:21.3977026Z STATS: call_* op count: 60 | FakeTensorMode.__torch_dispatch__:3067 | FakeTensor.__torch_dispatch__:58 | ProxyTorchDispatchMode.__torch_dispatch__:1304
2024-08-20T22:54:21.3979291Z Dynamo produced 6 graphs covering 60 ops with 6 graph breaks (5 unique)
2024-08-20T22:54:25.3308613Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:54:25.3310257Z   warnings.warn(
2024-08-20T22:54:25.5077713Z 
2024-08-20T22:54:28.9715556Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:54:28.9716029Z loading model: 0it [00:03, ?it/s]
2024-08-20T22:54:28.9716496Z cuda train PLBartForConditionalGeneration     
2024-08-20T22:54:29.0251883Z WARNING:common:fp64 golden ref were not generated for PLBartForConditionalGeneration. Setting accuracy check to cosine
2024-08-20T22:54:34.6256518Z pass
2024-08-20T22:54:34.6966665Z TIMING: entire_frame_compile:4.05686 backend_compile:2.46461 total_wall_time:4.05686
2024-08-20T22:54:34.6969271Z STATS: call_* op count: 137 | FakeTensorMode.__torch_dispatch__:6716 | FakeTensor.__torch_dispatch__:134 | ProxyTorchDispatchMode.__torch_dispatch__:3161
2024-08-20T22:54:34.6971447Z Dynamo produced 8 graphs covering 137 ops with 8 graph breaks (5 unique)
2024-08-20T22:54:38.6803936Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:54:38.6805578Z   warnings.warn(
2024-08-20T22:54:38.8680121Z 
2024-08-20T22:54:45.5203498Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:54:45.5204531Z loading model: 0it [00:06, ?it/s]
2024-08-20T22:54:45.5205428Z cuda train PegasusForCausalLM                 
2024-08-20T22:54:45.5509618Z WARNING:common:fp64 golden ref were not generated for PegasusForCausalLM. Setting accuracy check to cosine
2024-08-20T22:54:49.0639139Z pass
2024-08-20T22:54:49.1503116Z TIMING: entire_frame_compile:2.47138 backend_compile:1.65273 total_wall_time:2.47138
2024-08-20T22:54:49.1505556Z STATS: call_* op count: 58 | FakeTensorMode.__torch_dispatch__:2936 | FakeTensor.__torch_dispatch__:56 | ProxyTorchDispatchMode.__torch_dispatch__:1276
2024-08-20T22:54:49.1506688Z Dynamo produced 6 graphs covering 58 ops with 6 graph breaks (5 unique)
2024-08-20T22:54:53.0618404Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:54:53.0620060Z   warnings.warn(
2024-08-20T22:54:53.2656295Z 
2024-08-20T22:55:03.5666438Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:55:03.5667116Z loading model: 0it [00:10, ?it/s]
2024-08-20T22:55:03.5667748Z cuda train PegasusForConditionalGeneration    
2024-08-20T22:55:03.6412530Z WARNING:common:fp64 golden ref were not generated for PegasusForConditionalGeneration. Setting accuracy check to cosine
2024-08-20T22:55:09.7274285Z pass
2024-08-20T22:55:09.9342774Z TIMING: entire_frame_compile:3.8754 backend_compile:2.3517 total_wall_time:3.8754
2024-08-20T22:55:09.9344162Z STATS: call_* op count: 121 | FakeTensorMode.__torch_dispatch__:6137 | ProxyTorchDispatchMode.__torch_dispatch__:2966 | FakeTensor.__torch_dispatch__:122
2024-08-20T22:55:09.9345269Z Dynamo produced 7 graphs covering 121 ops with 7 graph breaks (4 unique)
2024-08-20T22:55:13.8960865Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:55:13.8962546Z   warnings.warn(
2024-08-20T22:55:14.1195073Z 
2024-08-20T22:55:14.1224554Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`
2024-08-20T22:55:16.2732120Z We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked.
2024-08-20T22:55:16.2734033Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded.
2024-08-20T22:55:16.5511833Z 
2024-08-20T22:55:16.5512328Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:55:16.5514559Z cuda train RobertaForCausalLM                 
2024-08-20T22:55:31.8324858Z W0820 22:55:31.831000 9365 torch/_logging/_internal.py:1058] [4/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T22:55:49.5898751Z pass
2024-08-20T22:55:49.7004435Z TIMING: entire_frame_compile:29.67529 backend_compile:15.97357 total_wall_time:29.67529
2024-08-20T22:55:49.7008411Z STATS: call_* op count: 1007 | FakeTensorMode.__torch_dispatch__:46201 | ProxyTorchDispatchMode.__torch_dispatch__:21552 | FakeTensor.__torch_dispatch__:3686
2024-08-20T22:55:49.7009631Z Dynamo produced 2 graphs covering 1007 ops with 5 graph breaks (4 unique)
2024-08-20T22:55:54.1550236Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:55:54.1551955Z   warnings.warn(
2024-08-20T22:55:54.3807043Z 
2024-08-20T22:55:56.1135047Z loading model: 0it [00:00, ?it/s]We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked.
2024-08-20T22:55:56.1137134Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded.
2024-08-20T22:55:56.4234701Z 
2024-08-20T22:55:56.4235334Z loading model: 0it [00:02, ?it/s]
2024-08-20T22:55:56.4235996Z cuda train RobertaForQuestionAnswering        
2024-08-20T22:56:11.1361519Z W0820 22:56:11.135000 9461 torch/_logging/_internal.py:1058] [4/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T22:56:28.7820571Z pass
2024-08-20T22:56:28.9092067Z TIMING: entire_frame_compile:29.55153 backend_compile:15.91157 total_wall_time:29.55153
2024-08-20T22:56:28.9093271Z STATS: call_* op count: 1000 | FakeTensorMode.__torch_dispatch__:45794 | ProxyTorchDispatchMode.__torch_dispatch__:21334 | FakeTensor.__torch_dispatch__:3638
2024-08-20T22:56:28.9097330Z Dynamo produced 2 graphs covering 1000 ops with 5 graph breaks (4 unique)
2024-08-20T22:56:33.3734127Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:56:33.3735840Z   warnings.warn(
2024-08-20T22:56:33.5597009Z 
2024-08-20T22:56:34.1782291Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:56:34.1782763Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:56:34.1783216Z cuda train Speech2Text2ForCausalLM            
2024-08-20T22:56:34.2866232Z WARNING:common:fp64 golden ref were not generated for Speech2Text2ForCausalLM. Setting accuracy check to cosine
2024-08-20T22:56:37.3418462Z pass
2024-08-20T22:56:37.3475920Z TIMING: entire_frame_compile:2.52951 backend_compile:1.53669 total_wall_time:2.52951
2024-08-20T22:56:37.3477317Z STATS: call_* op count: 68 | FakeTensorMode.__torch_dispatch__:3128 | FakeTensor.__torch_dispatch__:58 | ProxyTorchDispatchMode.__torch_dispatch__:1342
2024-08-20T22:56:37.3478432Z Dynamo produced 6 graphs covering 68 ops with 6 graph breaks (5 unique)
2024-08-20T22:56:41.2519827Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:56:41.2521516Z   warnings.warn(
2024-08-20T22:56:41.4529813Z 
2024-08-20T22:56:42.9643532Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:56:42.9644433Z loading model: 0it [00:01, ?it/s]
2024-08-20T22:56:42.9645349Z cuda train T5ForConditionalGeneration         
2024-08-20T22:57:02.2823902Z W0820 22:57:02.281000 9653 torch/_logging/_internal.py:1058] [4/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T22:57:15.1928482Z pass
2024-08-20T22:57:15.2554675Z TIMING: entire_frame_compile:28.19124 backend_compile:16.00636 total_wall_time:28.19124
2024-08-20T22:57:15.2556170Z STATS: call_* op count: 1223 | FakeTensorMode.__torch_dispatch__:50300 | ProxyTorchDispatchMode.__torch_dispatch__:24715 | FakeTensor.__torch_dispatch__:2452
2024-08-20T22:57:15.2560983Z Dynamo produced 2 graphs covering 1223 ops with 5 graph breaks (4 unique)
2024-08-20T22:57:19.6799738Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:57:19.6801443Z   warnings.warn(
2024-08-20T22:57:19.9466616Z 
2024-08-20T22:57:21.5573185Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:57:21.5573654Z loading model: 0it [00:01, ?it/s]
2024-08-20T22:57:21.5574084Z cuda train T5Small                            
2024-08-20T22:57:40.6809766Z W0820 22:57:40.680000 9749 torch/_logging/_internal.py:1058] [4/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T22:57:53.4610519Z pass
2024-08-20T22:57:53.5287001Z TIMING: entire_frame_compile:27.90861 backend_compile:15.7872 total_wall_time:27.90861
2024-08-20T22:57:53.5288211Z STATS: call_* op count: 1223 | FakeTensorMode.__torch_dispatch__:50300 | ProxyTorchDispatchMode.__torch_dispatch__:24715 | FakeTensor.__torch_dispatch__:2452
2024-08-20T22:57:53.5292307Z Dynamo produced 2 graphs covering 1223 ops with 5 graph breaks (4 unique)
2024-08-20T22:57:57.9808240Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:57:57.9809869Z   warnings.warn(
2024-08-20T22:57:58.1650334Z 
2024-08-20T22:58:02.1658559Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:58:02.1659283Z loading model: 0it [00:04, ?it/s]
2024-08-20T22:58:02.1660021Z cuda train TrOCRForCausalLM                   
2024-08-20T22:58:02.1977814Z WARNING:common:fp64 golden ref were not generated for TrOCRForCausalLM. Setting accuracy check to cosine
2024-08-20T22:58:05.7382070Z pass
2024-08-20T22:58:05.8111882Z TIMING: entire_frame_compile:2.50361 backend_compile:1.64369 total_wall_time:2.50361
2024-08-20T22:58:05.8113129Z STATS: call_* op count: 59 | FakeTensorMode.__torch_dispatch__:3025 | ProxyTorchDispatchMode.__torch_dispatch__:1295 | FakeTensor.__torch_dispatch__:58
2024-08-20T22:58:05.8114213Z Dynamo produced 6 graphs covering 59 ops with 6 graph breaks (5 unique)
2024-08-20T22:58:09.7071377Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:58:09.7073118Z   warnings.warn(
2024-08-20T22:58:09.9141549Z 
2024-08-20T22:58:19.8890732Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:58:19.8891209Z loading model: 0it [00:09, ?it/s]
2024-08-20T22:58:19.8891660Z cuda train XGLMForCausalLM                    
2024-08-20T22:58:19.9411450Z WARNING:common:fp64 golden ref were not generated for XGLMForCausalLM. Setting accuracy check to cosine
2024-08-20T22:58:24.5535422Z pass
2024-08-20T22:58:24.7221420Z TIMING: entire_frame_compile:2.60861 backend_compile:1.73102 total_wall_time:2.60861
2024-08-20T22:58:24.7223703Z STATS: call_* op count: 67 | FakeTensorMode.__torch_dispatch__:3219 | FakeTensor.__torch_dispatch__:61 | ProxyTorchDispatchMode.__torch_dispatch__:1397
2024-08-20T22:58:24.7224840Z Dynamo produced 6 graphs covering 67 ops with 6 graph breaks (5 unique)
2024-08-20T22:58:28.6815002Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T22:58:28.6817104Z   warnings.warn(
2024-08-20T22:58:28.8972236Z 
2024-08-20T22:58:34.5345214Z loading model: 0it [00:00, ?it/s]
2024-08-20T22:58:34.5345666Z loading model: 0it [00:05, ?it/s]
2024-08-20T22:58:34.5346109Z cuda train XLNetLMHeadModel                   
2024-08-20T22:59:16.7819141Z W0820 22:59:16.780000 10037 torch/_logging/_internal.py:1058] [4/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T22:59:54.3647711Z pass
2024-08-20T22:59:54.7429797Z TIMING: entire_frame_compile:70.00575 backend_compile:38.45201 total_wall_time:70.00575
2024-08-20T22:59:54.7431010Z STATS: call_* op count: 1873 | FakeTensorMode.__torch_dispatch__:120728 | ProxyTorchDispatchMode.__torch_dispatch__:55611 | FakeTensor.__torch_dispatch__:6562
2024-08-20T22:59:54.7432145Z Dynamo produced 2 graphs covering 1873 ops with 5 graph breaks (4 unique)
2024-08-20T23:00:00.1551436Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1150: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-08-20T23:00:00.1553091Z   warnings.warn(
2024-08-20T23:00:00.3450627Z 
2024-08-20T23:00:02.6147377Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:00:02.6147844Z loading model: 0it [00:02, ?it/s]
2024-08-20T23:00:02.6148289Z cuda train YituTechConvBert                   
2024-08-20T23:00:23.9881904Z W0820 23:00:23.987000 10133 torch/_logging/_internal.py:1058] [4/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T23:00:50.6835636Z pass
2024-08-20T23:00:50.7825023Z TIMING: entire_frame_compile:44.65562 backend_compile:24.37588 total_wall_time:44.65562
2024-08-20T23:00:50.7828691Z STATS: call_* op count: 1511 | FakeTensorMode.__torch_dispatch__:67175 | ProxyTorchDispatchMode.__torch_dispatch__:31640 | FakeTensor.__torch_dispatch__:5226
2024-08-20T23:00:50.7829877Z Dynamo produced 2 graphs covering 1511 ops with 5 graph breaks (4 unique)
2024-08-20T23:00:52.6873486Z accuracy            pass_rate=93.48%
2024-08-20T23:00:52.6876567Z calls_captured      gmean=0.00x mean=750.522x
2024-08-20T23:00:52.6879808Z unique_graphs       gmean=0.00x mean=3.348x
2024-08-20T23:00:52.6883320Z graph_breaks        gmean=0.00x mean=5.196x
2024-08-20T23:00:52.6886802Z unique_graph_breaks gmean=0.00x mean=4.043x
2024-08-20T23:00:52.6890184Z autograd_captures   gmean=0.00x mean=0.000x
2024-08-20T23:00:52.6893482Z autograd_compiles   gmean=0.00x mean=0.000x
2024-08-20T23:00:52.6896854Z cudagraph_skips     gmean=0.00x mean=0.000x
2024-08-20T23:00:53.2986789Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_aot_eager_huggingface_training.csv
2024-08-20T23:00:53.5602955Z AlbertForMaskedLM                   PASS
2024-08-20T23:00:53.5607028Z AlbertForQuestionAnswering          PASS
2024-08-20T23:00:53.5611941Z AllenaiLongformerBase               PASS
2024-08-20T23:00:53.5616889Z BartForCausalLM                     PASS
2024-08-20T23:00:53.5622489Z BartForConditionalGeneration        PASS
2024-08-20T23:00:53.5627303Z BertForMaskedLM                     PASS
2024-08-20T23:00:53.5632190Z BertForQuestionAnswering            PASS
2024-08-20T23:00:53.5637134Z BlenderbotForCausalLM               XFAIL
2024-08-20T23:00:53.5642085Z BlenderbotSmallForCausalLM          PASS
2024-08-20T23:00:53.5647164Z BlenderbotSmallForConditionalGeneration  PASS
2024-08-20T23:00:53.5652029Z CamemBert                           PASS
2024-08-20T23:00:53.5657278Z DebertaForMaskedLM                  PASS
2024-08-20T23:00:53.5662184Z DebertaForQuestionAnswering         PASS
2024-08-20T23:00:53.5667030Z DebertaV2ForMaskedLM                XFAIL
2024-08-20T23:00:53.5672700Z DebertaV2ForQuestionAnswering       XFAIL
2024-08-20T23:00:53.5677581Z DistilBertForMaskedLM               PASS
2024-08-20T23:00:53.5682533Z DistilBertForQuestionAnswering      PASS
2024-08-20T23:00:53.5687423Z DistillGPT2                         PASS
2024-08-20T23:00:53.5692090Z ElectraForCausalLM                  PASS
2024-08-20T23:00:53.5696743Z ElectraForQuestionAnswering         PASS
2024-08-20T23:00:53.5701550Z GPT2ForSequenceClassification       PASS
2024-08-20T23:00:53.5706039Z GoogleFnet                          PASS
2024-08-20T23:00:53.5710519Z LayoutLMForMaskedLM                 PASS
2024-08-20T23:00:53.5715155Z LayoutLMForSequenceClassification   PASS
2024-08-20T23:00:53.5719397Z M2M100ForConditionalGeneration      PASS
2024-08-20T23:00:53.5723875Z MBartForCausalLM                    PASS
2024-08-20T23:00:53.5728344Z MBartForConditionalGeneration       PASS
2024-08-20T23:00:53.5732912Z MT5ForConditionalGeneration         PASS
2024-08-20T23:00:53.5737521Z MegatronBertForCausalLM             PASS
2024-08-20T23:00:53.5742122Z MegatronBertForQuestionAnswering    PASS
2024-08-20T23:00:53.5746785Z MobileBertForMaskedLM               PASS
2024-08-20T23:00:53.5751032Z MobileBertForQuestionAnswering      PASS
2024-08-20T23:00:53.5755730Z OPTForCausalLM                      PASS
2024-08-20T23:00:53.5760137Z PLBartForCausalLM                   PASS
2024-08-20T23:00:53.5764532Z PLBartForConditionalGeneration      PASS
2024-08-20T23:00:53.5769375Z PegasusForCausalLM                  PASS
2024-08-20T23:00:53.5775070Z PegasusForConditionalGeneration     PASS
2024-08-20T23:00:53.5779710Z RobertaForCausalLM                  PASS
2024-08-20T23:00:53.5784565Z RobertaForQuestionAnswering         PASS
2024-08-20T23:00:53.5788850Z Speech2Text2ForCausalLM             PASS
2024-08-20T23:00:53.5793272Z T5ForConditionalGeneration          PASS
2024-08-20T23:00:53.5797731Z T5Small                             PASS
2024-08-20T23:00:53.5802334Z TrOCRForCausalLM                    PASS
2024-08-20T23:00:53.5806898Z XGLMForCausalLM                     PASS
2024-08-20T23:00:53.5811472Z XLNetLMHeadModel                    PASS
2024-08-20T23:00:53.5815980Z YituTechConvBert                    PASS
2024-08-20T23:00:53.6229663Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_aot_eager_huggingface_training.csv
2024-08-20T23:00:53.8843466Z AlbertForMaskedLM                   PASS
2024-08-20T23:00:53.8847105Z AlbertForQuestionAnswering          PASS
2024-08-20T23:00:53.8851873Z AllenaiLongformerBase               PASS
2024-08-20T23:00:53.8856715Z BartForCausalLM                     PASS
2024-08-20T23:00:53.8861739Z BartForConditionalGeneration        PASS
2024-08-20T23:00:53.8866521Z BertForMaskedLM                     PASS
2024-08-20T23:00:53.8872024Z BertForQuestionAnswering            PASS
2024-08-20T23:00:53.8876747Z BlenderbotForCausalLM               PASS
2024-08-20T23:00:53.8881710Z BlenderbotSmallForCausalLM          PASS
2024-08-20T23:00:53.8886554Z BlenderbotSmallForConditionalGeneration  PASS
2024-08-20T23:00:53.8891370Z CamemBert                           PASS
2024-08-20T23:00:53.8896105Z DebertaForMaskedLM                  PASS
2024-08-20T23:00:53.8901250Z DebertaForQuestionAnswering         PASS
2024-08-20T23:00:53.8906049Z DebertaV2ForMaskedLM                PASS
2024-08-20T23:00:53.8911027Z DebertaV2ForQuestionAnswering       PASS
2024-08-20T23:00:53.8915853Z DistilBertForMaskedLM               PASS
2024-08-20T23:00:53.8921162Z DistilBertForQuestionAnswering      PASS
2024-08-20T23:00:53.8925740Z DistillGPT2                         PASS
2024-08-20T23:00:53.8930669Z ElectraForCausalLM                  PASS
2024-08-20T23:00:53.8935477Z ElectraForQuestionAnswering         PASS
2024-08-20T23:00:53.8940536Z GPT2ForSequenceClassification       PASS
2024-08-20T23:00:53.8945407Z GoogleFnet                          PASS
2024-08-20T23:00:53.8950497Z LayoutLMForMaskedLM                 PASS
2024-08-20T23:00:53.8955215Z LayoutLMForSequenceClassification   PASS
2024-08-20T23:00:53.8960156Z M2M100ForConditionalGeneration      PASS
2024-08-20T23:00:53.8964960Z MBartForCausalLM                    PASS
2024-08-20T23:00:53.8970137Z MBartForConditionalGeneration       PASS
2024-08-20T23:00:53.8978851Z MT5ForConditionalGeneration         PASS
2024-08-20T23:00:53.8983964Z MegatronBertForCausalLM             PASS
2024-08-20T23:00:53.8988693Z MegatronBertForQuestionAnswering    PASS
2024-08-20T23:00:53.8993646Z MobileBertForMaskedLM               PASS
2024-08-20T23:00:53.8998386Z MobileBertForQuestionAnswering      PASS
2024-08-20T23:00:53.9003435Z OPTForCausalLM                      PASS
2024-08-20T23:00:53.9008218Z PLBartForCausalLM                   PASS
2024-08-20T23:00:53.9013292Z PLBartForConditionalGeneration      PASS
2024-08-20T23:00:53.9018005Z PegasusForCausalLM                  PASS
2024-08-20T23:00:53.9023166Z PegasusForConditionalGeneration     PASS
2024-08-20T23:00:53.9027956Z RobertaForCausalLM                  PASS
2024-08-20T23:00:53.9032888Z RobertaForQuestionAnswering         PASS
2024-08-20T23:00:53.9037923Z Speech2Text2ForCausalLM             PASS
2024-08-20T23:00:53.9042704Z T5ForConditionalGeneration          PASS
2024-08-20T23:00:53.9047557Z T5Small                             PASS
2024-08-20T23:00:53.9052627Z TrOCRForCausalLM                    PASS
2024-08-20T23:00:53.9057423Z XGLMForCausalLM                     PASS
2024-08-20T23:00:53.9062574Z XLNetLMHeadModel                    PASS
2024-08-20T23:00:53.9067423Z YituTechConvBert                    PASS
2024-08-20T23:00:53.9488272Z + cleanup_workspace
2024-08-20T23:00:53.9490895Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.'
2024-08-20T23:00:53.9491987Z sudo may print the following warning message that can be ignored. The chown command will still run.
2024-08-20T23:00:53.9492902Z + echo '    sudo: setrlimit(RLIMIT_STACK): Operation not permitted'
2024-08-20T23:00:53.9493535Z     sudo: setrlimit(RLIMIT_STACK): Operation not permitted
2024-08-20T23:00:53.9494343Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42'
2024-08-20T23:00:53.9495238Z For more details refer to https://github.com/sudo-project/sudo/issues/42
2024-08-20T23:00:53.9495950Z + sudo chown -R 1000 /var/lib/jenkins/workspace
2024-08-20T23:00:54.6848709Z ##[group]Run cat test/**/*_toprint.log || true
2024-08-20T23:00:54.6849233Z cat test/**/*_toprint.log || true
2024-08-20T23:00:54.6863034Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T23:00:54.6863542Z env:
2024-08-20T23:00:54.6863812Z   GIT_DEFAULT_BRANCH: main
2024-08-20T23:00:54.6864265Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T23:00:54.6864993Z   DOCKER_CONTAINER_ID: 0069e58a274b850ea9dc89e5a84233030f0d0aca44320cdb9963abb2a7d1ef4f
2024-08-20T23:00:54.6865631Z ##[endgroup]
2024-08-20T23:00:54.6959241Z cat: 'test/**/*_toprint.log': No such file or directory
2024-08-20T23:00:54.6991596Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2024-08-20T23:00:54.6992062Z kill "$MONITOR_SCRIPT_PID"
2024-08-20T23:00:54.7000553Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T23:00:54.7001055Z env:
2024-08-20T23:00:54.7001328Z   GIT_DEFAULT_BRANCH: main
2024-08-20T23:00:54.7001784Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T23:00:54.7002512Z   DOCKER_CONTAINER_ID: 0069e58a274b850ea9dc89e5a84233030f0d0aca44320cdb9963abb2a7d1ef4f
2024-08-20T23:00:54.7003164Z   MONITOR_SCRIPT_PID: 52479
2024-08-20T23:00:54.7003511Z ##[endgroup]
2024-08-20T23:00:54.7198326Z Prepare all required actions
2024-08-20T23:00:54.7198770Z Getting action download info
2024-08-20T23:00:54.9137700Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32)
2024-08-20T23:00:55.0733034Z ##[group]Run ./.github/actions/upload-test-artifacts
2024-08-20T23:00:55.0733495Z with:
2024-08-20T23:00:55.0734061Z   file-suffix: test-dynamic_aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026300273
2024-08-20T23:00:55.0734761Z   s3-bucket: gha-artifacts
2024-08-20T23:00:55.0735283Z env:
2024-08-20T23:00:55.0735554Z   GIT_DEFAULT_BRANCH: main
2024-08-20T23:00:55.0736007Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T23:00:55.0736791Z   DOCKER_CONTAINER_ID: 0069e58a274b850ea9dc89e5a84233030f0d0aca44320cdb9963abb2a7d1ef4f
2024-08-20T23:00:55.0737424Z ##[endgroup]
2024-08-20T23:00:55.0763799Z ##[group]Run # Remove any previous test jsons if they exist
2024-08-20T23:00:55.0764444Z # Remove any previous test jsons if they exist
2024-08-20T23:00:55.0764946Z rm -f test-jsons-*.zip
2024-08-20T23:00:55.0765454Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'
2024-08-20T23:00:55.0775699Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T23:00:55.0776197Z env:
2024-08-20T23:00:55.0776473Z   GIT_DEFAULT_BRANCH: main
2024-08-20T23:00:55.0776922Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T23:00:55.0777649Z   DOCKER_CONTAINER_ID: 0069e58a274b850ea9dc89e5a84233030f0d0aca44320cdb9963abb2a7d1ef4f
2024-08-20T23:00:55.0778589Z   FILE_SUFFIX: test-dynamic_aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026300273
2024-08-20T23:00:55.0779389Z ##[endgroup]
2024-08-20T23:00:55.1132070Z   adding: test/allowlist_for_publicAPI.json (deflated 79%)
2024-08-20T23:00:55.1161779Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2024-08-20T23:00:55.1163155Z   adding: test/minioptest_failures_dict.json (deflated 70%)
2024-08-20T23:00:55.1167991Z   adding: test/profiler/profiler_utils_mock_events.json (deflated 87%)
2024-08-20T23:00:55.1174075Z   adding: test/slow_tests.json (deflated 82%)
2024-08-20T23:00:55.1205892Z ##[group]Run # Remove any previous test reports if they exist
2024-08-20T23:00:55.1206548Z # Remove any previous test reports if they exist
2024-08-20T23:00:55.1207104Z rm -f test-reports-*.zip
2024-08-20T23:00:55.1207692Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv'
2024-08-20T23:00:55.1216543Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T23:00:55.1217054Z env:
2024-08-20T23:00:55.1217330Z   GIT_DEFAULT_BRANCH: main
2024-08-20T23:00:55.1217779Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T23:00:55.1218511Z   DOCKER_CONTAINER_ID: 0069e58a274b850ea9dc89e5a84233030f0d0aca44320cdb9963abb2a7d1ef4f
2024-08-20T23:00:55.1219554Z   FILE_SUFFIX: test-dynamic_aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026300273
2024-08-20T23:00:55.1220244Z ##[endgroup]
2024-08-20T23:00:55.1482430Z   adding: test/test-reports/inference_huggingface.csv (deflated 76%)
2024-08-20T23:00:55.1483638Z   adding: test/test-reports/inference_huggingface_graph_breaks.csv (deflated 86%)
2024-08-20T23:00:55.1484789Z   adding: test/test-reports/inference_huggingface_graph_break_deduped.csv (deflated 72%)
2024-08-20T23:00:55.1485670Z   adding: test/test-reports/training_huggingface.csv (deflated 74%)
2024-08-20T23:00:55.1491736Z   adding: test/test-reports/training_huggingface_graph_breaks.csv (deflated 97%)
2024-08-20T23:00:55.1492940Z   adding: test/test-reports/training_huggingface_graph_break_deduped.csv (deflated 74%)
2024-08-20T23:00:55.1524644Z ##[group]Run # Remove any previous usage logs if they exist
2024-08-20T23:00:55.1525254Z # Remove any previous usage logs if they exist
2024-08-20T23:00:55.1525742Z rm -f logs-*.zip
2024-08-20T23:00:55.1526592Z # this workflow is also run in bazel build test, but we dont generate usage reports for it
2024-08-20T23:00:55.1527382Z # so check to see if the file exists first
2024-08-20T23:00:55.1527874Z if [ -f 'usage_log.txt' ]; then
2024-08-20T23:00:55.1528397Z     zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt'
2024-08-20T23:00:55.1528873Z fi
2024-08-20T23:00:55.1529228Z if ls test/**/*.log 1> /dev/null 2>&1; then
2024-08-20T23:00:55.1529794Z     zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log'
2024-08-20T23:00:55.1530285Z fi
2024-08-20T23:00:55.1539122Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T23:00:55.1539612Z env:
2024-08-20T23:00:55.1539887Z   GIT_DEFAULT_BRANCH: main
2024-08-20T23:00:55.1540331Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T23:00:55.1541058Z   DOCKER_CONTAINER_ID: 0069e58a274b850ea9dc89e5a84233030f0d0aca44320cdb9963abb2a7d1ef4f
2024-08-20T23:00:55.1541988Z   FILE_SUFFIX: test-dynamic_aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026300273
2024-08-20T23:00:55.1542676Z ##[endgroup]
2024-08-20T23:00:55.1615320Z   adding: usage_log.txt (deflated 92%)
2024-08-20T23:00:55.2137022Z ##[group]Run # Remove any previous debugging artifacts if they exist
2024-08-20T23:00:55.2137736Z # Remove any previous debugging artifacts if they exist
2024-08-20T23:00:55.2138277Z rm -f debug-*.zip
2024-08-20T23:00:55.2138800Z if [ -d 'test/debug' ]; then
2024-08-20T23:00:55.2139299Z   zip -r "debug-${FILE_SUFFIX}.zip" test/debug
2024-08-20T23:00:55.2139785Z fi
2024-08-20T23:00:55.2148621Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T23:00:55.2149120Z env:
2024-08-20T23:00:55.2149398Z   GIT_DEFAULT_BRANCH: main
2024-08-20T23:00:55.2149848Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T23:00:55.2150576Z   DOCKER_CONTAINER_ID: 0069e58a274b850ea9dc89e5a84233030f0d0aca44320cdb9963abb2a7d1ef4f
2024-08-20T23:00:55.2151521Z   FILE_SUFFIX: test-dynamic_aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026300273
2024-08-20T23:00:55.2152199Z ##[endgroup]
2024-08-20T23:00:55.2251818Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-08-20T23:00:55.2252243Z with:
2024-08-20T23:00:55.2252528Z   s3-bucket: gha-artifacts
2024-08-20T23:00:55.2252953Z   s3-prefix: pytorch/pytorch/10479310721/1/artifact

2024-08-20T23:00:55.2253419Z   retention-days: 14
2024-08-20T23:00:55.2253742Z   if-no-files-found: warn
2024-08-20T23:00:55.2254089Z   path: test-jsons-*.zip
2024-08-20T23:00:55.2254430Z   name: artifact
2024-08-20T23:00:55.2254720Z   region: us-east-1
2024-08-20T23:00:55.2255016Z env:
2024-08-20T23:00:55.2255285Z   GIT_DEFAULT_BRANCH: main
2024-08-20T23:00:55.2255724Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T23:00:55.2256436Z   DOCKER_CONTAINER_ID: 0069e58a274b850ea9dc89e5a84233030f0d0aca44320cdb9963abb2a7d1ef4f
2024-08-20T23:00:55.2257118Z ##[endgroup]
2024-08-20T23:00:55.5796877Z NOTE: s3-prefix specified, ignoring name parameter
2024-08-20T23:00:55.5797734Z With the provided path, there will be 1 file uploaded
2024-08-20T23:00:55.5798512Z Uploading to s3 prefix: pytorch/pytorch/10479310721/1/artifact
2024-08-20T23:00:55.6687867Z Starting upload of test-jsons-test-dynamic_aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026300273.zip
2024-08-20T23:00:55.7979710Z Finished upload of test-jsons-test-dynamic_aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026300273.zip
2024-08-20T23:00:55.8263246Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-08-20T23:00:55.8263694Z with:
2024-08-20T23:00:55.8263976Z   s3-bucket: gha-artifacts
2024-08-20T23:00:55.8264410Z   s3-prefix: pytorch/pytorch/10479310721/1/artifact

2024-08-20T23:00:55.8264881Z   retention-days: 14
2024-08-20T23:00:55.8265210Z   if-no-files-found: error
2024-08-20T23:00:55.8265574Z   path: test-reports-*.zip
2024-08-20T23:00:55.8265920Z   name: artifact
2024-08-20T23:00:55.8266215Z   region: us-east-1
2024-08-20T23:00:55.8266517Z env:
2024-08-20T23:00:55.8266985Z   GIT_DEFAULT_BRANCH: main
2024-08-20T23:00:55.8267439Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T23:00:55.8268158Z   DOCKER_CONTAINER_ID: 0069e58a274b850ea9dc89e5a84233030f0d0aca44320cdb9963abb2a7d1ef4f
2024-08-20T23:00:55.8268801Z ##[endgroup]
2024-08-20T23:00:56.1778442Z NOTE: s3-prefix specified, ignoring name parameter
2024-08-20T23:00:56.1779130Z With the provided path, there will be 1 file uploaded
2024-08-20T23:00:56.1779742Z Uploading to s3 prefix: pytorch/pytorch/10479310721/1/artifact
2024-08-20T23:00:56.1835633Z Starting upload of test-reports-test-dynamic_aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026300273.zip
2024-08-20T23:00:56.3140318Z Finished upload of test-reports-test-dynamic_aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026300273.zip
2024-08-20T23:00:56.3417507Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-08-20T23:00:56.3417940Z with:
2024-08-20T23:00:56.3418225Z   s3-bucket: gha-artifacts
2024-08-20T23:00:56.3418792Z   s3-prefix: pytorch/pytorch/10479310721/1/artifact

2024-08-20T23:00:56.3419268Z   retention-days: 14
2024-08-20T23:00:56.3419597Z   if-no-files-found: ignore
2024-08-20T23:00:56.3419953Z   path: logs-*.zip
2024-08-20T23:00:56.3420258Z   name: artifact
2024-08-20T23:00:56.3420552Z   region: us-east-1
2024-08-20T23:00:56.3420854Z env:
2024-08-20T23:00:56.3421125Z   GIT_DEFAULT_BRANCH: main
2024-08-20T23:00:56.3421571Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T23:00:56.3422288Z   DOCKER_CONTAINER_ID: 0069e58a274b850ea9dc89e5a84233030f0d0aca44320cdb9963abb2a7d1ef4f
2024-08-20T23:00:56.3422929Z ##[endgroup]
2024-08-20T23:00:56.6977741Z NOTE: s3-prefix specified, ignoring name parameter
2024-08-20T23:00:56.6978441Z With the provided path, there will be 1 file uploaded
2024-08-20T23:00:56.6979136Z Uploading to s3 prefix: pytorch/pytorch/10479310721/1/artifact
2024-08-20T23:00:56.7034002Z Starting upload of logs-test-dynamic_aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026300273.zip
2024-08-20T23:00:56.8299981Z Finished upload of logs-test-dynamic_aot_eager_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_29026300273.zip
2024-08-20T23:00:56.8573872Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-08-20T23:00:56.8574305Z with:
2024-08-20T23:00:56.8574593Z   s3-bucket: gha-artifacts
2024-08-20T23:00:56.8575022Z   s3-prefix: pytorch/pytorch/10479310721/1/artifact

2024-08-20T23:00:56.8575491Z   retention-days: 14
2024-08-20T23:00:56.8575821Z   if-no-files-found: ignore
2024-08-20T23:00:56.8576177Z   path: debug-*.zip
2024-08-20T23:00:56.8576511Z   name: artifact
2024-08-20T23:00:56.8576808Z   region: us-east-1
2024-08-20T23:00:56.8577158Z env:
2024-08-20T23:00:56.8577438Z   GIT_DEFAULT_BRANCH: main
2024-08-20T23:00:56.8577888Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T23:00:56.8578609Z   DOCKER_CONTAINER_ID: 0069e58a274b850ea9dc89e5a84233030f0d0aca44320cdb9963abb2a7d1ef4f
2024-08-20T23:00:56.8579300Z ##[endgroup]
2024-08-20T23:00:57.2017233Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded.
2024-08-20T23:00:57.2291805Z ##[group]Run # shellcheck disable=SC2156
2024-08-20T23:00:57.2292291Z # shellcheck disable=SC2156
2024-08-20T23:00:57.2293081Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2024-08-20T23:00:57.2302260Z shell: /usr/bin/bash -e {0}
2024-08-20T23:00:57.2302614Z env:
2024-08-20T23:00:57.2302886Z   GIT_DEFAULT_BRANCH: main
2024-08-20T23:00:57.2303331Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T23:00:57.2304065Z   DOCKER_CONTAINER_ID: 0069e58a274b850ea9dc89e5a84233030f0d0aca44320cdb9963abb2a7d1ef4f
2024-08-20T23:00:57.2304703Z ##[endgroup]
2024-08-20T23:00:57.4947020Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main
2024-08-20T23:00:57.4947621Z with:
2024-08-20T23:00:57.4947873Z env:
2024-08-20T23:00:57.4948145Z   GIT_DEFAULT_BRANCH: main
2024-08-20T23:00:57.4948581Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T23:00:57.4949307Z   DOCKER_CONTAINER_ID: 0069e58a274b850ea9dc89e5a84233030f0d0aca44320cdb9963abb2a7d1ef4f
2024-08-20T23:00:57.4949934Z ##[endgroup]
2024-08-20T23:00:57.4989902Z ##[group]Run set -eou pipefail
2024-08-20T23:00:57.4990306Z set -eou pipefail
2024-08-20T23:00:57.4990654Z 
2024-08-20T23:00:57.4991159Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2024-08-20T23:00:57.4991788Z for _ in $(seq 1440); do
2024-08-20T23:00:57.4992416Z     # Break if no ssh session exists anymore
2024-08-20T23:00:57.4992914Z     if [ "$(who)" = "" ]; then
2024-08-20T23:00:57.4993321Z       break
2024-08-20T23:00:57.4993656Z     fi
2024-08-20T23:00:57.4993949Z     echo "."
2024-08-20T23:00:57.4994275Z     sleep 5
2024-08-20T23:00:57.4994590Z done
2024-08-20T23:00:57.5003292Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T23:00:57.5003791Z env:
2024-08-20T23:00:57.5004065Z   GIT_DEFAULT_BRANCH: main
2024-08-20T23:00:57.5004515Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T23:00:57.5005238Z   DOCKER_CONTAINER_ID: 0069e58a274b850ea9dc89e5a84233030f0d0aca44320cdb9963abb2a7d1ef4f
2024-08-20T23:00:57.5005878Z ##[endgroup]
2024-08-20T23:00:57.5035800Z Holding runner for 2 hours until all ssh sessions have logged out
2024-08-20T23:00:57.5109351Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2024-08-20T23:00:57.5110115Z # ignore expansion of "docker ps -q" since it could be empty
2024-08-20T23:00:57.5110692Z # shellcheck disable=SC2046
2024-08-20T23:00:57.5111160Z docker stop $(docker ps -q) || true
2024-08-20T23:00:57.5111631Z # Prune all of the docker images
2024-08-20T23:00:57.5112083Z docker system prune -af
2024-08-20T23:00:57.5120813Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T23:00:57.5121306Z env:
2024-08-20T23:00:57.5121576Z   GIT_DEFAULT_BRANCH: main
2024-08-20T23:00:57.5122022Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T23:00:57.5122744Z   DOCKER_CONTAINER_ID: 0069e58a274b850ea9dc89e5a84233030f0d0aca44320cdb9963abb2a7d1ef4f
2024-08-20T23:00:57.5123372Z ##[endgroup]
2024-08-20T23:00:58.0754196Z 0069e58a274b
2024-08-20T23:00:58.8235121Z Deleted Containers:
2024-08-20T23:00:58.8235722Z 0069e58a274b850ea9dc89e5a84233030f0d0aca44320cdb9963abb2a7d1ef4f
2024-08-20T23:00:58.8236152Z 
2024-08-20T23:01:09.7716422Z Deleted Images:
2024-08-20T23:01:09.7718411Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T23:01:09.7721078Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402
2024-08-20T23:01:09.7722591Z deleted: sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb
2024-08-20T23:01:09.7723427Z deleted: sha256:c2ce10014dd98f6abbfbf2265f9780a8204c2a7a27463e61ee565573a23d141d
2024-08-20T23:01:09.7724266Z deleted: sha256:ae249c0e6e6013bd6ffea2ca0b0160c015c9ebf7c03070bc2f983a9ff2415bf5
2024-08-20T23:01:09.7725084Z deleted: sha256:8a73883c1067c0e29ff128f553518682b47ed9707d2f35e7d5138e2caf029d61
2024-08-20T23:01:09.7725883Z deleted: sha256:9c040db729fa084584aaf5573082408a2196bc64e5952270c79dc610ed4680b6
2024-08-20T23:01:09.7726675Z deleted: sha256:17311eeeb8b71b16699880c0f646e185107496c4d41da5d23370603768ec030c
2024-08-20T23:01:09.7727536Z deleted: sha256:07cc10bece8501b1c8035d4385ad9e9867dd6e92fa32218a4dcb3e42af7e0411
2024-08-20T23:01:09.7728705Z deleted: sha256:f3e32d05099d1a3115a4722bc2e94a48752c3e965446773673808252671bb967
2024-08-20T23:01:09.7729531Z deleted: sha256:be7c3f41475e3e72ffe3ac2f92ab2f4cf49d5496b2aae738942ac3bb687d742c
2024-08-20T23:01:09.7730366Z deleted: sha256:4fcac490be825129b468cf458089f1fcf8fd56afc44ef4ab27c2eff41ab1ac90
2024-08-20T23:01:09.7731212Z deleted: sha256:31c66d80a2a4792897f267d363c0cf84fb821c5d4a275ab1636f778fd0284c08
2024-08-20T23:01:09.7732088Z deleted: sha256:8a8bcb6085e7e438e5533e8a5613e2ae5888543853568d023af3e2623a1593a9
2024-08-20T23:01:09.7732973Z deleted: sha256:2f6cb60029c79d11567b738f5bdbfb9cb1838cc0f194c4a5e6ae685178d2ed34
2024-08-20T23:01:09.7733894Z deleted: sha256:e546391804d3313f558e1ac16e93cb29a417a92e4eccaf0d4bd12af3bbcf39e3
2024-08-20T23:01:09.7734789Z deleted: sha256:1623b2aa16f2bdc6b572f6799498d9a92fa8114d07c440599b17aea72e22f8b1
2024-08-20T23:01:09.7735831Z deleted: sha256:c0386ca58b7cb7afee015b7e58d2f70525eaf73d6da2cc63868b20e14b1f9497
2024-08-20T23:01:09.7736662Z deleted: sha256:dcbf534ff289186e246531925fed90f72516eb3d63663bad234d611cdc025d62
2024-08-20T23:01:09.7737470Z deleted: sha256:0bab537af579f56c9cd51c7073fd53635fffb424b363236e0de38a7dd0e60ae8
2024-08-20T23:01:09.7738283Z deleted: sha256:d85a852ae1ee027e2fc17240ed68b87957b0a6d210c9119752d8979a0dcd55cf
2024-08-20T23:01:09.7739215Z deleted: sha256:f3821d761b555362e44cf9a1447d8fd2d95b37d54aea67c7519cd9e5db0c4d7e
2024-08-20T23:01:09.7740047Z deleted: sha256:c05e84a545db5dfcb0847c73d2b0f4749fa67aabce05705632ff7f76c84d4688
2024-08-20T23:01:09.7740885Z deleted: sha256:27054eae7ca67b2bd5ca0fd3f4488bf5c5f78e95487d8c45616f3d753d48a8b5
2024-08-20T23:01:09.7741704Z deleted: sha256:9804139f6ae97796aaa993fe4b89ea0cf111eb4d30a79f4c87c339c2e1e04fa0
2024-08-20T23:01:09.7742511Z deleted: sha256:ad6ae2fc594aaa359ff4e287323836570454eb24dd162930655e414d1d77c6dd
2024-08-20T23:01:09.7743330Z deleted: sha256:9cfbb36e328af52a248853a46aab88e266c09e2b44a03e0a42194a031b9b9453
2024-08-20T23:01:09.7744161Z deleted: sha256:a8a842bd7f37b8d50b6d12d6f95f46bbba57953db466fcbde9ba522625f024ef
2024-08-20T23:01:09.7744983Z deleted: sha256:da9738654b3c86f89ebeed2560f1bae093f1219f1eff90b0394b7f211ecabd04
2024-08-20T23:01:09.7745805Z deleted: sha256:cc683bf0cb245394e0f568984b2dd4fce1b6a3a7fe7c44734bae1aa1caaa3251
2024-08-20T23:01:09.7746617Z deleted: sha256:965f220e71c7d1ec19aaf41b1529ab287beb9d86eb8482c4c14094485ab34475
2024-08-20T23:01:09.7747454Z deleted: sha256:da32ecec41a73ce7dd29464f90c7951510a18cd378753659a78cd7fabd10dc36
2024-08-20T23:01:09.7748415Z deleted: sha256:d828d187cee9ad48dbc3702fb61d359619dd8a1ed582878680b51686bf117e5a
2024-08-20T23:01:09.7749383Z deleted: sha256:712111f4b12ff8e2787597d5fdcb043bd0824e55ddd97f262ef550506ca195f0
2024-08-20T23:01:09.7750448Z deleted: sha256:a064eea405958ad14dc6c794e5df9fd1d4515424272801ec559d45a59317309a
2024-08-20T23:01:09.7751371Z deleted: sha256:e124382d136bd5e9a8b409ca79c1f1ef76458deaf444b1b2060d3fa99dbeb506
2024-08-20T23:01:09.7752257Z deleted: sha256:6e10fe757cc190a1b31b47010f4f7610e02cb1b4b438f190dcd301aa4fe2278e
2024-08-20T23:01:09.7753253Z deleted: sha256:4cea352e10f46ca4729f96ac6c4600fc53d4c36e7658c2e0f19f4f309cd4a2a7
2024-08-20T23:01:09.7754260Z deleted: sha256:518bba00fe1d0933053066d21cac0235c9fb485abbc41557ec184b293095c924
2024-08-20T23:01:09.7755266Z deleted: sha256:a64f9ca42e9f27ff2690c15a5c84312e6efaccad45668d9b496cb542165cfb5a
2024-08-20T23:01:09.7756260Z deleted: sha256:096ea0fa14a6c420658f5276b8a8f5d264c0de7e18161ceda2cfc2f4a92cd9bf
2024-08-20T23:01:09.7757136Z deleted: sha256:6f91c94ce6e3f8034c91c66ee25c66a3d85dba9d813b749bd6d2000942a99edc
2024-08-20T23:01:09.7758350Z deleted: sha256:8f8d255eeae546cead8f9a94d237fa57a99940c0f5d51bc29a9d41c7df9cffd9
2024-08-20T23:01:09.7759287Z deleted: sha256:fe05276f4a49f91eef56bfbc3da87cfff5eb116a827fa2d30efe42ce1c134b40
2024-08-20T23:01:09.7760295Z deleted: sha256:5f83cd75d0669c813d4f58978c92e14305cd8f6a07eb9a60665a583460fa7b59
2024-08-20T23:01:09.7761159Z deleted: sha256:8f7ab6366169d39d8953129f650f1927d2fb1328b3f11e2fd9076adb5f46b93a
2024-08-20T23:01:09.7762190Z deleted: sha256:4146cfd5dba0f5308954749f98596b14a1d8f6d23bd93f7ad86c6c3d72239328
2024-08-20T23:01:09.7763210Z deleted: sha256:c6d83cb9ccc606ff63bb3c4d5a5e1ff1450c619b5aae494303a94ae25971cca0
2024-08-20T23:01:09.7764133Z deleted: sha256:71e037e0a3e0963c4565534a7387c082afcb04475d3ac629acf7a0ab2f49fadc
2024-08-20T23:01:09.7764989Z deleted: sha256:4b31135c564686bdaf23262b901237880aea635655721cc7e8c95dc42b05e06d
2024-08-20T23:01:09.7765971Z deleted: sha256:d09bd5c2f35cedcb846543941ee64ce1f684b52e4285696c93b3fb2032da8579
2024-08-20T23:01:09.7766899Z deleted: sha256:4de7fb7d4fc22212aae5fc1ceb637526916d0cced240d7c93d5edd6b0ea28013
2024-08-20T23:01:09.7767840Z deleted: sha256:69890e3e82746e10aff2f1084f11afb4b95a4a6c26b30f64b7001bc0a5fe32bf
2024-08-20T23:01:09.7768755Z deleted: sha256:0156df53b3224e645336ea52b4bb816f1e8d8c9b591748cd645e5e5649fa507b
2024-08-20T23:01:09.7770147Z deleted: sha256:0664624cb9cc4a386f31bf49919a8d8a11d9fea428d05585d9e6228f754bc7cd
2024-08-20T23:01:09.7771085Z deleted: sha256:8c2ddd5fa535c768c058a23f5cc4b6c64cdd71bb7458491282aabc7f7bb94d69
2024-08-20T23:01:09.7772122Z deleted: sha256:8dd03dbe0e1c5f47dd333074dff4bd00415f2f63727a65bf17866774e2ebbd0d
2024-08-20T23:01:09.7773162Z deleted: sha256:3c56b63e094bd18fad9da7a541d537210bca3b730c590d3641213259f6908d6f
2024-08-20T23:01:09.7774133Z deleted: sha256:51484fef07bd1266d4d6ff3d767751172660fc4cc6a1ffe883523a63c985e31e
2024-08-20T23:01:09.7793573Z deleted: sha256:03e83c94023982ab7dd8dc65a6df2024e342f1b760c91ec7a680ee45824869c7
2024-08-20T23:01:09.7794443Z deleted: sha256:58440ab3e8175b5de94a76485c8689a6dc186015475ebf4b386ebba309c5da34
2024-08-20T23:01:09.7795275Z deleted: sha256:9a980f3383e8aa511f0ef68982cb6dfc53b38d1ac8c3d673cb3ba3d45158c91e
2024-08-20T23:01:09.7796114Z deleted: sha256:cb0269c9cf1c47b6c4188ca7be4db261b5d3fc4b4a5153629e222dba7b1f9f98
2024-08-20T23:01:09.7796972Z deleted: sha256:41f1ba870cdffc05e0bf8d87b640fb9d4c153d7f4f3b775f37f94e9f3d584aad
2024-08-20T23:01:09.7797814Z deleted: sha256:7b502ab352f48e8b6eb5e6824a29f308952a5ee7c7d987562e1df7e47639d7f1
2024-08-20T23:01:09.7798621Z deleted: sha256:6ba5217e18738660e6ca8b7a7ec88845e70b0fd781301520bf46c71d99df3144
2024-08-20T23:01:09.7799451Z deleted: sha256:dd5a1c233594780163726bdbbf5de00f7b6b8dca1dace8a001340714c54baedb
2024-08-20T23:01:09.7800295Z deleted: sha256:c27f0db27da73e01f67f6cb78d10f355e350d829479ba4351d82055bb29f6597
2024-08-20T23:01:09.7801126Z deleted: sha256:4e3ce3c854f35c08be3dd95e0d6e718512a885e4614291d7d8b7cd23f48b7dd8
2024-08-20T23:01:09.7801961Z deleted: sha256:c2d6fb088bbec16e60cf0ad74b016f7287c92e659ec2db01271bc309b4dc6d83
2024-08-20T23:01:09.7802797Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8
2024-08-20T23:01:09.7803619Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00
2024-08-20T23:01:09.7804428Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f
2024-08-20T23:01:09.7805248Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64
2024-08-20T23:01:09.7806086Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88
2024-08-20T23:01:09.7806912Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d
2024-08-20T23:01:09.7807752Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25
2024-08-20T23:01:09.7808588Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9
2024-08-20T23:01:09.7809432Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29
2024-08-20T23:01:09.7810272Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037
2024-08-20T23:01:09.7811095Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9
2024-08-20T23:01:09.7811583Z 
2024-08-20T23:01:09.7811744Z Total reclaimed space: 39.92GB
2024-08-20T23:01:09.7879794Z Post job cleanup.
2024-08-20T23:01:09.7930276Z Post job cleanup.
2024-08-20T23:01:09.8813924Z [command]/usr/bin/git version
2024-08-20T23:01:09.8863718Z git version 2.40.1
2024-08-20T23:01:09.8902923Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/edfc486c-7fe9-4bab-a714-28d1367e3078' before making global git config changes
2024-08-20T23:01:09.8904278Z Adding repository directory to the temporary git global config as a safe directory
2024-08-20T23:01:09.8907980Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-08-20T23:01:09.8956225Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2024-08-20T23:01:09.9000916Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2024-08-20T23:01:09.9372448Z Entering 'android/libs/fbjni'
2024-08-20T23:01:09.9442158Z Entering 'third_party/FP16'
2024-08-20T23:01:09.9511315Z Entering 'third_party/FXdiv'
2024-08-20T23:01:09.9579453Z Entering 'third_party/NNPACK'
2024-08-20T23:01:09.9651399Z Entering 'third_party/VulkanMemoryAllocator'
2024-08-20T23:01:09.9720484Z Entering 'third_party/XNNPACK'
2024-08-20T23:01:09.9803407Z Entering 'third_party/benchmark'
2024-08-20T23:01:09.9873848Z Entering 'third_party/cpp-httplib'
2024-08-20T23:01:09.9940239Z Entering 'third_party/cpuinfo'
2024-08-20T23:01:10.0009593Z Entering 'third_party/cudnn_frontend'
2024-08-20T23:01:10.0082405Z Entering 'third_party/cutlass'
2024-08-20T23:01:10.0156595Z Entering 'third_party/eigen'
2024-08-20T23:01:10.0231120Z Entering 'third_party/fbgemm'
2024-08-20T23:01:10.0301633Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-08-20T23:01:10.0366575Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-08-20T23:01:10.0434518Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-08-20T23:01:10.0508402Z Entering 'third_party/fbgemm/third_party/googletest'
2024-08-20T23:01:10.0575050Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-08-20T23:01:10.0645045Z Entering 'third_party/flatbuffers'
2024-08-20T23:01:10.0719399Z Entering 'third_party/fmt'
2024-08-20T23:01:10.0788517Z Entering 'third_party/gemmlowp/gemmlowp'
2024-08-20T23:01:10.0858111Z Entering 'third_party/gloo'
2024-08-20T23:01:10.0928498Z Entering 'third_party/googletest'
2024-08-20T23:01:10.0998244Z Entering 'third_party/ideep'
2024-08-20T23:01:10.1064949Z Entering 'third_party/ideep/mkl-dnn'
2024-08-20T23:01:10.1140549Z Entering 'third_party/ittapi'
2024-08-20T23:01:10.1210300Z Entering 'third_party/kineto'
2024-08-20T23:01:10.1278822Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-08-20T23:01:10.1343957Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-08-20T23:01:10.1414073Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-08-20T23:01:10.1481830Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-08-20T23:01:10.1549963Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-08-20T23:01:10.1616229Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-08-20T23:01:10.1686883Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-08-20T23:01:10.1755064Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-08-20T23:01:10.1823985Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-08-20T23:01:10.1893354Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-08-20T23:01:10.1964035Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-08-20T23:01:10.2031149Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-08-20T23:01:10.2105905Z Entering 'third_party/mimalloc'
2024-08-20T23:01:10.2175343Z Entering 'third_party/nccl/nccl'
2024-08-20T23:01:10.2245026Z Entering 'third_party/nlohmann'
2024-08-20T23:01:10.2316108Z Entering 'third_party/onnx'
2024-08-20T23:01:10.2399377Z Entering 'third_party/onnx/third_party/benchmark'
2024-08-20T23:01:10.2472732Z Entering 'third_party/onnx/third_party/pybind11'
2024-08-20T23:01:10.2550723Z Entering 'third_party/opentelemetry-cpp'
2024-08-20T23:01:10.2620350Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-08-20T23:01:10.2686021Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-08-20T23:01:10.2753152Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-08-20T23:01:10.2819454Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-08-20T23:01:10.2887161Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-08-20T23:01:10.2953993Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-08-20T23:01:10.3019961Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-08-20T23:01:10.3088646Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-08-20T23:01:10.3157475Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-08-20T23:01:10.3227322Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-08-20T23:01:10.3315906Z Entering 'third_party/pocketfft'
2024-08-20T23:01:10.3384954Z Entering 'third_party/protobuf'
2024-08-20T23:01:10.3454347Z Entering 'third_party/protobuf/third_party/benchmark'
2024-08-20T23:01:10.3523885Z Entering 'third_party/protobuf/third_party/googletest'
2024-08-20T23:01:10.3593813Z Entering 'third_party/psimd'
2024-08-20T23:01:10.3661671Z Entering 'third_party/pthreadpool'
2024-08-20T23:01:10.3729645Z Entering 'third_party/pybind11'
2024-08-20T23:01:10.3798662Z Entering 'third_party/python-peachpy'
2024-08-20T23:01:10.3867458Z Entering 'third_party/sleef'
2024-08-20T23:01:10.3936650Z Entering 'third_party/tensorpipe'
2024-08-20T23:01:10.4004571Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-08-20T23:01:10.4073277Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-08-20T23:01:10.4137522Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-08-20T23:01:10.4203862Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-08-20T23:01:10.4269792Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-08-20T23:01:10.4365376Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2024-08-20T23:01:10.4402107Z http.https://github.com/.extraheader
2024-08-20T23:01:10.4412819Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2024-08-20T23:01:10.4460550Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2024-08-20T23:01:10.4825734Z Entering 'android/libs/fbjni'
2024-08-20T23:01:10.4869657Z http.https://github.com/.extraheader
2024-08-20T23:01:10.4913930Z Entering 'third_party/FP16'
2024-08-20T23:01:10.4957601Z http.https://github.com/.extraheader
2024-08-20T23:01:10.5001264Z Entering 'third_party/FXdiv'
2024-08-20T23:01:10.5044059Z http.https://github.com/.extraheader
2024-08-20T23:01:10.5088711Z Entering 'third_party/NNPACK'
2024-08-20T23:01:10.5136359Z http.https://github.com/.extraheader
2024-08-20T23:01:10.5180390Z Entering 'third_party/VulkanMemoryAllocator'
2024-08-20T23:01:10.5223995Z http.https://github.com/.extraheader
2024-08-20T23:01:10.5267299Z Entering 'third_party/XNNPACK'
2024-08-20T23:01:10.5310856Z http.https://github.com/.extraheader
2024-08-20T23:01:10.5370666Z Entering 'third_party/benchmark'
2024-08-20T23:01:10.5413324Z http.https://github.com/.extraheader
2024-08-20T23:01:10.5457923Z Entering 'third_party/cpp-httplib'
2024-08-20T23:01:10.5501648Z http.https://github.com/.extraheader
2024-08-20T23:01:10.5544651Z Entering 'third_party/cpuinfo'
2024-08-20T23:01:10.5588635Z http.https://github.com/.extraheader
2024-08-20T23:01:10.5633400Z Entering 'third_party/cudnn_frontend'
2024-08-20T23:01:10.5677269Z http.https://github.com/.extraheader
2024-08-20T23:01:10.5720002Z Entering 'third_party/cutlass'
2024-08-20T23:01:10.5763590Z http.https://github.com/.extraheader
2024-08-20T23:01:10.5814192Z Entering 'third_party/eigen'
2024-08-20T23:01:10.5856770Z http.https://github.com/.extraheader
2024-08-20T23:01:10.5902013Z Entering 'third_party/fbgemm'
2024-08-20T23:01:10.5945514Z http.https://github.com/.extraheader
2024-08-20T23:01:10.5989126Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-08-20T23:01:10.6031712Z http.https://github.com/.extraheader
2024-08-20T23:01:10.6075225Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-08-20T23:01:10.6116781Z http.https://github.com/.extraheader
2024-08-20T23:01:10.6161226Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-08-20T23:01:10.6205124Z http.https://github.com/.extraheader
2024-08-20T23:01:10.6254719Z Entering 'third_party/fbgemm/third_party/googletest'
2024-08-20T23:01:10.6296875Z http.https://github.com/.extraheader
2024-08-20T23:01:10.6339346Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-08-20T23:01:10.6382725Z http.https://github.com/.extraheader
2024-08-20T23:01:10.6428896Z Entering 'third_party/flatbuffers'
2024-08-20T23:01:10.6477950Z http.https://github.com/.extraheader
2024-08-20T23:01:10.6524166Z Entering 'third_party/fmt'
2024-08-20T23:01:10.6567801Z http.https://github.com/.extraheader
2024-08-20T23:01:10.6615159Z Entering 'third_party/gemmlowp/gemmlowp'
2024-08-20T23:01:10.6661800Z http.https://github.com/.extraheader
2024-08-20T23:01:10.6705183Z Entering 'third_party/gloo'
2024-08-20T23:01:10.6749392Z http.https://github.com/.extraheader
2024-08-20T23:01:10.6792601Z Entering 'third_party/googletest'
2024-08-20T23:01:10.6835261Z http.https://github.com/.extraheader
2024-08-20T23:01:10.6878592Z Entering 'third_party/ideep'
2024-08-20T23:01:10.6921750Z http.https://github.com/.extraheader
2024-08-20T23:01:10.6963316Z Entering 'third_party/ideep/mkl-dnn'
2024-08-20T23:01:10.7006706Z http.https://github.com/.extraheader
2024-08-20T23:01:10.7058468Z Entering 'third_party/ittapi'
2024-08-20T23:01:10.7103049Z http.https://github.com/.extraheader
2024-08-20T23:01:10.7145170Z Entering 'third_party/kineto'
2024-08-20T23:01:10.7189054Z http.https://github.com/.extraheader
2024-08-20T23:01:10.7230656Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-08-20T23:01:10.7274667Z http.https://github.com/.extraheader
2024-08-20T23:01:10.7318247Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-08-20T23:01:10.7365212Z http.https://github.com/.extraheader
2024-08-20T23:01:10.7411071Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-08-20T23:01:10.7461681Z http.https://github.com/.extraheader
2024-08-20T23:01:10.7506260Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-08-20T23:01:10.7548131Z http.https://github.com/.extraheader
2024-08-20T23:01:10.7592268Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-08-20T23:01:10.7639953Z http.https://github.com/.extraheader
2024-08-20T23:01:10.7683488Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-08-20T23:01:10.7727187Z http.https://github.com/.extraheader
2024-08-20T23:01:10.7776845Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-08-20T23:01:10.7818043Z http.https://github.com/.extraheader
2024-08-20T23:01:10.7861928Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-08-20T23:01:10.7905716Z http.https://github.com/.extraheader
2024-08-20T23:01:10.7949174Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-08-20T23:01:10.7993993Z http.https://github.com/.extraheader
2024-08-20T23:01:10.8039297Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-08-20T23:01:10.8083213Z http.https://github.com/.extraheader
2024-08-20T23:01:10.8130652Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-08-20T23:01:10.8174259Z http.https://github.com/.extraheader
2024-08-20T23:01:10.8217044Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-08-20T23:01:10.8259199Z http.https://github.com/.extraheader
2024-08-20T23:01:10.8311176Z Entering 'third_party/mimalloc'
2024-08-20T23:01:10.8353751Z http.https://github.com/.extraheader
2024-08-20T23:01:10.8396555Z Entering 'third_party/nccl/nccl'
2024-08-20T23:01:10.8438866Z http.https://github.com/.extraheader
2024-08-20T23:01:10.8482149Z Entering 'third_party/nlohmann'
2024-08-20T23:01:10.8524667Z http.https://github.com/.extraheader
2024-08-20T23:01:10.8568268Z Entering 'third_party/onnx'
2024-08-20T23:01:10.8611710Z http.https://github.com/.extraheader
2024-08-20T23:01:10.8668131Z Entering 'third_party/onnx/third_party/benchmark'
2024-08-20T23:01:10.8712711Z http.https://github.com/.extraheader
2024-08-20T23:01:10.8756265Z Entering 'third_party/onnx/third_party/pybind11'
2024-08-20T23:01:10.8800699Z http.https://github.com/.extraheader
2024-08-20T23:01:10.8849255Z Entering 'third_party/opentelemetry-cpp'
2024-08-20T23:01:10.8893063Z http.https://github.com/.extraheader
2024-08-20T23:01:10.8936830Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-08-20T23:01:10.8979560Z http.https://github.com/.extraheader
2024-08-20T23:01:10.9022315Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-08-20T23:01:10.9064770Z http.https://github.com/.extraheader
2024-08-20T23:01:10.9108852Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-08-20T23:01:10.9151045Z http.https://github.com/.extraheader
2024-08-20T23:01:10.9194779Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-08-20T23:01:10.9236532Z http.https://github.com/.extraheader
2024-08-20T23:01:10.9281006Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-08-20T23:01:10.9323257Z http.https://github.com/.extraheader
2024-08-20T23:01:10.9366261Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-08-20T23:01:10.9409379Z http.https://github.com/.extraheader
2024-08-20T23:01:10.9452443Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-08-20T23:01:10.9495189Z http.https://github.com/.extraheader
2024-08-20T23:01:10.9536893Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-08-20T23:01:10.9579378Z http.https://github.com/.extraheader
2024-08-20T23:01:10.9624285Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-08-20T23:01:10.9667850Z http.https://github.com/.extraheader
2024-08-20T23:01:10.9715272Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-08-20T23:01:10.9756805Z http.https://github.com/.extraheader
2024-08-20T23:01:10.9820789Z Entering 'third_party/pocketfft'
2024-08-20T23:01:10.9864237Z http.https://github.com/.extraheader
2024-08-20T23:01:10.9907813Z Entering 'third_party/protobuf'
2024-08-20T23:01:10.9950997Z http.https://github.com/.extraheader
2024-08-20T23:01:11.0350540Z Entering 'third_party/protobuf/third_party/benchmark'
2024-08-20T23:01:11.0394357Z http.https://github.com/.extraheader
2024-08-20T23:01:11.0437973Z Entering 'third_party/protobuf/third_party/googletest'
2024-08-20T23:01:11.0480573Z http.https://github.com/.extraheader
2024-08-20T23:01:11.0527393Z Entering 'third_party/psimd'
2024-08-20T23:01:11.0573238Z http.https://github.com/.extraheader
2024-08-20T23:01:11.0616152Z Entering 'third_party/pthreadpool'
2024-08-20T23:01:11.0658872Z http.https://github.com/.extraheader
2024-08-20T23:01:11.0702983Z Entering 'third_party/pybind11'
2024-08-20T23:01:11.0745930Z http.https://github.com/.extraheader
2024-08-20T23:01:11.0789362Z Entering 'third_party/python-peachpy'
2024-08-20T23:01:11.0832296Z http.https://github.com/.extraheader
2024-08-20T23:01:11.0875134Z Entering 'third_party/sleef'
2024-08-20T23:01:11.0918587Z http.https://github.com/.extraheader
2024-08-20T23:01:11.0962263Z Entering 'third_party/tensorpipe'
2024-08-20T23:01:11.1005558Z http.https://github.com/.extraheader
2024-08-20T23:01:11.1047219Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-08-20T23:01:11.1090604Z http.https://github.com/.extraheader
2024-08-20T23:01:11.1132461Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-08-20T23:01:11.1175194Z http.https://github.com/.extraheader
2024-08-20T23:01:11.1216554Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-08-20T23:01:11.1258054Z http.https://github.com/.extraheader
2024-08-20T23:01:11.1300446Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-08-20T23:01:11.1341297Z http.https://github.com/.extraheader
2024-08-20T23:01:11.1488554Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-08-20T23:01:11.1530312Z http.https://github.com/.extraheader
2024-08-20T23:01:11.1690729Z A job completed hook has been configured by the self-hosted runner administrator
2024-08-20T23:01:11.1712599Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh'
2024-08-20T23:01:11.1720826Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T23:01:11.1721344Z ##[endgroup]
2024-08-20T23:01:17.9863962Z Cleaning up orphan processes