2024-12-18T00:18:00.0222935Z Current runner version: '2.321.0'
2024-12-18T00:18:00.0229748Z Runner name: 'i-019aa8c64fc4447ce'
2024-12-18T00:18:00.0230719Z Runner group name: 'Default'
2024-12-18T00:18:00.0231803Z Machine name: 'ip-10-0-57-4'
2024-12-18T00:18:00.0237489Z ##[group]GITHUB_TOKEN Permissions
2024-12-18T00:18:00.0240307Z Actions: read
2024-12-18T00:18:00.0241042Z Attestations: read
2024-12-18T00:18:00.0241847Z Checks: read
2024-12-18T00:18:00.0242551Z Contents: read
2024-12-18T00:18:00.0243221Z Deployments: read
2024-12-18T00:18:00.0244026Z Discussions: read
2024-12-18T00:18:00.0244790Z Issues: read
2024-12-18T00:18:00.0245510Z Metadata: read
2024-12-18T00:18:00.0246338Z Packages: read
2024-12-18T00:18:00.0247001Z Pages: read
2024-12-18T00:18:00.0247722Z PullRequests: read
2024-12-18T00:18:00.0248518Z RepositoryProjects: read
2024-12-18T00:18:00.0249303Z SecurityEvents: read
2024-12-18T00:18:00.0250024Z Statuses: read
2024-12-18T00:18:00.0250774Z ##[endgroup]
2024-12-18T00:18:00.0254151Z Secret source: Actions
2024-12-18T00:18:00.0255337Z Prepare workflow directory
2024-12-18T00:18:00.0923966Z Prepare all required actions
2024-12-18T00:18:00.0962064Z Getting action download info
2024-12-18T00:18:00.3065018Z Download action repository 'pytorch/test-infra@release/2.6' (SHA:eb0adf5a84668865394af69e26428b32c8105c1c)
2024-12-18T00:18:02.2991342Z Download action repository 'pytorch/pytorch@release/2.6' (SHA:0cdf8b1d09254cfda66191d1bd01e3041c3c76f7)
2024-12-18T00:18:16.8201814Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0)
2024-12-18T00:18:17.0300178Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2024-12-18T00:18:17.3310973Z Getting action download info
2024-12-18T00:18:17.4654789Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c)
2024-12-18T00:18:17.7195634Z Getting action download info
2024-12-18T00:18:17.8410517Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e)
2024-12-18T00:18:18.0395376Z Getting action download info
2024-12-18T00:18:18.1603132Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482)
2024-12-18T00:18:18.3378864Z Getting action download info
2024-12-18T00:18:18.4970229Z Download action repository 'pytorch/test-infra@main' (SHA:a07505a74641a4ff5123d635defac481ef28ef1e)
2024-12-18T00:18:20.0234177Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/release/2.6 (0cdf8b1d09254cfda66191d1bd01e3041c3c76f7)
2024-12-18T00:18:20.0236051Z ##[group] Inputs
2024-12-18T00:18:20.0236447Z   build-environment: linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-12-18T00:18:20.0238428Z   test-matrix: {"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}
2024-12-18T00:18:20.0240936Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:18:20.0241815Z   sync-tag: 
2024-12-18T00:18:20.0242564Z   timeout-minutes: 240
2024-12-18T00:18:20.0242838Z   use-gha: 
2024-12-18T00:18:20.0243074Z   dashboard-tag: 
2024-12-18T00:18:20.0243329Z   s3-bucket: gha-artifacts
2024-12-18T00:18:20.0243604Z   aws-role-to-assume: 
2024-12-18T00:18:20.0244264Z   disable-monitor: false
2024-12-18T00:18:20.0244558Z ##[endgroup]
2024-12-18T00:18:20.0245323Z Complete job name: cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:18:20.0775110Z A job started hook has been configured by the self-hosted runner administrator
2024-12-18T00:18:20.0875053Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh'
2024-12-18T00:18:20.0886604Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:18:20.0887202Z ##[endgroup]
2024-12-18T00:18:22.1508235Z Runner Type: linux.g5.4xlarge.nvidia.gpu
2024-12-18T00:18:22.1508759Z Instance Type: g5.4xlarge
2024-12-18T00:18:22.1509067Z AMI Name: unknown
2024-12-18T00:18:22.1547691Z AMI ID: ami-0fff1b9a61dec8a5f
2024-12-18T00:18:28.0347847Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@release/2.6
2024-12-18T00:18:28.0348307Z with:
2024-12-18T00:18:28.0348931Z   github-secret: ***
2024-12-18T00:18:28.0349618Z   instructions: All testing is done inside the container, to start an interactive session run:
  docker exec -it $(docker container ps --format '{{.ID}}') bash

2024-12-18T00:18:28.0350405Z   activate-with-label: false
2024-12-18T00:18:28.0350703Z   label: with-ssh
2024-12-18T00:18:28.0350965Z   remove-existing-keys: true
2024-12-18T00:18:28.0351254Z   fail-silently: true
2024-12-18T00:18:28.0351499Z env:
2024-12-18T00:18:28.0351722Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:18:28.0352005Z ##[endgroup]
2024-12-18T00:18:28.1549513Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info.
2024-12-18T00:18:28.1550861Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys
2024-12-18T00:18:28.1737878Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@release/2.6
2024-12-18T00:18:28.1738337Z with:
2024-12-18T00:18:28.1738566Z   no-sudo: true
2024-12-18T00:18:28.1738808Z   submodules: recursive
2024-12-18T00:18:28.1739063Z   fetch-depth: 0
2024-12-18T00:18:28.1739289Z env:
2024-12-18T00:18:28.1739503Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:18:28.1739761Z ##[endgroup]
2024-12-18T00:18:28.1819474Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T00:18:28.1820451Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T00:18:28.1832093Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:18:28.1832481Z env:
2024-12-18T00:18:28.1832704Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:18:28.1832979Z ##[endgroup]
2024-12-18T00:18:28.1930824Z ##[group]Run retry () {
2024-12-18T00:18:28.1931117Z retry () {
2024-12-18T00:18:28.1931481Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2024-12-18T00:18:28.1931906Z }
2024-12-18T00:18:28.1932157Z echo "${GITHUB_WORKSPACE}"
2024-12-18T00:18:28.1932486Z if [ -z "${NO_SUDO}" ]; then
2024-12-18T00:18:28.1932846Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2024-12-18T00:18:28.1933177Z else
2024-12-18T00:18:28.1933459Z   retry rm -rf "${GITHUB_WORKSPACE}"
2024-12-18T00:18:28.1933786Z fi
2024-12-18T00:18:28.1934036Z mkdir "${GITHUB_WORKSPACE}"
2024-12-18T00:18:28.1942275Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:18:28.1942651Z env:
2024-12-18T00:18:28.1942868Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:18:28.1943137Z   NO_SUDO: true
2024-12-18T00:18:28.1943368Z ##[endgroup]
2024-12-18T00:18:28.1971766Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-12-18T00:18:32.5772789Z ##[group]Run malfet/checkout@silent-checkout
2024-12-18T00:18:32.5773128Z with:
2024-12-18T00:18:32.5773384Z   ref: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:18:32.5773718Z   fetch-depth: 0
2024-12-18T00:18:32.5773966Z   submodules: recursive
2024-12-18T00:18:32.5774226Z   quiet-checkout: true
2024-12-18T00:18:32.5774502Z   repository: pytorch/pytorch
2024-12-18T00:18:32.5774879Z   token: ***
2024-12-18T00:18:32.5775108Z   ssh-strict: true
2024-12-18T00:18:32.5775631Z   persist-credentials: true
2024-12-18T00:18:32.5775909Z   clean: true
2024-12-18T00:18:32.5776159Z   sparse-checkout-cone-mode: true
2024-12-18T00:18:32.5776453Z   lfs: false
2024-12-18T00:18:32.5776694Z   set-safe-directory: true
2024-12-18T00:18:32.5776960Z env:
2024-12-18T00:18:32.5777173Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:18:32.5777437Z ##[endgroup]
2024-12-18T00:18:32.6666222Z Syncing repository: pytorch/pytorch
2024-12-18T00:18:32.6667388Z ##[group]Getting Git version info
2024-12-18T00:18:32.6667874Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2024-12-18T00:18:32.6668522Z [command]/usr/bin/git version
2024-12-18T00:18:32.6668809Z git version 2.40.1
2024-12-18T00:18:32.6681819Z ##[endgroup]
2024-12-18T00:18:32.6694972Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/19b2eafc-6df1-41e9-a1aa-6209c3409546' before making global git config changes
2024-12-18T00:18:32.6695891Z Adding repository directory to the temporary git global config as a safe directory
2024-12-18T00:18:32.6698774Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-12-18T00:18:32.6733227Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2024-12-18T00:18:32.6735998Z ##[group]Initializing the repository
2024-12-18T00:18:32.6738408Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-12-18T00:18:32.6769536Z hint: Using 'master' as the name for the initial branch. This default branch name
2024-12-18T00:18:32.6770170Z hint: is subject to change. To configure the initial branch name to use in all
2024-12-18T00:18:32.6770739Z hint: of your new repositories, which will suppress this warning, call:
2024-12-18T00:18:32.6771164Z hint: 
2024-12-18T00:18:32.6771482Z hint: 	git config --global init.defaultBranch <name>
2024-12-18T00:18:32.6771835Z hint: 
2024-12-18T00:18:32.6772185Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2024-12-18T00:18:32.6772774Z hint: 'development'. The just-created branch can be renamed via this command:
2024-12-18T00:18:32.6773216Z hint: 
2024-12-18T00:18:32.6773452Z hint: 	git branch -m <name>
2024-12-18T00:18:32.6774002Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2024-12-18T00:18:32.6782078Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2024-12-18T00:18:32.6811466Z ##[endgroup]
2024-12-18T00:18:32.6811935Z ##[group]Disabling automatic garbage collection
2024-12-18T00:18:32.6814303Z [command]/usr/bin/git config --local gc.auto 0
2024-12-18T00:18:32.6841660Z ##[endgroup]
2024-12-18T00:18:32.6842073Z ##[group]Setting up auth
2024-12-18T00:18:32.6846305Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2024-12-18T00:18:32.6873404Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2024-12-18T00:18:32.7216725Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2024-12-18T00:18:32.7243680Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2024-12-18T00:18:32.7581007Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2024-12-18T00:18:32.7624749Z ##[endgroup]
2024-12-18T00:18:32.7625189Z ##[group]Fetching the repository
2024-12-18T00:18:32.7630366Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2024-12-18T00:18:35.6440583Z remote: Enumerating objects: 1056419        
2024-12-18T00:18:35.6441219Z remote: Enumerating objects: 1057277, done.        
2024-12-18T00:18:35.6441706Z remote: Counting objects:   0% (1/858)        
2024-12-18T00:18:35.6442475Z remote: Counting objects:   1% (9/858)        
2024-12-18T00:18:35.6442987Z remote: Counting objects:   2% (18/858)        
2024-12-18T00:18:35.6443501Z remote: Counting objects:   3% (26/858)        
2024-12-18T00:18:35.6443991Z remote: Counting objects:   4% (35/858)        
2024-12-18T00:18:35.6444499Z remote: Counting objects:   5% (43/858)        
2024-12-18T00:18:35.6445000Z remote: Counting objects:   6% (52/858)        
2024-12-18T00:18:35.6445546Z remote: Counting objects:   7% (61/858)        
2024-12-18T00:18:35.6446065Z remote: Counting objects:   8% (69/858)        
2024-12-18T00:18:35.6446511Z remote: Counting objects:   9% (78/858)        
2024-12-18T00:18:35.6446875Z remote: Counting objects:  10% (86/858)        
2024-12-18T00:18:35.6447300Z remote: Counting objects:  11% (95/858)        
2024-12-18T00:18:35.6447808Z remote: Counting objects:  12% (103/858)        
2024-12-18T00:18:35.6448320Z remote: Counting objects:  13% (112/858)        
2024-12-18T00:18:35.6448831Z remote: Counting objects:  14% (121/858)        
2024-12-18T00:18:35.6449327Z remote: Counting objects:  15% (129/858)        
2024-12-18T00:18:35.6449859Z remote: Counting objects:  16% (138/858)        
2024-12-18T00:18:35.6450363Z remote: Counting objects:  17% (146/858)        
2024-12-18T00:18:35.6450882Z remote: Counting objects:  18% (155/858)        
2024-12-18T00:18:35.6451381Z remote: Counting objects:  19% (164/858)        
2024-12-18T00:18:35.6452077Z remote: Counting objects:  20% (172/858)        
2024-12-18T00:18:35.6452602Z remote: Counting objects:  21% (181/858)        
2024-12-18T00:18:35.6453106Z remote: Counting objects:  22% (189/858)        
2024-12-18T00:18:35.6453614Z remote: Counting objects:  23% (198/858)        
2024-12-18T00:18:35.6454112Z remote: Counting objects:  24% (206/858)        
2024-12-18T00:18:35.6454610Z remote: Counting objects:  25% (215/858)        
2024-12-18T00:18:35.6455100Z remote: Counting objects:  26% (224/858)        
2024-12-18T00:18:35.6455647Z remote: Counting objects:  27% (232/858)        
2024-12-18T00:18:35.6456009Z remote: Counting objects:  28% (241/858)        
2024-12-18T00:18:35.6456369Z remote: Counting objects:  29% (249/858)        
2024-12-18T00:18:35.6456725Z remote: Counting objects:  30% (258/858)        
2024-12-18T00:18:35.6457077Z remote: Counting objects:  31% (266/858)        
2024-12-18T00:18:35.6457436Z remote: Counting objects:  32% (275/858)        
2024-12-18T00:18:35.6457803Z remote: Counting objects:  33% (284/858)        
2024-12-18T00:18:35.6458161Z remote: Counting objects:  34% (292/858)        
2024-12-18T00:18:35.6458520Z remote: Counting objects:  35% (301/858)        
2024-12-18T00:18:35.6458872Z remote: Counting objects:  36% (309/858)        
2024-12-18T00:18:35.6459229Z remote: Counting objects:  37% (318/858)        
2024-12-18T00:18:35.6459592Z remote: Counting objects:  38% (327/858)        
2024-12-18T00:18:35.6460023Z remote: Counting objects:  39% (335/858)        
2024-12-18T00:18:35.6460391Z remote: Counting objects:  40% (344/858)        
2024-12-18T00:18:35.6460751Z remote: Counting objects:  41% (352/858)        
2024-12-18T00:18:35.6461102Z remote: Counting objects:  42% (361/858)        
2024-12-18T00:18:35.6461488Z remote: Counting objects:  43% (369/858)        
2024-12-18T00:18:35.6461870Z remote: Counting objects:  44% (378/858)        
2024-12-18T00:18:35.6462230Z remote: Counting objects:  45% (387/858)        
2024-12-18T00:18:35.6462594Z remote: Counting objects:  46% (395/858)        
2024-12-18T00:18:35.6462949Z remote: Counting objects:  47% (404/858)        
2024-12-18T00:18:35.6463356Z remote: Counting objects:  48% (412/858)        
2024-12-18T00:18:35.6463717Z remote: Counting objects:  49% (421/858)        
2024-12-18T00:18:35.6464077Z remote: Counting objects:  50% (429/858)        
2024-12-18T00:18:35.6464437Z remote: Counting objects:  51% (438/858)        
2024-12-18T00:18:35.6464797Z remote: Counting objects:  52% (447/858)        
2024-12-18T00:18:35.6465277Z remote: Counting objects:  53% (455/858)        
2024-12-18T00:18:35.6465647Z remote: Counting objects:  54% (464/858)        
2024-12-18T00:18:35.6466010Z remote: Counting objects:  55% (472/858)        
2024-12-18T00:18:35.6466370Z remote: Counting objects:  56% (481/858)        
2024-12-18T00:18:35.6466729Z remote: Counting objects:  57% (490/858)        
2024-12-18T00:18:35.6467079Z remote: Counting objects:  58% (498/858)        
2024-12-18T00:18:35.6467443Z remote: Counting objects:  59% (507/858)        
2024-12-18T00:18:35.6467799Z remote: Counting objects:  60% (515/858)        
2024-12-18T00:18:35.6468160Z remote: Counting objects:  61% (524/858)        
2024-12-18T00:18:35.6468515Z remote: Counting objects:  62% (532/858)        
2024-12-18T00:18:35.6468909Z remote: Counting objects:  63% (541/858)        
2024-12-18T00:18:35.6469315Z remote: Counting objects:  64% (550/858)        
2024-12-18T00:18:35.6469668Z remote: Counting objects:  65% (558/858)        
2024-12-18T00:18:35.6470027Z remote: Counting objects:  66% (567/858)        
2024-12-18T00:18:35.6470392Z remote: Counting objects:  67% (575/858)        
2024-12-18T00:18:35.6470753Z remote: Counting objects:  68% (584/858)        
2024-12-18T00:18:35.6471112Z remote: Counting objects:  69% (593/858)        
2024-12-18T00:18:35.6471490Z remote: Counting objects:  70% (601/858)        
2024-12-18T00:18:35.6471878Z remote: Counting objects:  71% (610/858)        
2024-12-18T00:18:35.6472236Z remote: Counting objects:  72% (618/858)        
2024-12-18T00:18:35.6472701Z remote: Counting objects:  73% (627/858)        
2024-12-18T00:18:35.6473065Z remote: Counting objects:  74% (635/858)        
2024-12-18T00:18:35.6473428Z remote: Counting objects:  75% (644/858)        
2024-12-18T00:18:35.6473779Z remote: Counting objects:  76% (653/858)        
2024-12-18T00:18:35.6474141Z remote: Counting objects:  77% (661/858)        
2024-12-18T00:18:35.6474509Z remote: Counting objects:  78% (670/858)        
2024-12-18T00:18:35.6474871Z remote: Counting objects:  79% (678/858)        
2024-12-18T00:18:35.6475234Z remote: Counting objects:  80% (687/858)        
2024-12-18T00:18:35.6475594Z remote: Counting objects:  81% (695/858)        
2024-12-18T00:18:35.6475954Z remote: Counting objects:  82% (704/858)        
2024-12-18T00:18:35.6476314Z remote: Counting objects:  83% (713/858)        
2024-12-18T00:18:35.6476677Z remote: Counting objects:  84% (721/858)        
2024-12-18T00:18:35.6477038Z remote: Counting objects:  85% (730/858)        
2024-12-18T00:18:35.6477400Z remote: Counting objects:  86% (738/858)        
2024-12-18T00:18:35.6477760Z remote: Counting objects:  87% (747/858)        
2024-12-18T00:18:35.6478123Z remote: Counting objects:  88% (756/858)        
2024-12-18T00:18:35.6478484Z remote: Counting objects:  89% (764/858)        
2024-12-18T00:18:35.6478845Z remote: Counting objects:  90% (773/858)        
2024-12-18T00:18:35.6479379Z remote: Counting objects:  91% (781/858)        
2024-12-18T00:18:35.6479735Z remote: Counting objects:  92% (790/858)        
2024-12-18T00:18:35.6480104Z remote: Counting objects:  93% (798/858)        
2024-12-18T00:18:35.6480465Z remote: Counting objects:  94% (807/858)        
2024-12-18T00:18:35.6480826Z remote: Counting objects:  95% (816/858)        
2024-12-18T00:18:35.6481185Z remote: Counting objects:  96% (824/858)        
2024-12-18T00:18:35.6481541Z remote: Counting objects:  97% (833/858)        
2024-12-18T00:18:35.6481902Z remote: Counting objects:  98% (841/858)        
2024-12-18T00:18:35.6482274Z remote: Counting objects:  99% (850/858)        
2024-12-18T00:18:35.6482637Z remote: Counting objects: 100% (858/858)        
2024-12-18T00:18:35.6483023Z remote: Counting objects: 100% (858/858), done.        
2024-12-18T00:18:35.6538538Z remote: Compressing objects:   0% (1/311)        
2024-12-18T00:18:35.6595123Z remote: Compressing objects:   1% (4/311)        
2024-12-18T00:18:35.6632831Z remote: Compressing objects:   2% (7/311)        
2024-12-18T00:18:35.6835415Z remote: Compressing objects:   3% (10/311)        
2024-12-18T00:18:35.7146199Z remote: Compressing objects:   4% (13/311)        
2024-12-18T00:18:35.7631914Z remote: Compressing objects:   5% (16/311)        
2024-12-18T00:18:35.8204127Z remote: Compressing objects:   6% (19/311)        
2024-12-18T00:18:35.8667097Z remote: Compressing objects:   7% (22/311)        
2024-12-18T00:18:35.9104956Z remote: Compressing objects:   8% (25/311)        
2024-12-18T00:18:35.9573142Z remote: Compressing objects:   9% (28/311)        
2024-12-18T00:18:35.9704481Z remote: Compressing objects:  10% (32/311)        
2024-12-18T00:18:35.9831266Z remote: Compressing objects:  11% (35/311)        
2024-12-18T00:18:35.9899899Z remote: Compressing objects:  12% (38/311)        
2024-12-18T00:18:35.9924535Z remote: Compressing objects:  13% (41/311)        
2024-12-18T00:18:35.9925091Z remote: Compressing objects:  14% (44/311)        
2024-12-18T00:18:35.9927594Z remote: Compressing objects:  15% (47/311)        
2024-12-18T00:18:35.9930414Z remote: Compressing objects:  16% (50/311)        
2024-12-18T00:18:35.9934525Z remote: Compressing objects:  17% (53/311)        
2024-12-18T00:18:35.9938836Z remote: Compressing objects:  18% (56/311)        
2024-12-18T00:18:35.9941017Z remote: Compressing objects:  19% (60/311)        
2024-12-18T00:18:35.9941798Z remote: Compressing objects:  20% (63/311)        
2024-12-18T00:18:35.9944500Z remote: Compressing objects:  21% (66/311)        
2024-12-18T00:18:35.9948186Z remote: Compressing objects:  22% (69/311)        
2024-12-18T00:18:35.9948778Z remote: Compressing objects:  23% (72/311)        
2024-12-18T00:18:35.9952585Z remote: Compressing objects:  24% (75/311)        
2024-12-18T00:18:35.9955380Z remote: Compressing objects:  25% (78/311)        
2024-12-18T00:18:35.9962893Z remote: Compressing objects:  26% (81/311)        
2024-12-18T00:18:35.9971817Z remote: Compressing objects:  27% (84/311)        
2024-12-18T00:18:35.9974781Z remote: Compressing objects:  28% (88/311)        
2024-12-18T00:18:35.9977227Z remote: Compressing objects:  29% (91/311)        
2024-12-18T00:18:35.9983947Z remote: Compressing objects:  30% (94/311)        
2024-12-18T00:18:35.9984847Z remote: Compressing objects:  31% (97/311)        
2024-12-18T00:18:35.9991598Z remote: Compressing objects:  32% (100/311)        
2024-12-18T00:18:36.0000045Z remote: Compressing objects:  33% (103/311)        
2024-12-18T00:18:36.0000725Z remote: Compressing objects:  34% (106/311)        
2024-12-18T00:18:36.0005232Z remote: Compressing objects:  35% (109/311)        
2024-12-18T00:18:36.0008207Z remote: Compressing objects:  36% (112/311)        
2024-12-18T00:18:36.0012440Z remote: Compressing objects:  37% (116/311)        
2024-12-18T00:18:36.0012974Z remote: Compressing objects:  38% (119/311)        
2024-12-18T00:18:36.0017973Z remote: Compressing objects:  39% (122/311)        
2024-12-18T00:18:36.0024141Z remote: Compressing objects:  40% (125/311)        
2024-12-18T00:18:36.0030855Z remote: Compressing objects:  41% (128/311)        
2024-12-18T00:18:36.0031528Z remote: Compressing objects:  42% (131/311)        
2024-12-18T00:18:36.0035688Z remote: Compressing objects:  43% (134/311)        
2024-12-18T00:18:36.0039369Z remote: Compressing objects:  44% (137/311)        
2024-12-18T00:18:36.0043328Z remote: Compressing objects:  45% (140/311)        
2024-12-18T00:18:36.0046595Z remote: Compressing objects:  46% (144/311)        
2024-12-18T00:18:36.0050772Z remote: Compressing objects:  47% (147/311)        
2024-12-18T00:18:36.0052401Z remote: Compressing objects:  48% (150/311)        
2024-12-18T00:18:36.0055349Z remote: Compressing objects:  49% (153/311)        
2024-12-18T00:18:36.0055874Z remote: Compressing objects:  50% (156/311)        
2024-12-18T00:18:36.0056736Z remote: Compressing objects:  51% (159/311)        
2024-12-18T00:18:36.0059135Z remote: Compressing objects:  52% (162/311)        
2024-12-18T00:18:36.0060193Z remote: Compressing objects:  53% (165/311)        
2024-12-18T00:18:36.0063111Z remote: Compressing objects:  54% (168/311)        
2024-12-18T00:18:36.0064411Z remote: Compressing objects:  55% (172/311)        
2024-12-18T00:18:36.0066253Z remote: Compressing objects:  56% (175/311)        
2024-12-18T00:18:36.0067092Z remote: Compressing objects:  57% (178/311)        
2024-12-18T00:18:36.0069382Z remote: Compressing objects:  58% (181/311)        
2024-12-18T00:18:36.0072037Z remote: Compressing objects:  59% (184/311)        
2024-12-18T00:18:36.0073070Z remote: Compressing objects:  60% (187/311)        
2024-12-18T00:18:36.0073637Z remote: Compressing objects:  61% (190/311)        
2024-12-18T00:18:36.0075590Z remote: Compressing objects:  62% (193/311)        
2024-12-18T00:18:36.0081856Z remote: Compressing objects:  63% (196/311)        
2024-12-18T00:18:36.0085392Z remote: Compressing objects:  64% (200/311)        
2024-12-18T00:18:36.0087504Z remote: Compressing objects:  65% (203/311)        
2024-12-18T00:18:36.0089832Z remote: Compressing objects:  66% (206/311)        
2024-12-18T00:18:36.0092389Z remote: Compressing objects:  67% (209/311)        
2024-12-18T00:18:36.0096224Z remote: Compressing objects:  68% (212/311)        
2024-12-18T00:18:36.0098070Z remote: Compressing objects:  69% (215/311)        
2024-12-18T00:18:36.0100268Z remote: Compressing objects:  70% (218/311)        
2024-12-18T00:18:36.0100809Z remote: Compressing objects:  71% (221/311)        
2024-12-18T00:18:36.0103682Z remote: Compressing objects:  72% (224/311)        
2024-12-18T00:18:36.0104432Z remote: Compressing objects:  73% (228/311)        
2024-12-18T00:18:36.0105201Z remote: Compressing objects:  74% (231/311)        
2024-12-18T00:18:36.0105736Z remote: Compressing objects:  75% (234/311)        
2024-12-18T00:18:36.0106669Z remote: Compressing objects:  76% (237/311)        
2024-12-18T00:18:36.0107971Z remote: Compressing objects:  77% (240/311)        
2024-12-18T00:18:36.0108651Z remote: Compressing objects:  78% (243/311)        
2024-12-18T00:18:36.0109588Z remote: Compressing objects:  79% (246/311)        
2024-12-18T00:18:36.0110384Z remote: Compressing objects:  80% (249/311)        
2024-12-18T00:18:36.0112544Z remote: Compressing objects:  81% (252/311)        
2024-12-18T00:18:36.0114426Z remote: Compressing objects:  82% (256/311)        
2024-12-18T00:18:36.0116615Z remote: Compressing objects:  83% (259/311)        
2024-12-18T00:18:36.0117988Z remote: Compressing objects:  84% (262/311)        
2024-12-18T00:18:36.0119814Z remote: Compressing objects:  85% (265/311)        
2024-12-18T00:18:36.0121073Z remote: Compressing objects:  86% (268/311)        
2024-12-18T00:18:36.0122710Z remote: Compressing objects:  87% (271/311)        
2024-12-18T00:18:36.0123443Z remote: Compressing objects:  88% (274/311)        
2024-12-18T00:18:36.0124650Z remote: Compressing objects:  89% (277/311)        
2024-12-18T00:18:36.0125984Z remote: Compressing objects:  90% (280/311)        
2024-12-18T00:18:36.0127248Z remote: Compressing objects:  91% (284/311)        
2024-12-18T00:18:36.0128242Z remote: Compressing objects:  92% (287/311)        
2024-12-18T00:18:36.0128794Z remote: Compressing objects:  93% (290/311)        
2024-12-18T00:18:36.0130448Z remote: Compressing objects:  94% (293/311)        
2024-12-18T00:18:36.0130972Z remote: Compressing objects:  95% (296/311)        
2024-12-18T00:18:36.0131506Z remote: Compressing objects:  96% (299/311)        
2024-12-18T00:18:36.0132072Z remote: Compressing objects:  97% (302/311)        
2024-12-18T00:18:36.0132604Z remote: Compressing objects:  98% (305/311)        
2024-12-18T00:18:36.0133134Z remote: Compressing objects:  99% (308/311)        
2024-12-18T00:18:36.0133666Z remote: Compressing objects: 100% (311/311)        
2024-12-18T00:18:36.0134215Z remote: Compressing objects: 100% (311/311), done.        
2024-12-18T00:18:59.6874504Z remote: Total 1057277 (delta 705), reused 551 (delta 547), pack-reused 1056419 (from 2)        
2024-12-18T00:19:22.2661023Z [command]/usr/bin/git rev-parse --verify --quiet 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7^{object}
2024-12-18T00:19:22.2685239Z 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:19:22.2690803Z ##[endgroup]
2024-12-18T00:19:22.2691361Z ##[group]Determining the checkout info
2024-12-18T00:19:22.2692147Z ##[endgroup]
2024-12-18T00:19:22.2692620Z ##[group]Checking out the ref
2024-12-18T00:19:22.2695232Z [command]/usr/bin/git checkout --quiet --force 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:19:24.0236063Z ##[endgroup]
2024-12-18T00:19:24.0236546Z ##[group]Setting up auth for fetching submodules
2024-12-18T00:19:24.0240523Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2024-12-18T00:19:24.0288740Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2024-12-18T00:19:24.0316159Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2024-12-18T00:19:24.0345099Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2024-12-18T00:19:24.0375698Z ##[endgroup]
2024-12-18T00:19:24.0376101Z ##[group]Fetching submodules
2024-12-18T00:19:24.0378164Z [command]/usr/bin/git submodule sync --recursive
2024-12-18T00:19:24.0750625Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2024-12-18T00:19:24.1106713Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2024-12-18T00:19:24.1109423Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2024-12-18T00:19:24.1112824Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2024-12-18T00:19:24.1116611Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2024-12-18T00:19:24.1120680Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX'
2024-12-18T00:19:24.1124909Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2024-12-18T00:19:24.1128455Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2024-12-18T00:19:24.1132574Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2024-12-18T00:19:24.1136721Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel'
2024-12-18T00:19:24.1140885Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib'
2024-12-18T00:19:24.1145079Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2024-12-18T00:19:24.1149550Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2024-12-18T00:19:24.1153819Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2024-12-18T00:19:24.1158342Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2024-12-18T00:19:24.1163296Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2024-12-18T00:19:24.1169256Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2024-12-18T00:19:24.1177804Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2024-12-18T00:19:24.1182780Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2024-12-18T00:19:24.1187467Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2024-12-18T00:19:24.1192788Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2024-12-18T00:19:24.1197778Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2024-12-18T00:19:24.1203128Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2024-12-18T00:19:24.1208711Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2024-12-18T00:19:24.1214017Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc'
2024-12-18T00:19:24.1219307Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2024-12-18T00:19:24.1224747Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2024-12-18T00:19:24.1230105Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2024-12-18T00:19:24.1235888Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp'
2024-12-18T00:19:24.1241560Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2024-12-18T00:19:24.1247469Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2024-12-18T00:19:24.1252913Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2024-12-18T00:19:24.1258797Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2024-12-18T00:19:24.1264612Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2024-12-18T00:19:24.1274273Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2024-12-18T00:19:24.1280392Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2024-12-18T00:19:24.1286594Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2024-12-18T00:19:24.1323490Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2024-12-18T00:19:24.4394900Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2024-12-18T00:19:24.6484004Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2024-12-18T00:19:24.8660216Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2024-12-18T00:19:25.1366300Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'...
2024-12-18T00:19:25.4705376Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2024-12-18T00:19:27.4504379Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2024-12-18T00:19:38.6446796Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2024-12-18T00:19:39.0788053Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'...
2024-12-18T00:19:41.2829712Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'...
2024-12-18T00:19:41.8093131Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2024-12-18T00:19:42.4419051Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2024-12-18T00:19:43.6634920Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2024-12-18T00:19:45.8326778Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2024-12-18T00:19:51.7522693Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2024-12-18T00:19:53.4109263Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2024-12-18T00:19:54.7062174Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2024-12-18T00:19:56.0569921Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2024-12-18T00:19:56.5330064Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2024-12-18T00:19:56.9222185Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2024-12-18T00:19:58.2350275Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2024-12-18T00:19:58.6030693Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2024-12-18T00:19:58.8625489Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2024-12-18T00:20:00.4777738Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'...
2024-12-18T00:20:01.3454501Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2024-12-18T00:20:01.7435639Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2024-12-18T00:20:09.0099123Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2024-12-18T00:20:11.0744425Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'...
2024-12-18T00:20:15.9981710Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2024-12-18T00:20:16.2372475Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2024-12-18T00:20:25.5474545Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2024-12-18T00:20:25.7659658Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2024-12-18T00:20:25.9754470Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2024-12-18T00:20:27.0800549Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2024-12-18T00:20:27.4043227Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2024-12-18T00:20:28.0110487Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2024-12-18T00:20:28.4538605Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2024-12-18T00:20:28.4689444Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2024-12-18T00:20:28.4806661Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2024-12-18T00:20:28.5119306Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2024-12-18T00:20:28.5558605Z Submodule path 'third_party/NVTX': checked out 'e170594ac7cf1dac584da473d4ca9301087090c1'
2024-12-18T00:20:28.6017081Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2024-12-18T00:20:29.5268273Z Submodule path 'third_party/XNNPACK': checked out '4ea82e595b36106653175dcb04b2aa532660d0d8'
2024-12-18T00:20:29.5555990Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2024-12-18T00:20:29.8490732Z Submodule path 'third_party/composable_kernel': checked out '50ee4267e27b875d149e642f4cebd47be1dc3b57'
2024-12-18T00:20:29.9052311Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006'
2024-12-18T00:20:30.0169484Z Submodule path 'third_party/cpuinfo': checked out '1e83a2fdd3102f65c6f1fb602c1b320486218a99'
2024-12-18T00:20:30.0568275Z Submodule path 'third_party/cudnn_frontend': checked out '936021bfed8c91dc416af1588b2c4eca631a9e45'
2024-12-18T00:20:30.6815083Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49'
2024-12-18T00:20:30.9694079Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2024-12-18T00:20:31.0769007Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e'
2024-12-18T00:20:31.0792470Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2024-12-18T00:20:31.0795845Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T00:20:31.0799386Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass'
2024-12-18T00:20:31.0803110Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2024-12-18T00:20:31.0807088Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T00:20:31.0839735Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2024-12-18T00:20:32.0367322Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2024-12-18T00:20:32.6447580Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'...
2024-12-18T00:20:34.8052194Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2024-12-18T00:20:35.8224418Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2024-12-18T00:20:36.2355980Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2024-12-18T00:20:36.3451112Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2024-12-18T00:20:36.8547310Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6'
2024-12-18T00:20:36.9257300Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2024-12-18T00:20:36.9411271Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1'
2024-12-18T00:20:37.1076104Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b'
2024-12-18T00:20:37.1525051Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592'
2024-12-18T00:20:37.1978814Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2024-12-18T00:20:37.2304171Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509'
2024-12-18T00:20:37.2808160Z Submodule path 'third_party/googletest': checked out 'b514bdc898e2951020cbdca1304b75f5950d1f59'
2024-12-18T00:20:37.2977960Z Submodule path 'third_party/ideep': checked out 'c7ccd5bdbe5434ba156f4e856dcef0601637334b'
2024-12-18T00:20:37.2998785Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2024-12-18T00:20:37.3029257Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2024-12-18T00:20:52.0670212Z Submodule path 'third_party/ideep/mkl-dnn': checked out '66f0cb9eb66affd2da3bf5f8d897376f04aae6af'
2024-12-18T00:20:52.0899374Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2024-12-18T00:20:52.1887134Z Submodule path 'third_party/kineto': checked out '338140f58a28d599da3434ced4fd2d75dd1a213d'
2024-12-18T00:20:52.1911663Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T00:20:52.1914932Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T00:20:52.1918482Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T00:20:52.1950098Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'...
2024-12-18T00:20:52.9787688Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2024-12-18T00:20:54.2062328Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2024-12-18T00:20:55.3383129Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e'
2024-12-18T00:20:55.3403810Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T00:20:55.3407592Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T00:20:55.3410776Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T00:20:55.3414340Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T00:20:55.3418668Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T00:20:55.3422444Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T00:20:55.3426251Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T00:20:55.3430150Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T00:20:55.3462698Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'...
2024-12-18T00:20:56.2683477Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'...
2024-12-18T00:20:56.6818395Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'...
2024-12-18T00:20:57.9122870Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'...
2024-12-18T00:20:58.2204344Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'...
2024-12-18T00:20:58.6983043Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'...
2024-12-18T00:20:59.7361930Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'...
2024-12-18T00:21:07.2158164Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'...
2024-12-18T00:21:07.6378720Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9'
2024-12-18T00:21:07.6617068Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400'
2024-12-18T00:21:07.7056253Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2024-12-18T00:21:07.7227315Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067'
2024-12-18T00:21:07.7247229Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T00:21:07.7278927Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'...
2024-12-18T00:21:08.0744849Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4'
2024-12-18T00:21:08.0977103Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446'
2024-12-18T00:21:08.1460806Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850'
2024-12-18T00:21:08.2638732Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5'
2024-12-18T00:21:08.2847072Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150'
2024-12-18T00:21:08.3296765Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164'
2024-12-18T00:21:08.3967931Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2024-12-18T00:21:08.4429864Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78'
2024-12-18T00:21:08.4787419Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290'
2024-12-18T00:21:08.6142583Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2024-12-18T00:21:09.1138677Z Submodule path 'third_party/onnx': checked out 'b8baa8446686496da4cc8fda09f2b6fe65c2a02c'
2024-12-18T00:21:09.1178829Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2024-12-18T00:21:09.1213336Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2024-12-18T00:21:10.2960391Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b'
2024-12-18T00:21:10.3857490Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878'
2024-12-18T00:21:10.3881119Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T00:21:10.3884094Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T00:21:10.3887482Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T00:21:10.3891013Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T00:21:10.3894843Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T00:21:10.3898406Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T00:21:10.3902107Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T00:21:10.3906179Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T00:21:10.3938118Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'...
2024-12-18T00:21:10.8937104Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'...
2024-12-18T00:21:11.9261249Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'...
2024-12-18T00:21:12.2650221Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'...
2024-12-18T00:21:19.4405072Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'...
2024-12-18T00:21:19.7455433Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'...
2024-12-18T00:21:19.9653469Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'...
2024-12-18T00:21:20.2883094Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'...
2024-12-18T00:21:27.1897084Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2'
2024-12-18T00:21:27.2375517Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1'
2024-12-18T00:21:27.2572085Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa'
2024-12-18T00:21:27.3882380Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d'
2024-12-18T00:21:27.4047554Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce'
2024-12-18T00:21:27.4237833Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5'
2024-12-18T00:21:27.4443047Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d'
2024-12-18T00:21:27.4462153Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T00:21:27.4465786Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T00:21:27.4497067Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'...
2024-12-18T00:21:29.3109305Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'...
2024-12-18T00:21:30.6439826Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4'
2024-12-18T00:21:30.6978397Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2024-12-18T00:21:31.3577566Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50'
2024-12-18T00:21:31.3730571Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe'
2024-12-18T00:21:31.7037518Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2024-12-18T00:21:31.7064052Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2024-12-18T00:21:31.7067239Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2024-12-18T00:21:31.7098959Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2024-12-18T00:21:32.1740132Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2024-12-18T00:21:33.2555080Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2024-12-18T00:21:33.3381010Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2024-12-18T00:21:33.3500551Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2024-12-18T00:21:33.3661762Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8'
2024-12-18T00:21:33.4108196Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4'
2024-12-18T00:21:33.4462187Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2024-12-18T00:21:33.4967543Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b'
2024-12-18T00:21:33.5317037Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2024-12-18T00:21:33.5338691Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2024-12-18T00:21:33.5342636Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2024-12-18T00:21:33.5345569Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2024-12-18T00:21:33.5349719Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T00:21:33.5381062Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2024-12-18T00:21:34.5514557Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2024-12-18T00:21:34.7944658Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2024-12-18T00:21:37.0771788Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2024-12-18T00:21:38.2372444Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2024-12-18T00:21:38.2570351Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2024-12-18T00:21:38.3298198Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2024-12-18T00:21:38.3641834Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2024-12-18T00:21:38.3662453Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T00:21:38.3693752Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2024-12-18T00:21:38.5840397Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2024-12-18T00:21:38.5886897Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2024-12-18T00:21:38.6251398Z Entering 'android/libs/fbjni'
2024-12-18T00:21:38.6304310Z Entering 'third_party/FP16'
2024-12-18T00:21:38.6356865Z Entering 'third_party/FXdiv'
2024-12-18T00:21:38.6409715Z Entering 'third_party/NNPACK'
2024-12-18T00:21:38.6462399Z Entering 'third_party/NVTX'
2024-12-18T00:21:38.6515962Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T00:21:38.6568460Z Entering 'third_party/XNNPACK'
2024-12-18T00:21:38.6638526Z Entering 'third_party/benchmark'
2024-12-18T00:21:38.6691851Z Entering 'third_party/composable_kernel'
2024-12-18T00:21:38.6751885Z Entering 'third_party/cpp-httplib'
2024-12-18T00:21:38.6804375Z Entering 'third_party/cpuinfo'
2024-12-18T00:21:38.6858104Z Entering 'third_party/cudnn_frontend'
2024-12-18T00:21:38.6911837Z Entering 'third_party/cutlass'
2024-12-18T00:21:38.6972448Z Entering 'third_party/eigen'
2024-12-18T00:21:38.7027155Z Entering 'third_party/fbgemm'
2024-12-18T00:21:38.7077555Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T00:21:38.7129030Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T00:21:38.7180657Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T00:21:38.7239614Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T00:21:38.7290826Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T00:21:38.7344692Z Entering 'third_party/flatbuffers'
2024-12-18T00:21:38.7398611Z Entering 'third_party/fmt'
2024-12-18T00:21:38.7450816Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T00:21:38.7502438Z Entering 'third_party/gloo'
2024-12-18T00:21:38.7554679Z Entering 'third_party/googletest'
2024-12-18T00:21:38.7606841Z Entering 'third_party/ideep'
2024-12-18T00:21:38.7657082Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T00:21:38.7717245Z Entering 'third_party/ittapi'
2024-12-18T00:21:38.7769541Z Entering 'third_party/kineto'
2024-12-18T00:21:38.7825119Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T00:21:38.7875035Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T00:21:38.7927897Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T00:21:38.7979985Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T00:21:38.8031772Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T00:21:38.8087556Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T00:21:38.8144711Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T00:21:38.8196210Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T00:21:38.8248303Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T00:21:38.8300830Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T00:21:38.8362067Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T00:21:38.8412237Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T00:21:38.8464893Z Entering 'third_party/mimalloc'
2024-12-18T00:21:38.8520329Z Entering 'third_party/nccl/nccl'
2024-12-18T00:21:38.8572412Z Entering 'third_party/nlohmann'
2024-12-18T00:21:38.8624830Z Entering 'third_party/onnx'
2024-12-18T00:21:38.8691445Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T00:21:38.8748341Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T00:21:38.8801766Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T00:21:38.8855639Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T00:21:38.8907448Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T00:21:38.8957189Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T00:21:38.9010789Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T00:21:38.9059942Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T00:21:38.9114228Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T00:21:38.9164441Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T00:21:38.9216117Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T00:21:38.9268574Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T00:21:38.9340670Z Entering 'third_party/pocketfft'
2024-12-18T00:21:38.9394632Z Entering 'third_party/protobuf'
2024-12-18T00:21:38.9449068Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T00:21:38.9498924Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T00:21:38.9556418Z Entering 'third_party/psimd'
2024-12-18T00:21:38.9609002Z Entering 'third_party/pthreadpool'
2024-12-18T00:21:38.9659951Z Entering 'third_party/pybind11'
2024-12-18T00:21:38.9713519Z Entering 'third_party/python-peachpy'
2024-12-18T00:21:38.9766736Z Entering 'third_party/sleef'
2024-12-18T00:21:38.9819826Z Entering 'third_party/tensorpipe'
2024-12-18T00:21:38.9870692Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T00:21:38.9920422Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T00:21:38.9970625Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T00:21:39.0022856Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T00:21:39.0070475Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T00:21:39.0143466Z ##[endgroup]
2024-12-18T00:21:39.0146746Z ##[group]Persisting credentials for submodules
2024-12-18T00:21:39.0149161Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :"
2024-12-18T00:21:39.0505324Z Entering 'android/libs/fbjni'
2024-12-18T00:21:39.0573649Z Entering 'third_party/FP16'
2024-12-18T00:21:39.0641103Z Entering 'third_party/FXdiv'
2024-12-18T00:21:39.0708447Z Entering 'third_party/NNPACK'
2024-12-18T00:21:39.0776104Z Entering 'third_party/NVTX'
2024-12-18T00:21:39.0845376Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T00:21:39.0912861Z Entering 'third_party/XNNPACK'
2024-12-18T00:21:39.0996161Z Entering 'third_party/benchmark'
2024-12-18T00:21:39.1062246Z Entering 'third_party/composable_kernel'
2024-12-18T00:21:39.1136421Z Entering 'third_party/cpp-httplib'
2024-12-18T00:21:39.1203772Z Entering 'third_party/cpuinfo'
2024-12-18T00:21:39.1272367Z Entering 'third_party/cudnn_frontend'
2024-12-18T00:21:39.1339349Z Entering 'third_party/cutlass'
2024-12-18T00:21:39.1414007Z Entering 'third_party/eigen'
2024-12-18T00:21:39.1483304Z Entering 'third_party/fbgemm'
2024-12-18T00:21:39.1550494Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T00:21:39.1616774Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T00:21:39.1683386Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T00:21:39.1756511Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T00:21:39.1823988Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T00:21:39.1892818Z Entering 'third_party/flatbuffers'
2024-12-18T00:21:39.1963318Z Entering 'third_party/fmt'
2024-12-18T00:21:39.2031125Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T00:21:39.2099089Z Entering 'third_party/gloo'
2024-12-18T00:21:39.2168282Z Entering 'third_party/googletest'
2024-12-18T00:21:39.2236373Z Entering 'third_party/ideep'
2024-12-18T00:21:39.2302502Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T00:21:39.2381845Z Entering 'third_party/ittapi'
2024-12-18T00:21:39.2449333Z Entering 'third_party/kineto'
2024-12-18T00:21:39.2515305Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T00:21:39.2580723Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T00:21:39.2650810Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T00:21:39.2719615Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T00:21:39.2786896Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T00:21:39.2853733Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T00:21:39.2925296Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T00:21:39.2993715Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T00:21:39.3062966Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T00:21:39.3132319Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T00:21:39.3202707Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T00:21:39.3270116Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T00:21:39.3339972Z Entering 'third_party/mimalloc'
2024-12-18T00:21:39.3409328Z Entering 'third_party/nccl/nccl'
2024-12-18T00:21:39.3477986Z Entering 'third_party/nlohmann'
2024-12-18T00:21:39.3548360Z Entering 'third_party/onnx'
2024-12-18T00:21:39.3629747Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T00:21:39.3701814Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T00:21:39.3772102Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T00:21:39.3843046Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T00:21:39.3910731Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T00:21:39.3976756Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T00:21:39.4046069Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T00:21:39.4113243Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T00:21:39.4180135Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T00:21:39.4245302Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T00:21:39.4316342Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T00:21:39.4387670Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T00:21:39.4475480Z Entering 'third_party/pocketfft'
2024-12-18T00:21:39.4544243Z Entering 'third_party/protobuf'
2024-12-18T00:21:39.4612719Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T00:21:39.4680015Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T00:21:39.4750753Z Entering 'third_party/psimd'
2024-12-18T00:21:39.4818317Z Entering 'third_party/pthreadpool'
2024-12-18T00:21:39.4886158Z Entering 'third_party/pybind11'
2024-12-18T00:21:39.4955633Z Entering 'third_party/python-peachpy'
2024-12-18T00:21:39.5022414Z Entering 'third_party/sleef'
2024-12-18T00:21:39.5089558Z Entering 'third_party/tensorpipe'
2024-12-18T00:21:39.5155921Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T00:21:39.5222746Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T00:21:39.5288252Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T00:21:39.5355454Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T00:21:39.5418714Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T00:21:39.5510513Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url"
2024-12-18T00:21:39.5866023Z Entering 'android/libs/fbjni'
2024-12-18T00:21:39.5933341Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2024-12-18T00:21:39.5954991Z Entering 'third_party/FP16'
2024-12-18T00:21:39.6018697Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2024-12-18T00:21:39.6040914Z Entering 'third_party/FXdiv'
2024-12-18T00:21:39.6101914Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2024-12-18T00:21:39.6124688Z Entering 'third_party/NNPACK'
2024-12-18T00:21:39.6186538Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2024-12-18T00:21:39.6209348Z Entering 'third_party/NVTX'
2024-12-18T00:21:39.6270522Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config	remote.origin.url
2024-12-18T00:21:39.6294392Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T00:21:39.6355083Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2024-12-18T00:21:39.6376858Z Entering 'third_party/XNNPACK'
2024-12-18T00:21:39.6437319Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2024-12-18T00:21:39.6473591Z Entering 'third_party/benchmark'
2024-12-18T00:21:39.6535747Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2024-12-18T00:21:39.6557630Z Entering 'third_party/composable_kernel'
2024-12-18T00:21:39.6620586Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config	remote.origin.url
2024-12-18T00:21:39.6648654Z Entering 'third_party/cpp-httplib'
2024-12-18T00:21:39.6710779Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config	remote.origin.url
2024-12-18T00:21:39.6732613Z Entering 'third_party/cpuinfo'
2024-12-18T00:21:39.6794051Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2024-12-18T00:21:39.6816754Z Entering 'third_party/cudnn_frontend'
2024-12-18T00:21:39.6878054Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2024-12-18T00:21:39.6900120Z Entering 'third_party/cutlass'
2024-12-18T00:21:39.6962357Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2024-12-18T00:21:39.6991210Z Entering 'third_party/eigen'
2024-12-18T00:21:39.7052997Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2024-12-18T00:21:39.7076487Z Entering 'third_party/fbgemm'
2024-12-18T00:21:39.7142543Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2024-12-18T00:21:39.7164320Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T00:21:39.7227389Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2024-12-18T00:21:39.7248726Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T00:21:39.7312150Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2024-12-18T00:21:39.7333551Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T00:21:39.7395485Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config	remote.origin.url
2024-12-18T00:21:39.7422996Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T00:21:39.7488237Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:39.7508841Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T00:21:39.7570501Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2024-12-18T00:21:39.7594043Z Entering 'third_party/flatbuffers'
2024-12-18T00:21:39.7656148Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2024-12-18T00:21:39.7680398Z Entering 'third_party/fmt'
2024-12-18T00:21:39.7742683Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2024-12-18T00:21:39.7763684Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T00:21:39.7826995Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2024-12-18T00:21:39.7849023Z Entering 'third_party/gloo'
2024-12-18T00:21:39.7912092Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2024-12-18T00:21:39.7933848Z Entering 'third_party/googletest'
2024-12-18T00:21:39.7995963Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:39.8020223Z Entering 'third_party/ideep'
2024-12-18T00:21:39.8085900Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2024-12-18T00:21:39.8107404Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T00:21:39.8168500Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2024-12-18T00:21:39.8198783Z Entering 'third_party/ittapi'
2024-12-18T00:21:39.8262701Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2024-12-18T00:21:39.8284630Z Entering 'third_party/kineto'
2024-12-18T00:21:39.8349102Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2024-12-18T00:21:39.8369817Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T00:21:39.8433047Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config	remote.origin.url
2024-12-18T00:21:39.8452766Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T00:21:39.8516366Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config	remote.origin.url
2024-12-18T00:21:39.8538837Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T00:21:39.8601246Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config	remote.origin.url
2024-12-18T00:21:39.8622932Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T00:21:39.8685540Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config	remote.origin.url
2024-12-18T00:21:39.8707779Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T00:21:39.8770789Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config	remote.origin.url
2024-12-18T00:21:39.8789382Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T00:21:39.8853663Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config	remote.origin.url
2024-12-18T00:21:39.8878355Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T00:21:39.8943710Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config	remote.origin.url
2024-12-18T00:21:39.8963921Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T00:21:39.9025609Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:39.9046631Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T00:21:39.9110431Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config	remote.origin.url
2024-12-18T00:21:39.9131903Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T00:21:39.9194005Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config	remote.origin.url
2024-12-18T00:21:39.9219193Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T00:21:39.9280703Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2024-12-18T00:21:39.9301516Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T00:21:39.9362750Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:39.9385798Z Entering 'third_party/mimalloc'
2024-12-18T00:21:39.9449220Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config	remote.origin.url
2024-12-18T00:21:39.9470443Z Entering 'third_party/nccl/nccl'
2024-12-18T00:21:39.9532266Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2024-12-18T00:21:39.9553923Z Entering 'third_party/nlohmann'
2024-12-18T00:21:39.9615734Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2024-12-18T00:21:39.9638816Z Entering 'third_party/onnx'
2024-12-18T00:21:39.9701456Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2024-12-18T00:21:39.9737961Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T00:21:39.9799890Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2024-12-18T00:21:39.9828664Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T00:21:39.9893364Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config	remote.origin.url
2024-12-18T00:21:39.9915806Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T00:21:39.9981240Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config	remote.origin.url
2024-12-18T00:21:40.0002537Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T00:21:40.0063449Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:40.0085364Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T00:21:40.0147115Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config	remote.origin.url
2024-12-18T00:21:40.0168212Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T00:21:40.0230458Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config	remote.origin.url
2024-12-18T00:21:40.0253594Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T00:21:40.0319094Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config	remote.origin.url
2024-12-18T00:21:40.0339841Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T00:21:40.0401119Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config	remote.origin.url
2024-12-18T00:21:40.0426971Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T00:21:40.0487737Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config	remote.origin.url
2024-12-18T00:21:40.0506947Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T00:21:40.0570176Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2024-12-18T00:21:40.0592703Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T00:21:40.0655693Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2024-12-18T00:21:40.0679829Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T00:21:40.0742054Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config	remote.origin.url
2024-12-18T00:21:40.0783906Z Entering 'third_party/pocketfft'
2024-12-18T00:21:40.0848662Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2024-12-18T00:21:40.0870419Z Entering 'third_party/protobuf'
2024-12-18T00:21:40.0933911Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2024-12-18T00:21:40.0957497Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T00:21:40.1018179Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2024-12-18T00:21:40.1039065Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T00:21:40.1101053Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:40.1126839Z Entering 'third_party/psimd'
2024-12-18T00:21:40.1192738Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2024-12-18T00:21:40.1214530Z Entering 'third_party/pthreadpool'
2024-12-18T00:21:40.1277115Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2024-12-18T00:21:40.1299290Z Entering 'third_party/pybind11'
2024-12-18T00:21:40.1362125Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2024-12-18T00:21:40.1383795Z Entering 'third_party/python-peachpy'
2024-12-18T00:21:40.1448096Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2024-12-18T00:21:40.1469449Z Entering 'third_party/sleef'
2024-12-18T00:21:40.1532478Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2024-12-18T00:21:40.1554735Z Entering 'third_party/tensorpipe'
2024-12-18T00:21:40.1618328Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2024-12-18T00:21:40.1639056Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T00:21:40.1700415Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:40.1721770Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T00:21:40.1786499Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2024-12-18T00:21:40.1807409Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T00:21:40.1871294Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2024-12-18T00:21:40.1892214Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T00:21:40.1953007Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2024-12-18T00:21:40.1971805Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T00:21:40.2034744Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2024-12-18T00:21:40.2782520Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2024-12-18T00:21:40.3143139Z Entering 'android/libs/fbjni'
2024-12-18T00:21:40.3199523Z Entering 'third_party/FP16'
2024-12-18T00:21:40.3252779Z Entering 'third_party/FXdiv'
2024-12-18T00:21:40.3304699Z Entering 'third_party/NNPACK'
2024-12-18T00:21:40.3357018Z Entering 'third_party/NVTX'
2024-12-18T00:21:40.3409739Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T00:21:40.3462788Z Entering 'third_party/XNNPACK'
2024-12-18T00:21:40.3529386Z Entering 'third_party/benchmark'
2024-12-18T00:21:40.3581411Z Entering 'third_party/composable_kernel'
2024-12-18T00:21:40.3642258Z Entering 'third_party/cpp-httplib'
2024-12-18T00:21:40.3693811Z Entering 'third_party/cpuinfo'
2024-12-18T00:21:40.3747551Z Entering 'third_party/cudnn_frontend'
2024-12-18T00:21:40.3800262Z Entering 'third_party/cutlass'
2024-12-18T00:21:40.3859964Z Entering 'third_party/eigen'
2024-12-18T00:21:40.3913887Z Entering 'third_party/fbgemm'
2024-12-18T00:21:40.3965562Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T00:21:40.4019318Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T00:21:40.4069206Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T00:21:40.4131899Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T00:21:40.4183325Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T00:21:40.4236664Z Entering 'third_party/flatbuffers'
2024-12-18T00:21:40.4291185Z Entering 'third_party/fmt'
2024-12-18T00:21:40.4343664Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T00:21:40.4395646Z Entering 'third_party/gloo'
2024-12-18T00:21:40.4448083Z Entering 'third_party/googletest'
2024-12-18T00:21:40.4499930Z Entering 'third_party/ideep'
2024-12-18T00:21:40.4551960Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T00:21:40.4611533Z Entering 'third_party/ittapi'
2024-12-18T00:21:40.4663237Z Entering 'third_party/kineto'
2024-12-18T00:21:40.4714376Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T00:21:40.4763136Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T00:21:40.4815453Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T00:21:40.4865786Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T00:21:40.4916512Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T00:21:40.4965299Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T00:21:40.5022375Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T00:21:40.5072315Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T00:21:40.5123396Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T00:21:40.5174288Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T00:21:40.5228105Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T00:21:40.5278022Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T00:21:40.5335728Z Entering 'third_party/mimalloc'
2024-12-18T00:21:40.5388189Z Entering 'third_party/nccl/nccl'
2024-12-18T00:21:40.5440887Z Entering 'third_party/nlohmann'
2024-12-18T00:21:40.5493955Z Entering 'third_party/onnx'
2024-12-18T00:21:40.5560740Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T00:21:40.5617145Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T00:21:40.5670874Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T00:21:40.5728377Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T00:21:40.5778153Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T00:21:40.5830027Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T00:21:40.5882191Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T00:21:40.5931850Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T00:21:40.5986558Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T00:21:40.6035964Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T00:21:40.6089455Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T00:21:40.6143136Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T00:21:40.6216044Z Entering 'third_party/pocketfft'
2024-12-18T00:21:40.6267734Z Entering 'third_party/protobuf'
2024-12-18T00:21:40.6322412Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T00:21:40.6372407Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T00:21:40.6427298Z Entering 'third_party/psimd'
2024-12-18T00:21:40.6479169Z Entering 'third_party/pthreadpool'
2024-12-18T00:21:40.6531647Z Entering 'third_party/pybind11'
2024-12-18T00:21:40.6583820Z Entering 'third_party/python-peachpy'
2024-12-18T00:21:40.6636561Z Entering 'third_party/sleef'
2024-12-18T00:21:40.6688729Z Entering 'third_party/tensorpipe'
2024-12-18T00:21:40.6744533Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T00:21:40.6795258Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T00:21:40.6845105Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T00:21:40.6895255Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T00:21:40.6946206Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T00:21:40.7023070Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2024-12-18T00:21:40.7382458Z Entering 'android/libs/fbjni'
2024-12-18T00:21:40.7438012Z Entering 'third_party/FP16'
2024-12-18T00:21:40.7489921Z Entering 'third_party/FXdiv'
2024-12-18T00:21:40.7541959Z Entering 'third_party/NNPACK'
2024-12-18T00:21:40.7593672Z Entering 'third_party/NVTX'
2024-12-18T00:21:40.7650419Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T00:21:40.7700450Z Entering 'third_party/XNNPACK'
2024-12-18T00:21:40.7765916Z Entering 'third_party/benchmark'
2024-12-18T00:21:40.7816550Z Entering 'third_party/composable_kernel'
2024-12-18T00:21:40.7873961Z Entering 'third_party/cpp-httplib'
2024-12-18T00:21:40.7923401Z Entering 'third_party/cpuinfo'
2024-12-18T00:21:40.7974247Z Entering 'third_party/cudnn_frontend'
2024-12-18T00:21:40.8025288Z Entering 'third_party/cutlass'
2024-12-18T00:21:40.8083741Z Entering 'third_party/eigen'
2024-12-18T00:21:40.8135881Z Entering 'third_party/fbgemm'
2024-12-18T00:21:40.8185780Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T00:21:40.8240532Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T00:21:40.8290747Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T00:21:40.8350517Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T00:21:40.8399713Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T00:21:40.8453810Z Entering 'third_party/flatbuffers'
2024-12-18T00:21:40.8510266Z Entering 'third_party/fmt'
2024-12-18T00:21:40.8561790Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T00:21:40.8615166Z Entering 'third_party/gloo'
2024-12-18T00:21:40.8667352Z Entering 'third_party/googletest'
2024-12-18T00:21:40.8720800Z Entering 'third_party/ideep'
2024-12-18T00:21:40.8769739Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T00:21:40.8829867Z Entering 'third_party/ittapi'
2024-12-18T00:21:40.8880910Z Entering 'third_party/kineto'
2024-12-18T00:21:40.8931652Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T00:21:40.8980933Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T00:21:40.9036265Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T00:21:40.9088263Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T00:21:40.9139809Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T00:21:40.9187848Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T00:21:40.9246105Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T00:21:40.9326015Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T00:21:40.9351721Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T00:21:40.9404891Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T00:21:40.9460122Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T00:21:40.9510539Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T00:21:40.9566554Z Entering 'third_party/mimalloc'
2024-12-18T00:21:40.9618851Z Entering 'third_party/nccl/nccl'
2024-12-18T00:21:40.9670469Z Entering 'third_party/nlohmann'
2024-12-18T00:21:40.9723389Z Entering 'third_party/onnx'
2024-12-18T00:21:40.9791274Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T00:21:40.9848526Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T00:21:40.9900224Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T00:21:40.9950525Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T00:21:41.0005567Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T00:21:41.0055019Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T00:21:41.0106072Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T00:21:41.0155839Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T00:21:41.0206187Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T00:21:41.0254068Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T00:21:41.0309067Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T00:21:41.0362492Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T00:21:41.0443754Z Entering 'third_party/pocketfft'
2024-12-18T00:21:41.0496141Z Entering 'third_party/protobuf'
2024-12-18T00:21:41.0550773Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T00:21:41.0600442Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T00:21:41.0657506Z Entering 'third_party/psimd'
2024-12-18T00:21:41.0710101Z Entering 'third_party/pthreadpool'
2024-12-18T00:21:41.0761810Z Entering 'third_party/pybind11'
2024-12-18T00:21:41.0814516Z Entering 'third_party/python-peachpy'
2024-12-18T00:21:41.0866332Z Entering 'third_party/sleef'
2024-12-18T00:21:41.0919149Z Entering 'third_party/tensorpipe'
2024-12-18T00:21:41.0969049Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T00:21:41.1019363Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T00:21:41.1069140Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T00:21:41.1119245Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T00:21:41.1168414Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T00:21:41.1241913Z ##[endgroup]
2024-12-18T00:21:41.1278845Z [command]/usr/bin/git log -1 --format='%H'
2024-12-18T00:21:41.1302797Z '0cdf8b1d09254cfda66191d1bd01e3041c3c76f7'
2024-12-18T00:21:41.1476923Z Prepare all required actions
2024-12-18T00:21:41.1477381Z Getting action download info
2024-12-18T00:21:41.3000439Z ##[group]Run ./.github/actions/setup-linux
2024-12-18T00:21:41.3000761Z env:
2024-12-18T00:21:41.3000978Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:41.3001244Z ##[endgroup]
2024-12-18T00:21:41.3091218Z ##[group]Run set -euo pipefail
2024-12-18T00:21:41.3091561Z set -euo pipefail
2024-12-18T00:21:41.3091856Z function get_ec2_metadata() {
2024-12-18T00:21:41.3092234Z   # Pulled from instance metadata endpoint for EC2
2024-12-18T00:21:41.3092854Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2024-12-18T00:21:41.3093424Z   category=$1
2024-12-18T00:21:41.3093785Z   # If it is GCP runner (runner name contains gcp), do not run this
2024-12-18T00:21:41.3094217Z   runner_name_str=i-019aa8c64fc4447ce
2024-12-18T00:21:41.3094579Z   if [[ -f /.inarc ]]; then
2024-12-18T00:21:41.3094928Z     echo "ARC Runner, no info on ec2 metadata"
2024-12-18T00:21:41.3095310Z   elif [[ $runner_name_str == *"gcp"* ]]; then
2024-12-18T00:21:41.3095775Z     echo "Runner is from Google Cloud Platform, No info on ec2 metadata"
2024-12-18T00:21:41.3096198Z   else
2024-12-18T00:21:41.3097036Z     curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2024-12-18T00:21:41.3097931Z   fi
2024-12-18T00:21:41.3098154Z }
2024-12-18T00:21:41.3098423Z echo "ami-id: $(get_ec2_metadata ami-id)"
2024-12-18T00:21:41.3098851Z echo "instance-id: $(get_ec2_metadata instance-id)"
2024-12-18T00:21:41.3099321Z echo "instance-type: $(get_ec2_metadata instance-type)"
2024-12-18T00:21:41.3099731Z echo "system info $(uname -a)"
2024-12-18T00:21:41.3110510Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:41.3110883Z env:
2024-12-18T00:21:41.3111099Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:41.3111363Z ##[endgroup]
2024-12-18T00:21:41.3271007Z ami-id: ami-0fff1b9a61dec8a5f
2024-12-18T00:21:41.3385822Z instance-id: i-019aa8c64fc4447ce
2024-12-18T00:21:41.3496575Z instance-type: g5.4xlarge
2024-12-18T00:21:41.3509876Z system info Linux ip-10-0-57-4.ec2.internal 6.1.109-118.189.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 10 08:59:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
2024-12-18T00:21:41.3545312Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T00:21:41.3546220Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T00:21:41.3556028Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:41.3556408Z env:
2024-12-18T00:21:41.3556618Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:41.3556886Z ##[endgroup]
2024-12-18T00:21:41.3620738Z ##[group]Run if systemctl is-active --quiet docker; then
2024-12-18T00:21:41.3621172Z if systemctl is-active --quiet docker; then
2024-12-18T00:21:41.3621555Z     echo "Docker daemon is running...";
2024-12-18T00:21:41.3621895Z else
2024-12-18T00:21:41.3622277Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2024-12-18T00:21:41.3622692Z fi
2024-12-18T00:21:41.3630461Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:41.3631005Z env:
2024-12-18T00:21:41.3631229Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:41.3631490Z ##[endgroup]
2024-12-18T00:21:41.3715210Z Docker daemon is running...
2024-12-18T00:21:41.3764405Z ##[group]Run nick-fields/retry@v3.0.0
2024-12-18T00:21:41.3764715Z with:
2024-12-18T00:21:41.3764926Z   shell: bash
2024-12-18T00:21:41.3765364Z   timeout_minutes: 5
2024-12-18T00:21:41.3765621Z   max_attempts: 3
2024-12-18T00:21:41.3765866Z   retry_wait_seconds: 30
2024-12-18T00:21:41.3768037Z   command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \
    --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"

# For LF Runners we need to make sure we also login to Meta's ECR docker registry too.
META_AWS_ACCOUNT_ID=308535385114
if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then
    aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \
        --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
fi

2024-12-18T00:21:41.3770210Z   polling_interval_seconds: 1
2024-12-18T00:21:41.3770502Z   warning_on_retry: true
2024-12-18T00:21:41.3770774Z   continue_on_error: false
2024-12-18T00:21:41.3771039Z env:
2024-12-18T00:21:41.3771266Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:41.3771540Z   AWS_RETRY_MODE: standard
2024-12-18T00:21:41.3771811Z   AWS_MAX_ATTEMPTS: 5
2024-12-18T00:21:41.3772079Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T00:21:41.3772360Z ##[endgroup]
2024-12-18T00:21:42.6416715Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-12-18T00:21:42.6417621Z Configure a credential helper to remove this warning. See
2024-12-18T00:21:42.6418290Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-12-18T00:21:42.6418821Z 
2024-12-18T00:21:42.6418965Z Login Succeeded
2024-12-18T00:21:43.5202389Z Command completed after 1 attempt(s).
2024-12-18T00:21:43.5292919Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2024-12-18T00:21:43.5293701Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2024-12-18T00:21:43.5294256Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2024-12-18T00:21:43.5304609Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:43.5305082Z env:
2024-12-18T00:21:43.5305470Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:43.5305839Z ##[endgroup]
2024-12-18T00:21:43.5413642Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2024-12-18T00:21:43.5414348Z # ignore expansion of "docker ps -q" since it could be empty
2024-12-18T00:21:43.5414939Z # shellcheck disable=SC2046
2024-12-18T00:21:43.5415361Z docker stop $(docker ps -q) || true
2024-12-18T00:21:43.5415918Z # Prune all of the docker images
2024-12-18T00:21:43.5416431Z docker system prune -af
2024-12-18T00:21:43.5425805Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:43.5426426Z env:
2024-12-18T00:21:43.5426741Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:43.5427050Z ##[endgroup]
2024-12-18T00:21:43.5742691Z "docker stop" requires at least 1 argument.
2024-12-18T00:21:43.5743312Z See 'docker stop --help'.
2024-12-18T00:21:43.5743863Z 
2024-12-18T00:21:43.5744095Z Usage:  docker stop [OPTIONS] CONTAINER [CONTAINER...]
2024-12-18T00:21:43.5744392Z 
2024-12-18T00:21:43.5744574Z Stop one or more running containers
2024-12-18T00:21:48.5350174Z Deleted Images:
2024-12-18T00:21:48.5351048Z untagged: pytorch/manylinux2_28-builder:cuda12.6
2024-12-18T00:21:48.5351830Z untagged: pytorch/manylinux2_28-builder@sha256:d0fd3ebcdeb3a90e3635dd3179d04a887d02bd01e1d7684a040a9df5293cc371
2024-12-18T00:21:48.5352716Z deleted: sha256:ae03473ad9e35cccdc4c3892b71aef483bd85dd00103cf7f77ac37a345881dd6
2024-12-18T00:21:48.5353542Z deleted: sha256:9719fce6efa36cffadba5afe4eed723bdd05a28462c1377d351969f63638ebdf
2024-12-18T00:21:48.5354622Z deleted: sha256:dca2ca99e9ba36cec75217ec989c57aea010b7eaa57337a2d7d47dbb52be1948
2024-12-18T00:21:48.5355295Z deleted: sha256:c01b4b82da58b717943604e597010b14119892082f6213bd035de5911ec51d8f
2024-12-18T00:21:48.5356068Z deleted: sha256:59d2d3a5e4713148f1ee690a251029d2f359c8e7e167989a7679788a172d0c9d
2024-12-18T00:21:48.5356994Z deleted: sha256:5834644c6360df0dd9c9c15b7068e92e4f0ae4b86a67e9a6927ca28a4ccc976b
2024-12-18T00:21:48.5357688Z deleted: sha256:d7ee68fa44dfaef4207f39e9ec534b45de3995233231d8a34a48e9c27c144848
2024-12-18T00:21:48.5358541Z deleted: sha256:7c8ede63d0a4159d1b087df4a0bc8891af74f161f3ba0786fb365f7d344fde79
2024-12-18T00:21:48.5359254Z deleted: sha256:cc1092c9702f7d34b0bca7adc1b55abc184138f0d29822b27904250f84b91161
2024-12-18T00:21:48.5360231Z deleted: sha256:593907d67ac50b541d5f1445ccfdb59ad8b8116d4574517856370bb110e8249b
2024-12-18T00:21:48.5360927Z deleted: sha256:a5bced412fcd85eaa46c84f788a00efe03d86dabef6e9a102b4c5730ad56ff1a
2024-12-18T00:21:48.5361658Z deleted: sha256:27abeabceb607d47df1ec821f0e613e48698b8fa503ce86da72f4e74273e58b8
2024-12-18T00:21:48.5362460Z deleted: sha256:33e68e513da6f8b72bcdb153851f56929a21a9b6c6fbff2b85269688378c59ef
2024-12-18T00:21:48.5363187Z deleted: sha256:a9feb3cee33397c6199940667fc479a4ceba62e11b51c50d3af007a9c69dc859
2024-12-18T00:21:48.5363898Z deleted: sha256:c234b5fbc198234697407da6022bfe0101a4bbf6095c9150ccfa300c1b2cfd7b
2024-12-18T00:21:48.5364682Z deleted: sha256:3330d3ecc4709a009947480281727905fa702001e222a487ffbeee85be166e8a
2024-12-18T00:21:48.5365340Z deleted: sha256:b73a2173901932f2a2a54038a6b21a8c923bc1b82960fcab4983cff336d8ea9a
2024-12-18T00:21:48.5366061Z deleted: sha256:9448314da58e52dbc4d80c57e99919db44671202a15ea110b241c1db96416352
2024-12-18T00:21:48.5366958Z deleted: sha256:f8da575bedd0ab262db7aaac87db81ec0c374142fdc9149b0e0892a64568d967
2024-12-18T00:21:48.5367679Z deleted: sha256:2d70bd566ff6b70a523f7b785b71956721a5eff4a19dc8a0864aa84835a12b70
2024-12-18T00:21:48.5381362Z deleted: sha256:301a3ad675e5a693bddbf34dcffe7944856417be9e209d3b32ede42b41912cd5
2024-12-18T00:21:48.5382268Z deleted: sha256:5af47657331e321b120e1891fba4f340fc6371578abcce7d7688a622296e9395
2024-12-18T00:21:48.5383133Z deleted: sha256:d16f0c5acd523fff65eb6a4e8024c3edc4d4aab901b9b853122950407e10a668
2024-12-18T00:21:48.5383809Z deleted: sha256:f84f38e026bc650aaecbf5e03a929ef0238eb27a2dc393d8cf6d294069ba2ba3
2024-12-18T00:21:48.5384441Z deleted: sha256:20ee01833daa6a5c034d94fb30218452b3bc72b01deb4272de5d407af77fa4e9
2024-12-18T00:21:48.5385061Z deleted: sha256:8e59532bfe3e39590a3e9107bd9c6de176005aa0815f74a8dcaea4534939e1c2
2024-12-18T00:21:48.5385665Z deleted: sha256:42468fe632a54034dadfc4945917c9782d5778c4461523934a257e8265ce3866
2024-12-18T00:21:48.5386276Z deleted: sha256:9537758bff3cc9dfbed92d840551240e461ded49884d6a2ca278795834c626d3
2024-12-18T00:21:48.5386886Z deleted: sha256:a5d3eecefb86d1288d1094060fe133cd2ad1b68a11d4b08161e6b1a009390edf
2024-12-18T00:21:48.5387260Z 
2024-12-18T00:21:48.5387387Z Total reclaimed space: 11.49GB
2024-12-18T00:21:48.5433753Z ##[group]Run set +e
2024-12-18T00:21:48.5434053Z set +e
2024-12-18T00:21:48.5434295Z set -x
2024-12-18T00:21:48.5434574Z 
2024-12-18T00:21:48.5434829Z PT_DOMAIN=download.pytorch.org
2024-12-18T00:21:48.5435421Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400,
2024-12-18T00:21:48.5436165Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last
2024-12-18T00:21:48.5436689Z # one is returned at random
2024-12-18T00:21:48.5437100Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1)
2024-12-18T00:21:48.5437477Z 
2024-12-18T00:21:48.5437714Z if [ -z "${RESOLVED_IP}" ]; then
2024-12-18T00:21:48.5438147Z   echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..."
2024-12-18T00:21:48.5438664Z   RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1)
2024-12-18T00:21:48.5439056Z 
2024-12-18T00:21:48.5439621Z   if [ -z "${RESOLVED_IP}" ]; then
2024-12-18T00:21:48.5440009Z     echo "Couldn't resolve ${PT_DOMAIN}, exiting..."
2024-12-18T00:21:48.5440377Z     exit 1
2024-12-18T00:21:48.5440624Z   fi
2024-12-18T00:21:48.5440849Z fi
2024-12-18T00:21:48.5441237Z 
2024-12-18T00:21:48.5441506Z if grep -r "${PT_DOMAIN}" /etc/hosts; then
2024-12-18T00:21:48.5441886Z   # Clean up any old records first
2024-12-18T00:21:48.5442263Z   sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts
2024-12-18T00:21:48.5442595Z fi
2024-12-18T00:21:48.5442814Z 
2024-12-18T00:21:48.5443137Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts
2024-12-18T00:21:48.5443547Z cat /etc/hosts
2024-12-18T00:21:48.5453674Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:48.5454051Z env:
2024-12-18T00:21:48.5454278Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:48.5454560Z ##[endgroup]
2024-12-18T00:21:48.5483760Z + PT_DOMAIN=download.pytorch.org
2024-12-18T00:21:48.5490012Z ++ dig -4 +short download.pytorch.org
2024-12-18T00:21:48.5490418Z ++ tail -n1
2024-12-18T00:21:48.5866926Z + RESOLVED_IP=18.160.18.20
2024-12-18T00:21:48.5867364Z + '[' -z 18.160.18.20 ']'
2024-12-18T00:21:48.5867714Z + grep -r download.pytorch.org /etc/hosts
2024-12-18T00:21:48.5882206Z 3.167.99.125 download.pytorch.org
2024-12-18T00:21:48.5884481Z + sudo sed -i /download.pytorch.org/d /etc/hosts
2024-12-18T00:21:48.7007796Z + echo '18.160.18.20 download.pytorch.org'
2024-12-18T00:21:48.7008174Z + sudo tee -a /etc/hosts
2024-12-18T00:21:48.7341310Z 18.160.18.20 download.pytorch.org
2024-12-18T00:21:48.7363093Z + cat /etc/hosts
2024-12-18T00:21:48.7373954Z 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
2024-12-18T00:21:48.7379597Z ::1         localhost6 localhost6.localdomain6
2024-12-18T00:21:48.7380081Z 18.160.18.20 download.pytorch.org
2024-12-18T00:21:48.7535315Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@release/2.6
2024-12-18T00:21:48.7535810Z with:
2024-12-18T00:21:48.7536623Z   docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:48.7537544Z   docker-build-dir: .ci/docker
2024-12-18T00:21:48.7537837Z   working-directory: .
2024-12-18T00:21:48.7538196Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:48.7538593Z   force-push: false
2024-12-18T00:21:48.7538823Z env:
2024-12-18T00:21:48.7539043Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:48.7539308Z ##[endgroup]
2024-12-18T00:21:48.7568349Z ##[group]Run set -ex
2024-12-18T00:21:48.7568635Z set -ex
2024-12-18T00:21:48.7568862Z 
2024-12-18T00:21:48.7569247Z # If the docker build directory or the build script doesn't exist, the action will
2024-12-18T00:21:48.7569918Z # gracefully return the docker image name as it is.  Pulling docker image in Linux
2024-12-18T00:21:48.7570474Z # job could then download the pre-built image as usual
2024-12-18T00:21:48.7570972Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then
2024-12-18T00:21:48.7571431Z   echo "skip=true" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:48.7571867Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:48.7572264Z 
2024-12-18T00:21:48.7572611Z   echo "There is no Docker build script in ${REPO_NAME} repo, skipping..."
2024-12-18T00:21:48.7573034Z   exit 0
2024-12-18T00:21:48.7573257Z else
2024-12-18T00:21:48.7573524Z   echo "skip=false" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:48.7573845Z fi
2024-12-18T00:21:48.7574061Z 
2024-12-18T00:21:48.7574396Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then
2024-12-18T00:21:48.7575153Z   # The docker image name already includes the ECR prefix and tag, so we can just
2024-12-18T00:21:48.7575671Z   # use it as it is, but first let's extract the tag
2024-12-18T00:21:48.7576140Z   DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}')
2024-12-18T00:21:48.7576636Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:48.7577115Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:48.7577515Z else
2024-12-18T00:21:48.7577822Z   DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}")
2024-12-18T00:21:48.7578275Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:48.7578895Z   echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:48.7579436Z fi
2024-12-18T00:21:48.7589381Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:48.7589757Z env:
2024-12-18T00:21:48.7589968Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:48.7590239Z   REPO_NAME: pytorch
2024-12-18T00:21:48.7591068Z   DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:48.7591959Z   DOCKER_BUILD_DIR: .ci/docker
2024-12-18T00:21:48.7592331Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:48.7592713Z ##[endgroup]
2024-12-18T00:21:48.7623975Z + [[ ! -d .ci/docker ]]
2024-12-18T00:21:48.7624253Z + [[ ! -f .ci/docker/build.sh ]]
2024-12-18T00:21:48.7624542Z + echo skip=false
2024-12-18T00:21:48.7625797Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]]
2024-12-18T00:21:48.7635530Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:48.7636409Z ++ awk -F '[:,]' '{print $2}'
2024-12-18T00:21:48.7661629Z + DOCKER_TAG=45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:48.7662149Z + echo docker-tag=45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:48.7663098Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:48.7700629Z ##[group]Run set +e
2024-12-18T00:21:48.7700962Z set +e
2024-12-18T00:21:48.7701196Z set -x
2024-12-18T00:21:48.7701422Z 
2024-12-18T00:21:48.7701631Z login() {
2024-12-18T00:21:48.7702130Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2024-12-18T00:21:48.7702665Z }
2024-12-18T00:21:48.7702880Z 
2024-12-18T00:21:48.7703092Z retry () {
2024-12-18T00:21:48.7703682Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2024-12-18T00:21:48.7704004Z }
2024-12-18T00:21:48.7704215Z 
2024-12-18T00:21:48.7704455Z retry login "${DOCKER_REGISTRY}"
2024-12-18T00:21:48.7704762Z 
2024-12-18T00:21:48.7704980Z START_TIME=$(date +%s)
2024-12-18T00:21:48.7705275Z # Wait up to 90 minutes
2024-12-18T00:21:48.7705636Z while [[ $(( $(date +%s) - 5400 )) -lt $START_TIME ]]; do
2024-12-18T00:21:48.7706117Z   # Check if image already exists, if it does then skip building it
2024-12-18T00:21:48.7706599Z   if docker manifest inspect "${DOCKER_IMAGE}"; then
2024-12-18T00:21:48.7706957Z     exit 0
2024-12-18T00:21:48.7707188Z   fi
2024-12-18T00:21:48.7707404Z 
2024-12-18T00:21:48.7707791Z   # NB: This flag is used by Docker build workflow to push the image to ECR, so we can
2024-12-18T00:21:48.7708641Z   # use this to differentiate between the Docker build and regular build jobs. For the
2024-12-18T00:21:48.7709288Z   # latter, it will wait for the Docker images to become available before continuing
2024-12-18T00:21:48.7709801Z   if [ "${DOCKER_PUSH:-false}" == "true" ]; then
2024-12-18T00:21:48.7710194Z     # It's a Docker build job, let's build the image
2024-12-18T00:21:48.7710536Z     break
2024-12-18T00:21:48.7710773Z   else
2024-12-18T00:21:48.7711117Z     # It's a regular build job, wait for the image to become available
2024-12-18T00:21:48.7711526Z     sleep 300
2024-12-18T00:21:48.7711768Z   fi
2024-12-18T00:21:48.7711992Z done
2024-12-18T00:21:48.7712214Z 
2024-12-18T00:21:48.7712574Z # NB: This part requires a full checkout. Otherwise, the merge base will
2024-12-18T00:21:48.7713149Z # be empty.  The default action would be to continue rebuild the image
2024-12-18T00:21:48.7713652Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then
2024-12-18T00:21:48.7714114Z   # if we're on the base branch then use the parent commit
2024-12-18T00:21:48.7714515Z   MERGE_BASE=$(git rev-parse HEAD~)
2024-12-18T00:21:48.7714832Z else
2024-12-18T00:21:48.7715160Z   # otherwise we're on a PR, so use the most recent base commit
2024-12-18T00:21:48.7715632Z   MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION")
2024-12-18T00:21:48.7715987Z fi
2024-12-18T00:21:48.7716203Z 
2024-12-18T00:21:48.7716442Z if [[ -z "${MERGE_BASE}" ]]; then
2024-12-18T00:21:48.7716805Z   echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:48.7717135Z 
2024-12-18T00:21:48.7717760Z   echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..."
2024-12-18T00:21:48.7718309Z   exit 0
2024-12-18T00:21:48.7718535Z fi
2024-12-18T00:21:48.7718747Z 
2024-12-18T00:21:48.7719059Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then
2024-12-18T00:21:48.7719889Z   echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit"
2024-12-18T00:21:48.7720448Z   exit 1
2024-12-18T00:21:48.7720672Z fi
2024-12-18T00:21:48.7720883Z 
2024-12-18T00:21:48.7721241Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}")
2024-12-18T00:21:48.7721882Z # If no image exists but the hash is the same as the previous hash then we should error out here
2024-12-18T00:21:48.7722451Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then
2024-12-18T00:21:48.7723121Z   echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch"
2024-12-18T00:21:48.7723866Z   echo "         Will re-build docker image to store in local cache, TTS may be longer"
2024-12-18T00:21:48.7724311Z fi
2024-12-18T00:21:48.7724524Z 
2024-12-18T00:21:48.7724786Z echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:48.7733687Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:48.7734055Z env:
2024-12-18T00:21:48.7734278Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:48.7734559Z   DOCKER_BUILD_DIR: .ci/docker
2024-12-18T00:21:48.7734909Z   BASE_REVISION: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:21:48.7735842Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:48.7736749Z   DOCKER_TAG: 45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:48.7737180Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:48.7737651Z   DOCKER_PUSH: 
2024-12-18T00:21:48.7737886Z ##[endgroup]
2024-12-18T00:21:48.7766015Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:48.7766460Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:48.7769324Z + aws ecr get-login-password --region us-east-1
2024-12-18T00:21:48.7770318Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:49.3381366Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-12-18T00:21:49.3381961Z Configure a credential helper to remove this warning. See
2024-12-18T00:21:49.3382521Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-12-18T00:21:49.3382912Z 
2024-12-18T00:21:49.3383775Z Login Succeeded
2024-12-18T00:21:49.3410719Z ++ date +%s
2024-12-18T00:21:49.3423091Z + START_TIME=1734481309
2024-12-18T00:21:49.3427894Z ++ date +%s
2024-12-18T00:21:49.3439797Z + [[ 1734475909 -lt 1734481309 ]]
2024-12-18T00:21:49.3440733Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:49.5899219Z {
2024-12-18T00:21:49.5899512Z 	"schemaVersion": 2,
2024-12-18T00:21:49.5900013Z 	"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
2024-12-18T00:21:49.5900525Z 	"config": {
2024-12-18T00:21:49.5900855Z 		"mediaType": "application/vnd.docker.container.image.v1+json",
2024-12-18T00:21:49.5901257Z 		"size": 52680,
2024-12-18T00:21:49.5901664Z 		"digest": "sha256:67e93a8badaf799ceb664d7c453077c6c6785d56dc10c720be9358edeb91759b"
2024-12-18T00:21:49.5902126Z 	},
2024-12-18T00:21:49.5902323Z 	"layers": [
2024-12-18T00:21:49.5902525Z 		{
2024-12-18T00:21:49.5902854Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5903568Z 			"size": 28584506,
2024-12-18T00:21:49.5904350Z 			"digest": "sha256:80888bc6716fcbb8874e75ac88898d3e38e6f1bc55678f0e97ca9d706b7f3733"
2024-12-18T00:21:49.5904826Z 		},
2024-12-18T00:21:49.5905010Z 		{
2024-12-18T00:21:49.5905334Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5905744Z 			"size": 7944698,
2024-12-18T00:21:49.5906156Z 			"digest": "sha256:fbcd35dc5bc3a7bda41926aadd083020f942b001ebac6f1d30480f0f065394c0"
2024-12-18T00:21:49.5906621Z 		},
2024-12-18T00:21:49.5906804Z 		{
2024-12-18T00:21:49.5907129Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5907534Z 			"size": 57593527,
2024-12-18T00:21:49.5907952Z 			"digest": "sha256:c7232af9ae05f7de83f8d6171bd0c35a4dd0a85ebafb15b950dbc08f89ea5fb5"
2024-12-18T00:21:49.5908416Z 		},
2024-12-18T00:21:49.5908599Z 		{
2024-12-18T00:21:49.5908922Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5909326Z 			"size": 185,
2024-12-18T00:21:49.5909741Z 			"digest": "sha256:db6cdef1932a0d9ca6ef9a539e08d491f66d1b1ed81926ae1525375bdd8100cc"
2024-12-18T00:21:49.5910205Z 		},
2024-12-18T00:21:49.5910387Z 		{
2024-12-18T00:21:49.5910708Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5911111Z 			"size": 6886,
2024-12-18T00:21:49.5911503Z 			"digest": "sha256:56dc8550293751a1604e97ac949cfae82ba20cb2a28e034737bafd7382559609"
2024-12-18T00:21:49.5911946Z 		},
2024-12-18T00:21:49.5912128Z 		{
2024-12-18T00:21:49.5912451Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5912867Z 			"size": 1374287670,
2024-12-18T00:21:49.5913279Z 			"digest": "sha256:30c0ea6140d07e2a8deb70d780f277c63cf61836ff33d66eef944728a4bef6bd"
2024-12-18T00:21:49.5913788Z 		},
2024-12-18T00:21:49.5913976Z 		{
2024-12-18T00:21:49.5914298Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5914704Z 			"size": 62641,
2024-12-18T00:21:49.5915108Z 			"digest": "sha256:71bdb1a72c2d6dc97bbdbca82383f0260c4ee87556701e8e606c08a6bb0f0da5"
2024-12-18T00:21:49.5915569Z 		},
2024-12-18T00:21:49.5915913Z 		{
2024-12-18T00:21:49.5916235Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5916702Z 			"size": 1685,
2024-12-18T00:21:49.5917109Z 			"digest": "sha256:4829486be7c30f19f4136fa56adbb3de206ed0bbf0705b59fb2147406778ce38"
2024-12-18T00:21:49.5917562Z 		},
2024-12-18T00:21:49.5917746Z 		{
2024-12-18T00:21:49.5918068Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5918478Z 			"size": 1523,
2024-12-18T00:21:49.5918877Z 			"digest": "sha256:1f4e68d7b5e4224ba1da78ef461ff7f01e8d59c09d39281277521384105a9441"
2024-12-18T00:21:49.5919330Z 		},
2024-12-18T00:21:49.5919634Z 		{
2024-12-18T00:21:49.5919958Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5920367Z 			"size": 2626741021,
2024-12-18T00:21:49.5920775Z 			"digest": "sha256:7c373e2d9b7e82a6878d4a31293dd857915a0fe47d07dce541cea03b043d57fc"
2024-12-18T00:21:49.5921221Z 		},
2024-12-18T00:21:49.5921411Z 		{
2024-12-18T00:21:49.5921739Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5922146Z 			"size": 86545,
2024-12-18T00:21:49.5922533Z 			"digest": "sha256:6223811417458a3c93b84ee3b65f8b08d9e2828b926f0aed863041610d7d95d4"
2024-12-18T00:21:49.5922969Z 		},
2024-12-18T00:21:49.5923154Z 		{
2024-12-18T00:21:49.5923481Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5923934Z 			"size": 1824,
2024-12-18T00:21:49.5924337Z 			"digest": "sha256:f003d065e31e5e8c0cea490c88a912dd331bddcbd2f5ff22ccc692a5343bcbeb"
2024-12-18T00:21:49.5924792Z 		},
2024-12-18T00:21:49.5924975Z 		{
2024-12-18T00:21:49.5925295Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5925755Z 			"size": 246420139,
2024-12-18T00:21:49.5926167Z 			"digest": "sha256:cd6ce71168153e485b99493ce07032bbd9fd5cefceb68ef33ce088244306a653"
2024-12-18T00:21:49.5926619Z 		},
2024-12-18T00:21:49.5926803Z 		{
2024-12-18T00:21:49.5927229Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5927645Z 			"size": 703,
2024-12-18T00:21:49.5928033Z 			"digest": "sha256:e843066a8f68042266cfe814f1729975619fb8f733f371be115e8b2fe81cbc9d"
2024-12-18T00:21:49.5928473Z 		},
2024-12-18T00:21:49.5928657Z 		{
2024-12-18T00:21:49.5928982Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5929451Z 			"size": 1256,
2024-12-18T00:21:49.5929913Z 			"digest": "sha256:bac50ca2930bc4b1afa8a2f1b4b0d1d499d63b7922b784af690c3b31ec0c2f70"
2024-12-18T00:21:49.5930383Z 		},
2024-12-18T00:21:49.5930575Z 		{
2024-12-18T00:21:49.5930895Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5931302Z 			"size": 484,
2024-12-18T00:21:49.5931708Z 			"digest": "sha256:57e52c6751f4d865ada4cae919859fe563fe5dcbb93ca82cb4746f98caccbd3a"
2024-12-18T00:21:49.5932170Z 		},
2024-12-18T00:21:49.5932363Z 		{
2024-12-18T00:21:49.5932688Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5933103Z 			"size": 91715265,
2024-12-18T00:21:49.5933513Z 			"digest": "sha256:a9c5c14d77c035c7501566b1247024dfd241bd6bc12fbe2bca28ec7d90b26069"
2024-12-18T00:21:49.5933968Z 		},
2024-12-18T00:21:49.5934163Z 		{
2024-12-18T00:21:49.5934481Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5934889Z 			"size": 3706,
2024-12-18T00:21:49.5935293Z 			"digest": "sha256:b86099b8619ad88e8fe6c8e693180d52e0f2fbeac5927f0b403c8a9ef76d7f6b"
2024-12-18T00:21:49.5935749Z 		},
2024-12-18T00:21:49.5935941Z 		{
2024-12-18T00:21:49.5936257Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5936664Z 			"size": 1860,
2024-12-18T00:21:49.5937072Z 			"digest": "sha256:3c8ea8a600fff6cce4c1b36acc5f3949ecd664e55b5eaf4a50fe5499351574a6"
2024-12-18T00:21:49.5937536Z 		},
2024-12-18T00:21:49.5937731Z 		{
2024-12-18T00:21:49.5938048Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5938573Z 			"size": 701,
2024-12-18T00:21:49.5938986Z 			"digest": "sha256:bebd539a1d598fe86afc0da77bf16ae4c7ebfb9f031f21ed7634150fbe23f996"
2024-12-18T00:21:49.5939451Z 		},
2024-12-18T00:21:49.5939642Z 		{
2024-12-18T00:21:49.5939957Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5940365Z 			"size": 478,
2024-12-18T00:21:49.5940753Z 			"digest": "sha256:292b18ec2dab9bd60d00192387c1269d39b0b4886b782585d2c879808218fc66"
2024-12-18T00:21:49.5941198Z 		},
2024-12-18T00:21:49.5941387Z 		{
2024-12-18T00:21:49.5941712Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5942121Z 			"size": 2941869535,
2024-12-18T00:21:49.5942530Z 			"digest": "sha256:76e3817b4b4cec0a5b005161867dc257681b792271ba9d23c9b79b3a6801dcd9"
2024-12-18T00:21:49.5942976Z 		},
2024-12-18T00:21:49.5943159Z 		{
2024-12-18T00:21:49.5943481Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5943895Z 			"size": 380,
2024-12-18T00:21:49.5944296Z 			"digest": "sha256:50b1f245e9f5be8670d8ba5790e3b96450eee3428dcac7060367db48ae00234e"
2024-12-18T00:21:49.5944750Z 		},
2024-12-18T00:21:49.5944935Z 		{
2024-12-18T00:21:49.5945259Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5945667Z 			"size": 12901,
2024-12-18T00:21:49.5946075Z 			"digest": "sha256:ad93cf5205f24ada86b5f7eab5b63ed97ae7858368f6fdbce373343d2568d66e"
2024-12-18T00:21:49.5946537Z 		},
2024-12-18T00:21:49.5946721Z 		{
2024-12-18T00:21:49.5947047Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5947455Z 			"size": 863,
2024-12-18T00:21:49.5947856Z 			"digest": "sha256:8d17c59eaf7d84d49e3660380debcc70734b2a178e6cab86f0d4fd3e7a8100d1"
2024-12-18T00:21:49.5948312Z 		},
2024-12-18T00:21:49.5948497Z 		{
2024-12-18T00:21:49.5948824Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5949232Z 			"size": 106,
2024-12-18T00:21:49.5949742Z 			"digest": "sha256:b36a8ccf5c8c8ef0bf3e168ec9f568a0ada36c49103c9cacfef6dd072ddd681a"
2024-12-18T00:21:49.5950219Z 		},
2024-12-18T00:21:49.5950402Z 		{
2024-12-18T00:21:49.5950724Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5951132Z 			"size": 504,
2024-12-18T00:21:49.5951530Z 			"digest": "sha256:aa0a3b87e9e34ab15903206018bfd9fb1eb54fd21f05a61ef254557703890732"
2024-12-18T00:21:49.5951984Z 		},
2024-12-18T00:21:49.5952171Z 		{
2024-12-18T00:21:49.5952495Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5952909Z 			"size": 121477314,
2024-12-18T00:21:49.5953320Z 			"digest": "sha256:72a632052c5be4900c60ce09e32f8a0fd3b637ab0b7685dc5b07bda4776ddd15"
2024-12-18T00:21:49.5953772Z 		},
2024-12-18T00:21:49.5953954Z 		{
2024-12-18T00:21:49.5954275Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5954685Z 			"size": 109,
2024-12-18T00:21:49.5955097Z 			"digest": "sha256:ceed6316bc249ffffee9563cb7505023b3e70fba306f187bd612fece63508263"
2024-12-18T00:21:49.5955562Z 		},
2024-12-18T00:21:49.5955746Z 		{
2024-12-18T00:21:49.5956066Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5956471Z 			"size": 489,
2024-12-18T00:21:49.5956872Z 			"digest": "sha256:d0ca0f491f809a9f1dedd391f5b25cc3f59d8e95e7f76299f18ec3c931e56ed1"
2024-12-18T00:21:49.5957327Z 		},
2024-12-18T00:21:49.5957513Z 		{
2024-12-18T00:21:49.5957839Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5958244Z 			"size": 296,
2024-12-18T00:21:49.5958638Z 			"digest": "sha256:cf02c706cd567c781eeb567c90960801dc1ef78872b73fbc047df898951ecbb8"
2024-12-18T00:21:49.5959086Z 		},
2024-12-18T00:21:49.5959269Z 		{
2024-12-18T00:21:49.5959726Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5960133Z 			"size": 103,
2024-12-18T00:21:49.5960532Z 			"digest": "sha256:16073d3f107dab837d7edb5e6044449660fec7e2ded263ca81d5dcfea5beb486"
2024-12-18T00:21:49.5960995Z 		},
2024-12-18T00:21:49.5961272Z 		{
2024-12-18T00:21:49.5961593Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5962000Z 			"size": 1473,
2024-12-18T00:21:49.5962405Z 			"digest": "sha256:d234faf2f1bfb6cb9d4ba14103226d20d540e568b47690b67c2a2b2c5a49645f"
2024-12-18T00:21:49.5962863Z 		},
2024-12-18T00:21:49.5975042Z 		{
2024-12-18T00:21:49.5975417Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5975843Z 			"size": 430258758,
2024-12-18T00:21:49.5976272Z 			"digest": "sha256:3f0ade86af1a5bc72c9088735bb1a7ab9819fa20fdd96437fc080b2c711dfd3d"
2024-12-18T00:21:49.5976735Z 		},
2024-12-18T00:21:49.5976933Z 		{
2024-12-18T00:21:49.5977266Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5977683Z 			"size": 164,
2024-12-18T00:21:49.5978090Z 			"digest": "sha256:034b6a03f5f751ed12f7505175ce3443eb3ed7b34d02d1bb25de0ee52d93a44b"
2024-12-18T00:21:49.5978539Z 		},
2024-12-18T00:21:49.5978744Z 		{
2024-12-18T00:21:49.5979079Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5979491Z 			"size": 564,
2024-12-18T00:21:49.5979892Z 			"digest": "sha256:21f8f066852dd484ccbf79470452fe8005cde08773dd4fedc07fed783a0b9bf8"
2024-12-18T00:21:49.5980341Z 		},
2024-12-18T00:21:49.5980534Z 		{
2024-12-18T00:21:49.5980862Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5981268Z 			"size": 35865656,
2024-12-18T00:21:49.5981673Z 			"digest": "sha256:9dc1f315a979ec860a882d7df33584e02c1d18f86293521839ce42e21082e639"
2024-12-18T00:21:49.5982113Z 		},
2024-12-18T00:21:49.5982307Z 		{
2024-12-18T00:21:49.5982631Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5983034Z 			"size": 104,
2024-12-18T00:21:49.5983422Z 			"digest": "sha256:cd6a6063178b945c150fe21a31df6e33203295dd814802e0658835e810fb2bf9"
2024-12-18T00:21:49.5983857Z 		},
2024-12-18T00:21:49.5984047Z 		{
2024-12-18T00:21:49.5984491Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5984908Z 			"size": 425,
2024-12-18T00:21:49.5985298Z 			"digest": "sha256:2116738f4ea9299766a55629d39b881c4d33fc4fc0ce915965be1a850fcd7aaa"
2024-12-18T00:21:49.5985734Z 		},
2024-12-18T00:21:49.5985925Z 		{
2024-12-18T00:21:49.5986248Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5986652Z 			"size": 20262160,
2024-12-18T00:21:49.5987066Z 			"digest": "sha256:d7fa134316d91d2ec39bceb121619d44cdfdf1c7ac30c72768a9fd48f9763bd1"
2024-12-18T00:21:49.5987519Z 		},
2024-12-18T00:21:49.5987710Z 		{
2024-12-18T00:21:49.5988035Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5988451Z 			"size": 643,
2024-12-18T00:21:49.5988850Z 			"digest": "sha256:af03dc3fd15ea8f5f1fa41d514400f6a1d963985cf046b09d789f7f0359b2aba"
2024-12-18T00:21:49.5989300Z 		},
2024-12-18T00:21:49.5989494Z 		{
2024-12-18T00:21:49.5989822Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5990239Z 			"size": 701,
2024-12-18T00:21:49.5990654Z 			"digest": "sha256:bebd539a1d598fe86afc0da77bf16ae4c7ebfb9f031f21ed7634150fbe23f996"
2024-12-18T00:21:49.5991115Z 		},
2024-12-18T00:21:49.5991310Z 		{
2024-12-18T00:21:49.5991634Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5992039Z 			"size": 142,
2024-12-18T00:21:49.5992436Z 			"digest": "sha256:fd9de3dc425828e7b8984d4ced05f4102762137b81a5dc67104fc6b0805c570d"
2024-12-18T00:21:49.5992882Z 		},
2024-12-18T00:21:49.5993078Z 		{
2024-12-18T00:21:49.5993401Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5993805Z 			"size": 135,
2024-12-18T00:21:49.5994198Z 			"digest": "sha256:b85093661f7768ac691f4b3ee5f553b9ab14cb2a6877c8edc6133d74caa88f69"
2024-12-18T00:21:49.5994644Z 		},
2024-12-18T00:21:49.5994837Z 		{
2024-12-18T00:21:49.5995160Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5995577Z 			"size": 4995482131,
2024-12-18T00:21:49.5996120Z 			"digest": "sha256:ab6ef6c2ac4d23af7a3249cee1d4761f63a89513796ba720da7cc425c8afa6d2"
2024-12-18T00:21:49.5996571Z 		},
2024-12-18T00:21:49.5996760Z 		{
2024-12-18T00:21:49.5997084Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5997486Z 			"size": 196,
2024-12-18T00:21:49.5997874Z 			"digest": "sha256:56c896077120aaa4cbcc2203662d13b4ea14cf998e9657e91b252535ceae5348"
2024-12-18T00:21:49.5998313Z 		},
2024-12-18T00:21:49.5998502Z 		{
2024-12-18T00:21:49.5998824Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.5999230Z 			"size": 564,
2024-12-18T00:21:49.5999769Z 			"digest": "sha256:d88e9c5e7c70fe36bd1bd2e94d43d4192d94a1f28e323234a73a75a2d1da48f1"
2024-12-18T00:21:49.6000219Z 		},
2024-12-18T00:21:49.6000410Z 		{
2024-12-18T00:21:49.6000741Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6001151Z 			"size": 43164612,
2024-12-18T00:21:49.6001570Z 			"digest": "sha256:53617ee83302d7f97eb56f6756d78d18f8dc41bf252bf2d04cb643198f400f6f"
2024-12-18T00:21:49.6002033Z 		},
2024-12-18T00:21:49.6002229Z 		{
2024-12-18T00:21:49.6002553Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6002958Z 			"size": 106,
2024-12-18T00:21:49.6003742Z 			"digest": "sha256:3edbde10d4043bf2ebf32fddfa55c8d0923435465637df7dc8f4addcc03a0b84"
2024-12-18T00:21:49.6004210Z 		},
2024-12-18T00:21:49.6004404Z 		{
2024-12-18T00:21:49.6004725Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6005131Z 			"size": 1402,
2024-12-18T00:21:49.6005533Z 			"digest": "sha256:f4038f7524c3ed705e7560db6dd6efe76f548363f16bc7a93cfe360d5bc6dcaa"
2024-12-18T00:21:49.6005979Z 		},
2024-12-18T00:21:49.6006170Z 		{
2024-12-18T00:21:49.6006494Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6006901Z 			"size": 701,
2024-12-18T00:21:49.6007477Z 			"digest": "sha256:bebd539a1d598fe86afc0da77bf16ae4c7ebfb9f031f21ed7634150fbe23f996"
2024-12-18T00:21:49.6007948Z 		},
2024-12-18T00:21:49.6008141Z 		{
2024-12-18T00:21:49.6008463Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6008872Z 			"size": 137,
2024-12-18T00:21:49.6009268Z 			"digest": "sha256:4a51173f77576ab68fba2f862dd4d12b9ff1f250145c090520b05f88f5c81391"
2024-12-18T00:21:49.6009711Z 		},
2024-12-18T00:21:49.6009901Z 		{
2024-12-18T00:21:49.6010228Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6010637Z 			"size": 120,
2024-12-18T00:21:49.6011032Z 			"digest": "sha256:ed6a71d41274e5b1eea4025d24a4851d6880b902b08f7d4dd0618ea81154f575"
2024-12-18T00:21:49.6011477Z 		},
2024-12-18T00:21:49.6011669Z 		{
2024-12-18T00:21:49.6011993Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6012404Z 			"size": 4886226234,
2024-12-18T00:21:49.6012831Z 			"digest": "sha256:389dc939e2a414ef26b642b1e8657a710358afa6a6be3ebc4380b01ffacbbacc"
2024-12-18T00:21:49.6013279Z 		},
2024-12-18T00:21:49.6013472Z 		{
2024-12-18T00:21:49.6013791Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6014200Z 			"size": 174,
2024-12-18T00:21:49.6014600Z 			"digest": "sha256:0341eaeed33df462f8bd28003bb9b09739b0786abba58ccc17ba4cec535ee4ef"
2024-12-18T00:21:49.6015051Z 		},
2024-12-18T00:21:49.6015240Z 		{
2024-12-18T00:21:49.6015561Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6015963Z 			"size": 908,
2024-12-18T00:21:49.6016356Z 			"digest": "sha256:537cdb2d31685c0dbaf612d24633539a141cfcb69d90c5ce932923a6e4ab4474"
2024-12-18T00:21:49.6016799Z 		},
2024-12-18T00:21:49.6016989Z 		{
2024-12-18T00:21:49.6017307Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6017708Z 			"size": 701,
2024-12-18T00:21:49.6018109Z 			"digest": "sha256:bebd539a1d598fe86afc0da77bf16ae4c7ebfb9f031f21ed7634150fbe23f996"
2024-12-18T00:21:49.6018578Z 		},
2024-12-18T00:21:49.6018894Z 		{
2024-12-18T00:21:49.6019215Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6019617Z 			"size": 135,
2024-12-18T00:21:49.6020001Z 			"digest": "sha256:76d4a07024975dd727ff7b1c8873c513043bc9a5d310b469e51c3b110fdb6084"
2024-12-18T00:21:49.6020444Z 		},
2024-12-18T00:21:49.6020636Z 		{
2024-12-18T00:21:49.6020954Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6021361Z 			"size": 32,
2024-12-18T00:21:49.6021753Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-12-18T00:21:49.6022210Z 		},
2024-12-18T00:21:49.6022401Z 		{
2024-12-18T00:21:49.6022722Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6023128Z 			"size": 159,
2024-12-18T00:21:49.6023517Z 			"digest": "sha256:3287a539d61e3913b93941ecd49e55887b5122d63403e0fe9e7a3a63b6ed03c6"
2024-12-18T00:21:49.6023958Z 		},
2024-12-18T00:21:49.6024159Z 		{
2024-12-18T00:21:49.6024489Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6024893Z 			"size": 1932,
2024-12-18T00:21:49.6025271Z 			"digest": "sha256:9b763e24a7d42c731eb337815675c9478832146e73305559d0db1c88a0731492"
2024-12-18T00:21:49.6025709Z 		},
2024-12-18T00:21:49.6025899Z 		{
2024-12-18T00:21:49.6026223Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6026629Z 			"size": 196810750,
2024-12-18T00:21:49.6027027Z 			"digest": "sha256:f6ef3d4c814e330a230273579dc978970563774df75f73fba78181ddcf6ab98e"
2024-12-18T00:21:49.6027470Z 		},
2024-12-18T00:21:49.6027660Z 		{
2024-12-18T00:21:49.6027982Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6028386Z 			"size": 165,
2024-12-18T00:21:49.6028770Z 			"digest": "sha256:06914b00eb49b8005245cba4a631c34d0a2aaf8bf6291bc6477705206de11a06"
2024-12-18T00:21:49.6029217Z 		},
2024-12-18T00:21:49.6029405Z 		{
2024-12-18T00:21:49.6029831Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6030243Z 			"size": 7943,
2024-12-18T00:21:49.6030618Z 			"digest": "sha256:724553524224508e03de9422ca47b1a1c047e53cd305c2a1344147bb0d82d1d0"
2024-12-18T00:21:49.6031055Z 		},
2024-12-18T00:21:49.6031246Z 		{
2024-12-18T00:21:49.6031564Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6031967Z 			"size": 8070,
2024-12-18T00:21:49.6032361Z 			"digest": "sha256:47d9fb6dae26902b0fdb744916689adbd42add22a476d1c5ec89ae4e30b91e39"
2024-12-18T00:21:49.6032818Z 		},
2024-12-18T00:21:49.6033005Z 		{
2024-12-18T00:21:49.6033325Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6033727Z 			"size": 303,
2024-12-18T00:21:49.6034127Z 			"digest": "sha256:acce7b690dcb2fde77ba3ce65df70452fc7c98de73a42fe72652e477f86fc62d"
2024-12-18T00:21:49.6034588Z 		},
2024-12-18T00:21:49.6034779Z 		{
2024-12-18T00:21:49.6035111Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6035514Z 			"size": 7633763,
2024-12-18T00:21:49.6035916Z 			"digest": "sha256:58dc9f64def75bcb2abecdf39b798c3800e837d37da971e766a05731ea5537c2"
2024-12-18T00:21:49.6036367Z 		},
2024-12-18T00:21:49.6036554Z 		{
2024-12-18T00:21:49.6036871Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6037262Z 			"size": 108,
2024-12-18T00:21:49.6037661Z 			"digest": "sha256:94dfe5ef1b7e91ebd6a1a5acc1d7732e051334d5102d8a69c509c1d2f6e95598"
2024-12-18T00:21:49.6038116Z 		},
2024-12-18T00:21:49.6038305Z 		{
2024-12-18T00:21:49.6038624Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6039021Z 			"size": 54145665,
2024-12-18T00:21:49.6039508Z 			"digest": "sha256:40a2c9135b0683a7e2432748a9a6ff19926f6a2a49722d844c07e307320f7362"
2024-12-18T00:21:49.6039948Z 		},
2024-12-18T00:21:49.6040134Z 		{
2024-12-18T00:21:49.6040447Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6040849Z 			"size": 484,
2024-12-18T00:21:49.6041329Z 			"digest": "sha256:8839d6b0c8218eccb426367e5bfe066b172b8c6059bae8473e915e11c4e582a9"
2024-12-18T00:21:49.6041773Z 		},
2024-12-18T00:21:49.6041960Z 		{
2024-12-18T00:21:49.6042277Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6042675Z 			"size": 1374859271,
2024-12-18T00:21:49.6043073Z 			"digest": "sha256:1c588f0a9e97f3e192252a179d987d1ba1e5201ba4b6e74c429074e9e34df1b2"
2024-12-18T00:21:49.6043509Z 		},
2024-12-18T00:21:49.6043691Z 		{
2024-12-18T00:21:49.6044011Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6044407Z 			"size": 106,
2024-12-18T00:21:49.6044798Z 			"digest": "sha256:6727e7e279360b8b66f0137a15b357efede06e69bfd0800ebad6ed7c8ed71df4"
2024-12-18T00:21:49.6045247Z 		},
2024-12-18T00:21:49.6045433Z 		{
2024-12-18T00:21:49.6045750Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6046148Z 			"size": 568,
2024-12-18T00:21:49.6046541Z 			"digest": "sha256:3a55100d1957b103c43e314deb55697c459aa6fa225f91e7b1c69bc4c01dca4e"
2024-12-18T00:21:49.6046991Z 		},
2024-12-18T00:21:49.6047182Z 		{
2024-12-18T00:21:49.6047498Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6047893Z 			"size": 303706071,
2024-12-18T00:21:49.6048292Z 			"digest": "sha256:3d6f160d335161c5321e13ce2823ec68379a6a3401b07bd7378602a69c85b80c"
2024-12-18T00:21:49.6048728Z 		},
2024-12-18T00:21:49.6048915Z 		{
2024-12-18T00:21:49.6049234Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6049633Z 			"size": 111,
2024-12-18T00:21:49.6050027Z 			"digest": "sha256:b2a8bb947a87562144fe67a39c280d4dee50cbe9d8ff349140f5db6c377517e6"
2024-12-18T00:21:49.6050477Z 		},
2024-12-18T00:21:49.6050666Z 		{
2024-12-18T00:21:49.6050987Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6051391Z 			"size": 529,
2024-12-18T00:21:49.6051881Z 			"digest": "sha256:69a74b2360f4ede95654274ce7ef6cc1597a7234cc17e86d5cdf4c98286d97c3"
2024-12-18T00:21:49.6052337Z 		},
2024-12-18T00:21:49.6052531Z 		{
2024-12-18T00:21:49.6052854Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6053249Z 			"size": 48289052,
2024-12-18T00:21:49.6053656Z 			"digest": "sha256:f9e12a253d89c16863b9d4d90138ffa673447c9b7f27afcba54a2a679e58b5ab"
2024-12-18T00:21:49.6054101Z 		},
2024-12-18T00:21:49.6054289Z 		{
2024-12-18T00:21:49.6054610Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6055013Z 			"size": 106,
2024-12-18T00:21:49.6055407Z 			"digest": "sha256:8fdb06d51b96d538045b9566a381de3dc25e59d1adb9d0a4e567afcb3213f676"
2024-12-18T00:21:49.6055858Z 		},
2024-12-18T00:21:49.6056050Z 		{
2024-12-18T00:21:49.6056368Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6056765Z 			"size": 32,
2024-12-18T00:21:49.6057166Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-12-18T00:21:49.6057622Z 		},
2024-12-18T00:21:49.6057809Z 		{
2024-12-18T00:21:49.6058125Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6058519Z 			"size": 32,
2024-12-18T00:21:49.6058913Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-12-18T00:21:49.6059363Z 		},
2024-12-18T00:21:49.6059548Z 		{
2024-12-18T00:21:49.6059864Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6060258Z 			"size": 32,
2024-12-18T00:21:49.6060651Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-12-18T00:21:49.6061103Z 		},
2024-12-18T00:21:49.6061292Z 		{
2024-12-18T00:21:49.6061609Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:49.6062004Z 			"size": 32,
2024-12-18T00:21:49.6062396Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-12-18T00:21:49.6062850Z 		}
2024-12-18T00:21:49.6063131Z 	]
2024-12-18T00:21:49.6063321Z }
2024-12-18T00:21:49.6063513Z + exit 0
2024-12-18T00:21:49.6102417Z ##[group]Run set -eux
2024-12-18T00:21:49.6102706Z set -eux
2024-12-18T00:21:49.6104007Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin
2024-12-18T00:21:49.6113569Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:49.6113946Z env:
2024-12-18T00:21:49.6114166Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:49.6114431Z ##[endgroup]
2024-12-18T00:21:49.6146571Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token
2024-12-18T00:21:49.6147591Z + jq --raw-output .SecretString
2024-12-18T00:21:49.6149172Z + jq -r .docker_hub_readonly_token
2024-12-18T00:21:49.6151011Z + docker login --username pytorchbot --password-stdin
2024-12-18T00:21:50.2391169Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-12-18T00:21:50.2391895Z Configure a credential helper to remove this warning. See
2024-12-18T00:21:50.2392508Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-12-18T00:21:50.2392936Z 
2024-12-18T00:21:50.2394855Z Login Succeeded
2024-12-18T00:21:50.2495506Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/}
2024-12-18T00:21:50.2495875Z tag=${ECR_DOCKER_IMAGE##*/}
2024-12-18T00:21:50.2496271Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}"
2024-12-18T00:21:50.2505314Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:50.2505692Z env:
2024-12-18T00:21:50.2505912Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:50.2506771Z   ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:50.2507652Z ##[endgroup]
2024-12-18T00:21:50.2537829Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks-45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:50.2592429Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@release/2.6
2024-12-18T00:21:50.2592900Z with:
2024-12-18T00:21:50.2593690Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:50.2594671Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:50.2595065Z env:
2024-12-18T00:21:50.2595281Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:50.2595560Z ##[endgroup]
2024-12-18T00:21:50.2621303Z ##[group]Run set -x
2024-12-18T00:21:50.2621582Z set -x
2024-12-18T00:21:50.2621813Z set +e
2024-12-18T00:21:50.2622034Z 
2024-12-18T00:21:50.2622252Z login() {
2024-12-18T00:21:50.2622727Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2024-12-18T00:21:50.2623253Z }
2024-12-18T00:21:50.2623473Z 
2024-12-18T00:21:50.2623711Z retry () {
2024-12-18T00:21:50.2623992Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2024-12-18T00:21:50.2624303Z }
2024-12-18T00:21:50.2624520Z 
2024-12-18T00:21:50.2624764Z retry login "${DOCKER_REGISTRY}"
2024-12-18T00:21:50.2625074Z 
2024-12-18T00:21:50.2625292Z set -e
2024-12-18T00:21:50.2625625Z # ignore output since only exit code is used for conditional
2024-12-18T00:21:50.2626114Z # only pull docker image if it's not available locally
2024-12-18T00:21:50.2626653Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2024-12-18T00:21:50.2627146Z   retry docker pull "${DOCKER_IMAGE}"
2024-12-18T00:21:50.2627465Z fi
2024-12-18T00:21:50.2636180Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:50.2636729Z env:
2024-12-18T00:21:50.2636950Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:50.2637806Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:50.2638767Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:50.2639147Z ##[endgroup]
2024-12-18T00:21:50.2667538Z + set +e
2024-12-18T00:21:50.2668179Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:50.2668622Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:50.2671398Z + aws ecr get-login-password --region us-east-1
2024-12-18T00:21:50.2672370Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:50.8240256Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-12-18T00:21:50.8240865Z Configure a credential helper to remove this warning. See
2024-12-18T00:21:50.8241466Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-12-18T00:21:50.8241863Z 
2024-12-18T00:21:50.8243609Z Login Succeeded
2024-12-18T00:21:50.8270228Z + set -e
2024-12-18T00:21:50.8271333Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:50.8434652Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:50.8436090Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:51.0712361Z 45e1356b47a284893081276eff3000b7b534f3b1: Pulling from pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks
2024-12-18T00:21:51.0716919Z 80888bc6716f: Pulling fs layer
2024-12-18T00:21:51.0717497Z fbcd35dc5bc3: Pulling fs layer
2024-12-18T00:21:51.0718118Z c7232af9ae05: Pulling fs layer
2024-12-18T00:21:51.0718560Z db6cdef1932a: Pulling fs layer
2024-12-18T00:21:51.0718848Z 56dc85502937: Pulling fs layer
2024-12-18T00:21:51.0719167Z 30c0ea6140d0: Pulling fs layer
2024-12-18T00:21:51.0719745Z 71bdb1a72c2d: Pulling fs layer
2024-12-18T00:21:51.0720099Z 4829486be7c3: Pulling fs layer
2024-12-18T00:21:51.0720412Z 1f4e68d7b5e4: Pulling fs layer
2024-12-18T00:21:51.0720787Z 7c373e2d9b7e: Pulling fs layer
2024-12-18T00:21:51.0721077Z 622381141745: Pulling fs layer
2024-12-18T00:21:51.0721424Z f003d065e31e: Pulling fs layer
2024-12-18T00:21:51.0721770Z cd6ce7116815: Pulling fs layer
2024-12-18T00:21:51.0722058Z e843066a8f68: Pulling fs layer
2024-12-18T00:21:51.0722434Z bac50ca2930b: Pulling fs layer
2024-12-18T00:21:51.0722718Z 57e52c6751f4: Pulling fs layer
2024-12-18T00:21:51.0723004Z a9c5c14d77c0: Pulling fs layer
2024-12-18T00:21:51.0723390Z b86099b8619a: Pulling fs layer
2024-12-18T00:21:51.0723671Z 3c8ea8a600ff: Pulling fs layer
2024-12-18T00:21:51.0724015Z 30c0ea6140d0: Waiting
2024-12-18T00:21:51.0724312Z 71bdb1a72c2d: Waiting
2024-12-18T00:21:51.0724558Z 4829486be7c3: Waiting
2024-12-18T00:21:51.0724838Z 1f4e68d7b5e4: Waiting
2024-12-18T00:21:51.0725141Z bebd539a1d59: Pulling fs layer
2024-12-18T00:21:51.0725410Z 7c373e2d9b7e: Waiting
2024-12-18T00:21:51.0725693Z 292b18ec2dab: Pulling fs layer
2024-12-18T00:21:51.0726021Z 76e3817b4b4c: Pulling fs layer
2024-12-18T00:21:51.0726289Z 622381141745: Waiting
2024-12-18T00:21:51.0726575Z 50b1f245e9f5: Pulling fs layer
2024-12-18T00:21:51.0726899Z f003d065e31e: Waiting
2024-12-18T00:21:51.0727140Z cd6ce7116815: Waiting
2024-12-18T00:21:51.0727436Z ad93cf5205f2: Pulling fs layer
2024-12-18T00:21:51.0727987Z e843066a8f68: Waiting
2024-12-18T00:21:51.0728436Z 8d17c59eaf7d: Pulling fs layer
2024-12-18T00:21:51.0728842Z bac50ca2930b: Waiting
2024-12-18T00:21:51.0729303Z db6cdef1932a: Waiting
2024-12-18T00:21:51.0729887Z b36a8ccf5c8c: Pulling fs layer
2024-12-18T00:21:51.0730330Z 56dc85502937: Waiting
2024-12-18T00:21:51.0730776Z aa0a3b87e9e3: Pulling fs layer
2024-12-18T00:21:51.0731246Z 57e52c6751f4: Waiting
2024-12-18T00:21:51.0731641Z 72a632052c5b: Pulling fs layer
2024-12-18T00:21:51.0732069Z 3c8ea8a600ff: Waiting
2024-12-18T00:21:51.0732453Z ceed6316bc24: Pulling fs layer
2024-12-18T00:21:51.0732962Z d0ca0f491f80: Pulling fs layer
2024-12-18T00:21:51.0733354Z bebd539a1d59: Waiting
2024-12-18T00:21:51.0744146Z a9c5c14d77c0: Waiting
2024-12-18T00:21:51.0744435Z cf02c706cd56: Pulling fs layer
2024-12-18T00:21:51.0744722Z b86099b8619a: Waiting
2024-12-18T00:21:51.0744989Z 16073d3f107d: Pulling fs layer
2024-12-18T00:21:51.0745274Z 292b18ec2dab: Waiting
2024-12-18T00:21:51.0745531Z 76e3817b4b4c: Waiting
2024-12-18T00:21:51.0745798Z d234faf2f1bf: Pulling fs layer
2024-12-18T00:21:51.0746092Z 3f0ade86af1a: Pulling fs layer
2024-12-18T00:21:51.0746382Z b36a8ccf5c8c: Waiting
2024-12-18T00:21:51.0746657Z 034b6a03f5f7: Pulling fs layer
2024-12-18T00:21:51.0746942Z 8d17c59eaf7d: Waiting
2024-12-18T00:21:51.0747211Z 21f8f066852d: Pulling fs layer
2024-12-18T00:21:51.0747490Z aa0a3b87e9e3: Waiting
2024-12-18T00:21:51.0747745Z ceed6316bc24: Waiting
2024-12-18T00:21:51.0748000Z 72a632052c5b: Waiting
2024-12-18T00:21:51.0748251Z d0ca0f491f80: Waiting
2024-12-18T00:21:51.0748495Z 50b1f245e9f5: Waiting
2024-12-18T00:21:51.0748763Z 9dc1f315a979: Pulling fs layer
2024-12-18T00:21:51.0749055Z ad93cf5205f2: Waiting
2024-12-18T00:21:51.0749311Z cf02c706cd56: Waiting
2024-12-18T00:21:51.0749578Z cd6a6063178b: Pulling fs layer
2024-12-18T00:21:51.0749855Z d234faf2f1bf: Waiting
2024-12-18T00:21:51.0750116Z 3f0ade86af1a: Waiting
2024-12-18T00:21:51.0750382Z 2116738f4ea9: Pulling fs layer
2024-12-18T00:21:51.0750693Z 16073d3f107d: Waiting
2024-12-18T00:21:51.0750939Z 034b6a03f5f7: Waiting
2024-12-18T00:21:51.0751183Z 21f8f066852d: Waiting
2024-12-18T00:21:51.0751423Z 9dc1f315a979: Waiting
2024-12-18T00:21:51.0751690Z d7fa134316d9: Pulling fs layer
2024-12-18T00:21:51.0751975Z cd6a6063178b: Waiting
2024-12-18T00:21:51.0752405Z af03dc3fd15e: Pulling fs layer
2024-12-18T00:21:51.0752709Z fd9de3dc4258: Pulling fs layer
2024-12-18T00:21:51.0752984Z af03dc3fd15e: Waiting
2024-12-18T00:21:51.0753247Z b85093661f77: Pulling fs layer
2024-12-18T00:21:51.0753538Z ab6ef6c2ac4d: Pulling fs layer
2024-12-18T00:21:51.0753823Z fd9de3dc4258: Waiting
2024-12-18T00:21:51.0754082Z 56c896077120: Pulling fs layer
2024-12-18T00:21:51.0754351Z d7fa134316d9: Waiting
2024-12-18T00:21:51.0754611Z d88e9c5e7c70: Pulling fs layer
2024-12-18T00:21:51.0754901Z 53617ee83302: Pulling fs layer
2024-12-18T00:21:51.0755192Z 3edbde10d404: Pulling fs layer
2024-12-18T00:21:51.0755474Z ab6ef6c2ac4d: Waiting
2024-12-18T00:21:51.0755729Z f4038f7524c3: Pulling fs layer
2024-12-18T00:21:51.0756079Z 4a51173f7757: Pulling fs layer
2024-12-18T00:21:51.0756365Z ed6a71d41274: Pulling fs layer
2024-12-18T00:21:51.0756647Z 389dc939e2a4: Pulling fs layer
2024-12-18T00:21:51.0756942Z 0341eaeed33d: Pulling fs layer
2024-12-18T00:21:51.0757211Z 56c896077120: Waiting
2024-12-18T00:21:51.0757465Z 3edbde10d404: Waiting
2024-12-18T00:21:51.0757713Z f4038f7524c3: Waiting
2024-12-18T00:21:51.0757958Z d88e9c5e7c70: Waiting
2024-12-18T00:21:51.0758209Z 537cdb2d3168: Pulling fs layer
2024-12-18T00:21:51.0758485Z 4a51173f7757: Waiting
2024-12-18T00:21:51.0758727Z 53617ee83302: Waiting
2024-12-18T00:21:51.0758972Z 0341eaeed33d: Waiting
2024-12-18T00:21:51.0759222Z ed6a71d41274: Waiting
2024-12-18T00:21:51.0759580Z 389dc939e2a4: Waiting
2024-12-18T00:21:51.0759830Z 537cdb2d3168: Waiting
2024-12-18T00:21:51.0760087Z 76d4a0702497: Pulling fs layer
2024-12-18T00:21:51.0760373Z 4f4fb700ef54: Pulling fs layer
2024-12-18T00:21:51.0760658Z 3287a539d61e: Pulling fs layer
2024-12-18T00:21:51.0760933Z 9b763e24a7d4: Pulling fs layer
2024-12-18T00:21:51.0761218Z f6ef3d4c814e: Pulling fs layer
2024-12-18T00:21:51.0761500Z 06914b00eb49: Pulling fs layer
2024-12-18T00:21:51.0761783Z 724553524224: Pulling fs layer
2024-12-18T00:21:51.0762171Z 47d9fb6dae26: Pulling fs layer
2024-12-18T00:21:51.0762453Z acce7b690dcb: Pulling fs layer
2024-12-18T00:21:51.0762751Z 58dc9f64def7: Pulling fs layer
2024-12-18T00:21:51.0763030Z 94dfe5ef1b7e: Pulling fs layer
2024-12-18T00:21:51.0763300Z 76d4a0702497: Waiting
2024-12-18T00:21:51.0763537Z 06914b00eb49: Waiting
2024-12-18T00:21:51.0763772Z 47d9fb6dae26: Waiting
2024-12-18T00:21:51.0764024Z 40a2c9135b06: Pulling fs layer
2024-12-18T00:21:51.0764293Z 724553524224: Waiting
2024-12-18T00:21:51.0764531Z 4f4fb700ef54: Waiting
2024-12-18T00:21:51.0764778Z 8839d6b0c821: Pulling fs layer
2024-12-18T00:21:51.0765042Z f6ef3d4c814e: Waiting
2024-12-18T00:21:51.0765293Z 1c588f0a9e97: Pulling fs layer
2024-12-18T00:21:51.0765562Z 9b763e24a7d4: Waiting
2024-12-18T00:21:51.0765797Z 40a2c9135b06: Waiting
2024-12-18T00:21:51.0766043Z 6727e7e27936: Pulling fs layer
2024-12-18T00:21:51.0766315Z 94dfe5ef1b7e: Waiting
2024-12-18T00:21:51.0766561Z 3a55100d1957: Pulling fs layer
2024-12-18T00:21:51.0766834Z 8839d6b0c821: Waiting
2024-12-18T00:21:51.0767073Z acce7b690dcb: Waiting
2024-12-18T00:21:51.0767318Z 58dc9f64def7: Waiting
2024-12-18T00:21:51.0767562Z 3d6f160d3351: Pulling fs layer
2024-12-18T00:21:51.0767845Z b2a8bb947a87: Pulling fs layer
2024-12-18T00:21:51.0768126Z 69a74b2360f4: Pulling fs layer
2024-12-18T00:21:51.0768407Z f9e12a253d89: Pulling fs layer
2024-12-18T00:21:51.0768688Z 8fdb06d51b96: Pulling fs layer
2024-12-18T00:21:51.0768950Z 3d6f160d3351: Waiting
2024-12-18T00:21:51.0769187Z 69a74b2360f4: Waiting
2024-12-18T00:21:51.0769427Z f9e12a253d89: Waiting
2024-12-18T00:21:51.0769661Z 3a55100d1957: Waiting
2024-12-18T00:21:51.0769901Z 8fdb06d51b96: Waiting
2024-12-18T00:21:51.2013814Z fbcd35dc5bc3: Verifying Checksum
2024-12-18T00:21:51.2014150Z fbcd35dc5bc3: Download complete
2024-12-18T00:21:51.3209267Z db6cdef1932a: Verifying Checksum
2024-12-18T00:21:51.3209606Z db6cdef1932a: Download complete
2024-12-18T00:21:51.4076557Z 56dc85502937: Download complete
2024-12-18T00:21:51.4292348Z 80888bc6716f: Verifying Checksum
2024-12-18T00:21:51.4292740Z 80888bc6716f: Download complete
2024-12-18T00:21:51.4971927Z 71bdb1a72c2d: Verifying Checksum
2024-12-18T00:21:51.4972407Z 71bdb1a72c2d: Download complete
2024-12-18T00:21:51.5658771Z 4829486be7c3: Download complete
2024-12-18T00:21:51.6475548Z 1f4e68d7b5e4: Download complete
2024-12-18T00:21:51.7135747Z c7232af9ae05: Verifying Checksum
2024-12-18T00:21:51.7136243Z c7232af9ae05: Download complete
2024-12-18T00:21:51.7968667Z 622381141745: Verifying Checksum
2024-12-18T00:21:51.7968997Z 622381141745: Download complete
2024-12-18T00:21:51.8889567Z f003d065e31e: Verifying Checksum
2024-12-18T00:21:51.8890063Z f003d065e31e: Download complete
2024-12-18T00:21:52.5511144Z 80888bc6716f: Pull complete
2024-12-18T00:21:52.8436053Z fbcd35dc5bc3: Pull complete
2024-12-18T00:21:53.5919667Z c7232af9ae05: Pull complete
2024-12-18T00:21:53.6162685Z db6cdef1932a: Pull complete
2024-12-18T00:21:53.6406655Z 56dc85502937: Pull complete
2024-12-18T00:21:54.4215644Z cd6ce7116815: Verifying Checksum
2024-12-18T00:21:54.4215998Z cd6ce7116815: Download complete
2024-12-18T00:21:54.5145218Z e843066a8f68: Download complete
2024-12-18T00:21:54.6204358Z bac50ca2930b: Verifying Checksum
2024-12-18T00:21:54.6205394Z bac50ca2930b: Download complete
2024-12-18T00:21:54.7184363Z 57e52c6751f4: Verifying Checksum
2024-12-18T00:21:54.7184765Z 57e52c6751f4: Download complete
2024-12-18T00:21:55.6809027Z a9c5c14d77c0: Verifying Checksum
2024-12-18T00:21:55.6809866Z a9c5c14d77c0: Download complete
2024-12-18T00:21:55.7749198Z b86099b8619a: Download complete
2024-12-18T00:21:55.8583542Z 3c8ea8a600ff: Verifying Checksum
2024-12-18T00:21:55.8583881Z 3c8ea8a600ff: Download complete
2024-12-18T00:21:55.9467026Z bebd539a1d59: Verifying Checksum
2024-12-18T00:21:55.9467384Z bebd539a1d59: Download complete
2024-12-18T00:21:56.0471302Z 292b18ec2dab: Download complete
2024-12-18T00:22:05.1943707Z 30c0ea6140d0: Verifying Checksum
2024-12-18T00:22:05.1944206Z 30c0ea6140d0: Download complete
2024-12-18T00:22:05.2806258Z 50b1f245e9f5: Verifying Checksum
2024-12-18T00:22:05.2806739Z 50b1f245e9f5: Download complete
2024-12-18T00:22:05.3717012Z ad93cf5205f2: Verifying Checksum
2024-12-18T00:22:05.3717444Z ad93cf5205f2: Download complete
2024-12-18T00:22:05.4628562Z 8d17c59eaf7d: Verifying Checksum
2024-12-18T00:22:05.4629086Z 8d17c59eaf7d: Download complete
2024-12-18T00:22:05.5472727Z b36a8ccf5c8c: Download complete
2024-12-18T00:22:05.6422243Z aa0a3b87e9e3: Verifying Checksum
2024-12-18T00:22:05.6422715Z aa0a3b87e9e3: Download complete
2024-12-18T00:22:06.9258099Z 72a632052c5b: Verifying Checksum
2024-12-18T00:22:06.9258582Z 72a632052c5b: Download complete
2024-12-18T00:22:07.0106725Z ceed6316bc24: Verifying Checksum
2024-12-18T00:22:07.0107394Z ceed6316bc24: Download complete
2024-12-18T00:22:07.0859875Z d0ca0f491f80: Verifying Checksum
2024-12-18T00:22:07.0860408Z d0ca0f491f80: Download complete
2024-12-18T00:22:07.1587899Z cf02c706cd56: Verifying Checksum
2024-12-18T00:22:07.1588279Z cf02c706cd56: Download complete
2024-12-18T00:22:07.2277598Z 16073d3f107d: Download complete
2024-12-18T00:22:07.3027352Z d234faf2f1bf: Download complete
2024-12-18T00:22:11.6504779Z 3f0ade86af1a: Verifying Checksum
2024-12-18T00:22:11.6505154Z 3f0ade86af1a: Download complete
2024-12-18T00:22:11.7533673Z 034b6a03f5f7: Verifying Checksum
2024-12-18T00:22:11.7534030Z 034b6a03f5f7: Download complete
2024-12-18T00:22:11.8361341Z 21f8f066852d: Verifying Checksum
2024-12-18T00:22:11.8361667Z 21f8f066852d: Download complete
2024-12-18T00:22:12.2876425Z 9dc1f315a979: Verifying Checksum
2024-12-18T00:22:12.2876810Z 9dc1f315a979: Download complete
2024-12-18T00:22:12.3777599Z cd6a6063178b: Verifying Checksum
2024-12-18T00:22:12.3777953Z cd6a6063178b: Download complete
2024-12-18T00:22:12.4502435Z 2116738f4ea9: Verifying Checksum
2024-12-18T00:22:12.4502789Z 2116738f4ea9: Download complete
2024-12-18T00:22:12.7104860Z d7fa134316d9: Verifying Checksum
2024-12-18T00:22:12.7105651Z d7fa134316d9: Download complete
2024-12-18T00:22:12.8234397Z af03dc3fd15e: Verifying Checksum
2024-12-18T00:22:12.8234796Z af03dc3fd15e: Download complete
2024-12-18T00:22:12.9074079Z fd9de3dc4258: Verifying Checksum
2024-12-18T00:22:12.9074483Z fd9de3dc4258: Download complete
2024-12-18T00:22:12.9941235Z b85093661f77: Download complete
2024-12-18T00:22:17.1599936Z 30c0ea6140d0: Pull complete
2024-12-18T00:22:17.3635716Z 71bdb1a72c2d: Pull complete
2024-12-18T00:22:17.5385804Z 4829486be7c3: Pull complete
2024-12-18T00:22:17.7671550Z 1f4e68d7b5e4: Pull complete
2024-12-18T00:22:17.9831721Z 7c373e2d9b7e: Verifying Checksum
2024-12-18T00:22:17.9832251Z 7c373e2d9b7e: Download complete
2024-12-18T00:22:18.0616827Z 56c896077120: Verifying Checksum
2024-12-18T00:22:18.0617189Z 56c896077120: Download complete
2024-12-18T00:22:18.1335067Z d88e9c5e7c70: Verifying Checksum
2024-12-18T00:22:18.1335450Z d88e9c5e7c70: Download complete
2024-12-18T00:22:18.6316510Z 53617ee83302: Verifying Checksum
2024-12-18T00:22:18.6316926Z 53617ee83302: Download complete
2024-12-18T00:22:18.7077602Z 3edbde10d404: Download complete
2024-12-18T00:22:18.8187278Z f4038f7524c3: Verifying Checksum
2024-12-18T00:22:18.8187758Z f4038f7524c3: Download complete
2024-12-18T00:22:18.9152117Z 4a51173f7757: Verifying Checksum
2024-12-18T00:22:18.9155176Z 4a51173f7757: Download complete
2024-12-18T00:22:19.0091263Z ed6a71d41274: Verifying Checksum
2024-12-18T00:22:19.0091763Z ed6a71d41274: Download complete
2024-12-18T00:22:25.6786723Z 76e3817b4b4c: Verifying Checksum
2024-12-18T00:22:25.6787183Z 76e3817b4b4c: Download complete
2024-12-18T00:22:25.7540778Z 0341eaeed33d: Verifying Checksum
2024-12-18T00:22:25.7541121Z 0341eaeed33d: Download complete
2024-12-18T00:22:25.8670194Z 537cdb2d3168: Download complete
2024-12-18T00:22:25.9601231Z 76d4a0702497: Verifying Checksum
2024-12-18T00:22:25.9601705Z 76d4a0702497: Download complete
2024-12-18T00:22:25.9683568Z 4f4fb700ef54: Verifying Checksum
2024-12-18T00:22:25.9684847Z 4f4fb700ef54: Download complete
2024-12-18T00:22:26.0746846Z 3287a539d61e: Verifying Checksum
2024-12-18T00:22:26.0747713Z 3287a539d61e: Download complete
2024-12-18T00:22:26.1841749Z 9b763e24a7d4: Verifying Checksum
2024-12-18T00:22:26.1842276Z 9b763e24a7d4: Download complete
2024-12-18T00:22:28.2066737Z f6ef3d4c814e: Verifying Checksum
2024-12-18T00:22:28.2067150Z f6ef3d4c814e: Download complete
2024-12-18T00:22:28.3131751Z 06914b00eb49: Download complete
2024-12-18T00:22:28.4074437Z 724553524224: Verifying Checksum
2024-12-18T00:22:28.4074881Z 724553524224: Download complete
2024-12-18T00:22:28.5051580Z 47d9fb6dae26: Download complete
2024-12-18T00:22:28.5850596Z acce7b690dcb: Verifying Checksum
2024-12-18T00:22:28.5850937Z acce7b690dcb: Download complete
2024-12-18T00:22:28.7234381Z 58dc9f64def7: Verifying Checksum
2024-12-18T00:22:28.7234747Z 58dc9f64def7: Download complete
2024-12-18T00:22:28.8223105Z 94dfe5ef1b7e: Verifying Checksum
2024-12-18T00:22:28.8223561Z 94dfe5ef1b7e: Download complete
2024-12-18T00:22:29.4301002Z 40a2c9135b06: Verifying Checksum
2024-12-18T00:22:29.4301356Z 40a2c9135b06: Download complete
2024-12-18T00:22:29.5135867Z 8839d6b0c821: Verifying Checksum
2024-12-18T00:22:29.5136387Z 8839d6b0c821: Download complete
2024-12-18T00:23:03.8140005Z 1c588f0a9e97: Verifying Checksum
2024-12-18T00:23:03.8140405Z 1c588f0a9e97: Download complete
2024-12-18T00:23:03.8961408Z 6727e7e27936: Verifying Checksum
2024-12-18T00:23:03.8961810Z 6727e7e27936: Download complete
2024-12-18T00:23:03.9750305Z 3a55100d1957: Verifying Checksum
2024-12-18T00:23:03.9750681Z 3a55100d1957: Download complete
2024-12-18T00:23:12.3440858Z 3d6f160d3351: Verifying Checksum
2024-12-18T00:23:12.3441202Z 3d6f160d3351: Download complete
2024-12-18T00:23:12.4428354Z b2a8bb947a87: Verifying Checksum
2024-12-18T00:23:12.4428691Z b2a8bb947a87: Download complete
2024-12-18T00:23:12.5322746Z 69a74b2360f4: Verifying Checksum
2024-12-18T00:23:13.9632071Z f9e12a253d89: Verifying Checksum
2024-12-18T00:23:13.9632427Z f9e12a253d89: Download complete
2024-12-18T00:23:14.0519414Z 8fdb06d51b96: Download complete
2024-12-18T00:23:42.7333908Z ab6ef6c2ac4d: Verifying Checksum
2024-12-18T00:23:42.7334280Z ab6ef6c2ac4d: Download complete
2024-12-18T00:23:47.5635500Z 389dc939e2a4: Verifying Checksum
2024-12-18T00:23:47.5635882Z 389dc939e2a4: Download complete
2024-12-18T00:23:59.8708134Z 7c373e2d9b7e: Pull complete
2024-12-18T00:24:00.0719810Z 622381141745: Pull complete
2024-12-18T00:24:00.2654259Z f003d065e31e: Pull complete
2024-12-18T00:24:08.1454541Z cd6ce7116815: Pull complete
2024-12-18T00:24:08.3826584Z e843066a8f68: Pull complete
2024-12-18T00:24:08.6271581Z bac50ca2930b: Pull complete
2024-12-18T00:24:08.8603062Z 57e52c6751f4: Pull complete
2024-12-18T00:24:11.0185692Z a9c5c14d77c0: Pull complete
2024-12-18T00:24:11.1089213Z b86099b8619a: Pull complete
2024-12-18T00:24:11.2638232Z 3c8ea8a600ff: Pull complete
2024-12-18T00:24:11.4556431Z bebd539a1d59: Pull complete
2024-12-18T00:24:11.6751448Z 292b18ec2dab: Pull complete
2024-12-18T00:25:04.6441243Z 76e3817b4b4c: Pull complete
2024-12-18T00:25:04.7857694Z 50b1f245e9f5: Pull complete
2024-12-18T00:25:04.9710562Z ad93cf5205f2: Pull complete
2024-12-18T00:25:05.1977192Z 8d17c59eaf7d: Pull complete
2024-12-18T00:25:05.4130057Z b36a8ccf5c8c: Pull complete
2024-12-18T00:25:05.5985071Z aa0a3b87e9e3: Pull complete
2024-12-18T00:25:08.1450751Z 72a632052c5b: Pull complete
2024-12-18T00:25:08.3822002Z ceed6316bc24: Pull complete
2024-12-18T00:25:08.6214942Z d0ca0f491f80: Pull complete
2024-12-18T00:25:08.8526655Z cf02c706cd56: Pull complete
2024-12-18T00:25:09.0869929Z 16073d3f107d: Pull complete
2024-12-18T00:25:09.3242568Z d234faf2f1bf: Pull complete
2024-12-18T00:25:17.9191411Z 3f0ade86af1a: Pull complete
2024-12-18T00:25:18.2375198Z 034b6a03f5f7: Pull complete
2024-12-18T00:25:18.3847640Z 21f8f066852d: Pull complete
2024-12-18T00:25:19.0822685Z 9dc1f315a979: Pull complete
2024-12-18T00:25:19.2117629Z cd6a6063178b: Pull complete
2024-12-18T00:25:19.3525534Z 2116738f4ea9: Pull complete
2024-12-18T00:25:19.6927384Z d7fa134316d9: Pull complete
2024-12-18T00:25:19.8339088Z af03dc3fd15e: Pull complete
2024-12-18T00:25:20.1154105Z fd9de3dc4258: Pull complete
2024-12-18T00:25:20.2376893Z b85093661f77: Pull complete
2024-12-18T00:25:50.5764223Z ab6ef6c2ac4d: Pull complete
2024-12-18T00:25:50.8034338Z 56c896077120: Pull complete
2024-12-18T00:25:50.9975586Z d88e9c5e7c70: Pull complete
2024-12-18T00:25:53.5842874Z 53617ee83302: Pull complete
2024-12-18T00:25:53.8174039Z 3edbde10d404: Pull complete
2024-12-18T00:25:54.0561780Z f4038f7524c3: Pull complete
2024-12-18T00:25:54.4862291Z 4a51173f7757: Pull complete
2024-12-18T00:25:54.7354275Z ed6a71d41274: Pull complete
2024-12-18T00:27:23.4465446Z 389dc939e2a4: Pull complete
2024-12-18T00:27:23.6849783Z 0341eaeed33d: Pull complete
2024-12-18T00:27:23.9157890Z 537cdb2d3168: Pull complete
2024-12-18T00:27:24.3357509Z 76d4a0702497: Pull complete
2024-12-18T00:27:24.5651106Z 4f4fb700ef54: Pull complete
2024-12-18T00:27:24.8023883Z 3287a539d61e: Pull complete
2024-12-18T00:27:25.0497485Z 9b763e24a7d4: Pull complete
2024-12-18T00:27:32.1507650Z f6ef3d4c814e: Pull complete
2024-12-18T00:27:32.3759360Z 06914b00eb49: Pull complete
2024-12-18T00:27:32.6031806Z 724553524224: Pull complete
2024-12-18T00:27:32.8307420Z 47d9fb6dae26: Pull complete
2024-12-18T00:27:33.0659788Z acce7b690dcb: Pull complete
2024-12-18T00:27:34.4643129Z 58dc9f64def7: Pull complete
2024-12-18T00:27:34.6933697Z 94dfe5ef1b7e: Pull complete
2024-12-18T00:27:36.4033681Z 40a2c9135b06: Pull complete
2024-12-18T00:27:36.6425575Z 8839d6b0c821: Pull complete
2024-12-18T00:27:49.4976701Z 1c588f0a9e97: Pull complete
2024-12-18T00:27:49.7103754Z 6727e7e27936: Pull complete
2024-12-18T00:27:49.9014862Z 3a55100d1957: Pull complete
2024-12-18T00:27:52.8241004Z 3d6f160d3351: Pull complete
2024-12-18T00:27:53.0296713Z b2a8bb947a87: Pull complete
2024-12-18T00:27:53.2358060Z 69a74b2360f4: Pull complete
2024-12-18T00:27:53.8736407Z f9e12a253d89: Pull complete
2024-12-18T00:27:54.0954212Z 8fdb06d51b96: Pull complete
2024-12-18T00:27:55.0710527Z Digest: sha256:ff80d4c93d18e8ab7a2501f0e62b7c40cf97aa83fc9702e88b9c2d02cdcbecdb
2024-12-18T00:27:55.1031927Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:27:55.1273827Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:27:55.1336576Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T00:27:55.1337505Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T00:27:55.1347447Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:27:55.1347815Z env:
2024-12-18T00:27:55.1348037Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:27:55.1348310Z ##[endgroup]
2024-12-18T00:27:55.1676103Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@release/2.6
2024-12-18T00:27:55.1676571Z with:
2024-12-18T00:27:55.1676807Z   driver-version: 550.54.15
2024-12-18T00:27:55.1677071Z env:
2024-12-18T00:27:55.1677293Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:27:55.1677566Z ##[endgroup]
2024-12-18T00:27:55.1877968Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2024-12-18T00:27:55.1878383Z with:
2024-12-18T00:27:55.1878611Z   timeout_minutes: 10
2024-12-18T00:27:55.1878864Z   max_attempts: 3
2024-12-18T00:27:55.1904149Z   command: # Is it disgusting to have a full shell script here in this github action? Sure
# But is it the best way to make it so that this action relies on nothing else? Absolutely
set -eou pipefail

DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID)
DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run"

install_nvidia_docker2_amzn2() {
    (
        set -x
        # Needed for yum-config-manager
        sudo yum install -y yum-utils
        if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then
          YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo"
        else
          # Amazon Linux 2
          YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo"
        fi

        sudo yum-config-manager --add-repo "${YUM_REPO_URL}"
        sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2
        sudo systemctl restart docker
    )
}

install_nvidia_docker2_ubuntu20() {
    (
        set -x
        # Install nvidia-driver package if not installed
        status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)"
        if [ ! $? = 0 ] || [ ! "$status" = installed ]; then
          sudo apt-get install -y nvidia-docker2 nvidia-container-toolkit-1.16.2
          sudo systemctl restart docker
        fi
    )
}

pre_install_nvidia_driver_amzn2() {
    (
        # Purge any nvidia driver installed from RHEL repo
        sudo yum remove -y nvidia-driver-latest-dkms
    )
}

install_nvidia_driver_common() {
    (
        # Try to gather more information about the runner and its existing NVIDIA driver if any
        echo "Before installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        HAS_NVIDIA_DRIVER=0
        # Check if NVIDIA driver has already been installed
        if [ -x "$(command -v nvidia-smi)" ]; then
            set +e
            # The driver exists, check its version next. Also check only the first GPU if there are more than one of them
            # so that the same driver version is not print over multiple lines
            INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
            NVIDIA_SMI_STATUS=$?

            if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing"
            elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing"
            else
                HAS_NVIDIA_DRIVER=1
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation"
            fi
            set -e
        fi

        if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then
            # CAUTION: this may need to be updated in future
            if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then
                  sudo yum groupinstall -y "Development Tools"
                  # ensure our kernel install is the same as our underlying kernel,
                  # groupinstall "Development Tools" has a habit of mismatching kernel headers
                  sudo yum install -y "kernel-devel-uname-r == $(uname -r)"
                  sudo modprobe backlight
            fi
            sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN"

            set +e
            sudo /bin/bash /tmp/nvidia_driver -s --no-drm
            NVIDIA_INSTALLATION_STATUS=$?

            RESET_GPU=0
            if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then
                sudo cat /var/log/nvidia-installer.log
                # Fail to install NVIDIA driver, try to reset the GPU
                RESET_GPU=1
            elif [ -x "$(command -v nvidia-smi)" ]; then
                # Check again if nvidia-smi works even if the driver installation completes successfully
                INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
                NVIDIA_SMI_STATUS=$?

                if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                    RESET_GPU=1
                fi
            fi

            if [ "$RESET_GPU" -eq 1 ]; then
                NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1)
                # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this
                # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388
                for PCI_ID in $NVIDIA_DEVICES; do
                    DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable)

                    echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)"
                    # This requires sudo permission of course
                    echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset
                    sleep 1
                done
            fi

            sudo rm -fv /tmp/nvidia_driver
            set -e
        fi
    )
}

post_install_nvidia_driver_common() {
    (
        sudo modprobe nvidia || true
        echo "After installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        (
            set +e

            nvidia-smi
            # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in
            # the case where the driver has already crashed as it still can get the driver version
            # and some basic information like the bus ID.  However, the rest of the information
            # would be missing (ERR!), for example:
            #
            # +-----------------------------------------------------------------------------+
            # | NVIDIA-SMI 525.89.02    Driver Version: 525.89.02    CUDA Version: 12.0     |
            # |-------------------------------+----------------------+----------------------+
            # | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
            # | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
            # |                               |                      |               MIG M. |
            # |===============================+======================+======================|
            # |   0  ERR!                Off  | 00000000:00:1E.0 Off |                 ERR! |
            # |ERR!  ERR! ERR!    ERR! / ERR! |   4184MiB / 23028MiB |    ERR!      Default |
            # |                               |                      |                 ERR! |
            # +-------------------------------+----------------------+----------------------+
            #
            # +-----------------------------------------------------------------------------+
            # | Processes:                                                                  |
            # |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
            # |        ID   ID                                                   Usage      |
            # |=============================================================================|
            # +-----------------------------------------------------------------------------+
            #
            # This should be reported as a failure instead as it will guarantee to fail when
            # Docker tries to run with --gpus all
            #
            # So, the correct check here is to query one of the missing piece of info like
            # GPU name, so that the command can fail accordingly
            nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0
            NVIDIA_SMI_STATUS=$?

            # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285
            if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then
                echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}"
            else
                echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}"
                exit ${NVIDIA_SMI_STATUS}
            fi
            set -e
        )
    )
}

install_nvidia_driver_amzn2() {
    (
        set -x
        pre_install_nvidia_driver_amzn2
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

install_nvidia_driver_ubuntu20() {
    (
        set -x
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

echo "== Installing nvidia driver ${DRIVER_FN} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_driver_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_driver_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac

# Install container toolkit based on distribution
echo "== Installing nvidia container toolkit for ${DISTRIBUTION} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_docker2_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_docker2_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac
echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}"

# Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with
# more than one GPUs. This just needs to be run once. The command fails
# on subsequent runs and complains that the mode is already on, but that's
# ok
sudo nvidia-persistenced || true
# This should show persistence mode ON
nvidia-smi

2024-12-18T00:27:55.1929332Z   retry_wait_seconds: 10
2024-12-18T00:27:55.1929618Z   polling_interval_seconds: 1
2024-12-18T00:27:55.1929910Z   warning_on_retry: true
2024-12-18T00:27:55.1930182Z   continue_on_error: false
2024-12-18T00:27:55.1930441Z env:
2024-12-18T00:27:55.1930658Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:27:55.1930934Z   DRIVER_VERSION: 550.54.15
2024-12-18T00:27:55.1931207Z ##[endgroup]
2024-12-18T00:27:55.2958247Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run ==
2024-12-18T00:27:55.2958874Z + pre_install_nvidia_driver_amzn2
2024-12-18T00:27:55.2962635Z + sudo yum remove -y nvidia-driver-latest-dkms
2024-12-18T00:27:56.1617724Z No match for argument: nvidia-driver-latest-dkms
2024-12-18T00:27:56.1618148Z No packages marked for removal.
2024-12-18T00:27:56.2107216Z Dependencies resolved.
2024-12-18T00:27:56.2117240Z Nothing to do.
2024-12-18T00:27:56.2117638Z Complete!
2024-12-18T00:27:56.2753940Z + install_nvidia_driver_common
2024-12-18T00:27:56.2757331Z + echo 'Before installing NVIDIA driver'
2024-12-18T00:27:56.2758412Z Before installing NVIDIA driver
2024-12-18T00:27:56.2761048Z + lspci
2024-12-18T00:27:56.2902562Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma]
2024-12-18T00:27:56.2903525Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2024-12-18T00:27:56.2904325Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
2024-12-18T00:27:56.2905027Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111
2024-12-18T00:27:56.2905615Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller
2024-12-18T00:27:56.2906162Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2024-12-18T00:27:56.2906681Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1)
2024-12-18T00:27:56.2907186Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller
2024-12-18T00:27:56.2908681Z + lsmod
2024-12-18T00:27:56.2958603Z Module                  Size  Used by
2024-12-18T00:27:56.2958912Z xt_nat                 16384  0
2024-12-18T00:27:56.2959207Z ib_core               454656  0
2024-12-18T00:27:56.2959573Z veth                   36864  0
2024-12-18T00:27:56.2959846Z nvidia_modeset       1351680  0
2024-12-18T00:27:56.2960143Z video                  65536  1 nvidia_modeset
2024-12-18T00:27:56.2960457Z wmi                    36864  1 video
2024-12-18T00:27:56.2960744Z nvidia_uvm           4706304  0
2024-12-18T00:27:56.2961052Z nvidia              54071296  7 nvidia_uvm,nvidia_modeset
2024-12-18T00:27:56.2961394Z drm                   602112  1 nvidia
2024-12-18T00:27:56.2961716Z drm_panel_orientation_quirks    32768  1 drm
2024-12-18T00:27:56.2962095Z backlight              24576  3 video,drm,nvidia_modeset
2024-12-18T00:27:56.2962458Z i2c_core              106496  2 nvidia,drm
2024-12-18T00:27:56.2962757Z xt_conntrack           16384  1
2024-12-18T00:27:56.2963040Z nft_chain_nat          16384  3
2024-12-18T00:27:56.2963318Z xt_MASQUERADE          20480  1
2024-12-18T00:27:56.2963833Z nf_nat                 57344  3 xt_nat,nft_chain_nat,xt_MASQUERADE
2024-12-18T00:27:56.2964194Z nf_conntrack_netlink    57344  0
2024-12-18T00:27:56.2964633Z nf_conntrack          184320  5 xt_conntrack,nf_nat,xt_nat,nf_conntrack_netlink,xt_MASQUERADE
2024-12-18T00:27:56.2965112Z nf_defrag_ipv6         24576  1 nf_conntrack
2024-12-18T00:27:56.2965439Z nf_defrag_ipv4         16384  1 nf_conntrack
2024-12-18T00:27:56.2965751Z xfrm_user              57344  1
2024-12-18T00:27:56.2966031Z xfrm_algo              16384  1 xfrm_user
2024-12-18T00:27:56.2966329Z xt_addrtype            16384  2
2024-12-18T00:27:56.2966601Z nft_compat             20480  4
2024-12-18T00:27:56.2966922Z nf_tables             311296  57 nft_compat,nft_chain_nat
2024-12-18T00:27:56.2967352Z nfnetlink              20480  4 nft_compat,nf_conntrack_netlink,nf_tables
2024-12-18T00:27:56.2967748Z br_netfilter           36864  0
2024-12-18T00:27:56.2968043Z bridge                323584  1 br_netfilter
2024-12-18T00:27:56.2968361Z stp                    16384  1 bridge
2024-12-18T00:27:56.2968664Z llc                    16384  2 bridge,stp
2024-12-18T00:27:56.2968965Z overlay               167936  0
2024-12-18T00:27:56.2969234Z tls                   114688  0
2024-12-18T00:27:56.2969496Z sunrpc                692224  1
2024-12-18T00:27:56.2969762Z nls_ascii              16384  1
2024-12-18T00:27:56.2970029Z nls_cp437              20480  1
2024-12-18T00:27:56.2970292Z vfat                   24576  1
2024-12-18T00:27:56.2970557Z fat                    86016  1 vfat
2024-12-18T00:27:56.2970829Z ena                   167936  0
2024-12-18T00:27:56.2971086Z i8042                  45056  0
2024-12-18T00:27:56.2971351Z serio                  28672  3 i8042
2024-12-18T00:27:56.2971646Z ghash_clmulni_intel    16384  0
2024-12-18T00:27:56.2971930Z button                 24576  0
2024-12-18T00:27:56.2972192Z sch_fq_codel           20480  17
2024-12-18T00:27:56.2972472Z dm_mod                188416  0
2024-12-18T00:27:56.2972738Z fuse                  163840  1
2024-12-18T00:27:56.2973000Z loop                   36864  0
2024-12-18T00:27:56.2973266Z configfs               57344  1
2024-12-18T00:27:56.2973529Z dax                    45056  1 dm_mod
2024-12-18T00:27:56.2973816Z dmi_sysfs              20480  0
2024-12-18T00:27:56.2974084Z crc32_pclmul           16384  0
2024-12-18T00:27:56.2974483Z crc32c_intel           24576  0
2024-12-18T00:27:56.2974753Z efivarfs               24576  1
2024-12-18T00:27:56.2975015Z + modinfo nvidia
2024-12-18T00:27:56.2978576Z filename:       /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko
2024-12-18T00:27:56.2979065Z alias:          char-major-195-*
2024-12-18T00:27:56.2979347Z version:        550.54.15
2024-12-18T00:27:56.2979608Z supported:      external
2024-12-18T00:27:56.2979862Z license:        NVIDIA
2024-12-18T00:27:56.2980139Z firmware:       nvidia/550.54.15/gsp_tu10x.bin
2024-12-18T00:27:56.2980664Z firmware:       nvidia/550.54.15/gsp_ga10x.bin
2024-12-18T00:27:56.2981011Z srcversion:     833721318DA517F0C2FEC97
2024-12-18T00:27:56.2981344Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2024-12-18T00:27:56.2981702Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2024-12-18T00:27:56.2982047Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2024-12-18T00:27:56.2982373Z depends:        i2c-core,drm
2024-12-18T00:27:56.2982645Z retpoline:      Y
2024-12-18T00:27:56.2982879Z name:           nvidia
2024-12-18T00:27:56.2983257Z vermagic:       6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 
2024-12-18T00:27:56.2983751Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2024-12-18T00:27:56.2984216Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2024-12-18T00:27:56.2984665Z parm:           NVreg_ResmanDebugLevel:int
2024-12-18T00:27:56.2984990Z parm:           NVreg_RmLogonRC:int
2024-12-18T00:27:56.2985312Z parm:           NVreg_ModifyDeviceFiles:int
2024-12-18T00:27:56.2985640Z parm:           NVreg_DeviceFileUID:int
2024-12-18T00:27:56.2986189Z parm:           NVreg_DeviceFileGID:int
2024-12-18T00:27:56.2986522Z parm:           NVreg_DeviceFileMode:int
2024-12-18T00:27:56.2986906Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2024-12-18T00:27:56.2987318Z parm:           NVreg_UsePageAttributeTable:int
2024-12-18T00:27:56.2987673Z parm:           NVreg_EnablePCIeGen3:int
2024-12-18T00:27:56.2987986Z parm:           NVreg_EnableMSI:int
2024-12-18T00:27:56.2988294Z parm:           NVreg_TCEBypassMode:int
2024-12-18T00:27:56.2988631Z parm:           NVreg_EnableStreamMemOPs:int
2024-12-18T00:27:56.2989013Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2024-12-18T00:27:56.2989440Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2024-12-18T00:27:56.2989840Z parm:           NVreg_EnableS0ixPowerManagement:int
2024-12-18T00:27:56.2990279Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2024-12-18T00:27:56.2990721Z parm:           NVreg_DynamicPowerManagement:int
2024-12-18T00:27:56.2991173Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2024-12-18T00:27:56.2991609Z parm:           NVreg_EnableGpuFirmware:int
2024-12-18T00:27:56.2992020Z parm:           NVreg_EnableGpuFirmwareLogs:int
2024-12-18T00:27:56.2992562Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2024-12-18T00:27:56.2993036Z parm:           NVreg_EnableUserNUMAManagement:int
2024-12-18T00:27:56.2993398Z parm:           NVreg_MemoryPoolSize:int
2024-12-18T00:27:56.2993740Z parm:           NVreg_KMallocHeapMaxSize:int
2024-12-18T00:27:56.2994082Z parm:           NVreg_VMallocHeapMaxSize:int
2024-12-18T00:27:56.2994424Z parm:           NVreg_IgnoreMMIOCheck:int
2024-12-18T00:27:56.2994762Z parm:           NVreg_NvLinkDisable:int
2024-12-18T00:27:56.2995129Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2024-12-18T00:27:56.2995513Z parm:           NVreg_RegisterPCIDriver:int
2024-12-18T00:27:56.2995857Z parm:           NVreg_EnableResizableBar:int
2024-12-18T00:27:56.2996215Z parm:           NVreg_EnableDbgBreakpoint:int
2024-12-18T00:27:56.2996582Z parm:           NVreg_EnableNonblockingOpen:int
2024-12-18T00:27:56.2997091Z parm:           NVreg_RegistryDwords:charp
2024-12-18T00:27:56.2997458Z parm:           NVreg_RegistryDwordsPerDevice:charp
2024-12-18T00:27:56.2997807Z parm:           NVreg_RmMsg:charp
2024-12-18T00:27:56.2998108Z parm:           NVreg_GpuBlacklist:charp
2024-12-18T00:27:56.2998449Z parm:           NVreg_TemporaryFilePath:charp
2024-12-18T00:27:56.2998799Z parm:           NVreg_ExcludedGpus:charp
2024-12-18T00:27:56.2999136Z parm:           NVreg_DmaRemapPeerMmio:int
2024-12-18T00:27:56.2999634Z parm:           NVreg_RmNvlinkBandwidth:charp
2024-12-18T00:27:56.2999981Z parm:           NVreg_ImexChannelCount:int
2024-12-18T00:27:56.3000315Z parm:           rm_firmware_active:charp
2024-12-18T00:27:56.3000637Z + HAS_NVIDIA_DRIVER=0
2024-12-18T00:27:56.3000896Z ++ command -v nvidia-smi
2024-12-18T00:27:56.3001299Z + '[' -x /usr/bin/nvidia-smi ']'
2024-12-18T00:27:56.3001577Z + set +e
2024-12-18T00:27:56.3001907Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0
2024-12-18T00:27:56.3269279Z + INSTALLED_DRIVER_VERSION=550.54.15
2024-12-18T00:27:56.3269594Z + NVIDIA_SMI_STATUS=0
2024-12-18T00:27:56.3269847Z + '[' 0 -ne 0 ']'
2024-12-18T00:27:56.3270073Z + '[' 550.54.15 '!=' 550.54.15 ']'
2024-12-18T00:27:56.3270352Z + HAS_NVIDIA_DRIVER=1
2024-12-18T00:27:56.3271032Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation'
2024-12-18T00:27:56.3271998Z + set -e
2024-12-18T00:27:56.3272409Z + '[' 1 -eq 0 ']'
2024-12-18T00:27:56.3273199Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation
2024-12-18T00:27:56.3274156Z + post_install_nvidia_driver_common
2024-12-18T00:27:56.3276936Z + sudo modprobe nvidia
2024-12-18T00:27:56.4308233Z + echo 'After installing NVIDIA driver'
2024-12-18T00:27:56.4308583Z + lspci
2024-12-18T00:27:56.4308828Z After installing NVIDIA driver
2024-12-18T00:27:56.4428069Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma]
2024-12-18T00:27:56.4428589Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2024-12-18T00:27:56.4429170Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
2024-12-18T00:27:56.4429710Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111
2024-12-18T00:27:56.4430217Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller
2024-12-18T00:27:56.4430779Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2024-12-18T00:27:56.4431293Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1)
2024-12-18T00:27:56.4431798Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller
2024-12-18T00:27:56.4432232Z + lsmod
2024-12-18T00:27:56.4464431Z Module                  Size  Used by
2024-12-18T00:27:56.4464758Z xt_nat                 16384  0
2024-12-18T00:27:56.4465045Z ib_core               454656  0
2024-12-18T00:27:56.4465310Z veth                   36864  0
2024-12-18T00:27:56.4465591Z nvidia_modeset       1351680  0
2024-12-18T00:27:56.4465899Z video                  65536  1 nvidia_modeset
2024-12-18T00:27:56.4466219Z wmi                    36864  1 video
2024-12-18T00:27:56.4466510Z nvidia_uvm           4706304  0
2024-12-18T00:27:56.4466829Z nvidia              54071296  7 nvidia_uvm,nvidia_modeset
2024-12-18T00:27:56.4467178Z drm                   602112  1 nvidia
2024-12-18T00:27:56.4467504Z drm_panel_orientation_quirks    32768  1 drm
2024-12-18T00:27:56.4467888Z backlight              24576  3 video,drm,nvidia_modeset
2024-12-18T00:27:56.4468380Z i2c_core              106496  2 nvidia,drm
2024-12-18T00:27:56.4468685Z xt_conntrack           16384  1
2024-12-18T00:27:56.4468965Z nft_chain_nat          16384  3
2024-12-18T00:27:56.4469248Z xt_MASQUERADE          20480  1
2024-12-18T00:27:56.4469587Z nf_nat                 57344  3 xt_nat,nft_chain_nat,xt_MASQUERADE
2024-12-18T00:27:56.4469954Z nf_conntrack_netlink    57344  0
2024-12-18T00:27:56.4470398Z nf_conntrack          184320  5 xt_conntrack,nf_nat,xt_nat,nf_conntrack_netlink,xt_MASQUERADE
2024-12-18T00:27:56.4470879Z nf_defrag_ipv6         24576  1 nf_conntrack
2024-12-18T00:27:56.4471211Z nf_defrag_ipv4         16384  1 nf_conntrack
2024-12-18T00:27:56.4471522Z xfrm_user              57344  1
2024-12-18T00:27:56.4471808Z xfrm_algo              16384  1 xfrm_user
2024-12-18T00:27:56.4472119Z xt_addrtype            16384  2
2024-12-18T00:27:56.4472392Z nft_compat             20480  4
2024-12-18T00:27:56.4472719Z nf_tables             311296  57 nft_compat,nft_chain_nat
2024-12-18T00:27:56.4473155Z nfnetlink              20480  4 nft_compat,nf_conntrack_netlink,nf_tables
2024-12-18T00:27:56.4473549Z br_netfilter           36864  0
2024-12-18T00:27:56.4473845Z bridge                323584  1 br_netfilter
2024-12-18T00:27:56.4474353Z stp                    16384  1 bridge
2024-12-18T00:27:56.4474666Z llc                    16384  2 bridge,stp
2024-12-18T00:27:56.4474976Z overlay               167936  0
2024-12-18T00:27:56.4475248Z tls                   114688  0
2024-12-18T00:27:56.4475516Z sunrpc                692224  1
2024-12-18T00:27:56.4475781Z nls_ascii              16384  1
2024-12-18T00:27:56.4476055Z nls_cp437              20480  1
2024-12-18T00:27:56.4476325Z vfat                   24576  1
2024-12-18T00:27:56.4476594Z fat                    86016  1 vfat
2024-12-18T00:27:56.4476885Z ena                   167936  0
2024-12-18T00:27:56.4477144Z i8042                  45056  0
2024-12-18T00:27:56.4477415Z serio                  28672  3 i8042
2024-12-18T00:27:56.4477715Z ghash_clmulni_intel    16384  0
2024-12-18T00:27:56.4477999Z button                 24576  0
2024-12-18T00:27:56.4478268Z sch_fq_codel           20480  17
2024-12-18T00:27:56.4478548Z dm_mod                188416  0
2024-12-18T00:27:56.4478817Z fuse                  163840  1
2024-12-18T00:27:56.4479093Z loop                   36864  0
2024-12-18T00:27:56.4479655Z configfs               57344  1
2024-12-18T00:27:56.4479925Z dax                    45056  1 dm_mod
2024-12-18T00:27:56.4480320Z dmi_sysfs              20480  0
2024-12-18T00:27:56.4480599Z crc32_pclmul           16384  0
2024-12-18T00:27:56.4480876Z crc32c_intel           24576  0
2024-12-18T00:27:56.4481154Z efivarfs               24576  1
2024-12-18T00:27:56.4481424Z + modinfo nvidia
2024-12-18T00:27:56.4482069Z filename:       /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko
2024-12-18T00:27:56.4482558Z alias:          char-major-195-*
2024-12-18T00:27:56.4482847Z version:        550.54.15
2024-12-18T00:27:56.4483115Z supported:      external
2024-12-18T00:27:56.4483377Z license:        NVIDIA
2024-12-18T00:27:56.4483657Z firmware:       nvidia/550.54.15/gsp_tu10x.bin
2024-12-18T00:27:56.4484015Z firmware:       nvidia/550.54.15/gsp_ga10x.bin
2024-12-18T00:27:56.4484360Z srcversion:     833721318DA517F0C2FEC97
2024-12-18T00:27:56.4484698Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2024-12-18T00:27:56.4485064Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2024-12-18T00:27:56.4485418Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2024-12-18T00:27:56.4485752Z depends:        i2c-core,drm
2024-12-18T00:27:56.4486028Z retpoline:      Y
2024-12-18T00:27:56.4486288Z name:           nvidia
2024-12-18T00:27:56.4486727Z vermagic:       6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 
2024-12-18T00:27:56.4487421Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2024-12-18T00:27:56.4488080Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2024-12-18T00:27:56.4488693Z parm:           NVreg_ResmanDebugLevel:int
2024-12-18T00:27:56.4489029Z parm:           NVreg_RmLogonRC:int
2024-12-18T00:27:56.4489362Z parm:           NVreg_ModifyDeviceFiles:int
2024-12-18T00:27:56.4489701Z parm:           NVreg_DeviceFileUID:int
2024-12-18T00:27:56.4490036Z parm:           NVreg_DeviceFileGID:int
2024-12-18T00:27:56.4490370Z parm:           NVreg_DeviceFileMode:int
2024-12-18T00:27:56.4490764Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2024-12-18T00:27:56.4491233Z parm:           NVreg_UsePageAttributeTable:int
2024-12-18T00:27:56.4491586Z parm:           NVreg_EnablePCIeGen3:int
2024-12-18T00:27:56.4491914Z parm:           NVreg_EnableMSI:int
2024-12-18T00:27:56.4492231Z parm:           NVreg_TCEBypassMode:int
2024-12-18T00:27:56.4492576Z parm:           NVreg_EnableStreamMemOPs:int
2024-12-18T00:27:56.4492968Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2024-12-18T00:27:56.4493390Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2024-12-18T00:27:56.4493797Z parm:           NVreg_EnableS0ixPowerManagement:int
2024-12-18T00:27:56.4494236Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2024-12-18T00:27:56.4494674Z parm:           NVreg_DynamicPowerManagement:int
2024-12-18T00:27:56.4495250Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2024-12-18T00:27:56.4495698Z parm:           NVreg_EnableGpuFirmware:int
2024-12-18T00:27:56.4496058Z parm:           NVreg_EnableGpuFirmwareLogs:int
2024-12-18T00:27:56.4496458Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2024-12-18T00:27:56.4496863Z parm:           NVreg_EnableUserNUMAManagement:int
2024-12-18T00:27:56.4497230Z parm:           NVreg_MemoryPoolSize:int
2024-12-18T00:27:56.4497577Z parm:           NVreg_KMallocHeapMaxSize:int
2024-12-18T00:27:56.4497931Z parm:           NVreg_VMallocHeapMaxSize:int
2024-12-18T00:27:56.4498282Z parm:           NVreg_IgnoreMMIOCheck:int
2024-12-18T00:27:56.4498619Z parm:           NVreg_NvLinkDisable:int
2024-12-18T00:27:56.4498995Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2024-12-18T00:27:56.4499385Z parm:           NVreg_RegisterPCIDriver:int
2024-12-18T00:27:56.4499734Z parm:           NVreg_EnableResizableBar:int
2024-12-18T00:27:56.4500093Z parm:           NVreg_EnableDbgBreakpoint:int
2024-12-18T00:27:56.4500471Z parm:           NVreg_EnableNonblockingOpen:int
2024-12-18T00:27:56.4500929Z parm:           NVreg_RegistryDwords:charp
2024-12-18T00:27:56.4501296Z parm:           NVreg_RegistryDwordsPerDevice:charp
2024-12-18T00:27:56.4501647Z parm:           NVreg_RmMsg:charp
2024-12-18T00:27:56.4501962Z parm:           NVreg_GpuBlacklist:charp
2024-12-18T00:27:56.4502309Z parm:           NVreg_TemporaryFilePath:charp
2024-12-18T00:27:56.4502664Z parm:           NVreg_ExcludedGpus:charp
2024-12-18T00:27:56.4503004Z parm:           NVreg_DmaRemapPeerMmio:int
2024-12-18T00:27:56.4503657Z parm:           NVreg_RmNvlinkBandwidth:charp
2024-12-18T00:27:56.4504019Z parm:           NVreg_ImexChannelCount:int
2024-12-18T00:27:56.4504357Z parm:           rm_firmware_active:charp
2024-12-18T00:27:56.4504671Z + set +e
2024-12-18T00:27:56.4504887Z + nvidia-smi
2024-12-18T00:27:56.4698222Z Wed Dec 18 00:27:56 2024       
2024-12-18T00:27:56.4698723Z +-----------------------------------------------------------------------------------------+
2024-12-18T00:27:56.4699268Z | NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
2024-12-18T00:27:56.4699783Z |-----------------------------------------+------------------------+----------------------+
2024-12-18T00:27:56.4700304Z | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
2024-12-18T00:27:56.4700871Z | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
2024-12-18T00:27:56.4701340Z |                                         |                        |               MIG M. |
2024-12-18T00:27:56.4701692Z |=========================================+========================+======================|
2024-12-18T00:27:56.4937335Z |   0  NVIDIA A10G                    On  |   00000000:00:1E.0 Off |                    0 |
2024-12-18T00:27:56.4937897Z |  0%   22C    P8             15W /  300W |       0MiB /  23028MiB |      0%      Default |
2024-12-18T00:27:56.4938333Z |                                         |                        |                  N/A |
2024-12-18T00:27:56.4938755Z +-----------------------------------------+------------------------+----------------------+
2024-12-18T00:27:56.4941723Z                                                                                          
2024-12-18T00:27:56.4942246Z +-----------------------------------------------------------------------------------------+
2024-12-18T00:27:56.4942697Z | Processes:                                                                              |
2024-12-18T00:27:56.4943176Z |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
2024-12-18T00:27:56.4943625Z |        ID   ID                                                               Usage      |
2024-12-18T00:27:56.4943983Z |=========================================================================================|
2024-12-18T00:27:56.4948806Z |  No running processes found                                                             |
2024-12-18T00:27:56.4949321Z +-----------------------------------------------------------------------------------------+
2024-12-18T00:27:56.7846191Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0
2024-12-18T00:27:56.8031880Z NVIDIA A10G
2024-12-18T00:27:56.8094376Z + NVIDIA_SMI_STATUS=0
2024-12-18T00:27:56.8094640Z + '[' 0 -eq 0 ']'
2024-12-18T00:27:56.8094968Z + echo 'INFO: Ignoring allowed status 0'
2024-12-18T00:27:56.8095309Z + set -e
2024-12-18T00:27:56.8095537Z INFO: Ignoring allowed status 0
2024-12-18T00:27:56.8105137Z == Installing nvidia container toolkit for amzn2023 ==
2024-12-18T00:27:56.8108965Z + sudo yum install -y yum-utils
2024-12-18T00:27:57.4042868Z Last metadata expiration check: 2:22:02 ago on Tue Dec 17 22:05:55 2024.
2024-12-18T00:27:57.4265895Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed.
2024-12-18T00:27:57.4596441Z Dependencies resolved.
2024-12-18T00:27:57.4735985Z Nothing to do.
2024-12-18T00:27:57.4736374Z Complete!
2024-12-18T00:27:57.5508647Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]]
2024-12-18T00:27:57.5509690Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
2024-12-18T00:27:57.5510591Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
2024-12-18T00:27:57.7837129Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
2024-12-18T00:27:57.8646062Z + sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2
2024-12-18T00:27:58.3973103Z nvidia-container-toolkit                         11 kB/s | 833  B     00:00    
2024-12-18T00:27:58.4206988Z Package nvidia-docker2-2.14.0-1.noarch is already installed.
2024-12-18T00:27:58.4212869Z Package nvidia-container-toolkit-1.16.2-1.x86_64 is already installed.
2024-12-18T00:27:58.4536869Z Dependencies resolved.
2024-12-18T00:27:58.4676079Z Nothing to do.
2024-12-18T00:27:58.4676673Z Complete!
2024-12-18T00:27:58.5756947Z + sudo systemctl restart docker
2024-12-18T00:28:58.0975051Z nvidia-persistenced failed to initialize. Check syslog for more details.
2024-12-18T00:28:58.1216472Z Wed Dec 18 00:28:58 2024       
2024-12-18T00:28:58.1220308Z +-----------------------------------------------------------------------------------------+
2024-12-18T00:28:58.1221020Z | NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
2024-12-18T00:28:58.1221528Z |-----------------------------------------+------------------------+----------------------+
2024-12-18T00:28:58.1222053Z | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
2024-12-18T00:28:58.1222615Z | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
2024-12-18T00:28:58.1223082Z |                                         |                        |               MIG M. |
2024-12-18T00:28:58.1223462Z |=========================================+========================+======================|
2024-12-18T00:28:58.1429664Z |   0  NVIDIA A10G                    On  |   00000000:00:1E.0 Off |                    0 |
2024-12-18T00:28:58.1430153Z |  0%   21C    P8             15W /  300W |       0MiB /  23028MiB |      0%      Default |
2024-12-18T00:28:58.1430555Z |                                         |                        |                  N/A |
2024-12-18T00:28:58.1430963Z +-----------------------------------------+------------------------+----------------------+
2024-12-18T00:28:58.1434371Z                                                                                          
2024-12-18T00:28:58.1434857Z +-----------------------------------------------------------------------------------------+
2024-12-18T00:28:58.1435307Z | Processes:                                                                              |
2024-12-18T00:28:58.1435792Z |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
2024-12-18T00:28:58.1437882Z |        ID   ID                                                               Usage      |
2024-12-18T00:28:58.1438445Z |=========================================================================================|
2024-12-18T00:28:58.1441830Z |  No running processes found                                                             |
2024-12-18T00:28:58.1442420Z +-----------------------------------------------------------------------------------------+
2024-12-18T00:28:59.3385179Z Command completed after 1 attempt(s).
2024-12-18T00:28:59.3481595Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84
2024-12-18T00:28:59.3482167Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84
2024-12-18T00:28:59.3482665Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2024-12-18T00:28:59.3483134Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}"
2024-12-18T00:28:59.3497505Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:28:59.3497876Z env:
2024-12-18T00:28:59.3498303Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:28:59.3498642Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:28:59.3498991Z ##[endgroup]
2024-12-18T00:28:59.5971082Z Defaulting to user installation because normal site-packages is not writeable
2024-12-18T00:28:59.6141391Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.9/site-packages (5.9.1)
2024-12-18T00:28:59.6147545Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.9/site-packages (11.525.84)
2024-12-18T00:28:59.7519582Z Prepare all required actions
2024-12-18T00:28:59.7520462Z Getting action download info
2024-12-18T00:28:59.9495541Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6)
2024-12-18T00:29:00.1976981Z Download action repository 'actions/download-artifact@v4' (SHA:fa0a91b85d4f404e444e00e005971372dc801d16)
2024-12-18T00:29:00.5973197Z ##[group]Run ./.github/actions/download-build-artifacts
2024-12-18T00:29:00.5973580Z with:
2024-12-18T00:29:00.5973842Z   name: linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-12-18T00:29:00.5974179Z   s3-bucket: gha-artifacts
2024-12-18T00:29:00.5974514Z env:
2024-12-18T00:29:00.5974738Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:00.5975080Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:00.5975435Z ##[endgroup]
2024-12-18T00:29:00.6009738Z ##[group]Run seemethere/download-artifact-s3@v4
2024-12-18T00:29:00.6010076Z with:
2024-12-18T00:29:00.6010363Z   name: linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-12-18T00:29:00.6010697Z   s3-bucket: gha-artifacts
2024-12-18T00:29:00.6010960Z   region: us-east-1
2024-12-18T00:29:00.6011186Z env:
2024-12-18T00:29:00.6011404Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:00.6011739Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:00.6012091Z ##[endgroup]
2024-12-18T00:29:01.0972350Z (node:341304) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2024-12-18T00:29:01.0972876Z 
2024-12-18T00:29:01.0973075Z Please migrate your code to use AWS SDK for JavaScript (v3).
2024-12-18T00:29:01.0973617Z For more information, check the migration guide at https://a.co/7PzMCcy
2024-12-18T00:29:01.0974174Z (Use `node --trace-warnings ...` to show where the warning was created)
2024-12-18T00:29:01.1998655Z Found 1 objects with prefix pytorch/pytorch/12383255690/linux-focal-cuda12.4-py3.10-gcc9-sm86/
2024-12-18T00:29:01.1999593Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2024-12-18T00:29:08.9811802Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2024-12-18T00:29:08.9816824Z Artifact download has finished successfully
2024-12-18T00:29:09.0181587Z ##[group]Run unzip -o artifacts.zip
2024-12-18T00:29:09.0181940Z unzip -o artifacts.zip
2024-12-18T00:29:09.0191373Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:09.0191769Z env:
2024-12-18T00:29:09.0192000Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:09.0192348Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:09.0192713Z ##[endgroup]
2024-12-18T00:29:09.0250041Z Archive:  artifacts.zip
2024-12-18T00:29:09.0251565Z    creating: dist/
2024-12-18T00:29:11.2092765Z   inflating: dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl  
2024-12-18T00:29:11.2093273Z    creating: build/custom_test_artifacts/
2024-12-18T00:29:11.2093678Z    creating: build/custom_test_artifacts/custom-op-build/
2024-12-18T00:29:11.2094198Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2024-12-18T00:29:11.2094781Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/
2024-12-18T00:29:11.2102421Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml  
2024-12-18T00:29:11.2103082Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/
2024-12-18T00:29:11.2104110Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2024-12-18T00:29:11.2105097Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/
2024-12-18T00:29:11.2105780Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-12-18T00:29:11.2107935Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2024-12-18T00:29:11.2109612Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2024-12-18T00:29:11.2110334Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-12-18T00:29:11.2111039Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-12-18T00:29:11.2113396Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2024-12-18T00:29:11.2114858Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2024-12-18T00:29:11.2117662Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2024-12-18T00:29:11.2118562Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2024-12-18T00:29:11.2120694Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2024-12-18T00:29:11.2121853Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2024-12-18T00:29:11.2122596Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2024-12-18T00:29:11.2123304Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2024-12-18T00:29:11.2165161Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2024-12-18T00:29:11.2206647Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2024-12-18T00:29:11.2207851Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2024-12-18T00:29:11.2254854Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2024-12-18T00:29:11.2255865Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2024-12-18T00:29:11.2257115Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2024-12-18T00:29:11.2258183Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2024-12-18T00:29:11.2259195Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2024-12-18T00:29:11.2260196Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2024-12-18T00:29:11.2261188Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2024-12-18T00:29:11.2262174Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2024-12-18T00:29:11.2263165Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2024-12-18T00:29:11.2264096Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2024-12-18T00:29:11.2265254Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2024-12-18T00:29:11.2266354Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2024-12-18T00:29:11.2267543Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2024-12-18T00:29:11.2268515Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2024-12-18T00:29:11.2270025Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2024-12-18T00:29:11.2345891Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2024-12-18T00:29:11.2420454Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2024-12-18T00:29:11.2421689Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2024-12-18T00:29:11.2422715Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/
2024-12-18T00:29:11.2423614Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2024-12-18T00:29:11.2424559Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2024-12-18T00:29:11.2425529Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2024-12-18T00:29:11.2426635Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2024-12-18T00:29:11.2427854Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2024-12-18T00:29:11.2429031Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2024-12-18T00:29:11.2430112Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2024-12-18T00:29:11.2431233Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2024-12-18T00:29:11.2432353Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2024-12-18T00:29:11.2433478Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2024-12-18T00:29:11.2434654Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2024-12-18T00:29:11.2435782Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2024-12-18T00:29:11.2456197Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2024-12-18T00:29:11.2601652Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2024-12-18T00:29:11.2602732Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2024-12-18T00:29:11.2604263Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2024-12-18T00:29:11.2605555Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2024-12-18T00:29:11.2606810Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2024-12-18T00:29:11.2607976Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2024-12-18T00:29:11.2609170Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2024-12-18T00:29:11.2610392Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2024-12-18T00:29:11.2611574Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2024-12-18T00:29:11.2612787Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2024-12-18T00:29:11.2613986Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2024-12-18T00:29:11.2634736Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2024-12-18T00:29:11.2719433Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2024-12-18T00:29:11.2720692Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2024-12-18T00:29:11.2721820Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2024-12-18T00:29:11.2723054Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2024-12-18T00:29:11.2724018Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2024-12-18T00:29:11.2725217Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2024-12-18T00:29:11.2726158Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2024-12-18T00:29:11.2729904Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2024-12-18T00:29:11.2730905Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2024-12-18T00:29:11.2732080Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2024-12-18T00:29:11.2854079Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2024-12-18T00:29:11.2918280Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2024-12-18T00:29:11.2919024Z    creating: build/custom_test_artifacts/jit-hook-build/
2024-12-18T00:29:11.2919911Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2024-12-18T00:29:11.2920765Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/
2024-12-18T00:29:11.2928163Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml  
2024-12-18T00:29:11.2929108Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/
2024-12-18T00:29:11.2930018Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2024-12-18T00:29:11.2931043Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/
2024-12-18T00:29:11.2932038Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-12-18T00:29:11.2933875Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2024-12-18T00:29:11.2936401Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2024-12-18T00:29:11.2937421Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-12-18T00:29:11.2938447Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-12-18T00:29:11.2940546Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2024-12-18T00:29:11.2942701Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2024-12-18T00:29:11.2945437Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2024-12-18T00:29:11.2946676Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2024-12-18T00:29:11.2948591Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2024-12-18T00:29:11.2950332Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2024-12-18T00:29:11.2951380Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2024-12-18T00:29:11.2952438Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2024-12-18T00:29:11.2994895Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2024-12-18T00:29:11.3037305Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2024-12-18T00:29:11.3039075Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2024-12-18T00:29:11.3086904Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2024-12-18T00:29:11.3088386Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2024-12-18T00:29:11.3090083Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2024-12-18T00:29:11.3091631Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2024-12-18T00:29:11.3093095Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2024-12-18T00:29:11.3094553Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2024-12-18T00:29:11.3096023Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2024-12-18T00:29:11.3097458Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2024-12-18T00:29:11.3098853Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2024-12-18T00:29:11.3100196Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2024-12-18T00:29:11.3101501Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2024-12-18T00:29:11.3102777Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2024-12-18T00:29:11.3104288Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2024-12-18T00:29:11.3105529Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2024-12-18T00:29:11.3106903Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2024-12-18T00:29:11.3181169Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2024-12-18T00:29:11.3257627Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2024-12-18T00:29:11.3258836Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2024-12-18T00:29:11.3259846Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/
2024-12-18T00:29:11.3260708Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2024-12-18T00:29:11.3261626Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2024-12-18T00:29:11.3262590Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2024-12-18T00:29:11.3263699Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2024-12-18T00:29:11.3264947Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2024-12-18T00:29:11.3266142Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2024-12-18T00:29:11.3267311Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2024-12-18T00:29:11.3268476Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2024-12-18T00:29:11.3269641Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2024-12-18T00:29:11.3270999Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2024-12-18T00:29:11.3272177Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2024-12-18T00:29:11.3273333Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2024-12-18T00:29:11.3294088Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2024-12-18T00:29:11.3359581Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2024-12-18T00:29:11.3360818Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2024-12-18T00:29:11.3361928Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2024-12-18T00:29:11.3362935Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2024-12-18T00:29:11.3363885Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2024-12-18T00:29:11.3365291Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2024-12-18T00:29:11.3366622Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2024-12-18T00:29:11.3369203Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2024-12-18T00:29:11.3370040Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2024-12-18T00:29:11.3371219Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2024-12-18T00:29:11.3421818Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2024-12-18T00:29:11.3422355Z    creating: build/custom_test_artifacts/custom-backend-build/
2024-12-18T00:29:11.3422882Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2024-12-18T00:29:11.3423509Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/
2024-12-18T00:29:11.3431330Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml  
2024-12-18T00:29:11.3432038Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/
2024-12-18T00:29:11.3432732Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2024-12-18T00:29:11.3433467Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/
2024-12-18T00:29:11.3434202Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-12-18T00:29:11.3437077Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2024-12-18T00:29:11.3438693Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2024-12-18T00:29:11.3439528Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-12-18T00:29:11.3440280Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-12-18T00:29:11.3443449Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2024-12-18T00:29:11.3445021Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2024-12-18T00:29:11.3447533Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2024-12-18T00:29:11.3448396Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2024-12-18T00:29:11.3450663Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2024-12-18T00:29:11.3452137Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2024-12-18T00:29:11.3452926Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2024-12-18T00:29:11.3454035Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2024-12-18T00:29:11.3496024Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2024-12-18T00:29:11.3537792Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2024-12-18T00:29:11.3539293Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2024-12-18T00:29:11.3587963Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2024-12-18T00:29:11.3590026Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2024-12-18T00:29:11.3592150Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2024-12-18T00:29:11.3594305Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2024-12-18T00:29:11.3596391Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2024-12-18T00:29:11.3597677Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2024-12-18T00:29:11.3598718Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2024-12-18T00:29:11.3599820Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2024-12-18T00:29:11.3600820Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2024-12-18T00:29:11.3601776Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2024-12-18T00:29:11.3602707Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2024-12-18T00:29:11.3603772Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2024-12-18T00:29:11.3604688Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2024-12-18T00:29:11.3605591Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2024-12-18T00:29:11.3607080Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2024-12-18T00:29:11.3681732Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2024-12-18T00:29:11.3756771Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2024-12-18T00:29:11.3758085Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2024-12-18T00:29:11.3759315Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/
2024-12-18T00:29:11.3760288Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2024-12-18T00:29:11.3761294Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2024-12-18T00:29:11.3762351Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2024-12-18T00:29:11.3763548Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2024-12-18T00:29:11.3764869Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2024-12-18T00:29:11.3766471Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2024-12-18T00:29:11.3767682Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2024-12-18T00:29:11.3768915Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2024-12-18T00:29:11.3770373Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2024-12-18T00:29:11.3771695Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2024-12-18T00:29:11.3772933Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2024-12-18T00:29:11.3774135Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2024-12-18T00:29:11.3776005Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2024-12-18T00:29:11.3899970Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2024-12-18T00:29:11.3901234Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2024-12-18T00:29:11.3902481Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2024-12-18T00:29:11.3904226Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2024-12-18T00:29:11.3905594Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2024-12-18T00:29:11.3906853Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2024-12-18T00:29:11.3908160Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2024-12-18T00:29:11.3909468Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2024-12-18T00:29:11.3910844Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2024-12-18T00:29:11.3912188Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2024-12-18T00:29:11.3913460Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2024-12-18T00:29:11.3932783Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2024-12-18T00:29:11.3988940Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2024-12-18T00:29:11.3990335Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2024-12-18T00:29:11.3991548Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2024-12-18T00:29:11.3992646Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2024-12-18T00:29:11.3993671Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2024-12-18T00:29:11.3994689Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2024-12-18T00:29:11.3995715Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2024-12-18T00:29:11.3999482Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2024-12-18T00:29:11.4000392Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2024-12-18T00:29:11.4001666Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2024-12-18T00:29:11.4103678Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2024-12-18T00:29:11.4147620Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2024-12-18T00:29:11.4148345Z    creating: build/lib/
2024-12-18T00:29:11.4238050Z   inflating: build/lib/libprotobuf-lite.a  
2024-12-18T00:29:11.4695517Z   inflating: build/lib/libprotobuf.a  
2024-12-18T00:29:11.4706080Z   inflating: build/lib/libpthreadpool.a  
2024-12-18T00:29:11.4714781Z   inflating: build/lib/libcpuinfo.a  
2024-12-18T00:29:11.4723703Z   inflating: build/lib/libcpuinfo_internals.a  
2024-12-18T00:29:11.4724445Z   inflating: build/lib/libclog.a     
2024-12-18T00:29:11.4743555Z   inflating: build/lib/libnnpack.a   
2024-12-18T00:29:11.4746694Z   inflating: build/lib/libnnpack_reference_layers.a  
2024-12-18T00:29:11.4935601Z   inflating: build/lib/libmicrokernels-prod.a  
2024-12-18T00:29:11.5004981Z   inflating: build/lib/libgtest.a    
2024-12-18T00:29:11.5080411Z   inflating: build/lib/libbenchmark.a  
2024-12-18T00:29:11.5145069Z   inflating: build/lib/libasmjit.a   
2024-12-18T00:29:11.5153226Z   inflating: build/lib/libittnotify.a  
2024-12-18T00:29:11.5181661Z   inflating: build/lib/libtensorpipe_uv.a  
2024-12-18T00:29:11.5309887Z   inflating: build/lib/libgloo.a     
2024-12-18T00:29:11.5331159Z   inflating: build/lib/libfmt.a      
2024-12-18T00:29:11.5431193Z   inflating: build/lib/libc10.so     
2024-12-18T00:29:11.5433280Z   inflating: build/lib/libcaffe2_nvrtc.so  
2024-12-18T00:29:11.5434844Z   inflating: build/lib/libtorch_global_deps.so  
2024-12-18T00:29:11.5940674Z   inflating: build/lib/libprotoc.a   
2024-12-18T00:29:11.5960415Z   inflating: build/lib/libpytorch_qnnpack.a  
2024-12-18T00:29:11.5978034Z   inflating: build/lib/libgmock.a    
2024-12-18T00:29:11.5978857Z   inflating: build/lib/libgtest_main.a  
2024-12-18T00:29:11.5980135Z   inflating: build/lib/libbenchmark_main.a  
2024-12-18T00:29:11.6556524Z   inflating: build/lib/libtensorpipe.a  
2024-12-18T00:29:11.6946483Z   inflating: build/lib/libgloo_cuda.a  
2024-12-18T00:29:12.7276637Z   inflating: build/lib/libdnnl.a     
2024-12-18T00:29:12.7336817Z   inflating: build/lib/libc10_cuda.so  
2024-12-18T00:29:12.7337562Z   inflating: build/lib/libgmock_main.a  
2024-12-18T00:29:12.8599413Z   inflating: build/lib/libfbgemm.a   
2024-12-18T00:29:12.8855827Z   inflating: build/lib/libtensorpipe_cuda.a  
2024-12-18T00:29:12.9365293Z   inflating: build/lib/libkineto.a   
2024-12-18T00:29:12.9455105Z   inflating: build/lib/libXNNPACK.a  
2024-12-18T00:29:12.9497949Z   inflating: build/lib/libonnx_proto.a  
2024-12-18T00:29:13.0232418Z   inflating: build/lib/libonnx.a     
2024-12-18T00:29:13.1105573Z   inflating: build/lib/libmicrokernels-all.a  
2024-12-18T00:29:15.6445922Z   inflating: build/lib/libtorch_cpu.so  
2024-12-18T00:29:15.6450521Z   inflating: build/lib/libunbox_lib.a  
2024-12-18T00:29:15.6454885Z   inflating: build/lib/libshm.so     
2024-12-18T00:29:17.7457333Z   inflating: build/lib/libtorch_cuda.so  
2024-12-18T00:29:17.7458346Z   inflating: build/lib/libtorch.so   
2024-12-18T00:29:17.7461308Z   inflating: build/lib/libc10d_cuda_test.so  
2024-12-18T00:29:18.6275669Z   inflating: build/lib/libtorch_cuda_linalg.so  
2024-12-18T00:29:18.8364494Z   inflating: build/lib/libtorch_python.so  
2024-12-18T00:29:18.8435799Z   inflating: build/lib/libtorchbind_test.so  
2024-12-18T00:29:18.8455586Z   inflating: build/lib/libjitbackend_test.so  
2024-12-18T00:29:18.8480598Z   inflating: build/lib/libbackend_with_compiler.so  
2024-12-18T00:29:18.8506205Z   inflating: build/lib/libaoti_custom_ops.so  
2024-12-18T00:29:18.8540652Z   inflating: build/lib/libnnapi_backend.so  
2024-12-18T00:29:18.8540994Z    creating: build/bin/
2024-12-18T00:29:18.8594955Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2024-12-18T00:29:18.8650767Z   inflating: build/bin/c10_DeviceGuard_test  
2024-12-18T00:29:18.8706747Z   inflating: build/bin/c10_Device_test  
2024-12-18T00:29:18.8769148Z   inflating: build/bin/c10_DispatchKeySet_test  
2024-12-18T00:29:18.8826530Z   inflating: build/bin/c10_Scalar_test  
2024-12-18T00:29:18.8878616Z   inflating: build/bin/c10_StreamGuard_test  
2024-12-18T00:29:18.8933939Z   inflating: build/bin/c10_SymInt_test  
2024-12-18T00:29:18.8991675Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2024-12-18T00:29:18.9051356Z   inflating: build/bin/c10_InlineStreamGuard_test  
2024-12-18T00:29:18.9111131Z   inflating: build/bin/c10_SizesAndStrides_test  
2024-12-18T00:29:18.9185600Z   inflating: build/bin/c10_cow_test  
2024-12-18T00:29:18.9239414Z   inflating: build/bin/c10_ArrayRef_test  
2024-12-18T00:29:18.9295859Z   inflating: build/bin/c10_Bitset_test  
2024-12-18T00:29:18.9348281Z   inflating: build/bin/c10_ConstexprCrc_test  
2024-12-18T00:29:18.9401518Z   inflating: build/bin/c10_DeadlockDetection_test  
2024-12-18T00:29:18.9455846Z   inflating: build/bin/c10_Half_test  
2024-12-18T00:29:18.9515788Z   inflating: build/bin/c10_LeftRight_test  
2024-12-18T00:29:18.9574686Z   inflating: build/bin/c10_Metaprogramming_test  
2024-12-18T00:29:18.9631252Z   inflating: build/bin/c10_NetworkFlow_test  
2024-12-18T00:29:18.9685228Z   inflating: build/bin/c10_Synchronized_test  
2024-12-18T00:29:18.9744360Z   inflating: build/bin/c10_ThreadLocal_test  
2024-12-18T00:29:18.9799105Z   inflating: build/bin/c10_TypeIndex_test  
2024-12-18T00:29:18.9854349Z   inflating: build/bin/c10_TypeList_test  
2024-12-18T00:29:18.9906703Z   inflating: build/bin/c10_TypeTraits_test  
2024-12-18T00:29:18.9962074Z   inflating: build/bin/c10_accumulate_test  
2024-12-18T00:29:19.0021702Z   inflating: build/bin/c10_bfloat16_test  
2024-12-18T00:29:19.0075899Z   inflating: build/bin/c10_bit_cast_test  
2024-12-18T00:29:19.0136453Z   inflating: build/bin/c10_complex_math_test  
2024-12-18T00:29:19.0194941Z   inflating: build/bin/c10_complex_test  
2024-12-18T00:29:19.0248031Z   inflating: build/bin/c10_error_test  
2024-12-18T00:29:19.0304400Z   inflating: build/bin/c10_exception_test  
2024-12-18T00:29:19.0358567Z   inflating: build/bin/c10_flags_test  
2024-12-18T00:29:19.0412758Z   inflating: build/bin/c10_generic_math_test  
2024-12-18T00:29:19.0585172Z   inflating: build/bin/c10_intrusive_ptr_test  
2024-12-18T00:29:19.0640249Z   inflating: build/bin/c10_irange_test  
2024-12-18T00:29:19.0696717Z   inflating: build/bin/c10_lazy_test  
2024-12-18T00:29:19.0758003Z   inflating: build/bin/c10_logging_test  
2024-12-18T00:29:19.0838330Z   inflating: build/bin/c10_optional_test  
2024-12-18T00:29:19.0904957Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2024-12-18T00:29:19.0962429Z   inflating: build/bin/c10_registry_test  
2024-12-18T00:29:19.1122346Z   inflating: build/bin/c10_small_vector_test  
2024-12-18T00:29:19.1177767Z   inflating: build/bin/c10_ssize_test  
2024-12-18T00:29:19.1233264Z   inflating: build/bin/c10_string_util_test  
2024-12-18T00:29:19.1297057Z   inflating: build/bin/c10_string_view_test  
2024-12-18T00:29:19.1351043Z   inflating: build/bin/c10_tempfile_test  
2024-12-18T00:29:19.1410734Z   inflating: build/bin/c10_typeid_test  
2024-12-18T00:29:19.1460196Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2024-12-18T00:29:19.1908823Z   inflating: build/bin/protoc-3.13.0.0  
2024-12-18T00:29:19.2357697Z   inflating: build/bin/protoc        
2024-12-18T00:29:19.2414319Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test  
2024-12-18T00:29:19.2470825Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream  
2024-12-18T00:29:19.2527272Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device  
2024-12-18T00:29:19.2582828Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes  
2024-12-18T00:29:19.2639323Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads  
2024-12-18T00:29:19.2695748Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks  
2024-12-18T00:29:19.2752612Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block  
2024-12-18T00:29:19.2805000Z   inflating: build/bin/c10_cuda_CUDATest  
2024-12-18T00:29:19.3206883Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2024-12-18T00:29:19.3619760Z   inflating: build/bin/vec_test_all_types_AVX512  
2024-12-18T00:29:19.4047034Z   inflating: build/bin/vec_test_all_types_AVX2  
2024-12-18T00:29:19.4103596Z   inflating: build/bin/BackoffTest   
2024-12-18T00:29:19.4160538Z   inflating: build/bin/FileStoreTest  
2024-12-18T00:29:19.4219479Z   inflating: build/bin/TCPStoreTest  
2024-12-18T00:29:19.4276617Z   inflating: build/bin/HashStoreTest  
2024-12-18T00:29:19.4290378Z   inflating: build/bin/ProcessGroupMPITest  
2024-12-18T00:29:19.4348660Z   inflating: build/bin/test_edge_op_registration  
2024-12-18T00:29:19.4353097Z   inflating: build/bin/torch_shm_manager  
2024-12-18T00:29:19.4356076Z   inflating: build/bin/example_allreduce  
2024-12-18T00:29:19.4415660Z   inflating: build/bin/test_dist_autograd  
2024-12-18T00:29:19.4488640Z   inflating: build/bin/test_cpp_rpc  
2024-12-18T00:29:19.4491229Z   inflating: build/bin/parallel_benchmark  
2024-12-18T00:29:19.4562584Z   inflating: build/bin/test_mobile_nnc  
2024-12-18T00:29:19.4571772Z   inflating: build/bin/aot_model_compiler_test  
2024-12-18T00:29:19.4927829Z   inflating: build/bin/test_lazy     
2024-12-18T00:29:19.6120845Z   inflating: build/bin/test_api      
2024-12-18T00:29:19.6198991Z   inflating: build/bin/Dict_test     
2024-12-18T00:29:19.6254901Z   inflating: build/bin/Dimname_test  
2024-12-18T00:29:19.6323917Z   inflating: build/bin/MaybeOwned_test  
2024-12-18T00:29:19.6385398Z   inflating: build/bin/NamedTensor_test  
2024-12-18T00:29:19.6448171Z   inflating: build/bin/apply_utils_test  
2024-12-18T00:29:19.6511390Z   inflating: build/bin/atest         
2024-12-18T00:29:19.6578954Z   inflating: build/bin/basic         
2024-12-18T00:29:19.6638087Z   inflating: build/bin/broadcast_test  
2024-12-18T00:29:19.6692773Z   inflating: build/bin/cpu_allocator_test  
2024-12-18T00:29:19.6755288Z   inflating: build/bin/cpu_generator_test  
2024-12-18T00:29:19.6812578Z   inflating: build/bin/cpu_profiling_allocator_test  
2024-12-18T00:29:19.6909184Z   inflating: build/bin/cpu_rng_test  
2024-12-18T00:29:19.6963179Z   inflating: build/bin/dispatch_key_set_test  
2024-12-18T00:29:19.7017611Z   inflating: build/bin/dlconvertor_test  
2024-12-18T00:29:19.7080032Z   inflating: build/bin/extension_backend_test  
2024-12-18T00:29:19.7139014Z   inflating: build/bin/half_test     
2024-12-18T00:29:19.7240008Z   inflating: build/bin/ivalue_test   
2024-12-18T00:29:19.7292770Z   inflating: build/bin/lazy_tensor_test  
2024-12-18T00:29:19.7350554Z   inflating: build/bin/math_kernel_test  
2024-12-18T00:29:19.7408486Z   inflating: build/bin/memory_format_test  
2024-12-18T00:29:19.7465140Z   inflating: build/bin/memory_overlapping_test  
2024-12-18T00:29:19.7521641Z   inflating: build/bin/mobile_memory_cleanup  
2024-12-18T00:29:19.7581752Z   inflating: build/bin/native_test   
2024-12-18T00:29:19.7636234Z   inflating: build/bin/operator_name_test  
2024-12-18T00:29:19.7691162Z   inflating: build/bin/operators_test  
2024-12-18T00:29:19.7747123Z   inflating: build/bin/packedtensoraccessor_test  
2024-12-18T00:29:19.7819486Z   inflating: build/bin/pow_test      
2024-12-18T00:29:19.7880867Z   inflating: build/bin/quantized_test  
2024-12-18T00:29:19.7934167Z   inflating: build/bin/reduce_ops_test  
2024-12-18T00:29:19.7988866Z   inflating: build/bin/reportMemoryUsage_test  
2024-12-18T00:29:19.8050088Z   inflating: build/bin/scalar_tensor_test  
2024-12-18T00:29:19.8112950Z   inflating: build/bin/scalar_test   
2024-12-18T00:29:19.8168127Z   inflating: build/bin/StorageUtils_test  
2024-12-18T00:29:19.8224731Z   inflating: build/bin/stride_properties_test  
2024-12-18T00:29:19.8308650Z   inflating: build/bin/tensor_iterator_test  
2024-12-18T00:29:19.8366982Z   inflating: build/bin/test_parallel  
2024-12-18T00:29:19.8370455Z   inflating: build/bin/thread_init_test  
2024-12-18T00:29:19.8429726Z   inflating: build/bin/type_ptr_test  
2024-12-18T00:29:19.8493850Z   inflating: build/bin/type_test     
2024-12-18T00:29:19.8550431Z   inflating: build/bin/undefined_tensor_test  
2024-12-18T00:29:19.8551727Z   inflating: build/bin/verify_api_visibility  
2024-12-18T00:29:19.8626103Z   inflating: build/bin/legacy_vmap_test  
2024-12-18T00:29:19.8681469Z   inflating: build/bin/weakref_test  
2024-12-18T00:29:19.8736857Z   inflating: build/bin/wrapdim_test  
2024-12-18T00:29:19.8792591Z   inflating: build/bin/xla_tensor_test  
2024-12-18T00:29:19.8856193Z   inflating: build/bin/IListRef_test  
2024-12-18T00:29:19.8968353Z   inflating: build/bin/List_test     
2024-12-18T00:29:19.9039574Z   inflating: build/bin/KernelFunction_test  
2024-12-18T00:29:19.9165803Z   inflating: build/bin/kernel_function_legacy_test  
2024-12-18T00:29:19.9267206Z   inflating: build/bin/kernel_function_test  
2024-12-18T00:29:19.9399742Z   inflating: build/bin/kernel_lambda_legacy_test  
2024-12-18T00:29:19.9508605Z   inflating: build/bin/kernel_lambda_test  
2024-12-18T00:29:19.9574048Z   inflating: build/bin/kernel_stackbased_test  
2024-12-18T00:29:19.9674916Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2024-12-18T00:29:19.9730217Z   inflating: build/bin/CppSignature_test  
2024-12-18T00:29:19.9790105Z   inflating: build/bin/backend_fallback_test  
2024-12-18T00:29:19.9842277Z   inflating: build/bin/op_allowlist_test  
2024-12-18T00:29:20.0151087Z   inflating: build/bin/op_registration_test  
2024-12-18T00:29:20.0219040Z   inflating: build/bin/inline_container_test  
2024-12-18T00:29:20.0275433Z   inflating: build/bin/cuda_apply_test  
2024-12-18T00:29:20.0331401Z   inflating: build/bin/cuda_allocator_test  
2024-12-18T00:29:20.0389731Z   inflating: build/bin/cuda_caching_host_allocator_test  
2024-12-18T00:29:20.0453257Z   inflating: build/bin/cuda_atomic_ops_test  
2024-12-18T00:29:20.0527670Z   inflating: build/bin/cuda_complex_math_test  
2024-12-18T00:29:20.0590418Z   inflating: build/bin/cuda_complex_test  
2024-12-18T00:29:20.0644268Z   inflating: build/bin/cuda_device_test  
2024-12-18T00:29:20.0706577Z   inflating: build/bin/cuda_cub_test  
2024-12-18T00:29:20.0761639Z   inflating: build/bin/cuda_dlconvertor_test  
2024-12-18T00:29:20.0831094Z   inflating: build/bin/cuda_distributions_test  
2024-12-18T00:29:20.0891586Z   inflating: build/bin/cuda_generator_test  
2024-12-18T00:29:20.0945863Z   inflating: build/bin/cuda_half_test  
2024-12-18T00:29:20.1000658Z   inflating: build/bin/cuda_integer_divider_test  
2024-12-18T00:29:20.1053737Z   inflating: build/bin/cuda_optional_test  
2024-12-18T00:29:20.1109610Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2024-12-18T00:29:20.1165881Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2024-12-18T00:29:20.1220046Z   inflating: build/bin/cuda_allocatorTraceTracker_test  
2024-12-18T00:29:20.1285025Z   inflating: build/bin/cuda_stream_test  
2024-12-18T00:29:20.1337918Z   inflating: build/bin/cuda_cudnn_test  
2024-12-18T00:29:20.1393633Z   inflating: build/bin/cuda_vectorized_test  
2024-12-18T00:29:20.1408531Z   inflating: build/bin/tutorial_tensorexpr  
2024-12-18T00:29:20.1478507Z   inflating: build/bin/ProcessGroupGlooTest  
2024-12-18T00:29:20.1540144Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2024-12-18T00:29:20.1608332Z   inflating: build/bin/ProcessGroupNCCLTest  
2024-12-18T00:29:20.1674570Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2024-12-18T00:29:20.2518000Z   inflating: build/bin/test_tensorexpr  
2024-12-18T00:29:20.3105766Z   inflating: build/bin/test_jit      
2024-12-18T00:29:20.3106123Z    creating: .additional_ci_files/
2024-12-18T00:29:20.3196904Z   inflating: .additional_ci_files/test-times.json  
2024-12-18T00:29:20.3558388Z   inflating: .additional_ci_files/test-class-times.json  
2024-12-18T00:29:20.3598186Z ##[group]Run rm artifacts.zip
2024-12-18T00:29:20.3598495Z rm artifacts.zip
2024-12-18T00:29:20.3607758Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:20.3608144Z env:
2024-12-18T00:29:20.3608371Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:20.3608908Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:20.3609287Z ##[endgroup]
2024-12-18T00:29:20.4762083Z ##[group]Run df -H
2024-12-18T00:29:20.4762341Z df -H
2024-12-18T00:29:20.4771143Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:20.4771519Z env:
2024-12-18T00:29:20.4771739Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:20.4772077Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:20.4772435Z ##[endgroup]
2024-12-18T00:29:20.4829744Z Filesystem        Size  Used Avail Use% Mounted on
2024-12-18T00:29:20.4830153Z devtmpfs          4.2M     0  4.2M   0% /dev
2024-12-18T00:29:20.4830490Z tmpfs              34G     0   34G   0% /dev/shm
2024-12-18T00:29:20.4830824Z tmpfs              14G  553k   14G   1% /run
2024-12-18T00:29:20.4831148Z /dev/nvme0n1p1    161G   60G  102G  37% /
2024-12-18T00:29:20.4831467Z tmpfs              34G   33k   34G   1% /tmp
2024-12-18T00:29:20.4831810Z /dev/nvme0n1p128   11M  1.4M  9.2M  13% /boot/efi
2024-12-18T00:29:20.4832180Z tmpfs             6.7G     0  6.7G   0% /run/user/0
2024-12-18T00:29:20.4865736Z Prepare all required actions
2024-12-18T00:29:20.4866120Z Getting action download info
2024-12-18T00:29:20.6454393Z ##[group]Run ./.github/actions/download-td-artifacts
2024-12-18T00:29:20.6454751Z with:
2024-12-18T00:29:20.6454957Z env:
2024-12-18T00:29:20.6455174Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:20.6455505Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:20.6455858Z ##[endgroup]
2024-12-18T00:29:20.6488102Z ##[group]Run seemethere/download-artifact-s3@v4
2024-12-18T00:29:20.6488446Z with:
2024-12-18T00:29:20.6488659Z   name: td_results
2024-12-18T00:29:20.6488902Z   s3-bucket: gha-artifacts
2024-12-18T00:29:20.6489169Z   region: us-east-1
2024-12-18T00:29:20.6489401Z env:
2024-12-18T00:29:20.6489618Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:20.6489957Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:20.6490324Z ##[endgroup]
2024-12-18T00:29:21.1033375Z (node:341325) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2024-12-18T00:29:21.1033860Z 
2024-12-18T00:29:21.1034053Z Please migrate your code to use AWS SDK for JavaScript (v3).
2024-12-18T00:29:21.1034578Z For more information, check the migration guide at https://a.co/7PzMCcy
2024-12-18T00:29:21.1035121Z (Use `node --trace-warnings ...` to show where the warning was created)
2024-12-18T00:29:21.1936065Z Found 0 objects with prefix pytorch/pytorch/12383255690/td_results/
2024-12-18T00:29:21.1942363Z Artifact download has finished successfully
2024-12-18T00:29:21.2292916Z ##[group]Run mkdir -p .additional_ci_files
2024-12-18T00:29:21.2293293Z mkdir -p .additional_ci_files
2024-12-18T00:29:21.2293722Z mv td_results.json .additional_ci_files/td_results.json || true
2024-12-18T00:29:21.2303728Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:21.2304123Z env:
2024-12-18T00:29:21.2304342Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:21.2304691Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:21.2305049Z ##[endgroup]
2024-12-18T00:29:21.2381958Z mv: cannot stat 'td_results.json': No such file or directory
2024-12-18T00:29:21.2416739Z ##[group]Run .github/scripts/parse_ref.py
2024-12-18T00:29:21.2417114Z .github/scripts/parse_ref.py
2024-12-18T00:29:21.2425012Z shell: /usr/bin/bash -e {0}
2024-12-18T00:29:21.2425288Z env:
2024-12-18T00:29:21.2425509Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:21.2425848Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:21.2426200Z ##[endgroup]
2024-12-18T00:29:21.2706642Z Prepare all required actions
2024-12-18T00:29:21.2740997Z ##[group]Run ./.github/actions/get-workflow-job-id
2024-12-18T00:29:21.2741346Z with:
2024-12-18T00:29:21.2741743Z   github-token: ***
2024-12-18T00:29:21.2741981Z env:
2024-12-18T00:29:21.2742198Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:21.2742755Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:21.2743114Z ##[endgroup]
2024-12-18T00:29:21.2778094Z ##[group]Run set -eux
2024-12-18T00:29:21.2778373Z set -eux
2024-12-18T00:29:21.2778812Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2024-12-18T00:29:21.2787590Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:21.2787970Z env:
2024-12-18T00:29:21.2788196Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:21.2788541Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:21.2789050Z   GITHUB_TOKEN: ***
2024-12-18T00:29:21.2789295Z ##[endgroup]
2024-12-18T00:29:21.2816751Z + python3 .github/scripts/get_workflow_job_id.py 12383255690 i-019aa8c64fc4447ce
2024-12-18T00:29:22.3496696Z setting job-id=34567152834
2024-12-18T00:29:22.3497222Z setting job-name=cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:22.3673476Z Prepare all required actions
2024-12-18T00:29:22.3673864Z Getting action download info
2024-12-18T00:29:22.5066040Z ##[group]Run ./.github/actions/filter-test-configs
2024-12-18T00:29:22.5066392Z with:
2024-12-18T00:29:22.5066810Z   github-token: ***
2024-12-18T00:29:22.5068525Z   test-matrix: {"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}
2024-12-18T00:29:22.5070488Z   job-name: cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:22.5070991Z env:
2024-12-18T00:29:22.5071424Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:22.5071786Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:22.5072142Z ##[endgroup]
2024-12-18T00:29:22.5112324Z ##[group]Run nick-fields/retry@v3.0.0
2024-12-18T00:29:22.5112695Z with:
2024-12-18T00:29:22.5112914Z   shell: bash
2024-12-18T00:29:22.5113140Z   timeout_minutes: 10
2024-12-18T00:29:22.5113394Z   max_attempts: 5
2024-12-18T00:29:22.5113670Z   retry_wait_seconds: 30
2024-12-18T00:29:22.5114453Z   command: set -eux
# PyYAML 6.0 doesn't work with MacOS x86 anymore
# This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2
python3 -m pip install requests==2.27.1 pyyaml==6.0.1

2024-12-18T00:29:22.5115261Z   polling_interval_seconds: 1
2024-12-18T00:29:22.5115549Z   warning_on_retry: true
2024-12-18T00:29:22.5115832Z   continue_on_error: false
2024-12-18T00:29:22.5116104Z env:
2024-12-18T00:29:22.5116341Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:22.5116691Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:22.5117243Z   GITHUB_TOKEN: ***
2024-12-18T00:29:22.5117485Z ##[endgroup]
2024-12-18T00:29:22.6099940Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1
2024-12-18T00:29:22.8535274Z Defaulting to user installation because normal site-packages is not writeable
2024-12-18T00:29:22.8711169Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.9/site-packages (2.27.1)
2024-12-18T00:29:22.8714907Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.9/site-packages (6.0.1)
2024-12-18T00:29:22.8827803Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10)
2024-12-18T00:29:22.8838164Z Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2.0.12)
2024-12-18T00:29:22.8846321Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10)
2024-12-18T00:29:22.8851613Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2024.12.14)
2024-12-18T00:29:23.5875260Z Command completed after 1 attempt(s).
2024-12-18T00:29:23.5948889Z ##[group]Run set -x
2024-12-18T00:29:23.5949202Z set -x
2024-12-18T00:29:23.5949436Z 
2024-12-18T00:29:23.5949817Z # Use relative path here as this could be checked out anywhere, not necessarily
2024-12-18T00:29:23.5950297Z # in runner workspace
2024-12-18T00:29:23.5950685Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py"
2024-12-18T00:29:23.5959550Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:23.5959922Z env:
2024-12-18T00:29:23.5960143Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:23.5960473Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:23.5960831Z ##[endgroup]
2024-12-18T00:29:23.5990555Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py
2024-12-18T00:29:23.6234916Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}"
2024-12-18T00:29:23.6235323Z echo "Workflow: ${GITHUB_WORKFLOW}"
2024-12-18T00:29:23.6235657Z echo "Job name: ${JOB_NAME}"
2024-12-18T00:29:23.6235943Z 
2024-12-18T00:29:23.6236326Z # Use relative path here as this could be checked out anywhere, not necessarily
2024-12-18T00:29:23.6236800Z # in runner workspace
2024-12-18T00:29:23.6237225Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \
2024-12-18T00:29:23.6237696Z   --workflow "${GITHUB_WORKFLOW}" \
2024-12-18T00:29:23.6238024Z   --job-name "${JOB_NAME}" \
2024-12-18T00:29:23.6240042Z   --test-matrix "{"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \
2024-12-18T00:29:23.6241864Z   --selected-test-configs "" \
2024-12-18T00:29:23.6242197Z   --pr-number "${PR_NUMBER}" \
2024-12-18T00:29:23.6242506Z   --tag "${TAG}" \
2024-12-18T00:29:23.6242796Z   --event-name "${EVENT_NAME}" \
2024-12-18T00:29:23.6243120Z   --schedule "${SCHEDULE}" \
2024-12-18T00:29:23.6243434Z   --branch "${HEAD_BRANCH}"
2024-12-18T00:29:23.6251727Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:23.6252087Z env:
2024-12-18T00:29:23.6252301Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:23.6252643Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:23.6253195Z   GITHUB_TOKEN: ***
2024-12-18T00:29:23.6253640Z   JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:23.6254139Z   PR_NUMBER: 
2024-12-18T00:29:23.6254359Z   TAG: 
2024-12-18T00:29:23.6254576Z   EVENT_NAME: push
2024-12-18T00:29:23.6254816Z   SCHEDULE: 
2024-12-18T00:29:23.6255039Z   HEAD_BRANCH: 
2024-12-18T00:29:23.6255262Z ##[endgroup]
2024-12-18T00:29:23.6282078Z Workflow: inductor
2024-12-18T00:29:23.6282708Z Job name: cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:23.8896891Z ##[group]Run echo "Filtered matrix:"
2024-12-18T00:29:23.8897237Z echo "Filtered matrix:"
2024-12-18T00:29:23.8899019Z echo "{"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}"
2024-12-18T00:29:23.8901021Z 
2024-12-18T00:29:23.8901224Z echo
2024-12-18T00:29:23.8901500Z echo "Is the current job unstable? False"
2024-12-18T00:29:23.8901834Z 
2024-12-18T00:29:23.8902042Z echo
2024-12-18T00:29:23.8902303Z echo "Is keep-going label set? False"
2024-12-18T00:29:23.8902614Z 
2024-12-18T00:29:23.8902821Z echo
2024-12-18T00:29:23.8903058Z echo "Renabled issues? "
2024-12-18T00:29:23.8912236Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:23.8912611Z env:
2024-12-18T00:29:23.8912832Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:23.8913366Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:23.8913744Z ##[endgroup]
2024-12-18T00:29:23.8942918Z Filtered matrix:
2024-12-18T00:29:23.8945057Z {include: [{config: inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}]}
2024-12-18T00:29:23.8946717Z 
2024-12-18T00:29:23.8946845Z Is the current job unstable? False
2024-12-18T00:29:23.8947060Z 
2024-12-18T00:29:23.8947171Z Is keep-going label set? False
2024-12-18T00:29:23.8947364Z 
2024-12-18T00:29:23.8947461Z Renabled issues? 
2024-12-18T00:29:23.8987547Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2024-12-18T00:29:23.8988054Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2024-12-18T00:29:23.8996266Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:23.8996638Z env:
2024-12-18T00:29:23.8996861Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:23.8997197Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:23.8997552Z   JOB_TIMEOUT: 240
2024-12-18T00:29:23.8997888Z ##[endgroup]
2024-12-18T00:29:23.9085452Z ##[group]Run set -x
2024-12-18T00:29:23.9085766Z set -x
2024-12-18T00:29:23.9085995Z 
2024-12-18T00:29:23.9086253Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2024-12-18T00:29:23.9086646Z   TEST_COMMAND=.ci/pytorch/multigpu-test.sh
2024-12-18T00:29:23.9087043Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2024-12-18T00:29:23.9087399Z   TEST_COMMAND=.ci/onnx/test.sh
2024-12-18T00:29:23.9087702Z else
2024-12-18T00:29:23.9087965Z   TEST_COMMAND=.ci/pytorch/test.sh
2024-12-18T00:29:23.9088279Z fi
2024-12-18T00:29:23.9088500Z 
2024-12-18T00:29:23.9088856Z # detached container should get cleaned up by teardown_ec2_linux
2024-12-18T00:29:23.9089434Z # TODO: Stop building test binaries as part of the build phase
2024-12-18T00:29:23.9089898Z # Used for GPU_FLAG since that doesn't play nice
2024-12-18T00:29:23.9090308Z # shellcheck disable=SC2086,SC2090
2024-12-18T00:29:23.9090642Z container_name=$(docker run \
2024-12-18T00:29:23.9090957Z   ${GPU_FLAG:-} \
2024-12-18T00:29:23.9091264Z   ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \
2024-12-18T00:29:23.9091612Z   -e BUILD_ENVIRONMENT \
2024-12-18T00:29:23.9091913Z   -e PR_NUMBER \
2024-12-18T00:29:23.9092180Z   -e GITHUB_ACTIONS \
2024-12-18T00:29:23.9092470Z   -e GITHUB_REPOSITORY \
2024-12-18T00:29:23.9092773Z   -e GITHUB_WORKFLOW \
2024-12-18T00:29:23.9093060Z   -e GITHUB_JOB \
2024-12-18T00:29:23.9093330Z   -e GITHUB_RUN_ID \
2024-12-18T00:29:23.9093789Z   -e GITHUB_RUN_NUMBER \
2024-12-18T00:29:23.9094092Z   -e GITHUB_RUN_ATTEMPT \
2024-12-18T00:29:23.9094385Z   -e JOB_ID \
2024-12-18T00:29:23.9094643Z   -e JOB_NAME \
2024-12-18T00:29:23.9094909Z   -e BASE_SHA \
2024-12-18T00:29:23.9095162Z   -e BRANCH \
2024-12-18T00:29:23.9095418Z   -e SHA1 \
2024-12-18T00:29:23.9095679Z   -e AWS_DEFAULT_REGION \
2024-12-18T00:29:23.9095980Z   -e IN_WHEEL_TEST \
2024-12-18T00:29:23.9096257Z   -e SHARD_NUMBER \
2024-12-18T00:29:23.9096526Z   -e TEST_CONFIG \
2024-12-18T00:29:23.9096804Z   -e NUM_TEST_SHARDS \
2024-12-18T00:29:23.9097096Z   -e REENABLED_ISSUES \
2024-12-18T00:29:23.9097405Z   -e CONTINUE_THROUGH_ERROR \
2024-12-18T00:29:23.9097719Z   -e VERBOSE_TEST_LOGS \
2024-12-18T00:29:23.9098010Z   -e TEST_SHOWLOCALS \
2024-12-18T00:29:23.9098301Z   -e NO_TEST_TIMEOUT \
2024-12-18T00:29:23.9098586Z   -e NO_TD \
2024-12-18T00:29:23.9098845Z   -e TD_DISTRIBUTED \
2024-12-18T00:29:23.9099131Z   -e PR_LABELS \
2024-12-18T00:29:23.9099430Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2024-12-18T00:29:23.9099755Z   -e SCCACHE_BUCKET \
2024-12-18T00:29:23.9100045Z   -e SCCACHE_REGION \
2024-12-18T00:29:23.9100338Z   -e SCCACHE_S3_KEY_PREFIX \
2024-12-18T00:29:23.9100642Z   -e XLA_CUDA \
2024-12-18T00:29:23.9100937Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2024-12-18T00:29:23.9101295Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2024-12-18T00:29:23.9101668Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2024-12-18T00:29:23.9102039Z   -e SKIP_SCCACHE_INITIALIZATION=1 \
2024-12-18T00:29:23.9102386Z   -e HUGGING_FACE_HUB_TOKEN \
2024-12-18T00:29:23.9102721Z   -e SCRIBE_GRAPHQL_ACCESS_TOKEN \
2024-12-18T00:29:23.9103038Z   -e DASHBOARD_TAG \
2024-12-18T00:29:23.9103814Z   -e IS_A100_RUNNER \
2024-12-18T00:29:23.9104189Z   -e ARTIFACTS_FILE_SUFFIX \
2024-12-18T00:29:23.9104558Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2024-12-18T00:29:23.9104956Z   --security-opt seccomp=unconfined \
2024-12-18T00:29:23.9105477Z   --cap-add=SYS_PTRACE \
2024-12-18T00:29:23.9105773Z   --ipc=host \
2024-12-18T00:29:23.9106051Z   --shm-size="${SHM_SIZE}" \
2024-12-18T00:29:23.9106342Z   --tty \
2024-12-18T00:29:23.9106581Z   --detach \
2024-12-18T00:29:23.9106849Z   --name="${container_name}" \
2024-12-18T00:29:23.9107153Z   --user jenkins \
2024-12-18T00:29:23.9107501Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2024-12-18T00:29:23.9107910Z   -w /var/lib/jenkins/workspace \
2024-12-18T00:29:23.9108226Z   "${DOCKER_IMAGE}"
2024-12-18T00:29:23.9108490Z )
2024-12-18T00:29:23.9108795Z # Propagate download.pytorch.org IP to container
2024-12-18T00:29:23.9109441Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts"
2024-12-18T00:29:23.9110127Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}"
2024-12-18T00:29:23.9110812Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"
2024-12-18T00:29:23.9119066Z shell: /usr/bin/bash -e {0}
2024-12-18T00:29:23.9119444Z env:
2024-12-18T00:29:23.9119674Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:23.9120018Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:23.9120468Z   BUILD_ENVIRONMENT: linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-12-18T00:29:23.9120858Z   PR_NUMBER: 
2024-12-18T00:29:23.9121121Z   GITHUB_REPOSITORY: pytorch/pytorch
2024-12-18T00:29:23.9121443Z   GITHUB_WORKFLOW: inductor
2024-12-18T00:29:23.9121719Z   GITHUB_JOB: test
2024-12-18T00:29:23.9122107Z   GITHUB_RUN_ID: 12383255690
2024-12-18T00:29:23.9122393Z   GITHUB_RUN_NUMBER: 109429
2024-12-18T00:29:23.9122671Z   GITHUB_RUN_ATTEMPT: 1
2024-12-18T00:29:23.9122937Z   JOB_ID: 34567152834
2024-12-18T00:29:23.9123390Z   JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:23.9123892Z   BRANCH: release/2.6
2024-12-18T00:29:23.9124185Z   SHA1: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:23.9124570Z   BASE_SHA: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:23.9124925Z   TEST_CONFIG: inductor_timm
2024-12-18T00:29:23.9125206Z   SHARD_NUMBER: 1
2024-12-18T00:29:23.9125440Z   NUM_TEST_SHARDS: 2
2024-12-18T00:29:23.9125697Z   REENABLED_ISSUES: 
2024-12-18T00:29:23.9125962Z   CONTINUE_THROUGH_ERROR: False
2024-12-18T00:29:23.9126260Z   VERBOSE_TEST_LOGS: False
2024-12-18T00:29:23.9126545Z   TEST_SHOWLOCALS: False
2024-12-18T00:29:23.9126812Z   NO_TEST_TIMEOUT: False
2024-12-18T00:29:23.9127074Z   NO_TD: False
2024-12-18T00:29:23.9127325Z   TD_DISTRIBUTED: False
2024-12-18T00:29:23.9127646Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2024-12-18T00:29:23.9128016Z   SCCACHE_REGION: us-east-1
2024-12-18T00:29:23.9128298Z   SCCACHE_S3_KEY_PREFIX: inductor
2024-12-18T00:29:23.9128590Z   SHM_SIZE: 2g
2024-12-18T00:29:23.9129390Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:29:23.9130249Z   XLA_CUDA: 
2024-12-18T00:29:23.9130614Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2024-12-18T00:29:23.9131071Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2024-12-18T00:29:23.9131393Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2024-12-18T00:29:23.9131706Z   DASHBOARD_TAG: 
2024-12-18T00:29:23.9132116Z   HUGGING_FACE_HUB_TOKEN: ***
2024-12-18T00:29:23.9132533Z   SCRIBE_GRAPHQL_ACCESS_TOKEN: ***
2024-12-18T00:29:23.9132839Z   IS_A100_RUNNER: 0
2024-12-18T00:29:23.9133274Z   ARTIFACTS_FILE_SUFFIX: test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_34567152834
2024-12-18T00:29:23.9133764Z ##[endgroup]
2024-12-18T00:29:23.9160198Z + [[ inductor_timm == \m\u\l\t\i\g\p\u ]]
2024-12-18T00:29:23.9160611Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *onnx* ]]
2024-12-18T00:29:23.9160992Z + TEST_COMMAND=.ci/pytorch/test.sh
2024-12-18T00:29:23.9170629Z +++ nproc --ignore=2
2024-12-18T00:29:23.9197049Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_REGION -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e IS_A100_RUNNER -e ARTIFACTS_FILE_SUFFIX --env-file=/tmp/github_env_12383255690 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:29:35.7189794Z + container_name=2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T00:29:35.7191437Z + grep download.pytorch.org /etc/hosts
2024-12-18T00:29:35.7192997Z + docker exec -i 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2 sudo bash -c '/bin/cat >> /etc/hosts'
2024-12-18T00:29:35.8565869Z + echo DOCKER_CONTAINER_ID=2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T00:29:35.8569622Z ++ echo dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl
2024-12-18T00:29:35.8572643Z + docker exec -t 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2 sh -c 'python3 -m pip install dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh'
2024-12-18T00:29:36.2567648Z Processing ./dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl (from torch==2.6.0a0+git0cdf8b1)
2024-12-18T00:29:36.5892600Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.16.1)
2024-12-18T00:29:36.5895592Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (4.12.2)
2024-12-18T00:29:36.5898642Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (2.8.8)
2024-12-18T00:29:36.5901803Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.1.4)
2024-12-18T00:29:36.5905011Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (2024.10.0)
2024-12-18T00:29:36.5910382Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.13.1)
2024-12-18T00:29:36.5927342Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.3.0)
2024-12-18T00:29:36.5940249Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.3.0)
2024-12-18T00:29:36.5957625Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.22.4)
2024-12-18T00:29:36.6341689Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.0.2)
2024-12-18T00:29:37.2389033Z Installing collected packages: torch
2024-12-18T00:29:47.9232994Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
2024-12-18T00:29:47.9234654Z timm 1.0.9.dev0 requires torchvision, which is not installed.
2024-12-18T00:29:47.9235237Z Successfully installed torch-2.6.0a0+git0cdf8b1
2024-12-18T00:29:48.0036148Z + export TERM=vt100
2024-12-18T00:29:48.0036514Z + TERM=vt100
2024-12-18T00:29:48.0040285Z ++ dirname .ci/pytorch/test.sh
2024-12-18T00:29:48.0049500Z + source .ci/pytorch/common.sh
2024-12-18T00:29:48.0053043Z +++ dirname .ci/pytorch/common.sh
2024-12-18T00:29:48.0062170Z ++ source .ci/pytorch/common_utils.sh
2024-12-18T00:29:48.0064072Z +++ declare -f -t trap_add
2024-12-18T00:29:48.0069983Z ++ set -ex
2024-12-18T00:29:48.0070596Z ++ [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *rocm* ]]
2024-12-18T00:29:48.0071031Z ++ BUILD_TEST_LIBTORCH=0
2024-12-18T00:29:48.0071394Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *rocm* ]]
2024-12-18T00:29:48.0071763Z + [[ -d /var/lib/jenkins/workspace ]]
2024-12-18T00:29:48.0074595Z ++ stat -c %u /var/lib/jenkins/workspace
2024-12-18T00:29:48.0096259Z + WORKSPACE_ORIGINAL_OWNER_ID=1000
2024-12-18T00:29:48.0096580Z + trap_add cleanup_workspace EXIT
2024-12-18T00:29:48.0096888Z + trap_add_cmd=cleanup_workspace
2024-12-18T00:29:48.0097159Z + shift
2024-12-18T00:29:48.0097768Z + for trap_add_name in "$@"
2024-12-18T00:29:48.0106108Z +++ trap -p EXIT
2024-12-18T00:29:48.0109410Z ++ eval 'extract_trap_cmd '
2024-12-18T00:29:48.0109795Z +++ extract_trap_cmd
2024-12-18T00:29:48.0110046Z +++ printf '%s\n' ''
2024-12-18T00:29:48.0110321Z ++ printf '%s\n' cleanup_workspace
2024-12-18T00:29:48.0113741Z + trap -- '
2024-12-18T00:29:48.0114131Z cleanup_workspace' EXIT
2024-12-18T00:29:48.0114481Z + sudo chown -R jenkins /var/lib/jenkins/workspace
2024-12-18T00:29:48.7375383Z + git config --global --add safe.directory /var/lib/jenkins/workspace
2024-12-18T00:29:48.7401317Z + echo 'Environment variables:'
2024-12-18T00:29:48.7402320Z Environment variables:
2024-12-18T00:29:48.7402628Z + env
2024-12-18T00:29:48.7412852Z INSTALLED_DB=yes
2024-12-18T00:29:48.7413400Z NV_LIBCUBLAS_VERSION=12.4.5.8-1
2024-12-18T00:29:48.7413873Z NVIDIA_VISIBLE_DEVICES=all
2024-12-18T00:29:48.7414345Z NV_NVML_DEV_VERSION=12.4.127-1
2024-12-18T00:29:48.7415011Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-12-18T00:29:48.7415594Z CONTINUE_THROUGH_ERROR=False
2024-12-18T00:29:48.7415972Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.21.5-1+cuda12.4
2024-12-18T00:29:48.7416336Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.21.5-1
2024-12-18T00:29:48.7416752Z BUILD_ENVIRONMENT=linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-12-18T00:29:48.7417151Z HOSTNAME=2ca6b3c5afad
2024-12-18T00:29:48.7417726Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_06940b47-d8fd-460b-9247-78cfe3db2778
2024-12-18T00:29:48.7418330Z GITHUB_ACTION=__self
2024-12-18T00:29:48.7418602Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2024-12-18T00:29:48.7423189Z NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536
2024-12-18T00:29:48.7427619Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-4=12.4.5.8-1
2024-12-18T00:29:48.7427983Z NV_NVTX_VERSION=12.4.127-1
2024-12-18T00:29:48.7428253Z GITHUB_RUN_NUMBER=109429
2024-12-18T00:29:48.7428518Z TEST_CONFIG=inductor_timm
2024-12-18T00:29:48.7428800Z GITHUB_REPOSITORY_OWNER_ID=21003710
2024-12-18T00:29:48.7429131Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2024-12-18T00:29:48.7429436Z IS_A100_RUNNER=0
2024-12-18T00:29:48.7429685Z NV_CUDA_CUDART_DEV_VERSION=12.4.127-1
2024-12-18T00:29:48.7429997Z NV_LIBCUSPARSE_VERSION=12.3.1.170-1
2024-12-18T00:29:48.7430508Z SCRIBE_GRAPHQL_ACCESS_TOKEN=***
2024-12-18T00:29:48.7430809Z NV_LIBNPP_VERSION=12.2.5.30-1
2024-12-18T00:29:48.7431102Z GITHUB_TRIGGERING_ACTOR=malfet
2024-12-18T00:29:48.7431441Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2024-12-18T00:29:48.7431794Z GITHUB_REF_TYPE=branch
2024-12-18T00:29:48.7432060Z TORCH_CUDA_ARCH_LIST=Maxwell
2024-12-18T00:29:48.7432335Z NCCL_VERSION=2.21.5-1
2024-12-18T00:29:48.7432635Z BASE_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:48.7432967Z XLA_CUDA=
2024-12-18T00:29:48.7433300Z HUGGING_FACE_HUB_TOKEN=***
2024-12-18T00:29:48.7435492Z ***
2024-12-18T00:29:48.7435717Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2024-12-18T00:29:48.7436228Z GITHUB_REPOSITORY_ID=65600975
2024-12-18T00:29:48.7436510Z GITHUB_ACTIONS=true
2024-12-18T00:29:48.7436774Z NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:48.7437113Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4=12.4.127-1
2024-12-18T00:29:48.7437479Z NV_LIBNPP_PACKAGE=libnpp-12-4=12.2.5.30-1
2024-12-18T00:29:48.7437852Z SHA1=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:48.7438202Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2024-12-18T00:29:48.7438567Z GITHUB_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:48.7439120Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/release/2.6
2024-12-18T00:29:48.7439754Z UCC_HOME=/usr
2024-12-18T00:29:48.7440002Z NV_LIBCUBLAS_DEV_VERSION=12.4.5.8-1
2024-12-18T00:29:48.7440305Z VERBOSE_TEST_LOGS=False
2024-12-18T00:29:48.7440572Z NVIDIA_PRODUCT_NAME=CUDA
2024-12-18T00:29:48.7440887Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-4
2024-12-18T00:29:48.7441239Z GITHUB_REF=refs/heads/release/2.6
2024-12-18T00:29:48.7441551Z NV_CUDA_CUDART_VERSION=12.4.127-1
2024-12-18T00:29:48.7441842Z SHARD_NUMBER=1
2024-12-18T00:29:48.7442084Z GITHUB_REF_PROTECTED=true
2024-12-18T00:29:48.7442353Z HOME=/var/lib/jenkins
2024-12-18T00:29:48.7442634Z GITHUB_API_URL=https://api.github.com
2024-12-18T00:29:48.7442970Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2024-12-18T00:29:48.7443327Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb
2024-12-18T00:29:48.7443694Z SCCACHE_S3_KEY_PREFIX=inductor
2024-12-18T00:29:48.7443979Z CUDA_VERSION=12.4.1
2024-12-18T00:29:48.7444280Z NV_LIBCUBLAS_PACKAGE=libcublas-12-4=12.4.5.8-1
2024-12-18T00:29:48.7444640Z NUM_TEST_SHARDS=2
2024-12-18T00:29:48.7444880Z UCX_HOME=/usr
2024-12-18T00:29:48.7445229Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-4=12.4.1-1
2024-12-18T00:29:48.7445970Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_06940b47-d8fd-460b-9247-78cfe3db2778
2024-12-18T00:29:48.7446780Z JOB_NAME=cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:48.7447582Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_06940b47-d8fd-460b-9247-78cfe3db2778
2024-12-18T00:29:48.7448411Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2024-12-18T00:29:48.7448904Z GITHUB_EVENT_NAME=push
2024-12-18T00:29:48.7449308Z DASHBOARD_TAG=
2024-12-18T00:29:48.7449562Z GITHUB_RUN_ID=12383255690
2024-12-18T00:29:48.7449867Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-4=12.2.5.30-1
2024-12-18T00:29:48.7450239Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-4
2024-12-18T00:29:48.7450913Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_06940b47-d8fd-460b-9247-78cfe3db2778
2024-12-18T00:29:48.7451562Z GITHUB_ACTOR=malfet
2024-12-18T00:29:48.7451822Z NV_LIBNPP_DEV_VERSION=12.2.5.30-1
2024-12-18T00:29:48.7452105Z PR_NUMBER=
2024-12-18T00:29:48.7452339Z GITHUB_RUN_ATTEMPT=1
2024-12-18T00:29:48.7452605Z ANACONDA_PYTHON_VERSION=3.10
2024-12-18T00:29:48.7452952Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2024-12-18T00:29:48.7453300Z TERM=vt100
2024-12-18T00:29:48.7453543Z NV_LIBCUSPARSE_DEV_VERSION=12.3.1.170-1
2024-12-18T00:29:48.7453866Z INSTALLED_VISION=yes
2024-12-18T00:29:48.7454117Z BRANCH=release/2.6
2024-12-18T00:29:48.7454371Z SCCACHE_REGION=us-east-1
2024-12-18T00:29:48.7454658Z OPENSSL_ROOT_DIR=/opt/openssl
2024-12-18T00:29:48.7454957Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2024-12-18T00:29:48.7455281Z CUDA_PATH=/usr/local/cuda
2024-12-18T00:29:48.7455796Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2024-12-18T00:29:48.7456474Z GITHUB_SERVER_URL=https://github.com
2024-12-18T00:29:48.7456853Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b
2024-12-18T00:29:48.7457243Z REENABLED_ISSUES=
2024-12-18T00:29:48.7457508Z SHLVL=1
2024-12-18T00:29:48.7457713Z MAX_JOBS=14
2024-12-18T00:29:48.7457937Z NV_CUDA_LIB_VERSION=12.4.1-1
2024-12-18T00:29:48.7458277Z NVARCH=x86_64
2024-12-18T00:29:48.7458627Z GITHUB_ACTOR_ID=2453524
2024-12-18T00:29:48.7459028Z GITHUB_WORKFLOW_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:48.7459452Z GITHUB_REF_NAME=release/2.6
2024-12-18T00:29:48.7459777Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-4
2024-12-18T00:29:48.7460280Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2024-12-18T00:29:48.7460774Z GITHUB_JOB=test
2024-12-18T00:29:48.7461059Z NV_LIBNCCL_PACKAGE=libnccl2=2.21.5-1+cuda12.4
2024-12-18T00:29:48.7461496Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2024-12-18T00:29:48.7461938Z NO_TEST_TIMEOUT=False
2024-12-18T00:29:48.7462188Z TD_DISTRIBUTED=False
2024-12-18T00:29:48.7462536Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.1-1
2024-12-18T00:29:48.7462870Z GITHUB_REPOSITORY=pytorch/pytorch
2024-12-18T00:29:48.7463191Z NV_NVPROF_VERSION=12.4.127-1
2024-12-18T00:29:48.7463543Z GITHUB_RETENTION_DAYS=90
2024-12-18T00:29:48.7463811Z OPENSSL_DIR=/opt/openssl
2024-12-18T00:29:48.7464119Z GITHUB_ACTION_REPOSITORY=
2024-12-18T00:29:48.7464943Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-12-18T00:29:48.7465730Z GITHUB_BASE_REF=
2024-12-18T00:29:48.7466183Z ARTIFACTS_FILE_SUFFIX=test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_34567152834
2024-12-18T00:29:48.7466698Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2024-12-18T00:29:48.7466974Z CI=true
2024-12-18T00:29:48.7467210Z NV_LIBNCCL_PACKAGE_VERSION=2.21.5-1
2024-12-18T00:29:48.7467533Z GITHUB_REPOSITORY_OWNER=pytorch
2024-12-18T00:29:48.7467817Z JOB_ID=34567152834
2024-12-18T00:29:48.7468068Z INSTALLED_PROTOBUF=yes
2024-12-18T00:29:48.7468326Z GITHUB_HEAD_REF=
2024-12-18T00:29:48.7468567Z GITHUB_ACTION_REF=
2024-12-18T00:29:48.7468868Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2024-12-18T00:29:48.7469235Z TEST_SHOWLOCALS=False
2024-12-18T00:29:48.7469499Z GITHUB_WORKFLOW=inductor
2024-12-18T00:29:48.7469772Z DEBIAN_FRONTEND=noninteractive
2024-12-18T00:29:48.7470400Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_06940b47-d8fd-460b-9247-78cfe3db2778
2024-12-18T00:29:48.7471028Z NO_TD=False
2024-12-18T00:29:48.7471270Z SKIP_SCCACHE_INITIALIZATION=1
2024-12-18T00:29:48.7471549Z _=/usr/bin/env
2024-12-18T00:29:48.7471987Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2024-12-18T00:29:48.7651240Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch
2024-12-18T00:29:48.7651966Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2024-12-18T00:29:48.7652750Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib
2024-12-18T00:29:48.7653337Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test
2024-12-18T00:29:48.7653776Z + BUILD_DIR=build
2024-12-18T00:29:48.7654028Z + BUILD_RENAMED_DIR=build_renamed
2024-12-18T00:29:48.7654330Z + BUILD_BIN_DIR=build/bin
2024-12-18T00:29:48.7654592Z + SHARD_NUMBER=1
2024-12-18T00:29:48.7654842Z + NUM_TEST_SHARDS=2
2024-12-18T00:29:48.7655085Z + export VALGRIND=ON
2024-12-18T00:29:48.7655327Z + VALGRIND=ON
2024-12-18T00:29:48.7655637Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *clang9* ]]
2024-12-18T00:29:48.7656068Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *xpu* ]]
2024-12-18T00:29:48.7656423Z + [[ 0 == \1 ]]
2024-12-18T00:29:48.7656675Z + [[ False == \1 ]]
2024-12-18T00:29:48.7657030Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *bazel* ]]
2024-12-18T00:29:48.7658165Z ++ realpath build/custom_test_artifacts
2024-12-18T00:29:48.7672091Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2024-12-18T00:29:48.7672587Z + [[ -n '' ]]
2024-12-18T00:29:48.7672829Z + echo 'Environment variables'
2024-12-18T00:29:48.7673108Z Environment variables
2024-12-18T00:29:48.7673350Z + env
2024-12-18T00:29:48.7681657Z INSTALLED_DB=yes
2024-12-18T00:29:48.7682061Z NV_LIBCUBLAS_VERSION=12.4.5.8-1
2024-12-18T00:29:48.7682468Z NVIDIA_VISIBLE_DEVICES=all
2024-12-18T00:29:48.7683175Z NV_NVML_DEV_VERSION=12.4.127-1
2024-12-18T00:29:48.7683735Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-12-18T00:29:48.7684353Z CONTINUE_THROUGH_ERROR=False
2024-12-18T00:29:48.7684707Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.21.5-1+cuda12.4
2024-12-18T00:29:48.7685089Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.21.5-1
2024-12-18T00:29:48.7685479Z BUILD_ENVIRONMENT=linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-12-18T00:29:48.7685867Z HOSTNAME=2ca6b3c5afad
2024-12-18T00:29:48.7686650Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_06940b47-d8fd-460b-9247-78cfe3db2778
2024-12-18T00:29:48.7687323Z GITHUB_ACTION=__self
2024-12-18T00:29:48.7687601Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2024-12-18T00:29:48.7691899Z NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536
2024-12-18T00:29:48.7696867Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-4=12.4.5.8-1
2024-12-18T00:29:48.7697256Z NV_NVTX_VERSION=12.4.127-1
2024-12-18T00:29:48.7697538Z GITHUB_RUN_NUMBER=109429
2024-12-18T00:29:48.7697815Z TEST_CONFIG=inductor_timm
2024-12-18T00:29:48.7698103Z GITHUB_REPOSITORY_OWNER_ID=21003710
2024-12-18T00:29:48.7698440Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2024-12-18T00:29:48.7698759Z IS_A100_RUNNER=0
2024-12-18T00:29:48.7699214Z NV_CUDA_CUDART_DEV_VERSION=12.4.127-1
2024-12-18T00:29:48.7699650Z NV_LIBCUSPARSE_VERSION=12.3.1.170-1
2024-12-18T00:29:48.7700108Z SCRIBE_GRAPHQL_ACCESS_TOKEN=***
2024-12-18T00:29:48.7700406Z NV_LIBNPP_VERSION=12.2.5.30-1
2024-12-18T00:29:48.7700765Z GITHUB_TRIGGERING_ACTOR=malfet
2024-12-18T00:29:48.7701134Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2024-12-18T00:29:48.7701491Z GITHUB_REF_TYPE=branch
2024-12-18T00:29:48.7701754Z TORCH_CUDA_ARCH_LIST=Maxwell
2024-12-18T00:29:48.7702037Z NCCL_VERSION=2.21.5-1
2024-12-18T00:29:48.7702430Z BASE_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:48.7702768Z XLA_CUDA=
2024-12-18T00:29:48.7703129Z HUGGING_FACE_HUB_TOKEN=***
2024-12-18T00:29:48.7704067Z ***
2024-12-18T00:29:48.7704303Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2024-12-18T00:29:48.7704621Z GITHUB_REPOSITORY_ID=65600975
2024-12-18T00:29:48.7704905Z GITHUB_ACTIONS=true
2024-12-18T00:29:48.7705172Z NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:48.7705523Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4=12.4.127-1
2024-12-18T00:29:48.7705922Z NV_LIBNPP_PACKAGE=libnpp-12-4=12.2.5.30-1
2024-12-18T00:29:48.7706273Z SHA1=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:48.7706641Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2024-12-18T00:29:48.7707009Z GITHUB_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:48.7707562Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/release/2.6
2024-12-18T00:29:48.7708069Z UCC_HOME=/usr
2024-12-18T00:29:48.7708315Z NV_LIBCUBLAS_DEV_VERSION=12.4.5.8-1
2024-12-18T00:29:48.7708621Z VERBOSE_TEST_LOGS=False
2024-12-18T00:29:48.7708892Z NVIDIA_PRODUCT_NAME=CUDA
2024-12-18T00:29:48.7709382Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-4
2024-12-18T00:29:48.7709752Z GITHUB_REF=refs/heads/release/2.6
2024-12-18T00:29:48.7710055Z NV_CUDA_CUDART_VERSION=12.4.127-1
2024-12-18T00:29:48.7710346Z SHARD_NUMBER=1
2024-12-18T00:29:48.7710593Z GITHUB_REF_PROTECTED=true
2024-12-18T00:29:48.7710869Z HOME=/var/lib/jenkins
2024-12-18T00:29:48.7711161Z GITHUB_API_URL=https://api.github.com
2024-12-18T00:29:48.7711496Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2024-12-18T00:29:48.7711858Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb
2024-12-18T00:29:48.7712224Z SCCACHE_S3_KEY_PREFIX=inductor
2024-12-18T00:29:48.7712515Z CUDA_VERSION=12.4.1
2024-12-18T00:29:48.7712803Z NV_LIBCUBLAS_PACKAGE=libcublas-12-4=12.4.5.8-1
2024-12-18T00:29:48.7713132Z NUM_TEST_SHARDS=2
2024-12-18T00:29:48.7713372Z UCX_HOME=/usr
2024-12-18T00:29:48.7713725Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-4=12.4.1-1
2024-12-18T00:29:48.7714470Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_06940b47-d8fd-460b-9247-78cfe3db2778
2024-12-18T00:29:48.7715291Z JOB_NAME=cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:48.7716082Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_06940b47-d8fd-460b-9247-78cfe3db2778
2024-12-18T00:29:48.7716865Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2024-12-18T00:29:48.7717360Z GITHUB_EVENT_NAME=push
2024-12-18T00:29:48.7717618Z DASHBOARD_TAG=
2024-12-18T00:29:48.7717856Z GITHUB_RUN_ID=12383255690
2024-12-18T00:29:48.7718165Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-4=12.2.5.30-1
2024-12-18T00:29:48.7718534Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-4
2024-12-18T00:29:48.7719324Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_06940b47-d8fd-460b-9247-78cfe3db2778
2024-12-18T00:29:48.7719999Z GITHUB_ACTOR=malfet
2024-12-18T00:29:48.7720257Z NV_LIBNPP_DEV_VERSION=12.2.5.30-1
2024-12-18T00:29:48.7720550Z PR_NUMBER=
2024-12-18T00:29:48.7720773Z GITHUB_RUN_ATTEMPT=1
2024-12-18T00:29:48.7721018Z VALGRIND=ON
2024-12-18T00:29:48.7721254Z ANACONDA_PYTHON_VERSION=3.10
2024-12-18T00:29:48.7721594Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2024-12-18T00:29:48.7721941Z TERM=vt100
2024-12-18T00:29:48.7722315Z NV_LIBCUSPARSE_DEV_VERSION=12.3.1.170-1
2024-12-18T00:29:48.7722638Z INSTALLED_VISION=yes
2024-12-18T00:29:48.7722886Z BRANCH=release/2.6
2024-12-18T00:29:48.7723135Z SCCACHE_REGION=us-east-1
2024-12-18T00:29:48.7723411Z OPENSSL_ROOT_DIR=/opt/openssl
2024-12-18T00:29:48.7723708Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2024-12-18T00:29:48.7724035Z CUDA_PATH=/usr/local/cuda
2024-12-18T00:29:48.7724602Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2024-12-18T00:29:48.7725171Z GITHUB_SERVER_URL=https://github.com
2024-12-18T00:29:48.7725531Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b
2024-12-18T00:29:48.7725876Z REENABLED_ISSUES=
2024-12-18T00:29:48.7726111Z SHLVL=1
2024-12-18T00:29:48.7726317Z MAX_JOBS=14
2024-12-18T00:29:48.7726547Z NV_CUDA_LIB_VERSION=12.4.1-1
2024-12-18T00:29:48.7726821Z NVARCH=x86_64
2024-12-18T00:29:48.7727048Z GITHUB_ACTOR_ID=2453524
2024-12-18T00:29:48.7727391Z GITHUB_WORKFLOW_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:48.7727787Z GITHUB_REF_NAME=release/2.6
2024-12-18T00:29:48.7728089Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-4
2024-12-18T00:29:48.7728539Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2024-12-18T00:29:48.7728960Z GITHUB_JOB=test
2024-12-18T00:29:48.7729232Z NV_LIBNCCL_PACKAGE=libnccl2=2.21.5-1+cuda12.4
2024-12-18T00:29:48.7729642Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2024-12-18T00:29:48.7730020Z NO_TEST_TIMEOUT=False
2024-12-18T00:29:48.7730277Z TD_DISTRIBUTED=False
2024-12-18T00:29:48.7730550Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.1-1
2024-12-18T00:29:48.7730880Z GITHUB_REPOSITORY=pytorch/pytorch
2024-12-18T00:29:48.7731283Z NV_NVPROF_VERSION=12.4.127-1
2024-12-18T00:29:48.7731564Z GITHUB_RETENTION_DAYS=90
2024-12-18T00:29:48.7731828Z OPENSSL_DIR=/opt/openssl
2024-12-18T00:29:48.7732104Z GITHUB_ACTION_REPOSITORY=
2024-12-18T00:29:48.7732861Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-12-18T00:29:48.7733635Z GITHUB_BASE_REF=
2024-12-18T00:29:48.7734062Z ARTIFACTS_FILE_SUFFIX=test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_34567152834
2024-12-18T00:29:48.7734621Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2024-12-18T00:29:48.7734893Z CI=true
2024-12-18T00:29:48.7735123Z NV_LIBNCCL_PACKAGE_VERSION=2.21.5-1
2024-12-18T00:29:48.7735438Z GITHUB_REPOSITORY_OWNER=pytorch
2024-12-18T00:29:48.7735719Z JOB_ID=34567152834
2024-12-18T00:29:48.7735961Z INSTALLED_PROTOBUF=yes
2024-12-18T00:29:48.7736211Z GITHUB_HEAD_REF=
2024-12-18T00:29:48.7736445Z GITHUB_ACTION_REF=
2024-12-18T00:29:48.7736744Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2024-12-18T00:29:48.7737106Z TEST_SHOWLOCALS=False
2024-12-18T00:29:48.7737360Z GITHUB_WORKFLOW=inductor
2024-12-18T00:29:48.7737641Z DEBIAN_FRONTEND=noninteractive
2024-12-18T00:29:48.7738253Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_06940b47-d8fd-460b-9247-78cfe3db2778
2024-12-18T00:29:48.7738877Z NO_TD=False
2024-12-18T00:29:48.7739120Z SKIP_SCCACHE_INITIALIZATION=1
2024-12-18T00:29:48.7739393Z _=/usr/bin/env
2024-12-18T00:29:48.7739629Z + echo 'Testing pytorch'
2024-12-18T00:29:48.7739895Z Testing pytorch
2024-12-18T00:29:48.7740140Z + export LANG=C.UTF-8
2024-12-18T00:29:48.7740390Z + LANG=C.UTF-8
2024-12-18T00:29:48.7740611Z + PR_NUMBER=
2024-12-18T00:29:48.7740853Z + [[ inductor_timm == \d\e\f\a\u\l\t ]]
2024-12-18T00:29:48.7741188Z + [[ inductor_timm == \d\i\s\t\r\i\b\u\t\e\d ]]
2024-12-18T00:29:48.7741527Z + [[ inductor_timm == \s\l\o\w ]]
2024-12-18T00:29:48.7741920Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *slow-gradcheck* ]]
2024-12-18T00:29:48.7742382Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *cuda* ]]
2024-12-18T00:29:48.7742772Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2024-12-18T00:29:48.7743124Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2024-12-18T00:29:48.7743444Z + [[ inductor_timm == *crossref* ]]
2024-12-18T00:29:48.7743930Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *rocm* ]]
2024-12-18T00:29:48.7744339Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *xpu* ]]
2024-12-18T00:29:48.7744766Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]]
2024-12-18T00:29:48.7745152Z + pip_install --user ninja==1.10.2
2024-12-18T00:29:48.7745550Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T00:29:48.7746035Z + python3 -m pip install --progress-bar off --user ninja==1.10.2
2024-12-18T00:29:49.9495687Z Collecting ninja==1.10.2
2024-12-18T00:29:49.9674656Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB)
2024-12-18T00:29:50.0292409Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2024-12-18T00:29:50.6451656Z Installing collected packages: ninja
2024-12-18T00:29:50.6530701Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2024-12-18T00:29:50.6531861Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2024-12-18T00:29:50.6810089Z Successfully installed ninja-1.10.2
2024-12-18T00:29:50.7616072Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-12-18T00:29:50.7617620Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-12-18T00:29:50.7618578Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *aarch64* ]]
2024-12-18T00:29:50.7619370Z + install_tlparse
2024-12-18T00:29:50.7619642Z + pip_install --user tlparse==0.3.25
2024-12-18T00:29:50.7620044Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T00:29:50.7620537Z + python3 -m pip install --progress-bar off --user tlparse==0.3.25
2024-12-18T00:29:51.1924952Z Collecting tlparse==0.3.25
2024-12-18T00:29:51.2078567Z   Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB)
2024-12-18T00:29:51.2426498Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB)
2024-12-18T00:29:51.8803780Z Installing collected packages: tlparse
2024-12-18T00:29:51.9155351Z Successfully installed tlparse-0.3.25
2024-12-18T00:29:51.9954090Z ++ python -m site --user-base
2024-12-18T00:29:52.0201206Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-12-18T00:29:52.0202995Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *asan* ]]
2024-12-18T00:29:52.0203817Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *-debug* ]]
2024-12-18T00:29:52.0204394Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]]
2024-12-18T00:29:52.0205043Z + echo 'We are not in debug mode: linux-focal-cuda12.4-py3.10-gcc9-sm86. Expect the assertion to pass'
2024-12-18T00:29:52.0205764Z We are not in debug mode: linux-focal-cuda12.4-py3.10-gcc9-sm86. Expect the assertion to pass
2024-12-18T00:29:52.0206942Z + cd test
2024-12-18T00:29:52.0207423Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)'
2024-12-18T00:29:53.7015802Z + [[ inductor_timm == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2024-12-18T00:29:53.7016223Z + [[ inductor_timm == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2024-12-18T00:29:53.7020623Z + DYNAMO_BENCHMARK_FLAGS=()
2024-12-18T00:29:53.7021056Z + [[ inductor_timm == *pr_time_benchmarks* ]]
2024-12-18T00:29:53.7021501Z + [[ inductor_timm == *dynamo_eager* ]]
2024-12-18T00:29:53.7021821Z + [[ inductor_timm == *aot_eager* ]]
2024-12-18T00:29:53.7022130Z + [[ inductor_timm == *aot_inductor* ]]
2024-12-18T00:29:53.7022446Z + [[ inductor_timm == *inductor* ]]
2024-12-18T00:29:53.7022741Z + [[ inductor_timm != *perf* ]]
2024-12-18T00:29:53.7024424Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor)
2024-12-18T00:29:53.7024757Z + [[ inductor_timm == *dynamic* ]]
2024-12-18T00:29:53.7025055Z + [[ inductor_timm == *cpu* ]]
2024-12-18T00:29:53.7025355Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda)
2024-12-18T00:29:53.7053492Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *libtorch* ]]
2024-12-18T00:29:53.7053951Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *-bazel-* ]]
2024-12-18T00:29:53.7057757Z + cd test
2024-12-18T00:29:53.7058270Z + python -c 'import torch; print(torch.__config__.show())'
2024-12-18T00:29:55.2064341Z PyTorch built with:
2024-12-18T00:29:55.2064796Z   - GCC 9.4
2024-12-18T00:29:55.2065132Z   - C++ Version: 201703
2024-12-18T00:29:55.2065994Z   - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2024-12-18T00:29:55.2067081Z   - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af)
2024-12-18T00:29:55.2067754Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2024-12-18T00:29:55.2068294Z   - LAPACK is enabled (usually provided by MKL)
2024-12-18T00:29:55.2068803Z   - NNPACK is enabled
2024-12-18T00:29:55.2069205Z   - CPU capability usage: AVX2
2024-12-18T00:29:55.2069629Z   - CUDA Runtime 12.4
2024-12-18T00:29:55.2070293Z   - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86
2024-12-18T00:29:55.2071227Z   - CuDNN 90.1
2024-12-18T00:29:55.2071742Z   - Magma 2.6.1
2024-12-18T00:29:55.2079697Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7, CUDA_VERSION=12.4, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.6.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 
2024-12-18T00:29:55.2105247Z 
2024-12-18T00:29:55.5406500Z + cd test
2024-12-18T00:29:55.5407035Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2024-12-18T00:29:56.9184806Z ATen/Parallel:
2024-12-18T00:29:56.9185292Z 	at::get_num_threads() : 8
2024-12-18T00:29:56.9185743Z 	at::get_num_interop_threads() : 16
2024-12-18T00:29:56.9186256Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2024-12-18T00:29:56.9186713Z 	omp_get_max_threads() : 8
2024-12-18T00:29:56.9187542Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2024-12-18T00:29:56.9188402Z 	mkl_get_max_threads() : 8
2024-12-18T00:29:56.9188972Z Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af)
2024-12-18T00:29:56.9189626Z std::thread::hardware_concurrency() : 16
2024-12-18T00:29:56.9190103Z Environment variables:
2024-12-18T00:29:56.9190494Z 	OMP_NUM_THREADS : [not set]
2024-12-18T00:29:56.9190902Z 	MKL_NUM_THREADS : [not set]
2024-12-18T00:29:56.9191342Z ATen parallel backend: OpenMP
2024-12-18T00:29:56.9191632Z 
2024-12-18T00:29:57.2086215Z + [[ inductor_timm == *numpy_2* ]]
2024-12-18T00:29:57.2086746Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *aarch64* ]]
2024-12-18T00:29:57.2087149Z + [[ inductor_timm == *backward* ]]
2024-12-18T00:29:57.2087460Z + [[ inductor_timm == *xla* ]]
2024-12-18T00:29:57.2088220Z + [[ inductor_timm == *executorch* ]]
2024-12-18T00:29:57.2088557Z + [[ inductor_timm == \j\i\t\_\l\e\g\a\c\y ]]
2024-12-18T00:29:57.2088952Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *libtorch* ]]
2024-12-18T00:29:57.2089339Z + [[ inductor_timm == distributed ]]
2024-12-18T00:29:57.2089680Z + [[ inductor_timm == *inductor_distributed* ]]
2024-12-18T00:29:57.2090025Z + [[ inductor_timm == *inductor-halide* ]]
2024-12-18T00:29:57.2090380Z + [[ inductor_timm == *inductor-triton-cpu* ]]
2024-12-18T00:29:57.2090763Z + [[ inductor_timm == *inductor-micro-benchmark* ]]
2024-12-18T00:29:57.2091129Z + [[ inductor_timm == *huggingface* ]]
2024-12-18T00:29:57.2091458Z + [[ inductor_timm == *timm* ]]
2024-12-18T00:29:57.2091740Z + install_torchvision
2024-12-18T00:29:57.2091995Z + local orig_preload
2024-12-18T00:29:57.2092242Z + local commit
2024-12-18T00:29:57.2092485Z ++ get_pinned_commit vision
2024-12-18T00:29:57.2092783Z ++ cat .github/ci_commit_pins/vision.txt
2024-12-18T00:29:57.2110381Z + commit=d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:29:57.2110818Z + orig_preload=
2024-12-18T00:29:57.2111067Z + '[' -n '' ']'
2024-12-18T00:29:57.2111620Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:29:57.2112315Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T00:29:57.2113100Z + python3 -m pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:29:57.5488040Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:29:57.5492698Z   Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-ay4gg7pe
2024-12-18T00:29:57.5525437Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-ay4gg7pe
2024-12-18T00:29:59.0436660Z   Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f'
2024-12-18T00:29:59.0463463Z   Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:30:00.4153886Z   Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:30:00.7152196Z   Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:30:03.3584435Z   Preparing metadata (setup.py) ... [?25l- \ done
2024-12-18T00:30:03.3616308Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4)
2024-12-18T00:30:03.3619211Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.6.0a0+git0cdf8b1)
2024-12-18T00:30:03.3623418Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (11.0.0)
2024-12-18T00:30:03.3691448Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.16.1)
2024-12-18T00:30:03.3694863Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2)
2024-12-18T00:30:03.3698355Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8)
2024-12-18T00:30:03.3701231Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4)
2024-12-18T00:30:03.3704575Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.10.0)
2024-12-18T00:30:03.3710087Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.1)
2024-12-18T00:30:03.3723833Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0)
2024-12-18T00:30:03.4206153Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (3.0.2)
2024-12-18T00:30:03.4271780Z Building wheels for collected packages: torchvision
2024-12-18T00:31:18.8185730Z   Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done
2024-12-18T00:31:18.8206066Z [?25h  Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2028085 sha256=b781ec6d800f8bc9e4fa4da373c24d0fd1796cf90511a07f0a63d780bfbcb09e
2024-12-18T00:31:18.8207559Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96
2024-12-18T00:31:18.8247152Z Successfully built torchvision
2024-12-18T00:31:19.3438481Z Installing collected packages: torchvision
2024-12-18T00:31:19.7561722Z Successfully installed torchvision-0.19.0a0+d23a6e1
2024-12-18T00:31:19.9020553Z + '[' -n '' ']'
2024-12-18T00:31:19.9020860Z + id=0
2024-12-18T00:31:19.9021100Z + test_dynamo_benchmark timm_models 0
2024-12-18T00:31:19.9023368Z ++ pwd
2024-12-18T00:31:19.9026608Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2024-12-18T00:31:19.9027765Z + local suite=timm_models
2024-12-18T00:31:19.9028065Z + shift
2024-12-18T00:31:19.9028279Z + local shard_id=0
2024-12-18T00:31:19.9028504Z + shift
2024-12-18T00:31:19.9028739Z + [[ inductor_timm == *perf_compare* ]]
2024-12-18T00:31:19.9029063Z + [[ inductor_timm == *perf* ]]
2024-12-18T00:31:19.9029363Z + [[ inductor_timm == *cpu* ]]
2024-12-18T00:31:19.9029727Z + [[ inductor_timm == *aot_inductor* ]]
2024-12-18T00:31:19.9030371Z + test_single_dynamo_benchmark inference timm_models 0 --inference --bfloat16
2024-12-18T00:31:19.9031556Z ++ pwd
2024-12-18T00:31:19.9034626Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2024-12-18T00:31:19.9035272Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports
2024-12-18T00:31:19.9097669Z + local name=inference
2024-12-18T00:31:19.9098046Z + shift
2024-12-18T00:31:19.9098366Z + local suite=timm_models
2024-12-18T00:31:19.9098708Z + shift
2024-12-18T00:31:19.9098990Z + local shard_id=0
2024-12-18T00:31:19.9099324Z + shift
2024-12-18T00:31:19.9099628Z + partition_flags=()
2024-12-18T00:31:19.9099981Z + local partition_flags
2024-12-18T00:31:19.9100324Z + [[ -n 2 ]]
2024-12-18T00:31:19.9100624Z + [[ -n 0 ]]
2024-12-18T00:31:19.9101036Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id")
2024-12-18T00:31:19.9101553Z + [[ inductor_timm == *perf_compare* ]]
2024-12-18T00:31:19.9101866Z + [[ inductor_timm == *perf* ]]
2024-12-18T00:31:19.9102151Z + [[ inductor_timm == *_avx2* ]]
2024-12-18T00:31:19.9102441Z + [[ inductor_timm == *_avx512* ]]
2024-12-18T00:31:19.9103859Z + python benchmarks/dynamo/timm_models.py --ci --accuracy --timing --explain --inductor --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_timm_models.csv
2024-12-18T00:31:27.1835205Z 
2024-12-18T00:31:27.5106075Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:31:27.5106321Z 
2024-12-18T00:31:27.6500857Z model.safetensors:   0% 0.00/95.5M [00:00<?, ?B/s]
2024-12-18T00:31:27.6501161Z 
2024-12-18T00:31:27.7754403Z model.safetensors:  33% 31.5M/95.5M [00:00<00:00, 226MB/s]
2024-12-18T00:31:27.7754710Z 
2024-12-18T00:31:27.9075125Z model.safetensors:  66% 62.9M/95.5M [00:00<00:00, 240MB/s]
2024-12-18T00:31:27.9075508Z 
2024-12-18T00:31:27.9192865Z model.safetensors:  99% 94.4M/95.5M [00:00<00:00, 239MB/s]
2024-12-18T00:31:27.9193776Z model.safetensors: 100% 95.5M/95.5M [00:00<00:00, 234MB/s]
2024-12-18T00:31:28.7229415Z 
2024-12-18T00:31:28.7229764Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:31:28.7343720Z cuda eval  adv_inception_v3                   
2024-12-18T00:32:01.5180613Z pass
2024-12-18T00:32:01.5200266Z TIMING: _recursive_pre_grad_passes:0.03663 _recursive_joint_graph_passes:0.25963 _recursive_post_grad_passes:0.67021 async_compile.wait:1.7618 code_gen:9.01031 inductor_compile:15.11541 backend_compile:22.4707 entire_frame_compile:30.09673 total_wall_time:30.09673
2024-12-18T00:32:01.5201768Z STATS: call_* op count: 628 | FakeTensorMode.__torch_dispatch__:26240 | ProxyTorchDispatchMode.__torch_dispatch__:12549 | FakeTensor.__torch_dispatch__:4868
2024-12-18T00:32:01.5202638Z Dynamo produced 1 graphs covering 628 ops with 0 graph breaks (0 unique)
2024-12-18T00:32:07.0921363Z 
2024-12-18T00:32:08.1208624Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:32:08.1208998Z 
2024-12-18T00:32:08.2866470Z model.safetensors:   0% 0.00/350M [00:00<?, ?B/s]
2024-12-18T00:32:08.2866752Z 
2024-12-18T00:32:08.3921902Z model.safetensors:   6% 21.0M/350M [00:00<00:02, 127MB/s]
2024-12-18T00:32:08.3922223Z 
2024-12-18T00:32:08.5217406Z model.safetensors:  12% 41.9M/350M [00:00<00:01, 161MB/s]
2024-12-18T00:32:08.5217723Z 
2024-12-18T00:32:08.6542377Z model.safetensors:  21% 73.4M/350M [00:00<00:01, 198MB/s]
2024-12-18T00:32:08.6542762Z 
2024-12-18T00:32:08.7866241Z model.safetensors:  30% 105M/350M [00:00<00:01, 214MB/s] 
2024-12-18T00:32:08.7866668Z 
2024-12-18T00:32:08.9263637Z model.safetensors:  39% 136M/350M [00:00<00:00, 222MB/s]
2024-12-18T00:32:08.9264360Z 
2024-12-18T00:32:09.0624012Z model.safetensors:  48% 168M/350M [00:00<00:00, 223MB/s]
2024-12-18T00:32:09.0624322Z 
2024-12-18T00:32:09.1948313Z model.safetensors:  57% 199M/350M [00:00<00:00, 226MB/s]
2024-12-18T00:32:09.1948723Z 
2024-12-18T00:32:09.3232058Z model.safetensors:  66% 231M/350M [00:01<00:00, 230MB/s]
2024-12-18T00:32:09.3233323Z 
2024-12-18T00:32:09.4536815Z model.safetensors:  75% 262M/350M [00:01<00:00, 234MB/s]
2024-12-18T00:32:09.4537106Z 
2024-12-18T00:32:09.5806495Z model.safetensors:  84% 294M/350M [00:01<00:00, 236MB/s]
2024-12-18T00:32:09.5806782Z 
2024-12-18T00:32:09.6875865Z model.safetensors:  93% 325M/350M [00:01<00:00, 240MB/s]
2024-12-18T00:32:09.6876148Z 
2024-12-18T00:32:09.6900425Z model.safetensors: 100% 350M/350M [00:01<00:00, 238MB/s]
2024-12-18T00:32:09.6900886Z model.safetensors: 100% 350M/350M [00:01<00:00, 223MB/s]
2024-12-18T00:32:10.3003145Z 
2024-12-18T00:32:10.3003874Z loading model: 0it [00:03, ?it/s]
2024-12-18T00:32:10.3058109Z cuda eval  beit_base_patch16_224              
2024-12-18T00:32:27.6447011Z pass
2024-12-18T00:32:27.6472770Z TIMING: _recursive_pre_grad_passes:0.022 _recursive_joint_graph_passes:0.36299 _recursive_post_grad_passes:0.40594 async_compile.wait:1.66405 code_gen:3.69047 inductor_compile:6.7375 backend_compile:10.87498 entire_frame_compile:14.22591 total_wall_time:14.22591
2024-12-18T00:32:27.6474246Z STATS: call_* op count: 742 | FakeTensorMode.__torch_dispatch__:19986 | FakeTensor.__torch_dispatch__:2086 | ProxyTorchDispatchMode.__torch_dispatch__:8902
2024-12-18T00:32:27.6475091Z Dynamo produced 1 graphs covering 742 ops with 0 graph breaks (0 unique)
2024-12-18T00:32:32.7427004Z 
2024-12-18T00:32:32.9566513Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:32:32.9566885Z 
2024-12-18T00:32:33.0796267Z model.safetensors:   0% 0.00/50.1M [00:00<?, ?B/s]
2024-12-18T00:32:33.0796666Z 
2024-12-18T00:32:33.1080924Z model.safetensors:  84% 41.9M/50.1M [00:00<00:00, 341MB/s]
2024-12-18T00:32:33.1081601Z model.safetensors: 100% 50.1M/50.1M [00:00<00:00, 331MB/s]
2024-12-18T00:32:33.7158154Z 
2024-12-18T00:32:33.7158655Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:32:33.7198398Z cuda eval  botnet26t_256                      
2024-12-18T00:32:52.9906834Z pass
2024-12-18T00:32:52.9919568Z TIMING: _recursive_pre_grad_passes:0.01846 _recursive_joint_graph_passes:1.3676 _recursive_post_grad_passes:0.15491 async_compile.wait:1.63683 code_gen:6.05691 inductor_compile:9.09563 backend_compile:13.59258 entire_frame_compile:16.96496 total_wall_time:16.96496
2024-12-18T00:32:52.9921082Z STATS: call_* op count: 458 | FakeTensorMode.__torch_dispatch__:14290 | ProxyTorchDispatchMode.__torch_dispatch__:6234 | FakeTensor.__torch_dispatch__:1866
2024-12-18T00:32:52.9921928Z Dynamo produced 1 graphs covering 458 ops with 0 graph breaks (0 unique)
2024-12-18T00:32:58.0802587Z 
2024-12-18T00:33:00.8976708Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:33:00.8976979Z 
2024-12-18T00:33:01.0289903Z model.safetensors:   0% 0.00/1.08G [00:00<?, ?B/s]
2024-12-18T00:33:01.0290199Z 
2024-12-18T00:33:01.1529979Z model.safetensors:   3% 31.5M/1.08G [00:00<00:04, 240MB/s]
2024-12-18T00:33:01.1530316Z 
2024-12-18T00:33:01.2791148Z model.safetensors:   6% 62.9M/1.08G [00:00<00:04, 248MB/s]
2024-12-18T00:33:01.2791469Z 
2024-12-18T00:33:01.4055757Z model.safetensors:   9% 94.4M/1.08G [00:00<00:03, 248MB/s]
2024-12-18T00:33:01.4056079Z 
2024-12-18T00:33:01.5333415Z model.safetensors:  12% 126M/1.08G [00:00<00:03, 249MB/s] 
2024-12-18T00:33:01.5333844Z 
2024-12-18T00:33:01.6628962Z model.safetensors:  14% 157M/1.08G [00:00<00:03, 248MB/s]
2024-12-18T00:33:01.6629283Z 
2024-12-18T00:33:01.7942778Z model.safetensors:  17% 189M/1.08G [00:00<00:03, 246MB/s]
2024-12-18T00:33:01.7943085Z 
2024-12-18T00:33:01.9283851Z model.safetensors:  20% 220M/1.08G [00:00<00:03, 244MB/s]
2024-12-18T00:33:01.9284156Z 
2024-12-18T00:33:02.0572817Z model.safetensors:  23% 252M/1.08G [00:01<00:03, 241MB/s]
2024-12-18T00:33:02.0573554Z 
2024-12-18T00:33:02.1951638Z model.safetensors:  26% 283M/1.08G [00:01<00:03, 242MB/s]
2024-12-18T00:33:02.1951943Z 
2024-12-18T00:33:02.3233078Z model.safetensors:  29% 315M/1.08G [00:01<00:03, 237MB/s]
2024-12-18T00:33:02.3233387Z 
2024-12-18T00:33:02.4523302Z model.safetensors:  32% 346M/1.08G [00:01<00:03, 240MB/s]
2024-12-18T00:33:02.4523609Z 
2024-12-18T00:33:02.5914806Z model.safetensors:  35% 377M/1.08G [00:01<00:02, 241MB/s]
2024-12-18T00:33:02.5915112Z 
2024-12-18T00:33:02.7219683Z model.safetensors:  38% 409M/1.08G [00:01<00:02, 236MB/s]
2024-12-18T00:33:02.7220000Z 
2024-12-18T00:33:02.8519125Z model.safetensors:  41% 440M/1.08G [00:01<00:02, 238MB/s]
2024-12-18T00:33:02.8519535Z 
2024-12-18T00:33:03.0019140Z model.safetensors:  43% 472M/1.08G [00:01<00:02, 239MB/s]
2024-12-18T00:33:03.0019574Z 
2024-12-18T00:33:03.1333765Z model.safetensors:  46% 503M/1.08G [00:02<00:02, 229MB/s]
2024-12-18T00:33:03.1334070Z 
2024-12-18T00:33:03.2625800Z model.safetensors:  49% 535M/1.08G [00:02<00:02, 232MB/s]
2024-12-18T00:33:03.2626116Z 
2024-12-18T00:33:03.4009494Z model.safetensors:  52% 566M/1.08G [00:02<00:02, 236MB/s]
2024-12-18T00:33:03.4009895Z 
2024-12-18T00:33:03.5262567Z model.safetensors:  55% 598M/1.08G [00:02<00:02, 233MB/s]
2024-12-18T00:33:03.5262881Z 
2024-12-18T00:33:03.6604651Z model.safetensors:  58% 629M/1.08G [00:02<00:01, 238MB/s]
2024-12-18T00:33:03.6605091Z 
2024-12-18T00:33:03.7851615Z model.safetensors:  61% 661M/1.08G [00:02<00:01, 237MB/s]
2024-12-18T00:33:03.7851931Z 
2024-12-18T00:33:03.9117933Z model.safetensors:  64% 692M/1.08G [00:02<00:01, 241MB/s]
2024-12-18T00:33:03.9118253Z 
2024-12-18T00:33:04.0443129Z model.safetensors:  67% 724M/1.08G [00:03<00:01, 243MB/s]
2024-12-18T00:33:04.0443453Z 
2024-12-18T00:33:04.1784200Z model.safetensors:  70% 755M/1.08G [00:03<00:01, 242MB/s]
2024-12-18T00:33:04.1784538Z 
2024-12-18T00:33:04.3165774Z model.safetensors:  72% 786M/1.08G [00:03<00:01, 239MB/s]
2024-12-18T00:33:04.3166103Z 
2024-12-18T00:33:04.4539668Z model.safetensors:  75% 818M/1.08G [00:03<00:01, 236MB/s]
2024-12-18T00:33:04.4539967Z 
2024-12-18T00:33:04.5833887Z model.safetensors:  78% 849M/1.08G [00:03<00:01, 234MB/s]
2024-12-18T00:33:04.5834190Z 
2024-12-18T00:33:04.7112883Z model.safetensors:  81% 881M/1.08G [00:03<00:00, 236MB/s]
2024-12-18T00:33:04.7113491Z 
2024-12-18T00:33:04.8442556Z model.safetensors:  84% 912M/1.08G [00:03<00:00, 239MB/s]
2024-12-18T00:33:04.8442864Z 
2024-12-18T00:33:04.9736239Z model.safetensors:  87% 944M/1.08G [00:03<00:00, 238MB/s]
2024-12-18T00:33:04.9736561Z 
2024-12-18T00:33:05.1024143Z model.safetensors:  90% 975M/1.08G [00:04<00:00, 240MB/s]
2024-12-18T00:33:05.1024445Z 
2024-12-18T00:33:05.2310122Z model.safetensors:  93% 1.01G/1.08G [00:04<00:00, 241MB/s]
2024-12-18T00:33:05.2310757Z 
2024-12-18T00:33:05.3595902Z model.safetensors:  96% 1.04G/1.08G [00:04<00:00, 242MB/s]
2024-12-18T00:33:05.3596327Z 
2024-12-18T00:33:05.4256348Z model.safetensors:  99% 1.07G/1.08G [00:04<00:00, 243MB/s]
2024-12-18T00:33:05.4257012Z model.safetensors: 100% 1.08G/1.08G [00:04<00:00, 240MB/s]
2024-12-18T00:33:06.7183209Z 
2024-12-18T00:33:06.7183833Z loading model: 0it [00:08, ?it/s]
2024-12-18T00:33:06.7348910Z cuda eval  cait_m36_384                       
2024-12-18T00:34:14.9134776Z pass
2024-12-18T00:34:14.9211410Z TIMING: _recursive_pre_grad_passes:0.06841 _recursive_joint_graph_passes:1.34756 _recursive_post_grad_passes:2.49645 async_compile.wait:1.3657 code_gen:8.02774 inductor_compile:19.52684 backend_compile:37.00358 entire_frame_compile:48.37414 total_wall_time:48.37414
2024-12-18T00:34:14.9213771Z STATS: call_* op count: 2510 | FakeTensorMode.__torch_dispatch__:87529 | FakeTensor.__torch_dispatch__:7916 | ProxyTorchDispatchMode.__torch_dispatch__:39903
2024-12-18T00:34:14.9215061Z Dynamo produced 1 graphs covering 2510 ops with 0 graph breaks (0 unique)
2024-12-18T00:34:21.5346113Z 
2024-12-18T00:34:21.7319081Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:34:21.7319987Z 
2024-12-18T00:34:21.8426229Z pytorch_model.bin:   0% 0.00/44.1M [00:00<?, ?B/s]
2024-12-18T00:34:21.8426752Z 
2024-12-18T00:34:21.9465012Z pytorch_model.bin:  48% 21.0M/44.1M [00:00<00:00, 190MB/s]
2024-12-18T00:34:21.9465299Z 
2024-12-18T00:34:21.9478838Z pytorch_model.bin: 100% 44.1M/44.1M [00:00<00:00, 209MB/s]
2024-12-18T00:34:21.9479423Z pytorch_model.bin: 100% 44.1M/44.1M [00:00<00:00, 204MB/s]
2024-12-18T00:34:22.5166248Z 
2024-12-18T00:34:22.5166493Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:34:22.5215158Z cuda eval  coat_lite_mini                     
2024-12-18T00:34:44.5125136Z pass
2024-12-18T00:34:44.5139042Z TIMING: _recursive_pre_grad_passes:0.02708 _recursive_joint_graph_passes:2.58162 _recursive_post_grad_passes:0.36077 async_compile.wait:0.66505 code_gen:5.35702 inductor_compile:9.53715 backend_compile:16.25684 entire_frame_compile:19.93022 total_wall_time:19.93022
2024-12-18T00:34:44.5140539Z STATS: call_* op count: 904 | FakeTensorMode.__torch_dispatch__:23194 | FakeTensor.__torch_dispatch__:1967 | ProxyTorchDispatchMode.__torch_dispatch__:9505
2024-12-18T00:34:44.5141400Z Dynamo produced 1 graphs covering 904 ops with 0 graph breaks (0 unique)
2024-12-18T00:34:49.9839694Z 
2024-12-18T00:34:50.9201731Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:34:50.9202024Z 
2024-12-18T00:34:51.0213154Z model.safetensors:   0% 0.00/346M [00:00<?, ?B/s]
2024-12-18T00:34:51.0213443Z 
2024-12-18T00:34:51.1480062Z model.safetensors:   6% 21.0M/346M [00:00<00:01, 207MB/s]
2024-12-18T00:34:51.1480371Z 
2024-12-18T00:34:51.2842810Z model.safetensors:  15% 52.4M/346M [00:00<00:01, 234MB/s]
2024-12-18T00:34:51.2843123Z 
2024-12-18T00:34:51.4115576Z model.safetensors:  24% 83.9M/346M [00:00<00:01, 232MB/s]
2024-12-18T00:34:51.4115897Z 
2024-12-18T00:34:51.5483193Z model.safetensors:  33% 115M/346M [00:00<00:00, 238MB/s] 
2024-12-18T00:34:51.5483507Z 
2024-12-18T00:34:51.6800784Z model.safetensors:  42% 147M/346M [00:00<00:00, 235MB/s]
2024-12-18T00:34:51.6801137Z 
2024-12-18T00:34:51.8134632Z model.safetensors:  51% 178M/346M [00:00<00:00, 236MB/s]
2024-12-18T00:34:51.8134940Z 
2024-12-18T00:34:51.9453268Z model.safetensors:  61% 210M/346M [00:00<00:00, 236MB/s]
2024-12-18T00:34:51.9453598Z 
2024-12-18T00:34:52.0747082Z model.safetensors:  70% 241M/346M [00:01<00:00, 237MB/s]
2024-12-18T00:34:52.0747899Z 
2024-12-18T00:34:52.2094863Z model.safetensors:  79% 273M/346M [00:01<00:00, 239MB/s]
2024-12-18T00:34:52.2095173Z 
2024-12-18T00:34:52.3414786Z model.safetensors:  88% 304M/346M [00:01<00:00, 237MB/s]
2024-12-18T00:34:52.3415090Z 
2024-12-18T00:34:52.3941714Z model.safetensors:  97% 336M/346M [00:01<00:00, 237MB/s]
2024-12-18T00:34:52.3942284Z model.safetensors: 100% 346M/346M [00:01<00:00, 235MB/s]
2024-12-18T00:34:52.7368027Z WARNING:common:Model convit_base does not support bfloat16, running with amp instead
2024-12-18T00:34:53.1023516Z 
2024-12-18T00:34:53.1023897Z loading model: 0it [00:03, ?it/s]
2024-12-18T00:34:53.1024404Z WARNING:common:Model convit_base does not support bfloat16, running with amp instead
2024-12-18T00:34:53.1025153Z cuda eval  convit_base                        
2024-12-18T00:34:53.1569079Z WARNING:common:fp64 golden ref were not generated for convit_base. Setting accuracy check to cosine
2024-12-18T00:34:53.2184020Z WARNING:common:Model convit_base does not support bfloat16, running with amp instead
2024-12-18T00:35:17.2951468Z pass
2024-12-18T00:35:17.3064013Z TIMING: _recursive_pre_grad_passes:0.03026 _recursive_joint_graph_passes:2.57034 _recursive_post_grad_passes:0.73066 async_compile.wait:0.70334 code_gen:5.07069 inductor_compile:9.10808 backend_compile:18.03639 entire_frame_compile:22.09711 total_wall_time:22.09711
2024-12-18T00:35:17.3065567Z STATS: call_* op count: 1140 | FakeTensorMode.__torch_dispatch__:36754 | FakeTensor.__torch_dispatch__:3323 | ProxyTorchDispatchMode.__torch_dispatch__:16633
2024-12-18T00:35:17.3066410Z Dynamo produced 1 graphs covering 1140 ops with 0 graph breaks (0 unique)
2024-12-18T00:35:22.7233554Z 
2024-12-18T00:35:22.9267777Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:35:22.9268066Z 
2024-12-18T00:35:23.0664008Z model.safetensors:   0% 0.00/84.9M [00:00<?, ?B/s]
2024-12-18T00:35:23.0664435Z 
2024-12-18T00:35:23.1921058Z model.safetensors:  37% 31.5M/84.9M [00:00<00:00, 225MB/s]
2024-12-18T00:35:23.1921377Z 
2024-12-18T00:35:23.2887405Z model.safetensors:  74% 62.9M/84.9M [00:00<00:00, 239MB/s]
2024-12-18T00:35:23.2887881Z model.safetensors: 100% 84.9M/84.9M [00:00<00:00, 235MB/s]
2024-12-18T00:35:23.8568970Z 
2024-12-18T00:35:23.8569552Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:35:23.8659883Z cuda eval  convmixer_768_32                   
2024-12-18T00:35:44.1964674Z pass
2024-12-18T00:35:44.1969084Z TIMING: _recursive_pre_grad_passes:0.02304 _recursive_joint_graph_passes:0.43658 _recursive_post_grad_passes:0.28239 async_compile.wait:0.44435 code_gen:1.72241 inductor_compile:5.68951 backend_compile:11.13388 entire_frame_compile:16.70297 total_wall_time:16.70297
2024-12-18T00:35:44.1971589Z STATS: call_* op count: 462 | FakeTensorMode.__torch_dispatch__:19468 | ProxyTorchDispatchMode.__torch_dispatch__:9515 | FakeTensor.__torch_dispatch__:3553
2024-12-18T00:35:44.1972889Z Dynamo produced 1 graphs covering 462 ops with 0 graph breaks (0 unique)
2024-12-18T00:35:49.3781751Z 
2024-12-18T00:35:50.3257400Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:35:50.3257708Z 
2024-12-18T00:35:50.4637216Z model.safetensors:   0% 0.00/354M [00:00<?, ?B/s]
2024-12-18T00:35:50.4637503Z 
2024-12-18T00:35:50.5664399Z model.safetensors:   6% 21.0M/354M [00:00<00:02, 152MB/s]
2024-12-18T00:35:50.5664695Z 
2024-12-18T00:35:50.7000499Z model.safetensors:  12% 41.9M/354M [00:00<00:01, 179MB/s]
2024-12-18T00:35:50.7000813Z 
2024-12-18T00:35:50.8283200Z model.safetensors:  21% 73.4M/354M [00:00<00:01, 207MB/s]
2024-12-18T00:35:50.8283510Z 
2024-12-18T00:35:50.9532598Z model.safetensors:  30% 105M/354M [00:00<00:01, 222MB/s] 
2024-12-18T00:35:50.9532920Z 
2024-12-18T00:35:51.0825668Z model.safetensors:  38% 136M/354M [00:00<00:00, 233MB/s]
2024-12-18T00:35:51.0826038Z 
2024-12-18T00:35:51.2105067Z model.safetensors:  47% 168M/354M [00:00<00:00, 236MB/s]
2024-12-18T00:35:51.2105391Z 
2024-12-18T00:35:51.3395299Z model.safetensors:  56% 199M/354M [00:00<00:00, 240MB/s]
2024-12-18T00:35:51.3395615Z 
2024-12-18T00:35:51.4656117Z model.safetensors:  65% 231M/354M [00:01<00:00, 241MB/s]
2024-12-18T00:35:51.4656443Z 
2024-12-18T00:35:51.5924934Z model.safetensors:  74% 262M/354M [00:01<00:00, 244MB/s]
2024-12-18T00:35:51.5925271Z 
2024-12-18T00:35:51.7217378Z model.safetensors:  83% 294M/354M [00:01<00:00, 245MB/s]
2024-12-18T00:35:51.7217676Z 
2024-12-18T00:35:51.8534074Z model.safetensors:  92% 325M/354M [00:01<00:00, 244MB/s]
2024-12-18T00:35:51.8534372Z 
2024-12-18T00:35:51.8586958Z model.safetensors: 100% 354M/354M [00:01<00:00, 238MB/s]
2024-12-18T00:35:51.8587393Z model.safetensors: 100% 354M/354M [00:01<00:00, 231MB/s]
2024-12-18T00:35:52.6193469Z 
2024-12-18T00:35:52.6194554Z loading model: 0it [00:03, ?it/s]
2024-12-18T00:35:52.6281934Z cuda eval  convnext_base                      
2024-12-18T00:36:17.4423926Z pass
2024-12-18T00:36:17.4435570Z TIMING: _recursive_pre_grad_passes:0.02844 _recursive_joint_graph_passes:0.45505 _recursive_post_grad_passes:0.79699 async_compile.wait:0.70107 code_gen:3.44162 inductor_compile:8.0875 backend_compile:14.3969 entire_frame_compile:20.52033 total_wall_time:20.52033
2024-12-18T00:36:17.4438210Z STATS: call_* op count: 910 | FakeTensorMode.__torch_dispatch__:26585 | FakeTensor.__torch_dispatch__:3259 | ProxyTorchDispatchMode.__torch_dispatch__:12573
2024-12-18T00:36:17.4439675Z Dynamo produced 1 graphs covering 910 ops with 0 graph breaks (0 unique)
2024-12-18T00:36:22.7608108Z 
2024-12-18T00:36:22.9496381Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:36:22.9496653Z 
2024-12-18T00:36:23.0946070Z model.safetensors:   0% 0.00/34.2M [00:00<?, ?B/s]
2024-12-18T00:36:23.0946366Z 
2024-12-18T00:36:23.1087627Z model.safetensors:  92% 31.5M/34.2M [00:00<00:00, 217MB/s]
2024-12-18T00:36:23.1088783Z model.safetensors: 100% 34.2M/34.2M [00:00<00:00, 215MB/s]
2024-12-18T00:36:23.7053608Z 
2024-12-18T00:36:23.7054108Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:36:23.7123747Z cuda eval  crossvit_9_240                     
2024-12-18T00:36:50.4641593Z pass
2024-12-18T00:36:50.4656960Z TIMING: _recursive_pre_grad_passes:0.02963 _recursive_joint_graph_passes:1.92956 _recursive_post_grad_passes:0.80948 async_compile.wait:1.21636 code_gen:6.14645 inductor_compile:11.01718 backend_compile:19.36498 entire_frame_compile:24.62454 total_wall_time:24.62454
2024-12-18T00:36:50.4659198Z STATS: call_* op count: 974 | FakeTensorMode.__torch_dispatch__:34917 | FakeTensor.__torch_dispatch__:3649 | ProxyTorchDispatchMode.__torch_dispatch__:16243
2024-12-18T00:36:50.4660056Z Dynamo produced 1 graphs covering 974 ops with 0 graph breaks (0 unique)
2024-12-18T00:36:55.9705500Z 
2024-12-18T00:36:56.3495387Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:36:56.3495681Z 
2024-12-18T00:36:56.4508108Z model.safetensors:   0% 0.00/111M [00:00<?, ?B/s]
2024-12-18T00:36:56.4508396Z 
2024-12-18T00:36:56.5788652Z model.safetensors:  19% 21.0M/111M [00:00<00:00, 208MB/s]
2024-12-18T00:36:56.5788956Z 
2024-12-18T00:36:56.7031685Z model.safetensors:  47% 52.4M/111M [00:00<00:00, 232MB/s]
2024-12-18T00:36:56.7032141Z 
2024-12-18T00:36:56.8113788Z model.safetensors:  76% 83.9M/111M [00:00<00:00, 242MB/s]
2024-12-18T00:36:56.8114237Z 
2024-12-18T00:36:56.8151959Z model.safetensors: 100% 111M/111M [00:00<00:00, 244MB/s] 
2024-12-18T00:36:56.8152560Z model.safetensors: 100% 111M/111M [00:00<00:00, 238MB/s]
2024-12-18T00:36:57.4644390Z 
2024-12-18T00:36:57.4644739Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:36:57.4723063Z cuda eval  cspdarknet53                       
2024-12-18T00:37:20.3070202Z pass
2024-12-18T00:37:20.3083043Z TIMING: _recursive_pre_grad_passes:0.0243 _recursive_joint_graph_passes:0.46405 _recursive_post_grad_passes:0.37852 async_compile.wait:0.46089 code_gen:3.48117 inductor_compile:7.84189 backend_compile:13.82225 entire_frame_compile:19.92907 total_wall_time:19.92907
2024-12-18T00:37:20.3084618Z STATS: call_* op count: 506 | FakeTensorMode.__torch_dispatch__:20099 | FakeTensor.__torch_dispatch__:3691 | ProxyTorchDispatchMode.__torch_dispatch__:10314
2024-12-18T00:37:20.3085780Z Dynamo produced 1 graphs covering 506 ops with 0 graph breaks (0 unique)
2024-12-18T00:37:25.6188961Z 
2024-12-18T00:37:26.5872083Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:37:26.5872339Z 
2024-12-18T00:37:26.7069092Z model.safetensors:   0% 0.00/349M [00:00<?, ?B/s]
2024-12-18T00:37:26.7069380Z 
2024-12-18T00:37:26.8100515Z model.safetensors:   6% 21.0M/349M [00:00<00:01, 175MB/s]
2024-12-18T00:37:26.8100822Z 
2024-12-18T00:37:26.9345852Z model.safetensors:  12% 41.9M/349M [00:00<00:01, 191MB/s]
2024-12-18T00:37:26.9346155Z 
2024-12-18T00:37:27.0717385Z model.safetensors:  21% 73.4M/349M [00:00<00:01, 221MB/s]
2024-12-18T00:37:27.0717720Z 
2024-12-18T00:37:27.1984070Z model.safetensors:  30% 105M/349M [00:00<00:01, 225MB/s] 
2024-12-18T00:37:27.1985522Z 
2024-12-18T00:37:27.3227167Z model.safetensors:  39% 136M/349M [00:00<00:00, 233MB/s]
2024-12-18T00:37:27.3227481Z 
2024-12-18T00:37:27.4520211Z model.safetensors:  48% 168M/349M [00:00<00:00, 240MB/s]
2024-12-18T00:37:27.4520551Z 
2024-12-18T00:37:27.5831601Z model.safetensors:  57% 199M/349M [00:00<00:00, 241MB/s]
2024-12-18T00:37:27.5831910Z 
2024-12-18T00:37:27.7036584Z model.safetensors:  66% 231M/349M [00:00<00:00, 241MB/s]
2024-12-18T00:37:27.7036893Z 
2024-12-18T00:37:27.8316650Z model.safetensors:  75% 262M/349M [00:01<00:00, 247MB/s]
2024-12-18T00:37:27.8316964Z 
2024-12-18T00:37:27.9623624Z model.safetensors:  84% 294M/349M [00:01<00:00, 247MB/s]
2024-12-18T00:37:27.9623955Z 
2024-12-18T00:37:28.0650902Z model.safetensors:  93% 325M/349M [00:01<00:00, 245MB/s]
2024-12-18T00:37:28.0651370Z model.safetensors: 100% 349M/349M [00:01<00:00, 236MB/s]
2024-12-18T00:37:28.6705539Z 
2024-12-18T00:37:28.6706628Z loading model: 0it [00:03, ?it/s]
2024-12-18T00:37:28.6747725Z cuda eval  deit_base_distilled_patch16_224    
2024-12-18T00:37:42.1202480Z pass
2024-12-18T00:37:42.1209950Z TIMING: _recursive_pre_grad_passes:0.01679 _recursive_joint_graph_passes:0.335 _recursive_post_grad_passes:0.40219 async_compile.wait:0.39514 code_gen:1.8146 inductor_compile:4.44782 backend_compile:7.74779 entire_frame_compile:10.38809 total_wall_time:10.38809
2024-12-18T00:37:42.1211427Z STATS: call_* op count: 534 | FakeTensorMode.__torch_dispatch__:15782 | FakeTensor.__torch_dispatch__:1729 | ProxyTorchDispatchMode.__torch_dispatch__:7476
2024-12-18T00:37:42.1212262Z Dynamo produced 1 graphs covering 534 ops with 0 graph breaks (0 unique)
2024-12-18T00:37:47.1009271Z 
2024-12-18T00:37:47.5189954Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:37:47.5190283Z 
2024-12-18T00:37:47.6349160Z model.safetensors:   0% 0.00/133M [00:00<?, ?B/s]
2024-12-18T00:37:47.6349468Z 
2024-12-18T00:37:47.7629614Z model.safetensors:  16% 21.0M/133M [00:00<00:00, 181MB/s]
2024-12-18T00:37:47.7629920Z 
2024-12-18T00:37:47.8999016Z model.safetensors:  39% 52.4M/133M [00:00<00:00, 221MB/s]
2024-12-18T00:37:47.8999484Z 
2024-12-18T00:37:48.0272751Z model.safetensors:  63% 83.9M/133M [00:00<00:00, 225MB/s]
2024-12-18T00:37:48.0273086Z 
2024-12-18T00:37:48.1075462Z model.safetensors:  86% 115M/133M [00:00<00:00, 234MB/s] 
2024-12-18T00:37:48.1075924Z model.safetensors: 100% 133M/133M [00:00<00:00, 227MB/s]
2024-12-18T00:37:48.7762387Z 
2024-12-18T00:37:48.7763213Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:37:48.7884506Z cuda eval  dla102                             
2024-12-18T00:38:18.4764127Z pass
2024-12-18T00:38:18.4774281Z TIMING: _recursive_pre_grad_passes:0.03567 _recursive_joint_graph_passes:0.2665 _recursive_post_grad_passes:0.72881 async_compile.wait:0.44478 code_gen:3.47058 inductor_compile:9.49103 backend_compile:17.78833 entire_frame_compile:26.66088 total_wall_time:26.66088
2024-12-18T00:38:18.4776233Z STATS: call_* op count: 760 | FakeTensorMode.__torch_dispatch__:29045 | ProxyTorchDispatchMode.__torch_dispatch__:14100 | FakeTensor.__torch_dispatch__:5528
2024-12-18T00:38:18.4777078Z Dynamo produced 1 graphs covering 760 ops with 0 graph breaks (0 unique)
2024-12-18T00:38:23.9791222Z 
2024-12-18T00:38:24.7681171Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:38:24.7681657Z 
2024-12-18T00:38:24.9076226Z model.safetensors:   0% 0.00/286M [00:00<?, ?B/s]
2024-12-18T00:38:24.9076502Z 
2024-12-18T00:38:25.0308627Z model.safetensors:  11% 31.5M/286M [00:00<00:01, 225MB/s]
2024-12-18T00:38:25.0308919Z 
2024-12-18T00:38:25.1559106Z model.safetensors:  22% 62.9M/286M [00:00<00:00, 242MB/s]
2024-12-18T00:38:25.1559390Z 
2024-12-18T00:38:25.2882561Z model.safetensors:  33% 94.4M/286M [00:00<00:00, 246MB/s]
2024-12-18T00:38:25.2882840Z 
2024-12-18T00:38:25.4267870Z model.safetensors:  44% 126M/286M [00:00<00:00, 243MB/s] 
2024-12-18T00:38:25.4268172Z 
2024-12-18T00:38:25.5732954Z model.safetensors:  55% 157M/286M [00:00<00:00, 237MB/s]
2024-12-18T00:38:25.5733257Z 
2024-12-18T00:38:25.7005076Z model.safetensors:  66% 189M/286M [00:00<00:00, 229MB/s]
2024-12-18T00:38:25.7005362Z 
2024-12-18T00:38:25.8321558Z model.safetensors:  77% 220M/286M [00:00<00:00, 235MB/s]
2024-12-18T00:38:25.8321839Z 
2024-12-18T00:38:25.9635473Z model.safetensors:  88% 252M/286M [00:01<00:00, 236MB/s]
2024-12-18T00:38:25.9635757Z 
2024-12-18T00:38:25.9786165Z model.safetensors:  99% 283M/286M [00:01<00:00, 237MB/s]
2024-12-18T00:38:25.9787004Z model.safetensors: 100% 286M/286M [00:01<00:00, 236MB/s]
2024-12-18T00:38:26.7340459Z 
2024-12-18T00:38:26.7340642Z loading model: 0it [00:02, ?it/s]
2024-12-18T00:38:26.7398499Z cuda eval  dm_nfnet_f0                        
2024-12-18T00:38:50.8404981Z pass
2024-12-18T00:38:50.8417160Z TIMING: _recursive_pre_grad_passes:0.03523 _recursive_joint_graph_passes:0.56633 _recursive_post_grad_passes:0.34482 async_compile.wait:0.46776 code_gen:4.23291 inductor_compile:8.37999 backend_compile:14.61556 entire_frame_compile:20.45507 total_wall_time:20.45507
2024-12-18T00:38:50.8422067Z STATS: call_* op count: 1180 | FakeTensorMode.__torch_dispatch__:24129 | ProxyTorchDispatchMode.__torch_dispatch__:10762 | FakeTensor.__torch_dispatch__:3546
2024-12-18T00:38:50.8423352Z Dynamo produced 1 graphs covering 1180 ops with 0 graph breaks (0 unique)
2024-12-18T00:38:56.1938441Z 
2024-12-18T00:38:56.7033499Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:38:56.7033757Z 
2024-12-18T00:38:56.8436706Z model.safetensors:   0% 0.00/349M [00:00<?, ?B/s]
2024-12-18T00:38:56.8436985Z 
2024-12-18T00:38:56.9611308Z model.safetensors:   9% 31.5M/349M [00:00<00:01, 224MB/s]
2024-12-18T00:38:56.9611625Z 
2024-12-18T00:38:57.0923814Z model.safetensors:  18% 62.9M/349M [00:00<00:01, 248MB/s]
2024-12-18T00:38:57.0924123Z 
2024-12-18T00:38:57.2223881Z model.safetensors:  27% 94.4M/349M [00:00<00:01, 244MB/s]
2024-12-18T00:38:57.2224190Z 
2024-12-18T00:38:57.3522726Z model.safetensors:  36% 126M/349M [00:00<00:00, 243MB/s] 
2024-12-18T00:38:57.3523044Z 
2024-12-18T00:38:57.4841956Z model.safetensors:  45% 157M/349M [00:00<00:00, 243MB/s]
2024-12-18T00:38:57.4842265Z 
2024-12-18T00:38:57.6142880Z model.safetensors:  54% 189M/349M [00:00<00:00, 241MB/s]
2024-12-18T00:38:57.6143196Z 
2024-12-18T00:38:57.7562729Z model.safetensors:  63% 220M/349M [00:00<00:00, 242MB/s]
2024-12-18T00:38:57.7563036Z 
2024-12-18T00:38:57.8833937Z model.safetensors:  72% 252M/349M [00:01<00:00, 235MB/s]
2024-12-18T00:38:57.8834261Z 
2024-12-18T00:38:58.0136481Z model.safetensors:  81% 283M/349M [00:01<00:00, 239MB/s]
2024-12-18T00:38:58.0136797Z 
2024-12-18T00:38:58.1430421Z model.safetensors:  90% 315M/349M [00:01<00:00, 240MB/s]
2024-12-18T00:38:58.1430727Z 
2024-12-18T00:38:58.1584420Z model.safetensors:  99% 346M/349M [00:01<00:00, 241MB/s]
2024-12-18T00:38:58.1584862Z model.safetensors: 100% 349M/349M [00:01<00:00, 240MB/s]
2024-12-18T00:38:59.0046273Z 
2024-12-18T00:38:59.0046801Z loading model: 0it [00:02, ?it/s]
2024-12-18T00:38:59.0180842Z cuda eval  dpn107                             
2024-12-18T00:39:39.0382417Z pass
2024-12-18T00:39:39.0414088Z TIMING: _recursive_pre_grad_passes:0.04637 _recursive_joint_graph_passes:0.30489 _recursive_post_grad_passes:0.50157 async_compile.wait:0.8349 code_gen:7.91995 inductor_compile:15.39652 backend_compile:25.57607 entire_frame_compile:35.84299 total_wall_time:35.84299
2024-12-18T00:39:39.0415591Z STATS: call_* op count: 1038 | FakeTensorMode.__torch_dispatch__:39619 | ProxyTorchDispatchMode.__torch_dispatch__:17933 | FakeTensor.__torch_dispatch__:6062
2024-12-18T00:39:39.0416448Z Dynamo produced 1 graphs covering 1038 ops with 0 graph breaks (0 unique)
2024-12-18T00:39:44.7991656Z 
2024-12-18T00:39:44.9953142Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:39:44.9953524Z 
2024-12-18T00:39:45.1228298Z model.safetensors:   0% 0.00/42.5M [00:00<?, ?B/s]
2024-12-18T00:39:45.1228577Z 
2024-12-18T00:39:45.1715067Z model.safetensors:  74% 31.5M/42.5M [00:00<00:00, 247MB/s]
2024-12-18T00:39:45.1715572Z model.safetensors: 100% 42.5M/42.5M [00:00<00:00, 241MB/s]
2024-12-18T00:39:45.7797409Z 
2024-12-18T00:39:45.7797648Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:39:45.7840804Z cuda eval  eca_botnext26ts_256                
2024-12-18T00:40:05.3339125Z pass
2024-12-18T00:40:05.3352339Z TIMING: _recursive_pre_grad_passes:0.02003 _recursive_joint_graph_passes:1.1432 _recursive_post_grad_passes:0.18219 async_compile.wait:1.15656 code_gen:5.71083 inductor_compile:9.16195 backend_compile:13.77876 entire_frame_compile:17.3794 total_wall_time:17.3794
2024-12-18T00:40:05.3353861Z STATS: call_* op count: 528 | FakeTensorMode.__torch_dispatch__:15615 | FakeTensor.__torch_dispatch__:2025 | ProxyTorchDispatchMode.__torch_dispatch__:6983
2024-12-18T00:40:05.3354691Z Dynamo produced 1 graphs covering 528 ops with 0 graph breaks (0 unique)
2024-12-18T00:40:10.5106266Z 
2024-12-18T00:40:10.7055514Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:40:10.7057012Z 
2024-12-18T00:40:10.8297183Z model.safetensors:   0% 0.00/43.2M [00:00<?, ?B/s]
2024-12-18T00:40:10.8297593Z 
2024-12-18T00:40:10.8855761Z model.safetensors:  73% 31.5M/43.2M [00:00<00:00, 254MB/s]
2024-12-18T00:40:10.8856521Z model.safetensors: 100% 43.2M/43.2M [00:00<00:00, 240MB/s]
2024-12-18T00:40:11.4958232Z 
2024-12-18T00:40:11.4959755Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:40:11.5004425Z cuda eval  eca_halonext26ts                   
2024-12-18T00:40:36.8564276Z pass
2024-12-18T00:40:36.8576917Z TIMING: _recursive_pre_grad_passes:0.02119 _recursive_joint_graph_passes:1.16943 _recursive_post_grad_passes:0.19142 async_compile.wait:1.06027 code_gen:10.83866 inductor_compile:14.51539 backend_compile:19.32174 entire_frame_compile:23.10426 total_wall_time:23.10426
2024-12-18T00:40:36.8579143Z STATS: call_* op count: 574 | FakeTensorMode.__torch_dispatch__:16290 | FakeTensor.__torch_dispatch__:2069 | ProxyTorchDispatchMode.__torch_dispatch__:7206
2024-12-18T00:40:36.8579997Z Dynamo produced 1 graphs covering 574 ops with 0 graph breaks (0 unique)
2024-12-18T00:40:42.0821565Z 
2024-12-18T00:40:42.2025655Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:40:42.2025916Z 
2024-12-18T00:40:42.3180064Z model.safetensors:   0% 0.00/26.2M [00:00<?, ?B/s]
2024-12-18T00:40:42.3180339Z 
2024-12-18T00:40:42.3214437Z model.safetensors: 100% 26.2M/26.2M [00:00<00:00, 228MB/s]
2024-12-18T00:40:42.3214893Z model.safetensors: 100% 26.2M/26.2M [00:00<00:00, 221MB/s]
2024-12-18T00:40:42.8707371Z 
2024-12-18T00:40:42.8707893Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:40:42.8742635Z cuda eval  ese_vovnet19b_dw                   
2024-12-18T00:40:54.8405803Z pass
2024-12-18T00:40:54.8416537Z TIMING: _recursive_pre_grad_passes:0.01237 _recursive_joint_graph_passes:0.34369 _recursive_post_grad_passes:0.12013 async_compile.wait:0.5734 code_gen:2.56618 inductor_compile:4.8418 backend_compile:7.18228 entire_frame_compile:10.06889 total_wall_time:10.06889
2024-12-18T00:40:54.8418033Z STATS: call_* op count: 220 | FakeTensorMode.__torch_dispatch__:8562 | ProxyTorchDispatchMode.__torch_dispatch__:3738 | FakeTensor.__torch_dispatch__:1345
2024-12-18T00:40:54.8418907Z Dynamo produced 1 graphs covering 220 ops with 0 graph breaks (0 unique)
2024-12-18T00:40:59.8637157Z 
2024-12-18T00:41:00.0270990Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:41:00.0271242Z 
2024-12-18T00:41:00.1246612Z model.safetensors:   0% 0.00/22.5M [00:00<?, ?B/s]
2024-12-18T00:41:00.1247068Z model.safetensors: 100% 22.5M/22.5M [00:00<00:00, 231MB/s]
2024-12-18T00:41:00.7330560Z 
2024-12-18T00:41:00.7330883Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:41:00.7408831Z cuda eval  fbnetc_100                         
2024-12-18T00:41:19.1548685Z pass
2024-12-18T00:41:19.1550427Z TIMING: _recursive_pre_grad_passes:0.02117 _recursive_joint_graph_passes:0.4303 _recursive_post_grad_passes:0.27646 async_compile.wait:0.28779 code_gen:1.84752 inductor_compile:5.68029 backend_compile:10.71878 entire_frame_compile:16.49146 total_wall_time:16.49146
2024-12-18T00:41:19.1551939Z STATS: call_* op count: 384 | FakeTensorMode.__torch_dispatch__:18168 | ProxyTorchDispatchMode.__torch_dispatch__:8536 | FakeTensor.__torch_dispatch__:3288
2024-12-18T00:41:19.1552779Z Dynamo produced 1 graphs covering 384 ops with 0 graph breaks (0 unique)
2024-12-18T00:41:24.3415149Z 
2024-12-18T00:41:24.5139870Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:41:24.5140142Z 
2024-12-18T00:41:24.6458534Z model.safetensors:   0% 0.00/34.6M [00:00<?, ?B/s]
2024-12-18T00:41:24.6458808Z 
2024-12-18T00:41:24.6600666Z model.safetensors:  91% 31.5M/34.6M [00:00<00:00, 239MB/s]
2024-12-18T00:41:24.6601113Z model.safetensors: 100% 34.6M/34.6M [00:00<00:00, 237MB/s]
2024-12-18T00:41:25.3388941Z 
2024-12-18T00:41:25.3389455Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:41:25.3510989Z cuda eval  fbnetv3_b                          
2024-12-18T00:41:56.4525373Z pass
2024-12-18T00:41:56.4527304Z TIMING: _recursive_pre_grad_passes:0.03428 _recursive_joint_graph_passes:0.55241 _recursive_post_grad_passes:0.54634 async_compile.wait:0.41492 code_gen:4.24836 inductor_compile:10.76812 backend_compile:20.18638 entire_frame_compile:28.91939 total_wall_time:28.91939
2024-12-18T00:41:56.4529224Z STATS: call_* op count: 736 | FakeTensorMode.__torch_dispatch__:30622 | FakeTensor.__torch_dispatch__:5870 | ProxyTorchDispatchMode.__torch_dispatch__:16729
2024-12-18T00:41:56.4530084Z Dynamo produced 1 graphs covering 736 ops with 0 graph breaks (0 unique)
2024-12-18T00:42:02.1487021Z 
2024-12-18T00:42:02.5848885Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:42:02.5849124Z 
2024-12-18T00:42:02.6859207Z model.safetensors:   0% 0.00/125M [00:00<?, ?B/s]
2024-12-18T00:42:02.6861092Z 
2024-12-18T00:42:02.8057503Z model.safetensors:  17% 21.0M/125M [00:00<00:00, 208MB/s]
2024-12-18T00:42:02.8057818Z 
2024-12-18T00:42:02.9314517Z model.safetensors:  42% 52.4M/125M [00:00<00:00, 242MB/s]
2024-12-18T00:42:02.9314829Z 
2024-12-18T00:42:03.0632161Z model.safetensors:  67% 83.9M/125M [00:00<00:00, 246MB/s]
2024-12-18T00:42:03.0632505Z 
2024-12-18T00:42:03.1069241Z model.safetensors:  92% 115M/125M [00:00<00:00, 243MB/s] 
2024-12-18T00:42:03.1069692Z model.safetensors: 100% 125M/125M [00:00<00:00, 239MB/s]
2024-12-18T00:42:03.7712627Z 
2024-12-18T00:42:03.7713286Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:42:03.7796101Z cuda eval  gernet_l                           
2024-12-18T00:42:21.0532405Z pass
2024-12-18T00:42:21.0545453Z TIMING: _recursive_pre_grad_passes:0.01944 _recursive_joint_graph_passes:0.20232 _recursive_post_grad_passes:0.25038 async_compile.wait:0.29675 code_gen:1.73224 inductor_compile:5.36708 backend_compile:9.68272 entire_frame_compile:14.8342 total_wall_time:14.8342
2024-12-18T00:42:21.0547627Z STATS: call_* op count: 378 | FakeTensorMode.__torch_dispatch__:15854 | ProxyTorchDispatchMode.__torch_dispatch__:7612 | FakeTensor.__torch_dispatch__:2957
2024-12-18T00:42:21.0548849Z Dynamo produced 1 graphs covering 378 ops with 0 graph breaks (0 unique)
2024-12-18T00:42:26.1571126Z 
2024-12-18T00:42:26.2488673Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:42:26.2488928Z 
2024-12-18T00:42:26.3533750Z model.safetensors:   0% 0.00/20.9M [00:00<?, ?B/s]
2024-12-18T00:42:26.3534189Z model.safetensors: 100% 20.9M/20.9M [00:00<00:00, 200MB/s]
2024-12-18T00:42:27.0016958Z 
2024-12-18T00:42:27.0017289Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:42:27.0121157Z cuda eval  ghostnet_100                       
2024-12-18T00:42:54.8129654Z pass
2024-12-18T00:42:54.8133567Z TIMING: _recursive_pre_grad_passes:0.03222 _recursive_joint_graph_passes:0.26425 _recursive_post_grad_passes:0.35838 async_compile.wait:0.77518 code_gen:4.49572 inductor_compile:9.9793 backend_compile:17.32834 entire_frame_compile:25.62298 total_wall_time:25.62298
2024-12-18T00:42:54.8135741Z STATS: call_* op count: 642 | FakeTensorMode.__torch_dispatch__:27227 | ProxyTorchDispatchMode.__torch_dispatch__:12517 | FakeTensor.__torch_dispatch__:4299
2024-12-18T00:42:54.8136706Z Dynamo produced 1 graphs covering 642 ops with 0 graph breaks (0 unique)
2024-12-18T00:43:00.3047991Z 
2024-12-18T00:43:00.6322161Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:43:00.6322424Z 
2024-12-18T00:43:00.7711960Z model.safetensors:   0% 0.00/95.5M [00:00<?, ?B/s]
2024-12-18T00:43:00.7712267Z 
2024-12-18T00:43:00.8931344Z model.safetensors:  33% 31.5M/95.5M [00:00<00:00, 227MB/s]
2024-12-18T00:43:00.8931771Z 
2024-12-18T00:43:01.0217947Z model.safetensors:  66% 62.9M/95.5M [00:00<00:00, 244MB/s]
2024-12-18T00:43:01.0218275Z 
2024-12-18T00:43:01.0333139Z model.safetensors:  99% 94.4M/95.5M [00:00<00:00, 244MB/s]
2024-12-18T00:43:01.0333608Z model.safetensors: 100% 95.5M/95.5M [00:00<00:00, 238MB/s]
2024-12-18T00:43:01.6959564Z 
2024-12-18T00:43:01.6960207Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:43:01.7080748Z cuda eval  gluon_inception_v3                 
2024-12-18T00:43:21.6956701Z pass
2024-12-18T00:43:21.6976548Z TIMING: _recursive_pre_grad_passes:0.0316 _recursive_joint_graph_passes:0.25355 async_compile.wait:0.21509 inductor_compile:2.22978 backend_compile:9.59407 entire_frame_compile:17.48017 total_wall_time:17.48017
2024-12-18T00:43:21.6978048Z STATS: call_* op count: 628 | FakeTensorMode.__torch_dispatch__:21617 | ProxyTorchDispatchMode.__torch_dispatch__:12549
2024-12-18T00:43:21.6978764Z Dynamo produced 1 graphs covering 628 ops with 0 graph breaks (0 unique)
2024-12-18T00:43:26.7087925Z 
2024-12-18T00:43:27.0636028Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:43:27.0636287Z 
2024-12-18T00:43:27.1973664Z model.safetensors:   0% 0.00/98.9M [00:00<?, ?B/s]
2024-12-18T00:43:27.1973948Z 
2024-12-18T00:43:27.3248212Z model.safetensors:  32% 31.5M/98.9M [00:00<00:00, 236MB/s]
2024-12-18T00:43:27.3248517Z 
2024-12-18T00:43:27.4579661Z model.safetensors:  64% 62.9M/98.9M [00:00<00:00, 242MB/s]
2024-12-18T00:43:27.4579969Z 
2024-12-18T00:43:27.4819356Z model.safetensors:  95% 94.4M/98.9M [00:00<00:00, 239MB/s]
2024-12-18T00:43:27.4819823Z model.safetensors: 100% 98.9M/98.9M [00:00<00:00, 237MB/s]
2024-12-18T00:43:28.0497450Z 
2024-12-18T00:43:28.0497867Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:43:28.0580011Z cuda eval  gmixer_24_224                      
2024-12-18T00:43:51.8837669Z pass
2024-12-18T00:43:51.8858798Z TIMING: _recursive_pre_grad_passes:0.03094 _recursive_joint_graph_passes:1.40653 _recursive_post_grad_passes:0.89045 async_compile.wait:0.37168 code_gen:4.64327 inductor_compile:8.90957 backend_compile:16.31504 entire_frame_compile:21.46545 total_wall_time:21.46545
2024-12-18T00:43:51.8861278Z STATS: call_* op count: 1166 | FakeTensorMode.__torch_dispatch__:30482 | FakeTensor.__torch_dispatch__:3465 | ProxyTorchDispatchMode.__torch_dispatch__:15219
2024-12-18T00:43:51.8862165Z Dynamo produced 1 graphs covering 1166 ops with 0 graph breaks (0 unique)
2024-12-18T00:43:57.3984023Z 
2024-12-18T00:43:57.6590528Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:43:57.6590767Z 
2024-12-18T00:43:57.7953799Z model.safetensors:   0% 0.00/77.7M [00:00<?, ?B/s]
2024-12-18T00:43:57.7954388Z 
2024-12-18T00:43:57.9368320Z model.safetensors:  27% 21.0M/77.7M [00:00<00:00, 154MB/s]
2024-12-18T00:43:57.9368675Z 
2024-12-18T00:43:58.0447041Z model.safetensors:  67% 52.4M/77.7M [00:00<00:00, 195MB/s]
2024-12-18T00:43:58.0447329Z 
2024-12-18T00:43:58.0474178Z model.safetensors: 100% 77.7M/77.7M [00:00<00:00, 210MB/s]
2024-12-18T00:43:58.0474941Z model.safetensors: 100% 77.7M/77.7M [00:00<00:00, 200MB/s]
2024-12-18T00:43:58.6094781Z 
2024-12-18T00:43:58.6095200Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:43:58.6172597Z cuda eval  gmlp_s16_224                       
2024-12-18T00:44:21.3995841Z pass
2024-12-18T00:44:21.4025819Z TIMING: _recursive_pre_grad_passes:0.02627 _recursive_joint_graph_passes:1.16803 _recursive_post_grad_passes:0.72785 async_compile.wait:0.3944 code_gen:4.24843 inductor_compile:8.45899 backend_compile:15.53785 entire_frame_compile:20.44813 total_wall_time:20.44813
2024-12-18T00:44:21.4027300Z STATS: call_* op count: 914 | FakeTensorMode.__torch_dispatch__:28537 | FakeTensor.__torch_dispatch__:3668 | ProxyTorchDispatchMode.__torch_dispatch__:14535
2024-12-18T00:44:21.4028156Z Dynamo produced 1 graphs covering 914 ops with 0 graph breaks (0 unique)
2024-12-18T00:44:26.7112135Z 
2024-12-18T00:44:27.0677637Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:44:27.0677929Z 
2024-12-18T00:44:27.2051815Z model.safetensors:   0% 0.00/85.6M [00:00<?, ?B/s]
2024-12-18T00:44:27.2052094Z 
2024-12-18T00:44:27.3294308Z model.safetensors:  37% 31.5M/85.6M [00:00<00:00, 229MB/s]
2024-12-18T00:44:27.3294599Z 
2024-12-18T00:44:27.4298571Z model.safetensors:  73% 62.9M/85.6M [00:00<00:00, 242MB/s]
2024-12-18T00:44:27.4299471Z model.safetensors: 100% 85.6M/85.6M [00:00<00:00, 236MB/s]
2024-12-18T00:44:28.6281881Z 
2024-12-18T00:44:28.6282218Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:44:28.6671209Z cuda eval  hrnet_w18                          
2024-12-18T00:45:54.5162873Z pass
2024-12-18T00:45:54.5176262Z TIMING: _recursive_pre_grad_passes:0.10254 _recursive_joint_graph_passes:0.62276 _recursive_post_grad_passes:1.58945 async_compile.wait:1.27576 code_gen:10.6622 inductor_compile:28.08431 backend_compile:54.16658 entire_frame_compile:81.60996 total_wall_time:81.60996
2024-12-18T00:45:54.5178409Z STATS: call_* op count: 2292 | FakeTensorMode.__torch_dispatch__:96584 | ProxyTorchDispatchMode.__torch_dispatch__:48000 | FakeTensor.__torch_dispatch__:18202
2024-12-18T00:45:54.5179377Z Dynamo produced 1 graphs covering 2292 ops with 0 graph breaks (0 unique)
2024-12-18T00:46:02.1333072Z 
2024-12-18T00:46:02.4554315Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:46:02.4554833Z 
2024-12-18T00:46:02.5977364Z model.safetensors:   0% 0.00/95.5M [00:00<?, ?B/s]
2024-12-18T00:46:02.5977638Z 
2024-12-18T00:46:02.7158843Z model.safetensors:  33% 31.5M/95.5M [00:00<00:00, 221MB/s]
2024-12-18T00:46:02.7159323Z 
2024-12-18T00:46:02.8500485Z model.safetensors:  66% 62.9M/95.5M [00:00<00:00, 245MB/s]
2024-12-18T00:46:02.8500828Z 
2024-12-18T00:46:02.8616149Z model.safetensors:  99% 94.4M/95.5M [00:00<00:00, 240MB/s]
2024-12-18T00:46:02.8616677Z model.safetensors: 100% 95.5M/95.5M [00:00<00:00, 235MB/s]
2024-12-18T00:46:03.5444284Z 
2024-12-18T00:46:03.5444497Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:46:03.5557019Z cuda eval  inception_v3                       
2024-12-18T00:46:23.5050576Z pass
2024-12-18T00:46:23.5070646Z TIMING: _recursive_pre_grad_passes:0.03153 _recursive_joint_graph_passes:0.25168 async_compile.wait:0.20585 inductor_compile:2.21741 backend_compile:9.52903 entire_frame_compile:17.43226 total_wall_time:17.43226
2024-12-18T00:46:23.5071834Z STATS: call_* op count: 628 | FakeTensorMode.__torch_dispatch__:21617 | ProxyTorchDispatchMode.__torch_dispatch__:12549
2024-12-18T00:46:23.5072559Z Dynamo produced 1 graphs covering 628 ops with 0 graph breaks (0 unique)
2024-12-18T00:46:28.5556389Z 
2024-12-18T00:46:29.3129821Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:46:29.3130075Z 
2024-12-18T00:46:29.4587422Z model.safetensors:   0% 0.00/271M [00:00<?, ?B/s]
2024-12-18T00:46:29.4587715Z 
2024-12-18T00:46:29.5851091Z model.safetensors:  12% 31.5M/271M [00:00<00:01, 216MB/s]
2024-12-18T00:46:29.5851387Z 
2024-12-18T00:46:29.7105784Z model.safetensors:  23% 62.9M/271M [00:00<00:00, 234MB/s]
2024-12-18T00:46:29.7106070Z 
2024-12-18T00:46:29.8432704Z model.safetensors:  35% 94.4M/271M [00:00<00:00, 241MB/s]
2024-12-18T00:46:29.8433369Z 
2024-12-18T00:46:29.9758158Z model.safetensors:  46% 126M/271M [00:00<00:00, 240MB/s] 
2024-12-18T00:46:29.9758443Z 
2024-12-18T00:46:30.1045814Z model.safetensors:  58% 157M/271M [00:00<00:00, 239MB/s]
2024-12-18T00:46:30.1046106Z 
2024-12-18T00:46:30.2324710Z model.safetensors:  70% 189M/271M [00:00<00:00, 241MB/s]
2024-12-18T00:46:30.2324995Z 
2024-12-18T00:46:30.3588839Z model.safetensors:  81% 220M/271M [00:00<00:00, 242MB/s]
2024-12-18T00:46:30.3589119Z 
2024-12-18T00:46:30.4438586Z model.safetensors:  93% 252M/271M [00:01<00:00, 244MB/s]
2024-12-18T00:46:30.4439022Z model.safetensors: 100% 271M/271M [00:01<00:00, 240MB/s]
2024-12-18T00:46:31.2194729Z 
2024-12-18T00:46:31.2195162Z loading model: 0it [00:02, ?it/s]
2024-12-18T00:46:31.2272711Z cuda eval  jx_nest_base                       
2024-12-18T00:47:05.7208448Z pass
2024-12-18T00:47:05.7225278Z TIMING: _recursive_pre_grad_passes:0.03187 _recursive_joint_graph_passes:0.80032 _recursive_post_grad_passes:1.96677 async_compile.wait:1.44938 code_gen:7.66949 inductor_compile:14.95616 backend_compile:24.0989 entire_frame_compile:30.34557 total_wall_time:30.34557
2024-12-18T00:47:05.7226771Z STATS: call_* op count: 1102 | FakeTensorMode.__torch_dispatch__:47451 | FakeTensor.__torch_dispatch__:4545 | ProxyTorchDispatchMode.__torch_dispatch__:21640
2024-12-18T00:47:05.7227616Z Dynamo produced 1 graphs covering 1102 ops with 0 graph breaks (0 unique)
2024-12-18T00:47:11.5349369Z 
2024-12-18T00:47:11.6487744Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:47:11.6487991Z 
2024-12-18T00:47:11.6885253Z model.safetensors:   0% 0.00/7.56M [00:00<?, ?B/s]
2024-12-18T00:47:11.6885695Z model.safetensors: 100% 7.56M/7.56M [00:00<00:00, 191MB/s]
2024-12-18T00:47:12.2051361Z 
2024-12-18T00:47:12.2051662Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:47:12.2088448Z cuda eval  lcnet_050                          
2024-12-18T00:47:23.0291386Z pass
2024-12-18T00:47:23.0301730Z TIMING: _recursive_pre_grad_passes:0.01201 _recursive_joint_graph_passes:0.17726 _recursive_post_grad_passes:0.16794 async_compile.wait:0.30871 code_gen:1.34498 inductor_compile:3.7398 backend_compile:6.54478 entire_frame_compile:9.11381 total_wall_time:9.11381
2024-12-18T00:47:23.0303599Z STATS: call_* op count: 196 | FakeTensorMode.__torch_dispatch__:9381 | FakeTensor.__torch_dispatch__:1746 | ProxyTorchDispatchMode.__torch_dispatch__:4998
2024-12-18T00:47:23.0304706Z Dynamo produced 1 graphs covering 196 ops with 0 graph breaks (0 unique)
2024-12-18T00:47:27.9679688Z 
2024-12-18T00:47:28.9399216Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:47:28.9399757Z 
2024-12-18T00:47:29.0753848Z pytorch_model.bin:   0% 0.00/37.2M [00:00<?, ?B/s]
2024-12-18T00:47:29.0754188Z 
2024-12-18T00:47:29.0993807Z pytorch_model.bin:  85% 31.5M/37.2M [00:00<00:00, 232MB/s]
2024-12-18T00:47:29.0994378Z pytorch_model.bin: 100% 37.2M/37.2M [00:00<00:00, 233MB/s]
2024-12-18T00:47:29.7602432Z 
2024-12-18T00:47:29.7602745Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:47:29.7682857Z cuda eval  levit_128                          
2024-12-18T00:47:57.2826085Z pass
2024-12-18T00:47:57.2844702Z TIMING: _recursive_pre_grad_passes:0.0338 _recursive_joint_graph_passes:2.74976 _recursive_post_grad_passes:0.38198 async_compile.wait:0.4539 code_gen:4.17508 inductor_compile:8.66385 backend_compile:18.5931 entire_frame_compile:25.48803 total_wall_time:25.48803
2024-12-18T00:47:57.2847018Z STATS: call_* op count: 1090 | FakeTensorMode.__torch_dispatch__:35247 | ProxyTorchDispatchMode.__torch_dispatch__:17171 | FakeTensor.__torch_dispatch__:4205
2024-12-18T00:47:57.2847909Z Dynamo produced 1 graphs covering 1090 ops with 0 graph breaks (0 unique)
2024-12-18T00:48:03.0421177Z 
2024-12-18T00:48:03.6733399Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:48:03.6733736Z 
2024-12-18T00:48:03.8097348Z model.safetensors:   0% 0.00/240M [00:00<?, ?B/s]
2024-12-18T00:48:03.8097647Z 
2024-12-18T00:48:03.9370785Z model.safetensors:  13% 31.5M/240M [00:00<00:00, 231MB/s]
2024-12-18T00:48:03.9371077Z 
2024-12-18T00:48:04.0660950Z model.safetensors:  26% 62.9M/240M [00:00<00:00, 240MB/s]
2024-12-18T00:48:04.0661258Z 
2024-12-18T00:48:04.1950777Z model.safetensors:  39% 94.4M/240M [00:00<00:00, 242MB/s]
2024-12-18T00:48:04.1951062Z 
2024-12-18T00:48:04.3323127Z model.safetensors:  53% 126M/240M [00:00<00:00, 243MB/s] 
2024-12-18T00:48:04.3323414Z 
2024-12-18T00:48:04.4640668Z model.safetensors:  66% 157M/240M [00:00<00:00, 238MB/s]
2024-12-18T00:48:04.4640950Z 
2024-12-18T00:48:04.5948503Z model.safetensors:  79% 189M/240M [00:00<00:00, 238MB/s]
2024-12-18T00:48:04.5948790Z 
2024-12-18T00:48:04.6777905Z model.safetensors:  92% 220M/240M [00:00<00:00, 239MB/s]
2024-12-18T00:48:04.6780516Z model.safetensors: 100% 240M/240M [00:01<00:00, 239MB/s]
2024-12-18T00:48:05.2263033Z 
2024-12-18T00:48:05.2263538Z loading model: 0it [00:02, ?it/s]
2024-12-18T00:48:05.2306284Z cuda eval  mixer_b16_224                      
2024-12-18T00:48:19.5482037Z pass
2024-12-18T00:48:19.5514471Z TIMING: _recursive_pre_grad_passes:0.01338 _recursive_joint_graph_passes:1.215 _recursive_post_grad_passes:0.42388 async_compile.wait:0.36536 code_gen:2.76378 inductor_compile:5.06505 backend_compile:9.11645 entire_frame_compile:11.60596 total_wall_time:11.60596
2024-12-18T00:48:19.5515946Z STATS: call_* op count: 398 | FakeTensorMode.__torch_dispatch__:14337 | FakeTensor.__torch_dispatch__:1833 | ProxyTorchDispatchMode.__torch_dispatch__:7275
2024-12-18T00:48:19.5516784Z Dynamo produced 1 graphs covering 398 ops with 0 graph breaks (0 unique)
2024-12-18T00:48:21.0876599Z accuracy            pass_rate=100.00%
2024-12-18T00:48:21.0877412Z calls_captured      gmean=700.95x mean=820.733x
2024-12-18T00:48:21.0881266Z unique_graphs       gmean=1.00x mean=1.000x
2024-12-18T00:48:21.0884849Z graph_breaks        gmean=0.00x mean=0.000x
2024-12-18T00:48:21.0888170Z unique_graph_breaks gmean=0.00x mean=0.000x
2024-12-18T00:48:21.0891621Z autograd_captures   gmean=0.00x mean=0.000x
2024-12-18T00:48:21.0895173Z autograd_compiles   gmean=0.00x mean=0.000x
2024-12-18T00:48:21.0898535Z cudagraph_skips     gmean=0.00x mean=0.000x
2024-12-18T00:48:22.2900329Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_timm_models.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_timm_inference.csv
2024-12-18T00:48:22.5567206Z adv_inception_v3                    PASS
2024-12-18T00:48:22.5571277Z beit_base_patch16_224               PASS
2024-12-18T00:48:22.5575393Z botnet26t_256                       PASS
2024-12-18T00:48:22.5579860Z cait_m36_384                        PASS
2024-12-18T00:48:22.5584495Z coat_lite_mini                      PASS
2024-12-18T00:48:22.5589079Z convit_base                         PASS
2024-12-18T00:48:22.5593501Z convmixer_768_32                    PASS
2024-12-18T00:48:22.5597917Z convnext_base                       PASS
2024-12-18T00:48:22.5602872Z crossvit_9_240                      PASS
2024-12-18T00:48:22.5607708Z cspdarknet53                        PASS
2024-12-18T00:48:22.5612237Z deit_base_distilled_patch16_224     PASS
2024-12-18T00:48:22.5616597Z dla102                              PASS
2024-12-18T00:48:22.5621127Z dm_nfnet_f0                         PASS
2024-12-18T00:48:22.5625689Z dpn107                              PASS
2024-12-18T00:48:22.5630310Z eca_botnext26ts_256                 PASS
2024-12-18T00:48:22.5634666Z eca_halonext26ts                    PASS
2024-12-18T00:48:22.5639268Z ese_vovnet19b_dw                    PASS
2024-12-18T00:48:22.5643893Z fbnetc_100                          PASS
2024-12-18T00:48:22.5648074Z fbnetv3_b                           PASS
2024-12-18T00:48:22.5652545Z gernet_l                            PASS
2024-12-18T00:48:22.5657148Z ghostnet_100                        PASS
2024-12-18T00:48:22.5661505Z gluon_inception_v3                  PASS
2024-12-18T00:48:22.5666004Z gmixer_24_224                       PASS
2024-12-18T00:48:22.5670401Z gmlp_s16_224                        PASS
2024-12-18T00:48:22.5675940Z hrnet_w18                           PASS
2024-12-18T00:48:22.5679440Z inception_v3                        PASS
2024-12-18T00:48:22.5684014Z jx_nest_base                        PASS
2024-12-18T00:48:22.5688341Z lcnet_050                           PASS
2024-12-18T00:48:22.5692865Z levit_128                           PASS
2024-12-18T00:48:22.5697300Z mixer_b16_224                       PASS
2024-12-18T00:48:22.6123033Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_timm_models.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_timm_inference.csv
2024-12-18T00:48:22.8784171Z adv_inception_v3                    PASS
2024-12-18T00:48:22.8789092Z beit_base_patch16_224               PASS
2024-12-18T00:48:22.8793066Z botnet26t_256                       PASS
2024-12-18T00:48:22.8796863Z cait_m36_384                        PASS
2024-12-18T00:48:22.8801808Z coat_lite_mini                      PASS
2024-12-18T00:48:22.8806559Z convit_base                         PASS
2024-12-18T00:48:22.8811396Z convmixer_768_32                    PASS
2024-12-18T00:48:22.8815544Z convnext_base                       PASS
2024-12-18T00:48:22.8820514Z crossvit_9_240                      PASS
2024-12-18T00:48:22.8825382Z cspdarknet53                        PASS
2024-12-18T00:48:22.8829339Z deit_base_distilled_patch16_224     PASS
2024-12-18T00:48:22.8833801Z dla102                              PASS
2024-12-18T00:48:22.8838376Z dm_nfnet_f0                         PASS
2024-12-18T00:48:22.8843284Z dpn107                              PASS
2024-12-18T00:48:22.8847717Z eca_botnext26ts_256                 PASS
2024-12-18T00:48:22.8852432Z eca_halonext26ts                    PASS
2024-12-18T00:48:22.8857051Z ese_vovnet19b_dw                    PASS
2024-12-18T00:48:22.8861441Z fbnetc_100                          PASS
2024-12-18T00:48:22.8865928Z fbnetv3_b                           PASS
2024-12-18T00:48:22.8870399Z gernet_l                            PASS
2024-12-18T00:48:22.8875006Z ghostnet_100                        PASS
2024-12-18T00:48:22.8879673Z gluon_inception_v3                  PASS
2024-12-18T00:48:22.8884392Z gmixer_24_224                       PASS
2024-12-18T00:48:22.8888908Z gmlp_s16_224                        PASS
2024-12-18T00:48:22.8893520Z hrnet_w18                           PASS
2024-12-18T00:48:22.8898072Z inception_v3                        PASS
2024-12-18T00:48:22.8902640Z jx_nest_base                        PASS
2024-12-18T00:48:22.8907792Z lcnet_050                           PASS
2024-12-18T00:48:22.8912152Z levit_128                           PASS
2024-12-18T00:48:22.8916579Z mixer_b16_224                       PASS
2024-12-18T00:48:22.9343311Z + test_single_dynamo_benchmark training timm_models 0 --training --amp
2024-12-18T00:48:22.9347359Z ++ pwd
2024-12-18T00:48:22.9350765Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2024-12-18T00:48:22.9351424Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports
2024-12-18T00:48:22.9373197Z + local name=training
2024-12-18T00:48:22.9373575Z + shift
2024-12-18T00:48:22.9373871Z + local suite=timm_models
2024-12-18T00:48:22.9374225Z + shift
2024-12-18T00:48:22.9374517Z + local shard_id=0
2024-12-18T00:48:22.9374818Z + shift
2024-12-18T00:48:22.9375027Z + partition_flags=()
2024-12-18T00:48:22.9375288Z + local partition_flags
2024-12-18T00:48:22.9375648Z + [[ -n 2 ]]
2024-12-18T00:48:22.9375919Z + [[ -n 0 ]]
2024-12-18T00:48:22.9376329Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id")
2024-12-18T00:48:22.9376898Z + [[ inductor_timm == *perf_compare* ]]
2024-12-18T00:48:22.9377225Z + [[ inductor_timm == *perf* ]]
2024-12-18T00:48:22.9377573Z + [[ inductor_timm == *_avx2* ]]
2024-12-18T00:48:22.9377905Z + [[ inductor_timm == *_avx512* ]]
2024-12-18T00:48:22.9379034Z + python benchmarks/dynamo/timm_models.py --ci --accuracy --timing --explain --inductor --device cuda --training --amp --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/training_timm_models.csv
2024-12-18T00:48:29.9136380Z 
2024-12-18T00:48:31.2975206Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:48:31.2975567Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:48:31.2975894Z cuda train adv_inception_v3                   
2024-12-18T00:50:18.4097009Z pass
2024-12-18T00:50:18.4316142Z TIMING: entire_frame_compile:55.43603 _recursive_pre_grad_passes:0.0237 _recursive_joint_graph_passes:0.58231 _recursive_post_grad_passes:1.30511 async_compile.wait:4.98472 code_gen:57.04043 inductor_compile:73.86316 backend_compile:44.05792 entire_backward_compile:44.53947 total_wall_time:99.97551
2024-12-18T00:50:18.4317814Z STATS: call_* op count: 412 | FakeTensorMode.__torch_dispatch__:44531 | FakeTensor.__torch_dispatch__:15772 | ProxyTorchDispatchMode.__torch_dispatch__:20409
2024-12-18T00:50:18.4318676Z Dynamo produced 2 graphs covering 412 ops with 6 graph breaks (5 unique)
2024-12-18T00:50:26.4829187Z 
2024-12-18T00:50:28.3104717Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:50:28.3105550Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:50:28.3106186Z cuda train beit_base_patch16_224              
2024-12-18T00:51:00.7276363Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T00:51:00.7277219Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 405, in torch_dynamo_resume_in_forward_and_backward_pass_at_403
2024-12-18T00:51:00.7277928Z     pred = mod(*cloned_inputs)
2024-12-18T00:51:00.7278462Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/beit.py", line 521, in forward
2024-12-18T00:51:00.7279011Z     x = self.forward_features(x)
2024-12-18T00:51:00.7279641Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/beit.py", line 509, in forward_features
2024-12-18T00:51:00.7280607Z     x = blk(x, shared_rel_pos_bias=rel_pos_bias)
2024-12-18T00:51:00.7281158Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/beit.py", line 248, in forward
2024-12-18T00:51:00.7281904Z     x = x + self.drop_path1(self.gamma_1 * self.attn(self.norm1(x), shared_rel_pos_bias=shared_rel_pos_bias))
2024-12-18T00:51:00.7282644Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/beit.py", line 156, in forward
2024-12-18T00:51:00.7283198Z     rel_pos_bias = self._get_rel_pos_bias()
2024-12-18T00:51:00.7283780Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/beit.py", line 131, in _get_rel_pos_bias
2024-12-18T00:51:00.7284451Z     relative_position_bias = self.relative_position_bias_table[
2024-12-18T00:51:00.7284750Z 
2024-12-18T00:51:00.8212956Z W1218 00:51:00.820000 12206 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T00:51:02.5161153Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][163].grad", "L['self'].param_groups[0]['params'][164].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][166].grad", "L['self'].param_groups[0]['params'][167].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][169].grad", "L['self'].param_groups[0]['params'][170].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][172].grad", "L['self'].param_groups[0]['params'][173].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][175].grad", "L['self'].param_groups[0]['params'][176].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][178].grad", "L['self'].param_groups[0]['params'][179].grad", "L['self'].param_groups[0]['params'][180].grad", "L['self'].param_groups[0]['params'][181].grad", "L['self'].param_groups[0]['params'][182].grad", "L['self'].param_groups[0]['params'][183].grad", "L['self'].param_groups[0]['params'][184].grad", "L['self'].param_groups[0]['params'][185].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][187].grad", "L['self'].param_groups[0]['params'][188].grad", "L['self'].param_groups[0]['params'][189].grad", "L['self'].param_groups[0]['params'][190].grad", "L['self'].param_groups[0]['params'][191].grad", "L['self'].param_groups[0]['params'][192].grad", "L['self'].param_groups[0]['params'][193].grad", "L['self'].param_groups[0]['params'][194].grad", "L['self'].param_groups[0]['params'][195].grad", "L['self'].param_groups[0]['params'][196].grad", "L['self'].param_groups[0]['params'][197].grad", "L['self'].param_groups[0]['params'][198].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T00:51:42.4288233Z pass
2024-12-18T00:51:42.4820467Z TIMING: entire_frame_compile:56.53679 _recursive_pre_grad_passes:0.06302 _recursive_joint_graph_passes:1.18309 _recursive_post_grad_passes:0.69816 async_compile.wait:6.1019 code_gen:22.13884 inductor_compile:36.34786 backend_compile:43.96071 entire_backward_compile:8.33708 total_wall_time:64.87387
2024-12-18T00:51:42.4822505Z STATS: call_* op count: 1389 | FakeTensorMode.__torch_dispatch__:52476 | FakeTensor.__torch_dispatch__:14083 | ProxyTorchDispatchMode.__torch_dispatch__:23544
2024-12-18T00:51:42.4823390Z Dynamo produced 3 graphs covering 1389 ops with 7 graph breaks (5 unique)
2024-12-18T00:51:49.6004358Z 
2024-12-18T00:51:50.6514317Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:51:50.6514668Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:51:50.6514989Z cuda train botnet26t_256                      
2024-12-18T00:52:36.7731194Z pass
2024-12-18T00:52:36.7836563Z TIMING: entire_frame_compile:27.04559 _recursive_pre_grad_passes:0.01324 _recursive_joint_graph_passes:3.05832 _recursive_post_grad_passes:0.54778 async_compile.wait:2.87077 code_gen:20.42434 inductor_compile:28.46423 backend_compile:22.3094 entire_backward_compile:14.73394 total_wall_time:41.77953
2024-12-18T00:52:36.7838162Z STATS: call_* op count: 264 | FakeTensorMode.__torch_dispatch__:21415 | FakeTensor.__torch_dispatch__:5541 | ProxyTorchDispatchMode.__torch_dispatch__:9035
2024-12-18T00:52:36.7839549Z Dynamo produced 2 graphs covering 264 ops with 6 graph breaks (5 unique)
2024-12-18T00:52:42.7300327Z 
2024-12-18T00:52:46.8707861Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:52:46.8708254Z loading model: 0it [00:04, ?it/s]
2024-12-18T00:52:46.8708574Z cuda train cait_m36_384                       
2024-12-18T00:52:46.8714187Z Traceback (most recent call last):
2024-12-18T00:52:46.8714756Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2732, in validate_model
2024-12-18T00:52:46.8715321Z     self.model_iter_fn(model, example_inputs)
2024-12-18T00:52:46.8715932Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 405, in forward_and_backward_pass
2024-12-18T00:52:46.8716532Z     pred = mod(*cloned_inputs)
2024-12-18T00:52:46.8717151Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T00:52:46.8717806Z     return self._call_impl(*args, **kwargs)
2024-12-18T00:52:46.8718402Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T00:52:46.8719006Z     return forward_call(*args, **kwargs)
2024-12-18T00:52:46.8719620Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 438, in forward
2024-12-18T00:52:46.8720165Z     x = self.forward_features(x)
2024-12-18T00:52:46.8720714Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 423, in forward_features
2024-12-18T00:52:46.8721286Z     x = self.blocks(x)
2024-12-18T00:52:46.8721861Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T00:52:46.8722501Z     return self._call_impl(*args, **kwargs)
2024-12-18T00:52:46.8723089Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T00:52:46.8723682Z     return forward_call(*args, **kwargs)
2024-12-18T00:52:46.8724275Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/container.py", line 250, in forward
2024-12-18T00:52:46.8724863Z     input = module(input)
2024-12-18T00:52:46.8725859Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T00:52:46.8726507Z     return self._call_impl(*args, **kwargs)
2024-12-18T00:52:46.8727092Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T00:52:46.8727684Z     return forward_call(*args, **kwargs)
2024-12-18T00:52:46.8728215Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 199, in forward
2024-12-18T00:52:46.8728832Z     x = x + self.drop_path(self.gamma_1 * self.attn(self.norm1(x)))
2024-12-18T00:52:46.8729535Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T00:52:46.8730175Z     return self._call_impl(*args, **kwargs)
2024-12-18T00:52:46.8730767Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T00:52:46.8731363Z     return forward_call(*args, **kwargs)
2024-12-18T00:52:46.8731900Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 153, in forward
2024-12-18T00:52:46.8732470Z     x = (attn @ v).transpose(1, 2).reshape(B, N, C)
2024-12-18T00:52:46.8734514Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 16.44 MiB is free. Process 356260 has 21.95 GiB memory in use. Of the allocated memory 21.21 GiB is allocated by PyTorch, and 430.66 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-12-18T00:52:46.8736582Z 
2024-12-18T00:52:46.8736797Z The above exception was the direct cause of the following exception:
2024-12-18T00:52:46.8737116Z 
2024-12-18T00:52:46.8737234Z Traceback (most recent call last):
2024-12-18T00:52:46.8737706Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run
2024-12-18T00:52:46.8738186Z     ) = runner.load_model(
2024-12-18T00:52:46.8738665Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 329, in load_model
2024-12-18T00:52:46.8739221Z     self.validate_model(model, example_inputs)
2024-12-18T00:52:46.8739766Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2734, in validate_model
2024-12-18T00:52:46.8740325Z     raise RuntimeError("Eager run failed") from e
2024-12-18T00:52:46.8740673Z RuntimeError: Eager run failed
2024-12-18T00:52:46.8740868Z 
2024-12-18T00:52:46.8740965Z eager_fail_to_run
2024-12-18T00:52:51.3203741Z 
2024-12-18T00:52:52.3289719Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:52:52.3290083Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:52:52.3290421Z cuda train coat_lite_mini                     
2024-12-18T00:53:40.4235845Z pass
2024-12-18T00:53:40.4354512Z TIMING: entire_frame_compile:26.76671 _recursive_pre_grad_passes:0.01689 _recursive_joint_graph_passes:5.00426 _recursive_post_grad_passes:0.78258 async_compile.wait:1.89879 code_gen:18.73103 inductor_compile:27.29121 backend_compile:21.49326 entire_backward_compile:17.27296 total_wall_time:44.03967
2024-12-18T00:53:40.4356649Z STATS: call_* op count: 456 | FakeTensorMode.__torch_dispatch__:34253 | FakeTensor.__torch_dispatch__:5921 | ProxyTorchDispatchMode.__torch_dispatch__:14386
2024-12-18T00:53:40.4357629Z Dynamo produced 2 graphs covering 456 ops with 6 graph breaks (5 unique)
2024-12-18T00:53:46.6815712Z 
2024-12-18T00:53:48.4112906Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:53:48.4113288Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:53:48.4113604Z cuda train convit_base                        
2024-12-18T00:53:48.4667642Z WARNING:common:fp64 golden ref were not generated for convit_base. Setting accuracy check to cosine
2024-12-18T00:55:09.6346562Z pass
2024-12-18T00:55:09.6598112Z TIMING: entire_frame_compile:51.38073 _recursive_pre_grad_passes:0.03453 _recursive_joint_graph_passes:6.17153 _recursive_post_grad_passes:2.04914 async_compile.wait:2.32559 code_gen:26.95348 inductor_compile:44.76059 backend_compile:41.49408 entire_backward_compile:26.83038 total_wall_time:78.21111
2024-12-18T00:55:09.6599903Z STATS: call_* op count: 1142 | FakeTensorMode.__torch_dispatch__:84745 | FakeTensor.__torch_dispatch__:15384 | ProxyTorchDispatchMode.__torch_dispatch__:40269
2024-12-18T00:55:09.6600764Z Dynamo produced 3 graphs covering 1142 ops with 7 graph breaks (5 unique)
2024-12-18T00:55:16.9987168Z 
2024-12-18T00:55:17.9461593Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:55:17.9461979Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:55:17.9462301Z cuda train convmixer_768_32                   
2024-12-18T00:56:10.6777156Z pass
2024-12-18T00:56:10.6965241Z TIMING: entire_frame_compile:32.34298 _recursive_pre_grad_passes:0.01808 _recursive_joint_graph_passes:0.47267 _recursive_post_grad_passes:0.9296 async_compile.wait:0.78158 code_gen:13.18288 inductor_compile:24.50654 backend_compile:24.01269 entire_backward_compile:10.93318 total_wall_time:43.27616
2024-12-18T00:56:10.6967327Z STATS: call_* op count: 300 | FakeTensorMode.__torch_dispatch__:32453 | FakeTensor.__torch_dispatch__:11054 | ProxyTorchDispatchMode.__torch_dispatch__:15494
2024-12-18T00:56:10.6968190Z Dynamo produced 2 graphs covering 300 ops with 5 graph breaks (4 unique)
2024-12-18T00:56:17.2891848Z 
2024-12-18T00:56:19.1689961Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:56:19.1690418Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:56:19.1690821Z cuda train convnext_base                      
2024-12-18T00:57:13.6771228Z W1218 00:57:13.676000 15538 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T00:57:17.6081296Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][311].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T00:58:28.8628261Z pass
2024-12-18T00:58:28.9135865Z TIMING: entire_frame_compile:97.14954 _recursive_pre_grad_passes:0.10348 _recursive_joint_graph_passes:1.62791 _recursive_post_grad_passes:1.32572 async_compile.wait:6.37495 code_gen:40.77273 inductor_compile:65.84485 backend_compile:73.68149 entire_backward_compile:19.24862 total_wall_time:116.39816
2024-12-18T00:58:28.9137573Z STATS: call_* op count: 2198 | FakeTensorMode.__torch_dispatch__:85378 | FakeTensor.__torch_dispatch__:25731 | ProxyTorchDispatchMode.__torch_dispatch__:38215
2024-12-18T00:58:28.9138411Z Dynamo produced 3 graphs covering 2198 ops with 7 graph breaks (5 unique)
2024-12-18T00:58:38.1793726Z 
2024-12-18T00:58:39.0775791Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:58:39.0776147Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:58:39.0776462Z cuda train crossvit_9_240                     
2024-12-18T00:59:31.2023683Z W1218 00:59:31.201000 16275 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T01:00:29.1612922Z pass
2024-12-18T01:00:29.1754636Z TIMING: entire_frame_compile:85.73037 _recursive_pre_grad_passes:0.08286 _recursive_joint_graph_passes:4.92416 _recursive_post_grad_passes:1.54361 async_compile.wait:6.61176 code_gen:34.70361 inductor_compile:56.15636 backend_compile:67.25939 entire_backward_compile:15.89625 total_wall_time:101.62662
2024-12-18T01:00:29.1756353Z STATS: call_* op count: 1850 | FakeTensorMode.__torch_dispatch__:80629 | FakeTensor.__torch_dispatch__:21652 | ProxyTorchDispatchMode.__torch_dispatch__:36504
2024-12-18T01:00:29.1757194Z Dynamo produced 3 graphs covering 1850 ops with 7 graph breaks (5 unique)
2024-12-18T01:00:37.6717335Z 
2024-12-18T01:00:38.9520523Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:00:38.9520989Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:00:38.9521441Z cuda train cspdarknet53                       
2024-12-18T01:01:36.8760134Z W1218 01:01:36.874000 17053 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T01:01:39.1759930Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][180].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][192].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T01:02:24.6422239Z pass
2024-12-18T01:02:24.6556233Z TIMING: entire_frame_compile:79.6788 _recursive_pre_grad_passes:0.06682 _recursive_joint_graph_passes:0.69285 _recursive_post_grad_passes:1.39683 async_compile.wait:6.22938 code_gen:33.33601 inductor_compile:55.3327 backend_compile:61.14105 entire_backward_compile:16.58747 total_wall_time:96.26627
2024-12-18T01:02:24.6559593Z STATS: call_* op count: 1358 | FakeTensorMode.__torch_dispatch__:57667 | FakeTensor.__torch_dispatch__:21024 | ProxyTorchDispatchMode.__torch_dispatch__:26247
2024-12-18T01:02:24.6561278Z Dynamo produced 3 graphs covering 1358 ops with 7 graph breaks (5 unique)
2024-12-18T01:02:33.1437120Z 
2024-12-18T01:02:34.8524781Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:02:34.8525131Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:02:34.8525468Z cuda train deit_base_distilled_patch16_224    
2024-12-18T01:03:02.7709258Z W1218 01:03:02.769000 18049 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T01:03:35.9712076Z pass
2024-12-18T01:03:36.0245062Z TIMING: entire_frame_compile:44.6308 _recursive_pre_grad_passes:0.04912 _recursive_joint_graph_passes:0.83154 _recursive_post_grad_passes:0.75643 async_compile.wait:4.24448 code_gen:17.54406 inductor_compile:29.89685 backend_compile:35.0425 entire_backward_compile:7.5944 total_wall_time:52.2252
2024-12-18T01:03:36.0246870Z STATS: call_* op count: 1065 | FakeTensorMode.__torch_dispatch__:42417 | FakeTensor.__torch_dispatch__:11903 | ProxyTorchDispatchMode.__torch_dispatch__:19375
2024-12-18T01:03:36.0247711Z Dynamo produced 3 graphs covering 1065 ops with 7 graph breaks (5 unique)
2024-12-18T01:03:42.6446913Z 
2024-12-18T01:03:44.0285232Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:03:44.0287742Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:03:44.0288232Z cuda train dla102                             
2024-12-18T01:05:05.5020260Z W1218 01:05:05.500000 18372 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T01:05:09.2047635Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][189].grad", "L['self'].param_groups[0]['params'][198].grad", "L['self'].param_groups[0]['params'][210].grad", "L['self'].param_groups[0]['params'][219].grad", "L['self'].param_groups[0]['params'][231].grad", "L['self'].param_groups[0]['params'][240].grad", "L['self'].param_groups[0]['params'][252].grad", "L['self'].param_groups[0]['params'][261].grad", "L['self'].param_groups[0]['params'][273].grad", "L['self'].param_groups[0]['params'][282].grad", "L['self'].param_groups[0]['params'][297].grad", "L['self'].param_groups[0]['params'][306].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T01:06:21.9957267Z pass
2024-12-18T01:06:22.0180362Z TIMING: entire_frame_compile:122.16742 _recursive_pre_grad_passes:0.10465 _recursive_joint_graph_passes:0.94221 _recursive_post_grad_passes:1.62259 async_compile.wait:6.8536 code_gen:51.21925 inductor_compile:82.60789 backend_compile:92.58164 entire_backward_compile:22.25864 total_wall_time:144.42607
2024-12-18T01:06:22.0182004Z STATS: call_* op count: 2093 | FakeTensorMode.__torch_dispatch__:85717 | FakeTensor.__torch_dispatch__:30649 | ProxyTorchDispatchMode.__torch_dispatch__:39529
2024-12-18T01:06:22.0182857Z Dynamo produced 3 graphs covering 2093 ops with 7 graph breaks (5 unique)
2024-12-18T01:06:32.6499603Z 
2024-12-18T01:06:34.5192847Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:06:34.5193292Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:06:34.5193623Z cuda train dm_nfnet_f0                        
2024-12-18T01:07:26.2814161Z pass
2024-12-18T01:07:26.3333746Z TIMING: entire_frame_compile:28.53878 _recursive_pre_grad_passes:0.02106 _recursive_joint_graph_passes:0.68844 _recursive_post_grad_passes:1.06882 async_compile.wait:1.71973 code_gen:14.76743 inductor_compile:26.7899 backend_compile:20.57625 entire_backward_compile:16.78397 total_wall_time:45.32275
2024-12-18T01:07:26.3336600Z STATS: call_* op count: 594 | FakeTensorMode.__torch_dispatch__:36073 | ProxyTorchDispatchMode.__torch_dispatch__:16005 | FakeTensor.__torch_dispatch__:11564
2024-12-18T01:07:26.3337469Z Dynamo produced 2 graphs covering 594 ops with 6 graph breaks (5 unique)
2024-12-18T01:07:32.6399392Z 
2024-12-18T01:07:34.3266601Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:07:34.3266945Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:07:34.3267262Z cuda train dpn107                             
2024-12-18T01:09:08.2789732Z pass
2024-12-18T01:09:08.3446298Z TIMING: entire_frame_compile:59.54454 _recursive_pre_grad_passes:0.03229 _recursive_joint_graph_passes:0.86294 _recursive_post_grad_passes:1.6888 async_compile.wait:2.63861 code_gen:29.43188 inductor_compile:47.96088 backend_compile:44.82899 entire_backward_compile:23.03856 total_wall_time:82.5831
2024-12-18T01:09:08.3448458Z STATS: call_* op count: 634 | FakeTensorMode.__torch_dispatch__:61063 | FakeTensor.__torch_dispatch__:18074 | ProxyTorchDispatchMode.__torch_dispatch__:28636
2024-12-18T01:09:08.3449311Z Dynamo produced 2 graphs covering 634 ops with 6 graph breaks (5 unique)
2024-12-18T01:09:16.4230886Z 
2024-12-18T01:09:17.4643505Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:09:17.4643878Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:09:17.4644197Z cuda train eca_botnext26ts_256                
2024-12-18T01:10:02.3379214Z W1218 01:10:02.336000 21454 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T01:10:28.8956993Z pass
2024-12-18T01:10:28.9075916Z TIMING: entire_frame_compile:50.32271 _recursive_pre_grad_passes:0.04137 _recursive_joint_graph_passes:2.57068 _recursive_post_grad_passes:0.77203 async_compile.wait:5.62132 code_gen:27.93447 inductor_compile:41.29414 backend_compile:40.12517 entire_backward_compile:15.60853 total_wall_time:65.93124
2024-12-18T01:10:28.9077609Z STATS: call_* op count: 848 | FakeTensorMode.__torch_dispatch__:36712 | FakeTensor.__torch_dispatch__:11889 | ProxyTorchDispatchMode.__torch_dispatch__:15987
2024-12-18T01:10:28.9078443Z Dynamo produced 3 graphs covering 848 ops with 7 graph breaks (5 unique)
2024-12-18T01:10:35.6765426Z 
2024-12-18T01:10:36.7893534Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:10:36.7893880Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:10:36.7894207Z cuda train eca_halonext26ts                   
2024-12-18T01:11:27.0711612Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T01:11:27.0714175Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 405, in torch_dynamo_resume_in_forward_and_backward_pass_at_403
2024-12-18T01:11:27.0715241Z     pred = mod(*cloned_inputs)
2024-12-18T01:11:27.0715831Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/byobnet.py", line 1433, in forward
2024-12-18T01:11:27.0716426Z     x = self.forward_features(x)
2024-12-18T01:11:27.0717016Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/byobnet.py", line 1425, in forward_features
2024-12-18T01:11:27.0717602Z     x = self.stages(x)
2024-12-18T01:11:27.0718092Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/byobnet.py", line 894, in forward
2024-12-18T01:11:27.0718646Z     x = self.self_attn(x)
2024-12-18T01:11:27.0719300Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/layers/halo_attn.py", line 191, in forward
2024-12-18T01:11:27.0720053Z     kv = kv.unfold(2, self.win_size, self.block_size).unfold(3, self.win_size, self.block_size).reshape(
2024-12-18T01:11:27.0720463Z 
2024-12-18T01:11:27.3242298Z W1218 01:11:27.323000 22341 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T01:11:28.6296543Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T01:11:53.7046132Z pass
2024-12-18T01:11:53.7127665Z TIMING: entire_frame_compile:52.87353 _recursive_pre_grad_passes:0.04298 _recursive_joint_graph_passes:2.669 _recursive_post_grad_passes:0.84173 async_compile.wait:5.49372 code_gen:32.25293 inductor_compile:46.29761 backend_compile:41.62759 entire_backward_compile:19.17747 total_wall_time:72.051
2024-12-18T01:11:53.7130789Z STATS: call_* op count: 886 | FakeTensorMode.__torch_dispatch__:37996 | FakeTensor.__torch_dispatch__:11675 | ProxyTorchDispatchMode.__torch_dispatch__:16623
2024-12-18T01:11:53.7132106Z Dynamo produced 3 graphs covering 886 ops with 7 graph breaks (5 unique)
2024-12-18T01:12:00.6895401Z 
2024-12-18T01:12:01.5878334Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:12:01.5878683Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:12:01.5879026Z cuda train ese_vovnet19b_dw                   
2024-12-18T01:12:35.5338166Z W1218 01:12:35.532000 23082 site-packages/torch/_logging/_internal.py:1089] [7/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T01:12:36.7279587Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T01:13:00.3965378Z pass
2024-12-18T01:13:00.4038028Z TIMING: entire_frame_compile:39.59899 _recursive_pre_grad_passes:0.03404 _recursive_joint_graph_passes:0.34933 _recursive_post_grad_passes:0.46617 async_compile.wait:4.92186 code_gen:24.3532 inductor_compile:34.77681 backend_compile:30.65516 entire_backward_compile:13.62644 total_wall_time:53.22543
2024-12-18T01:13:00.4040956Z STATS: call_* op count: 622 | FakeTensorMode.__torch_dispatch__:26070 | FakeTensor.__torch_dispatch__:8694 | ProxyTorchDispatchMode.__torch_dispatch__:11135
2024-12-18T01:13:00.4041798Z Dynamo produced 4 graphs covering 622 ops with 7 graph breaks (5 unique)
2024-12-18T01:13:06.7659628Z 
2024-12-18T01:13:07.7329382Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:13:07.7329744Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:13:07.7330055Z cuda train fbnetc_100                         
2024-12-18T01:13:56.8266200Z W1218 01:13:56.825000 23890 site-packages/torch/_logging/_internal.py:1089] [7/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T01:13:59.0637736Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T01:14:46.1769744Z pass
2024-12-18T01:14:46.1819941Z TIMING: entire_frame_compile:76.87867 _recursive_pre_grad_passes:0.06374 _recursive_joint_graph_passes:0.61313 _recursive_post_grad_passes:0.98667 async_compile.wait:6.08752 code_gen:30.97929 inductor_compile:51.28271 backend_compile:59.30942 entire_backward_compile:12.38305 total_wall_time:89.26171
2024-12-18T01:14:46.1822154Z STATS: call_* op count: 1266 | FakeTensorMode.__torch_dispatch__:53306 | FakeTensor.__torch_dispatch__:18880 | ProxyTorchDispatchMode.__torch_dispatch__:23658
2024-12-18T01:14:46.1823008Z Dynamo produced 4 graphs covering 1266 ops with 7 graph breaks (5 unique)
2024-12-18T01:14:54.5610677Z 
2024-12-18T01:14:55.7119999Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:14:55.7120397Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:14:55.7120762Z cuda train fbnetv3_b                          
2024-12-18T01:16:22.0106409Z pass
2024-12-18T01:16:22.0190631Z TIMING: entire_frame_compile:54.12049 _recursive_pre_grad_passes:0.02473 _recursive_joint_graph_passes:0.67692 _recursive_post_grad_passes:1.79979 async_compile.wait:2.20329 code_gen:29.72712 inductor_compile:50.55033 backend_compile:41.33343 entire_backward_compile:25.89783 total_wall_time:80.01832
2024-12-18T01:16:22.0192246Z STATS: call_* op count: 459 | FakeTensorMode.__torch_dispatch__:51664 | FakeTensor.__torch_dispatch__:19558 | ProxyTorchDispatchMode.__torch_dispatch__:23345
2024-12-18T01:16:22.0193095Z Dynamo produced 2 graphs covering 459 ops with 6 graph breaks (5 unique)
2024-12-18T01:16:30.1156682Z 
2024-12-18T01:16:31.4349050Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:16:31.4349424Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:16:31.4349744Z cuda train gernet_l                           
2024-12-18T01:17:15.3748510Z pass
2024-12-18T01:17:15.3956625Z TIMING: entire_frame_compile:28.86372 _recursive_pre_grad_passes:0.01512 _recursive_joint_graph_passes:0.39688 _recursive_post_grad_passes:0.71816 async_compile.wait:1.27922 code_gen:12.38208 inductor_compile:22.35258 backend_compile:21.02733 entire_backward_compile:10.08914 total_wall_time:38.95285
2024-12-18T01:17:15.3958480Z STATS: call_* op count: 250 | FakeTensorMode.__torch_dispatch__:25961 | FakeTensor.__torch_dispatch__:8890 | ProxyTorchDispatchMode.__torch_dispatch__:12116
2024-12-18T01:17:15.3959536Z Dynamo produced 2 graphs covering 250 ops with 6 graph breaks (5 unique)
2024-12-18T01:17:21.5396318Z 
2024-12-18T01:17:22.5699498Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:17:22.5699877Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:17:22.5700213Z cuda train ghostnet_100                       
2024-12-18T01:18:37.9191959Z pass
2024-12-18T01:18:37.9242899Z TIMING: entire_frame_compile:49.41369 _recursive_pre_grad_passes:0.02331 _recursive_joint_graph_passes:0.60817 _recursive_post_grad_passes:1.13915 async_compile.wait:2.78113 code_gen:26.9717 inductor_compile:43.90941 backend_compile:37.7303 entire_backward_compile:20.45676 total_wall_time:69.87045
2024-12-18T01:18:37.9244982Z STATS: call_* op count: 405 | FakeTensorMode.__torch_dispatch__:44851 | FakeTensor.__torch_dispatch__:13738 | ProxyTorchDispatchMode.__torch_dispatch__:19864
2024-12-18T01:18:37.9245847Z Dynamo produced 2 graphs covering 405 ops with 6 graph breaks (5 unique)
2024-12-18T01:18:45.4866380Z 
2024-12-18T01:18:46.8757685Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:18:46.8758105Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:18:46.8758426Z cuda train gluon_inception_v3                 
2024-12-18T01:20:04.1300545Z W1218 01:20:04.129000 28436 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T01:20:07.3533267Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][189].grad", "L['self'].param_groups[0]['params'][195].grad", "L['self'].param_groups[0]['params'][198].grad", "L['self'].param_groups[0]['params'][201].grad", "L['self'].param_groups[0]['params'][204].grad", "L['self'].param_groups[0]['params'][213].grad", "L['self'].param_groups[0]['params'][219].grad", "L['self'].param_groups[0]['params'][222].grad", "L['self'].param_groups[0]['params'][225].grad", "L['self'].param_groups[0]['params'][234].grad", "L['self'].param_groups[0]['params'][237].grad", "L['self'].param_groups[0]['params'][243].grad", "L['self'].param_groups[0]['params'][246].grad", "L['self'].param_groups[0]['params'][249].grad", "L['self'].param_groups[0]['params'][261].grad", "L['self'].param_groups[0]['params'][264].grad", "L['self'].param_groups[0]['params'][270].grad", "L['self'].param_groups[0]['params'][273].grad", "L['self'].param_groups[0]['params'][276].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T01:21:10.4690402Z pass
2024-12-18T01:21:10.4927614Z TIMING: entire_frame_compile:89.90427 _recursive_pre_grad_passes:0.09182 _recursive_joint_graph_passes:1.17604 async_compile.wait:5.68729 inductor_compile:77.32034 backend_compile:63.65643 _recursive_post_grad_passes:1.80708 code_gen:51.73301 entire_backward_compile:43.54227 total_wall_time:133.44654
2024-12-18T01:21:10.4930673Z STATS: call_* op count: 1851 | FakeTensorMode.__torch_dispatch__:72963 | FakeTensor.__torch_dispatch__:21040 | ProxyTorchDispatchMode.__torch_dispatch__:35484
2024-12-18T01:21:10.4931527Z Dynamo produced 3 graphs covering 1851 ops with 7 graph breaks (5 unique)
2024-12-18T01:21:19.2684549Z 
2024-12-18T01:21:20.2965753Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:21:20.2966310Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:21:20.2966828Z cuda train gmixer_24_224                      
2024-12-18T01:22:10.3760689Z pass
2024-12-18T01:22:10.4014216Z TIMING: entire_frame_compile:29.39484 _recursive_pre_grad_passes:0.01923 _recursive_joint_graph_passes:3.38581 _recursive_post_grad_passes:1.34984 async_compile.wait:0.60821 code_gen:14.5178 inductor_compile:24.71146 backend_compile:22.47556 entire_backward_compile:15.18421 total_wall_time:44.57904
2024-12-18T01:22:10.4016389Z STATS: call_* op count: 587 | FakeTensorMode.__torch_dispatch__:47582 | FakeTensor.__torch_dispatch__:10780 | ProxyTorchDispatchMode.__torch_dispatch__:23802
2024-12-18T01:22:10.4017288Z Dynamo produced 2 graphs covering 587 ops with 6 graph breaks (5 unique)
2024-12-18T01:22:16.8209664Z 
2024-12-18T01:22:17.7435518Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:22:17.7435907Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:22:17.7436260Z cuda train gmlp_s16_224                       
2024-12-18T01:23:06.3181512Z W1218 01:23:06.317000 28958 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T01:24:11.4481245Z pass
2024-12-18T01:24:11.4688544Z TIMING: entire_frame_compile:86.89312 _recursive_pre_grad_passes:0.09103 _recursive_joint_graph_passes:2.53815 _recursive_post_grad_passes:1.53343 async_compile.wait:4.89488 code_gen:34.0354 inductor_compile:57.1654 backend_compile:66.41146 entire_backward_compile:16.82332 total_wall_time:103.71644
2024-12-18T01:24:11.4691204Z STATS: call_* op count: 2010 | FakeTensorMode.__torch_dispatch__:81345 | FakeTensor.__torch_dispatch__:24724 | ProxyTorchDispatchMode.__torch_dispatch__:37841
2024-12-18T01:24:11.4692102Z Dynamo produced 3 graphs covering 2010 ops with 7 graph breaks (5 unique)
2024-12-18T01:24:20.2294790Z 
2024-12-18T01:24:22.1127109Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:24:22.1128026Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:24:22.1128352Z cuda train hrnet_w18                          
2024-12-18T01:27:49.1222895Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T01:27:49.1225420Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 405, in torch_dynamo_resume_in_forward_and_backward_pass_at_403
2024-12-18T01:27:49.1226148Z     pred = mod(*cloned_inputs)
2024-12-18T01:27:49.1226683Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 792, in forward
2024-12-18T01:27:49.1227229Z     y = self.forward_features(x)
2024-12-18T01:27:49.1227782Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 770, in forward_features
2024-12-18T01:27:49.1228375Z     yl = self.stages(x)
2024-12-18T01:27:49.1228851Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 751, in stages
2024-12-18T01:27:49.1229380Z     yl = self.stage2(xl)
2024-12-18T01:27:49.1229877Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 506, in forward
2024-12-18T01:27:49.1230397Z     x = module(x)
2024-12-18T01:27:49.1230855Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 484, in forward
2024-12-18T01:27:49.1231375Z     y = y + f(x[j])
2024-12-18T01:27:49.1231521Z 
2024-12-18T01:27:53.2641287Z pass
2024-12-18T01:27:53.2893298Z TIMING: entire_frame_compile:151.76931 _recursive_pre_grad_passes:0.07451 _recursive_joint_graph_passes:1.79807 _recursive_post_grad_passes:4.27054 async_compile.wait:3.97427 code_gen:61.21834 inductor_compile:109.06824 backend_compile:111.00798 entire_backward_compile:48.4687 total_wall_time:200.23801
2024-12-18T01:27:53.2895912Z STATS: call_* op count: 1475 | FakeTensorMode.__torch_dispatch__:144309 | FakeTensor.__torch_dispatch__:47012 | ProxyTorchDispatchMode.__torch_dispatch__:71305
2024-12-18T01:27:53.2896753Z Dynamo produced 2 graphs covering 1475 ops with 5 graph breaks (4 unique)
2024-12-18T01:28:07.0506945Z 
2024-12-18T01:28:08.4421433Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:28:08.4421976Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:28:08.4422484Z cuda train inception_v3                       
2024-12-18T01:28:46.8877260Z pass
2024-12-18T01:28:46.9097132Z TIMING: entire_frame_compile:29.41545 _recursive_pre_grad_passes:0.02375 _recursive_joint_graph_passes:0.58719 async_compile.wait:1.24428 inductor_compile:6.72282 backend_compile:18.06385 entire_backward_compile:3.53677 total_wall_time:32.95222
2024-12-18T01:28:46.9098627Z STATS: call_* op count: 412 | FakeTensorMode.__torch_dispatch__:32189 | FakeTensor.__torch_dispatch__:46 | ProxyTorchDispatchMode.__torch_dispatch__:20409
2024-12-18T01:28:46.9099474Z Dynamo produced 2 graphs covering 412 ops with 6 graph breaks (5 unique)
2024-12-18T01:28:52.3224601Z 
2024-12-18T01:28:54.0496544Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:28:54.0496909Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:28:54.0497257Z cuda train jx_nest_base                       
2024-12-18T01:30:07.6727193Z W1218 01:30:07.671000 30721 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T01:30:11.1553903Z ('Grad tensors ["L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][57].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T01:31:14.2068594Z pass
2024-12-18T01:31:14.2546129Z TIMING: entire_frame_compile:101.36646 _recursive_pre_grad_passes:0.10244 _recursive_joint_graph_passes:2.10701 _recursive_post_grad_passes:2.31854 async_compile.wait:7.9484 code_gen:47.65569 inductor_compile:74.90507 backend_compile:80.06119 entire_backward_compile:24.20477 total_wall_time:125.57122
2024-12-18T01:31:14.2548259Z STATS: call_* op count: 2283 | FakeTensorMode.__torch_dispatch__:102765 | FakeTensor.__torch_dispatch__:26270 | ProxyTorchDispatchMode.__torch_dispatch__:45950
2024-12-18T01:31:14.2549123Z Dynamo produced 3 graphs covering 2283 ops with 7 graph breaks (5 unique)
2024-12-18T01:31:23.5810971Z 
2024-12-18T01:31:24.3918255Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:31:24.3918818Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:31:24.3919458Z cuda train lcnet_050                          
2024-12-18T01:31:52.7305438Z pass
2024-12-18T01:31:52.7331108Z TIMING: entire_frame_compile:17.37049 _recursive_pre_grad_passes:0.00928 _recursive_joint_graph_passes:0.29509 _recursive_post_grad_passes:0.56062 async_compile.wait:1.02796 code_gen:9.36346 inductor_compile:16.61026 backend_compile:13.74739 entire_backward_compile:7.96007 total_wall_time:25.33057
2024-12-18T01:31:52.7333231Z STATS: call_* op count: 129 | FakeTensorMode.__torch_dispatch__:15913 | FakeTensor.__torch_dispatch__:5956 | ProxyTorchDispatchMode.__torch_dispatch__:6908
2024-12-18T01:31:52.7334094Z Dynamo produced 2 graphs covering 129 ops with 6 graph breaks (5 unique)
2024-12-18T01:31:58.4117794Z 
2024-12-18T01:31:59.5942032Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:31:59.5942376Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:31:59.5942692Z cuda train levit_128                          
2024-12-18T01:33:09.7392647Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T01:33:09.7393765Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 405, in torch_dynamo_resume_in_forward_and_backward_pass_at_403
2024-12-18T01:33:09.7394477Z     pred = mod(*cloned_inputs)
2024-12-18T01:33:09.7395010Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 718, in forward
2024-12-18T01:33:09.7396070Z     x = self.forward_features(x)
2024-12-18T01:33:09.7396638Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 709, in forward_features
2024-12-18T01:33:09.7397217Z     x = self.stages(x)
2024-12-18T01:33:09.7397719Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 520, in forward
2024-12-18T01:33:09.7398247Z     x = self.blocks(x)
2024-12-18T01:33:09.7398733Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 458, in forward
2024-12-18T01:33:09.7399420Z     x = x + self.drop_path1(self.attn(x))
2024-12-18T01:33:09.7399972Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 237, in forward
2024-12-18T01:33:09.7400591Z     attn = q @ k * self.scale + self.get_attention_biases(x.device)
2024-12-18T01:33:09.7401262Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 212, in get_attention_biases
2024-12-18T01:33:09.7401945Z     return self.attention_biases[:, self.attention_bias_idxs]
2024-12-18T01:33:09.7402240Z 
2024-12-18T01:33:10.1162205Z W1218 01:33:10.115000 32293 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T01:33:11.9770048Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][163].grad", "L['self'].param_groups[0]['params'][164].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][166].grad", "L['self'].param_groups[0]['params'][167].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][169].grad", "L['self'].param_groups[0]['params'][170].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][172].grad", "L['self'].param_groups[0]['params'][173].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][175].grad", "L['self'].param_groups[0]['params'][176].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][178].grad", "L['self'].param_groups[0]['params'][179].grad", "L['self'].param_groups[0]['params'][180].grad", "L['self'].param_groups[0]['params'][181].grad", "L['self'].param_groups[0]['params'][182].grad", "L['self'].param_groups[0]['params'][183].grad", "L['self'].param_groups[0]['params'][184].grad", "L['self'].param_groups[0]['params'][185].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][187].grad", "L['self'].param_groups[0]['params'][188].grad", "L['self'].param_groups[0]['params'][189].grad", "L['self'].param_groups[0]['params'][190].grad", "L['self'].param_groups[0]['params'][191].grad", "L['self'].param_groups[0]['params'][192].grad", "L['self'].param_groups[0]['params'][193].grad", "L['self'].param_groups[0]['params'][194].grad", "L['self'].param_groups[0]['params'][195].grad", "L['self'].param_groups[0]['params'][196].grad", "L['self'].param_groups[0]['params'][197].grad", "L['self'].param_groups[0]['params'][198].grad", "L['self'].param_groups[0]['params'][199].grad", "L['self'].param_groups[0]['params'][200].grad", "L['self'].param_groups[0]['params'][201].grad", "L['self'].param_groups[0]['params'][202].grad", "L['self'].param_groups[0]['params'][203].grad", "L['self'].param_groups[0]['params'][204].grad", "L['self'].param_groups[0]['params'][205].grad", "L['self'].param_groups[0]['params'][206].grad", "L['self'].param_groups[0]['params'][207].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T01:33:55.2670524Z pass
2024-12-18T01:33:55.2706594Z TIMING: entire_frame_compile:87.55953 _recursive_pre_grad_passes:0.07551 _recursive_joint_graph_passes:7.17731 _recursive_post_grad_passes:1.28261 async_compile.wait:6.77228 code_gen:39.21059 inductor_compile:62.07363 backend_compile:69.35726 entire_backward_compile:21.61052 total_wall_time:109.17005
2024-12-18T01:33:55.2708484Z STATS: call_* op count: 1679 | FakeTensorMode.__torch_dispatch__:73660 | FakeTensor.__torch_dispatch__:21164 | ProxyTorchDispatchMode.__torch_dispatch__:33794
2024-12-18T01:33:55.2709658Z Dynamo produced 3 graphs covering 1679 ops with 7 graph breaks (5 unique)
2024-12-18T01:34:04.5569978Z 
2024-12-18T01:34:05.7961573Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:34:05.7962214Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:34:05.7962822Z cuda train mixer_b16_224                      
2024-12-18T01:34:34.5870931Z W1218 01:34:34.586000 33335 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T01:35:05.8132298Z pass
2024-12-18T01:35:05.8564095Z TIMING: entire_frame_compile:44.258 _recursive_pre_grad_passes:0.04587 _recursive_joint_graph_passes:2.65141 _recursive_post_grad_passes:0.93102 async_compile.wait:4.03184 code_gen:17.60095 inductor_compile:29.36269 backend_compile:35.31417 entire_backward_compile:8.16825 total_wall_time:52.42625
2024-12-18T01:35:05.8566422Z STATS: call_* op count: 972 | FakeTensorMode.__torch_dispatch__:40783 | FakeTensor.__torch_dispatch__:12288 | ProxyTorchDispatchMode.__torch_dispatch__:19055
2024-12-18T01:35:05.8567780Z Dynamo produced 3 graphs covering 972 ops with 7 graph breaks (5 unique)
2024-12-18T01:35:08.9559886Z accuracy            pass_rate=96.67%
2024-12-18T01:35:08.9563694Z calls_captured      gmean=0.00x mean=996.300x
2024-12-18T01:35:08.9567325Z unique_graphs       gmean=0.00x mean=2.533x
2024-12-18T01:35:08.9571024Z graph_breaks        gmean=0.00x mean=6.267x
2024-12-18T01:35:08.9574764Z unique_graph_breaks gmean=0.00x mean=4.767x
2024-12-18T01:35:08.9578379Z autograd_captures   gmean=0.00x mean=0.000x
2024-12-18T01:35:08.9582927Z autograd_compiles   gmean=0.00x mean=0.000x
2024-12-18T01:35:08.9586946Z cudagraph_skips     gmean=0.00x mean=0.133x
2024-12-18T01:35:12.1219094Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_timm_models.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_timm_training.csv
2024-12-18T01:35:12.3882826Z adv_inception_v3                    PASS
2024-12-18T01:35:12.3887652Z beit_base_patch16_224               PASS
2024-12-18T01:35:12.3893139Z botnet26t_256                       PASS
2024-12-18T01:35:12.3898859Z cait_m36_384                        XFAIL
2024-12-18T01:35:12.3904973Z coat_lite_mini                      PASS
2024-12-18T01:35:12.3911677Z convit_base                         PASS
2024-12-18T01:35:12.3916874Z convmixer_768_32                    PASS
2024-12-18T01:35:12.3922607Z convnext_base                       PASS
2024-12-18T01:35:12.3928047Z crossvit_9_240                      PASS
2024-12-18T01:35:12.3933403Z cspdarknet53                        PASS
2024-12-18T01:35:12.3938411Z deit_base_distilled_patch16_224     PASS
2024-12-18T01:35:12.3942997Z dla102                              PASS
2024-12-18T01:35:12.3947768Z dm_nfnet_f0                         PASS
2024-12-18T01:35:12.3952425Z dpn107                              PASS
2024-12-18T01:35:12.3956990Z eca_botnext26ts_256                 PASS
2024-12-18T01:35:12.3962367Z eca_halonext26ts                    PASS
2024-12-18T01:35:12.3966984Z ese_vovnet19b_dw                    PASS
2024-12-18T01:35:12.3971346Z fbnetc_100                          PASS
2024-12-18T01:35:12.3975973Z fbnetv3_b                           PASS
2024-12-18T01:35:12.3980657Z gernet_l                            PASS
2024-12-18T01:35:12.3985296Z ghostnet_100                        PASS
2024-12-18T01:35:12.3990119Z gluon_inception_v3                  PASS
2024-12-18T01:35:12.3994613Z gmixer_24_224                       PASS
2024-12-18T01:35:12.3999424Z gmlp_s16_224                        PASS
2024-12-18T01:35:12.4004366Z hrnet_w18                           PASS
2024-12-18T01:35:12.4009123Z inception_v3                        PASS
2024-12-18T01:35:12.4013773Z jx_nest_base                        PASS
2024-12-18T01:35:12.4018412Z lcnet_050                           PASS
2024-12-18T01:35:12.4023121Z levit_128                           PASS
2024-12-18T01:35:12.4027619Z mixer_b16_224                       PASS
2024-12-18T01:35:12.4465496Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_timm_models.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_timm_training.csv
2024-12-18T01:35:12.7133430Z adv_inception_v3                    PASS
2024-12-18T01:35:12.7136999Z beit_base_patch16_224               PASS
2024-12-18T01:35:12.7141914Z botnet26t_256                       PASS
2024-12-18T01:35:12.7146752Z cait_m36_384                        PASS
2024-12-18T01:35:12.7151907Z coat_lite_mini                      PASS
2024-12-18T01:35:12.7156737Z convit_base                         PASS
2024-12-18T01:35:12.7162125Z convmixer_768_32                    PASS
2024-12-18T01:35:12.7167488Z convnext_base                       PASS
2024-12-18T01:35:12.7172330Z crossvit_9_240                      PASS
2024-12-18T01:35:12.7177361Z cspdarknet53                        PASS
2024-12-18T01:35:12.7182312Z deit_base_distilled_patch16_224     PASS
2024-12-18T01:35:12.7187388Z dla102                              PASS
2024-12-18T01:35:12.7192170Z dm_nfnet_f0                         PASS
2024-12-18T01:35:12.7197301Z dpn107                              PASS
2024-12-18T01:35:12.7202744Z eca_botnext26ts_256                 PASS
2024-12-18T01:35:12.7208088Z eca_halonext26ts                    PASS
2024-12-18T01:35:12.7213033Z ese_vovnet19b_dw                    PASS
2024-12-18T01:35:12.7218333Z fbnetc_100                          PASS
2024-12-18T01:35:12.7223291Z fbnetv3_b                           PASS
2024-12-18T01:35:12.7228346Z gernet_l                            PASS
2024-12-18T01:35:12.7233364Z ghostnet_100                        PASS
2024-12-18T01:35:12.7238503Z gluon_inception_v3                  PASS
2024-12-18T01:35:12.7243961Z gmixer_24_224                       PASS
2024-12-18T01:35:12.7248744Z gmlp_s16_224                        PASS
2024-12-18T01:35:12.7253737Z hrnet_w18                           PASS
2024-12-18T01:35:12.7259031Z inception_v3                        PASS
2024-12-18T01:35:12.7264296Z jx_nest_base                        PASS
2024-12-18T01:35:12.7269469Z lcnet_050                           PASS
2024-12-18T01:35:12.7274453Z levit_128                           PASS
2024-12-18T01:35:12.7279323Z mixer_b16_224                       PASS
2024-12-18T01:35:12.7720017Z + cleanup_workspace
2024-12-18T01:35:12.7720864Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.'
2024-12-18T01:35:12.7721642Z sudo may print the following warning message that can be ignored. The chown command will still run.
2024-12-18T01:35:12.7722251Z + echo '    sudo: setrlimit(RLIMIT_STACK): Operation not permitted'
2024-12-18T01:35:12.7722710Z     sudo: setrlimit(RLIMIT_STACK): Operation not permitted
2024-12-18T01:35:12.7723234Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42'
2024-12-18T01:35:12.7723919Z For more details refer to https://github.com/sudo-project/sudo/issues/42
2024-12-18T01:35:12.7724398Z + sudo chown -R 1000 /var/lib/jenkins/workspace
2024-12-18T01:35:13.5453373Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@release/2.6
2024-12-18T01:35:13.5453879Z with:
2024-12-18T01:35:13.5454142Z   benchmark-results-dir: test/test-reports
2024-12-18T01:35:13.5454471Z   dry-run: false
2024-12-18T01:35:13.5454714Z   schema-version: v3
2024-12-18T01:35:13.5455157Z   github-token: ***
2024-12-18T01:35:13.5455395Z env:
2024-12-18T01:35:13.5455616Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:35:13.5455956Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:35:13.5456493Z   DOCKER_CONTAINER_ID: 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T01:35:13.5456975Z ##[endgroup]
2024-12-18T01:35:13.5485733Z ##[group]Run set -eux
2024-12-18T01:35:13.5485997Z set -eux
2024-12-18T01:35:13.5486267Z python3 -mpip install boto3==1.35.33
2024-12-18T01:35:13.5500123Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:35:13.5500721Z env:
2024-12-18T01:35:13.5500970Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:35:13.5501309Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:35:13.5501848Z   DOCKER_CONTAINER_ID: 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T01:35:13.5502334Z ##[endgroup]
2024-12-18T01:35:13.5538539Z + python3 -mpip install boto3==1.35.33
2024-12-18T01:35:13.8563722Z Defaulting to user installation because normal site-packages is not writeable
2024-12-18T01:35:13.8759798Z Requirement already satisfied: boto3==1.35.33 in /home/ec2-user/.local/lib/python3.9/site-packages (1.35.33)
2024-12-18T01:35:13.8809493Z Requirement already satisfied: botocore<1.36.0,>=1.35.33 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.33) (1.35.83)
2024-12-18T01:35:13.8813827Z Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.4)
2024-12-18T01:35:13.8818030Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.0)
2024-12-18T01:35:13.8870494Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.8.1)
2024-12-18T01:35:13.8880483Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.25.10)
2024-12-18T01:35:13.8919929Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.15.0)
2024-12-18T01:35:14.0141369Z ##[group]Run set -eux
2024-12-18T01:35:14.0141649Z set -eux
2024-12-18T01:35:14.0141885Z 
2024-12-18T01:35:14.0142131Z if [[ -z "${GITHUB_TOKEN}" ]]; then
2024-12-18T01:35:14.0142490Z   echo "Missing github-token input"
2024-12-18T01:35:14.0142811Z   exit 1
2024-12-18T01:35:14.0143058Z fi
2024-12-18T01:35:14.0151762Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:35:14.0152136Z env:
2024-12-18T01:35:14.0152362Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:35:14.0152703Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:35:14.0153251Z   DOCKER_CONTAINER_ID: 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T01:35:14.0154136Z   GITHUB_TOKEN: ***
2024-12-18T01:35:14.0154393Z ##[endgroup]
2024-12-18T01:35:14.0182788Z + [[ -z *** ]]
2024-12-18T01:35:14.0235530Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main
2024-12-18T01:35:14.0235957Z with:
2024-12-18T01:35:14.0236297Z   github-token: ***
2024-12-18T01:35:14.0236533Z env:
2024-12-18T01:35:14.0236754Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:35:14.0237090Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:35:14.0237626Z   DOCKER_CONTAINER_ID: 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T01:35:14.0238115Z ##[endgroup]
2024-12-18T01:35:14.0263807Z ##[group]Run set -eux
2024-12-18T01:35:14.0264080Z set -eux
2024-12-18T01:35:14.0264313Z 
2024-12-18T01:35:14.0264771Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2024-12-18T01:35:14.0273079Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:35:14.0273449Z env:
2024-12-18T01:35:14.0273670Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:35:14.0274007Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:35:14.0274545Z   DOCKER_CONTAINER_ID: 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T01:35:14.0275329Z   GITHUB_TOKEN: ***
2024-12-18T01:35:14.0275569Z ##[endgroup]
2024-12-18T01:35:14.0302984Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 12383255690 i-019aa8c64fc4447ce
2024-12-18T01:35:15.2737322Z setting job-id=34567152834
2024-12-18T01:35:15.2737877Z setting job-name=cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T01:35:15.2848070Z ##[group]Run set -eux
2024-12-18T01:35:15.2848397Z set -eux
2024-12-18T01:35:15.2848632Z 
2024-12-18T01:35:15.2849013Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \
2024-12-18T01:35:15.2849521Z   --schema-version "${SCHEMA_VERSION}" \
2024-12-18T01:35:15.2849893Z   --repo "${REPO}" \
2024-12-18T01:35:15.2850228Z   --head-branch "${HEAD_BRANCH}" \
2024-12-18T01:35:15.2850563Z   --head-sha "${HEAD_SHA}" \
2024-12-18T01:35:15.2850900Z   --workflow-id "${WORKFLOW_RUN_ID}" \
2024-12-18T01:35:15.2851251Z   --run-attempt "${RUN_ATTEMPT}" \
2024-12-18T01:35:15.2851587Z   --job-id "${JOB_ID}" \
2024-12-18T01:35:15.2851890Z   --job-name "${JOB_NAME}"
2024-12-18T01:35:15.2860724Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:35:15.2861096Z env:
2024-12-18T01:35:15.2861316Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:35:15.2861643Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:35:15.2862190Z   DOCKER_CONTAINER_ID: 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T01:35:15.2862670Z   SCHEMA_VERSION: v3
2024-12-18T01:35:15.2862960Z   REPO: pytorch/pytorch
2024-12-18T01:35:15.2863230Z   HEAD_BRANCH: refs/heads/release/2.6
2024-12-18T01:35:15.2863587Z   HEAD_SHA: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T01:35:15.2863934Z   WORKFLOW_RUN_ID: 12383255690
2024-12-18T01:35:15.2864216Z   RUN_ATTEMPT: 1
2024-12-18T01:35:15.2864454Z   JOB_ID: 34567152834
2024-12-18T01:35:15.2864890Z   JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T01:35:15.2865380Z ##[endgroup]
2024-12-18T01:35:15.2895621Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/release/2.6/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/heads/release/2.6 --head-sha 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 --workflow-id 12383255690 --run-attempt 1 --job-id 34567152834 --job-name 'cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu)'
2024-12-18T01:35:15.3233923Z ##[group]Run set -eux
2024-12-18T01:35:15.3234208Z set -eux
2024-12-18T01:35:15.3234443Z 
2024-12-18T01:35:15.3234707Z # TODO (huydhn): Implement this part
2024-12-18T01:35:15.3235076Z echo "runners=[]" >> "${GITHUB_OUTPUT}"
2024-12-18T01:35:15.3244006Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:35:15.3244389Z env:
2024-12-18T01:35:15.3244616Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:35:15.3244968Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:35:15.3245724Z   DOCKER_CONTAINER_ID: 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T01:35:15.3246224Z ##[endgroup]
2024-12-18T01:35:15.3273109Z + echo 'runners=[]'
2024-12-18T01:35:15.3306726Z ##[group]Run set -eux
2024-12-18T01:35:15.3307042Z set -eux
2024-12-18T01:35:15.3307268Z 
2024-12-18T01:35:15.3307528Z # TODO (huydhn): Implement this part
2024-12-18T01:35:15.3307956Z echo "dependencies={}" >> "${GITHUB_OUTPUT}"
2024-12-18T01:35:15.3316714Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:35:15.3317087Z env:
2024-12-18T01:35:15.3317311Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:35:15.3317653Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:35:15.3318194Z   DOCKER_CONTAINER_ID: 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T01:35:15.3318674Z ##[endgroup]
2024-12-18T01:35:15.3344637Z + echo 'dependencies={}'
2024-12-18T01:35:15.3377417Z ##[group]Run set -eux
2024-12-18T01:35:15.3377706Z set -eux
2024-12-18T01:35:15.3378115Z 
2024-12-18T01:35:15.3378394Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then
2024-12-18T01:35:15.3378828Z   echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping"
2024-12-18T01:35:15.3379305Z   # We don't want the job to fail if the directory doesn't exist
2024-12-18T01:35:15.3379698Z   exit 0
2024-12-18T01:35:15.3379938Z fi
2024-12-18T01:35:15.3380164Z 
2024-12-18T01:35:15.3380407Z if [[ "${DRY_RUN}" == "true" ]]; then
2024-12-18T01:35:15.3380869Z   python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \
2024-12-18T01:35:15.3381418Z     --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \
2024-12-18T01:35:15.3381836Z     --metadata "${BENCHMARK_METADATA}" \
2024-12-18T01:35:15.3382190Z     --runners "${RUNNER_INFO}" \
2024-12-18T01:35:15.3382537Z     --dependencies "${DEPENDENCIES}" \
2024-12-18T01:35:15.3382867Z     --dry-run
2024-12-18T01:35:15.3383125Z else
2024-12-18T01:35:15.3383492Z   python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \
2024-12-18T01:35:15.3384030Z     --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \
2024-12-18T01:35:15.3384448Z     --metadata "${BENCHMARK_METADATA}" \
2024-12-18T01:35:15.3384802Z     --runners "${RUNNER_INFO}" \
2024-12-18T01:35:15.3385146Z     --dependencies "${DEPENDENCIES}"
2024-12-18T01:35:15.3385552Z fi
2024-12-18T01:35:15.3393862Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:35:15.3394347Z env:
2024-12-18T01:35:15.3394680Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:35:15.3395162Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:35:15.3395803Z   DOCKER_CONTAINER_ID: 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T01:35:15.3396392Z   BENCHMARK_RESULTS_DIR: test/test-reports
2024-12-18T01:35:15.3396865Z   DRY_RUN: false
2024-12-18T01:35:15.3398253Z   BENCHMARK_METADATA: {"timestamp": 1734485715, "schema_version": "v3", "name": "cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/release/2.6", "head_sha": "0cdf8b1d09254cfda66191d1bd01e3041c3c76f7", "workflow_id": 12383255690, "run_attempt": 1, "job_id": 34567152834}
2024-12-18T01:35:15.3399808Z   RUNNER_INFO: []
2024-12-18T01:35:15.3400214Z   DEPENDENCIES: {}
2024-12-18T01:35:15.3400570Z ##[endgroup]
2024-12-18T01:35:15.3428162Z + [[ ! -d test/test-reports ]]
2024-12-18T01:35:15.3428526Z + [[ false == \t\r\u\e ]]
2024-12-18T01:35:15.3431019Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/release/2.6/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1734485715, "schema_version": "v3", "name": "cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/release/2.6", "head_sha": "0cdf8b1d09254cfda66191d1bd01e3041c3c76f7", "workflow_id": 12383255690, "run_attempt": 1, "job_id": 34567152834}' --runners '[]' --dependencies '{}'
2024-12-18T01:35:15.5155011Z INFO:root:Upload test/test-reports/inference_timm_models.json to s3://ossci-benchmarks/v3/pytorch/pytorch/12383255690/34567152834/inference_timm_models.json
2024-12-18T01:35:15.5575616Z INFO:botocore.credentials:Found credentials from IAM Role: gh-ci-github-action-runners-runner-role
2024-12-18T01:35:15.7842438Z INFO:root:Upload test/test-reports/training_timm_models.json to s3://ossci-benchmarks/v3/pytorch/pytorch/12383255690/34567152834/training_timm_models.json
2024-12-18T01:35:15.9428176Z INFO:root:Upload test/test-reports/training_timm_model_graph_breaks.json to s3://ossci-benchmarks/v3/pytorch/pytorch/12383255690/34567152834/training_timm_model_graph_breaks.json
2024-12-18T01:35:16.1843999Z ##[group]Run cat test/**/*_toprint.log || true
2024-12-18T01:35:16.1844747Z cat test/**/*_toprint.log || true
2024-12-18T01:35:16.1853791Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:35:16.1854354Z env:
2024-12-18T01:35:16.1854665Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:35:16.1855152Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:35:16.1855792Z   DOCKER_CONTAINER_ID: 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T01:35:16.1856334Z ##[endgroup]
2024-12-18T01:35:16.1943290Z cat: 'test/**/*_toprint.log': No such file or directory
2024-12-18T01:35:16.1987308Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2024-12-18T01:35:16.1987673Z kill "$MONITOR_SCRIPT_PID"
2024-12-18T01:35:16.1996170Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:35:16.1996540Z env:
2024-12-18T01:35:16.1996755Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:35:16.1997095Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:35:16.1997663Z   DOCKER_CONTAINER_ID: 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T01:35:16.1998178Z   MONITOR_SCRIPT_PID: 341287
2024-12-18T01:35:16.1998444Z ##[endgroup]
2024-12-18T01:35:16.2143970Z Prepare all required actions
2024-12-18T01:35:16.2144375Z Getting action download info
2024-12-18T01:35:16.3582179Z Download action repository 'actions/upload-artifact@v4' (SHA:6f51ac03b9356f520e9adb1b1b7802705f340c2b)
2024-12-18T01:35:16.6586857Z ##[group]Run ./.github/actions/upload-test-artifacts
2024-12-18T01:35:16.6587220Z with:
2024-12-18T01:35:16.6587595Z   file-suffix: test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_34567152834
2024-12-18T01:35:16.6588060Z   s3-bucket: gha-artifacts
2024-12-18T01:35:16.6588320Z env:
2024-12-18T01:35:16.6588539Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:35:16.6588873Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:35:16.6589408Z   DOCKER_CONTAINER_ID: 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T01:35:16.6589904Z ##[endgroup]
2024-12-18T01:35:16.6619566Z ##[group]Run # Remove any previous test jsons if they exist
2024-12-18T01:35:16.6620040Z # Remove any previous test jsons if they exist
2024-12-18T01:35:16.6620406Z rm -f test-jsons-*.zip
2024-12-18T01:35:16.6620829Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json'
2024-12-18T01:35:16.6629560Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:35:16.6629933Z env:
2024-12-18T01:35:16.6630153Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:35:16.6630489Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:35:16.6631019Z   DOCKER_CONTAINER_ID: 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T01:35:16.6631652Z   FILE_SUFFIX: test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_34567152834
2024-12-18T01:35:16.6632098Z ##[endgroup]
2024-12-18T01:35:16.6733110Z   adding: test/test-reports/inference_timm_models.json (deflated 99%)
2024-12-18T01:35:16.6773156Z   adding: test/test-reports/training_timm_models.json (deflated 99%)
2024-12-18T01:35:16.6873794Z   adding: test/test-reports/training_timm_model_graph_breaks.json (deflated 99%)
2024-12-18T01:35:16.6905145Z ##[group]Run # Remove any previous test reports if they exist
2024-12-18T01:35:16.6905615Z # Remove any previous test reports if they exist
2024-12-18T01:35:16.6905996Z rm -f test-reports-*.zip
2024-12-18T01:35:16.6906462Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv'
2024-12-18T01:35:16.6914595Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:35:16.6914971Z env:
2024-12-18T01:35:16.6915201Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:35:16.6915542Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:35:16.6916086Z   DOCKER_CONTAINER_ID: 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T01:35:16.6916716Z   FILE_SUFFIX: test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_34567152834
2024-12-18T01:35:16.6917363Z ##[endgroup]
2024-12-18T01:35:16.6966775Z   adding: test/test-reports/inference_timm_models.csv (deflated 68%)
2024-12-18T01:35:16.6967327Z   adding: test/test-reports/training_timm_models.csv (deflated 65%)
2024-12-18T01:35:16.6971922Z   adding: test/test-reports/training_timm_model_graph_breaks.csv (deflated 97%)
2024-12-18T01:35:16.6972563Z   adding: test/test-reports/training_timm_model_graph_break_deduped.csv (deflated 73%)
2024-12-18T01:35:16.7001731Z ##[group]Run # Remove any previous usage logs if they exist
2024-12-18T01:35:16.7002191Z # Remove any previous usage logs if they exist
2024-12-18T01:35:16.7002558Z rm -f logs-*.zip
2024-12-18T01:35:16.7003008Z # this workflow is also run in bazel build test, but we dont generate usage reports for it
2024-12-18T01:35:16.7003875Z # so check to see if the file exists first
2024-12-18T01:35:16.7004238Z if [ -f 'usage_log.txt' ]; then
2024-12-18T01:35:16.7004620Z     zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt'
2024-12-18T01:35:16.7004961Z fi
2024-12-18T01:35:16.7005563Z if find "test/test-reports" -name "*.log" 2>/dev/null | grep -q .; then
2024-12-18T01:35:16.7006112Z     zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log'
2024-12-18T01:35:16.7006494Z fi
2024-12-18T01:35:16.7014455Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:35:16.7014827Z env:
2024-12-18T01:35:16.7015047Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:35:16.7015388Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:35:16.7015935Z   DOCKER_CONTAINER_ID: 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T01:35:16.7016567Z   FILE_SUFFIX: test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_34567152834
2024-12-18T01:35:16.7017021Z ##[endgroup]
2024-12-18T01:35:16.7253074Z   adding: usage_log.txt (deflated 98%)
2024-12-18T01:35:16.7308775Z ##[group]Run # Remove any previous debugging artifacts if they exist
2024-12-18T01:35:16.7309307Z # Remove any previous debugging artifacts if they exist
2024-12-18T01:35:16.7309721Z rm -f debug-*.zip
2024-12-18T01:35:16.7310002Z if [ -d 'test/debug' ]; then
2024-12-18T01:35:16.7310366Z   zip -r "debug-${FILE_SUFFIX}.zip" test/debug
2024-12-18T01:35:16.7310709Z fi
2024-12-18T01:35:16.7318670Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:35:16.7319036Z env:
2024-12-18T01:35:16.7319401Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:35:16.7319736Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:35:16.7320286Z   DOCKER_CONTAINER_ID: 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T01:35:16.7320921Z   FILE_SUFFIX: test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_34567152834
2024-12-18T01:35:16.7321370Z ##[endgroup]
2024-12-18T01:35:16.7407150Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-12-18T01:35:16.7407494Z with:
2024-12-18T01:35:16.7407714Z   s3-bucket: gha-artifacts
2024-12-18T01:35:16.7408039Z   s3-prefix: pytorch/pytorch/12383255690/1/artifact

2024-12-18T01:35:16.7408394Z   retention-days: 14
2024-12-18T01:35:16.7408660Z   if-no-files-found: warn
2024-12-18T01:35:16.7408935Z   path: test-jsons-*.zip
2024-12-18T01:35:16.7409196Z   name: artifact
2024-12-18T01:35:16.7409432Z   region: us-east-1
2024-12-18T01:35:16.7409664Z env:
2024-12-18T01:35:16.7409884Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:35:16.7410222Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:35:16.7410752Z   DOCKER_CONTAINER_ID: 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T01:35:16.7411232Z ##[endgroup]
2024-12-18T01:35:17.1071379Z NOTE: s3-prefix specified, ignoring name parameter
2024-12-18T01:35:17.1071842Z With the provided path, there will be 1 file uploaded
2024-12-18T01:35:17.1072287Z Uploading to s3 prefix: pytorch/pytorch/12383255690/1/artifact
2024-12-18T01:35:17.1131372Z Starting upload of test-jsons-test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_34567152834.zip
2024-12-18T01:35:17.2865237Z Finished upload of test-jsons-test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_34567152834.zip
2024-12-18T01:35:17.3168234Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-12-18T01:35:17.3168567Z with:
2024-12-18T01:35:17.3168796Z   s3-bucket: gha-artifacts
2024-12-18T01:35:17.3169123Z   s3-prefix: pytorch/pytorch/12383255690/1/artifact

2024-12-18T01:35:17.3169475Z   retention-days: 14
2024-12-18T01:35:17.3169742Z   if-no-files-found: error
2024-12-18T01:35:17.3170025Z   path: test-reports-*.zip
2024-12-18T01:35:17.3170299Z   name: artifact
2024-12-18T01:35:17.3170535Z   region: us-east-1
2024-12-18T01:35:17.3170768Z env:
2024-12-18T01:35:17.3170987Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:35:17.3171318Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:35:17.3171860Z   DOCKER_CONTAINER_ID: 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T01:35:17.3172356Z ##[endgroup]
2024-12-18T01:35:17.6478761Z NOTE: s3-prefix specified, ignoring name parameter
2024-12-18T01:35:17.6479557Z With the provided path, there will be 1 file uploaded
2024-12-18T01:35:17.6480023Z Uploading to s3 prefix: pytorch/pytorch/12383255690/1/artifact
2024-12-18T01:35:17.6538584Z Starting upload of test-reports-test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_34567152834.zip
2024-12-18T01:35:17.8229312Z Finished upload of test-reports-test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_34567152834.zip
2024-12-18T01:35:17.8532247Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-12-18T01:35:17.8532588Z with:
2024-12-18T01:35:17.8532816Z   s3-bucket: gha-artifacts
2024-12-18T01:35:17.8533134Z   s3-prefix: pytorch/pytorch/12383255690/1/artifact

2024-12-18T01:35:17.8533489Z   retention-days: 14
2024-12-18T01:35:17.8533756Z   if-no-files-found: ignore
2024-12-18T01:35:17.8534035Z   path: logs-*.zip
2024-12-18T01:35:17.8534278Z   name: artifact
2024-12-18T01:35:17.8534526Z   region: us-east-1
2024-12-18T01:35:17.8534761Z env:
2024-12-18T01:35:17.8534981Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:35:17.8535335Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:35:17.8535879Z   DOCKER_CONTAINER_ID: 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T01:35:17.8536349Z ##[endgroup]
2024-12-18T01:35:18.1841013Z NOTE: s3-prefix specified, ignoring name parameter
2024-12-18T01:35:18.1841459Z With the provided path, there will be 1 file uploaded
2024-12-18T01:35:18.1841918Z Uploading to s3 prefix: pytorch/pytorch/12383255690/1/artifact
2024-12-18T01:35:18.1898938Z Starting upload of logs-test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_34567152834.zip
2024-12-18T01:35:18.3139649Z Finished upload of logs-test-inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_34567152834.zip
2024-12-18T01:35:18.3447516Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-12-18T01:35:18.3447849Z with:
2024-12-18T01:35:18.3448068Z   s3-bucket: gha-artifacts
2024-12-18T01:35:18.3448412Z   s3-prefix: pytorch/pytorch/12383255690/1/artifact

2024-12-18T01:35:18.3448761Z   retention-days: 14
2024-12-18T01:35:18.3449030Z   if-no-files-found: ignore
2024-12-18T01:35:18.3449322Z   path: debug-*.zip
2024-12-18T01:35:18.3449554Z   name: artifact
2024-12-18T01:35:18.3449799Z   region: us-east-1
2024-12-18T01:35:18.3450028Z env:
2024-12-18T01:35:18.3450251Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:35:18.3450721Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:35:18.3451253Z   DOCKER_CONTAINER_ID: 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T01:35:18.3451731Z ##[endgroup]
2024-12-18T01:35:18.6720706Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded.
2024-12-18T01:35:18.7028489Z ##[group]Run # shellcheck disable=SC2156
2024-12-18T01:35:18.7028877Z # shellcheck disable=SC2156
2024-12-18T01:35:18.7029448Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2024-12-18T01:35:18.7038611Z shell: /usr/bin/bash -e {0}
2024-12-18T01:35:18.7038886Z env:
2024-12-18T01:35:18.7039106Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:35:18.7039584Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:35:18.7040125Z   DOCKER_CONTAINER_ID: 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T01:35:18.7040609Z ##[endgroup]
2024-12-18T01:35:18.9762867Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@release/2.6
2024-12-18T01:35:18.9763553Z with:
2024-12-18T01:35:18.9763865Z env:
2024-12-18T01:35:18.9764196Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:35:18.9764706Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:35:18.9765495Z   DOCKER_CONTAINER_ID: 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T01:35:18.9766277Z ##[endgroup]
2024-12-18T01:35:18.9801767Z ##[group]Run set -eou pipefail
2024-12-18T01:35:18.9802247Z set -eou pipefail
2024-12-18T01:35:18.9802672Z 
2024-12-18T01:35:18.9803550Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2024-12-18T01:35:18.9804233Z for _ in $(seq 1440); do
2024-12-18T01:35:18.9804610Z     # Break if no ssh session exists anymore
2024-12-18T01:35:18.9804955Z     if [ "$(who)" = "" ]; then
2024-12-18T01:35:18.9805250Z       break
2024-12-18T01:35:18.9805515Z     fi
2024-12-18T01:35:18.9805744Z     echo "."
2024-12-18T01:35:18.9805985Z     sleep 5
2024-12-18T01:35:18.9806225Z done
2024-12-18T01:35:18.9814938Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:35:18.9815309Z env:
2024-12-18T01:35:18.9815532Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:35:18.9815866Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:35:18.9816406Z   DOCKER_CONTAINER_ID: 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T01:35:18.9816895Z ##[endgroup]
2024-12-18T01:35:18.9843558Z Holding runner for 2 hours until all ssh sessions have logged out
2024-12-18T01:35:18.9918129Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2024-12-18T01:35:18.9918702Z # ignore expansion of "docker ps -q" since it could be empty
2024-12-18T01:35:18.9919306Z # shellcheck disable=SC2046
2024-12-18T01:35:18.9919660Z docker stop $(docker ps -q) || true
2024-12-18T01:35:18.9920003Z # Prune all of the docker images
2024-12-18T01:35:18.9920329Z docker system prune -af
2024-12-18T01:35:18.9929097Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:35:18.9929463Z env:
2024-12-18T01:35:18.9929685Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:35:18.9930011Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:35:18.9930550Z   DOCKER_CONTAINER_ID: 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T01:35:18.9931048Z ##[endgroup]
2024-12-18T01:35:19.5393362Z 2ca6b3c5afad
2024-12-18T01:35:21.0409761Z Deleted Containers:
2024-12-18T01:35:21.0410259Z 2ca6b3c5afad0006423814371199f660b35f2769adfa0ad80ffb8b4f97ea43a2
2024-12-18T01:35:21.0410594Z 
2024-12-18T01:35:33.1222440Z Deleted Images:
2024-12-18T01:35:33.1223752Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T01:35:33.1225454Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks@sha256:ff80d4c93d18e8ab7a2501f0e62b7c40cf97aa83fc9702e88b9c2d02cdcbecdb
2024-12-18T01:35:33.1226623Z deleted: sha256:67e93a8badaf799ceb664d7c453077c6c6785d56dc10c720be9358edeb91759b
2024-12-18T01:35:33.1227241Z deleted: sha256:7958cbbed07d0e7e404de5150d4669da383984a17de74e455563b7ecc409b2e8
2024-12-18T01:35:33.1227859Z deleted: sha256:dc6a8b10e3a8452ea47655ddad3b52816f12bcbe35298b3aa96aae8a0da9bc12
2024-12-18T01:35:33.1228877Z deleted: sha256:c92bf6e7cfbdecd0dfe86d452eac5dbe5aa5b2a6aa7ef2eaeffb21af2870cda7
2024-12-18T01:35:33.1229507Z deleted: sha256:f9597cb89d67c0c899e266adc5e5f04817474bad200712b3c256ed33c346b3e8
2024-12-18T01:35:33.1230120Z deleted: sha256:eb062308fcca86ab5a341555bf03af093d21f1d44dbe3dce6f9b7781b650b2f1
2024-12-18T01:35:33.1230946Z deleted: sha256:96f28e0a6369023dd4e75125d2454531ff686a14467f3e06d3b5c6da89893472
2024-12-18T01:35:33.1231553Z deleted: sha256:3e6e64b255956d1bf66c97393ae4325f7fc67414cd710e284c17190cb262c92d
2024-12-18T01:35:33.1232155Z deleted: sha256:a3c0a58b4f23e46b45dd1b385363731800042b3f04ca66f8867b99f3c9311f11
2024-12-18T01:35:33.1232760Z deleted: sha256:435e519973ffb9c85330ae8db1d1fb1fa75d673e2f27640c2ca7277c6352092b
2024-12-18T01:35:33.1233355Z deleted: sha256:69800681a8b6382d520ac4b8b8828bd5540a297961ddeb004e415bc46b33f80d
2024-12-18T01:35:33.1233958Z deleted: sha256:1e837ad662894f46afc56a73aa5ccb04064e4a974c9e3cad86ce9e5546b1d63d
2024-12-18T01:35:33.1234575Z deleted: sha256:688eccf2945022bd75f9f2782e2397ea54f1032c2435aa8f7dffa23cbcf2ae99
2024-12-18T01:35:33.1235185Z deleted: sha256:4855651bf4c3c8acad23029fda7d17f43bf32a69e8d575bb43f4491f0d653c1f
2024-12-18T01:35:33.1235803Z deleted: sha256:ca2d39907b7e10e3c7ae753b8bd230fac77c4e4b15980befd71173da8a36da93
2024-12-18T01:35:33.1236420Z deleted: sha256:d4f04a8342f272455eb816a413dc2dfcc75b9ebb9ef7f6fab32436621412ba6b
2024-12-18T01:35:33.1237035Z deleted: sha256:63ae01c22dc840b763ddc08bf4e145c14a04e0ff8a833b357ed45cb0962a8116
2024-12-18T01:35:33.1237650Z deleted: sha256:cf91e4695bf05d3607b7fa68e3ba3d6cc8762e6fac8773483f8116812f3d52c0
2024-12-18T01:35:33.1238251Z deleted: sha256:71baa7b72633376bb45367512dcb676c81b1e09964085a3151381530df7cdc28
2024-12-18T01:35:33.1238837Z deleted: sha256:68855433b69809d035da2a30cb22864ed1cd2795434571a0c77532de19344497
2024-12-18T01:35:33.1239526Z deleted: sha256:929280209b1f4e29860f386a519df32161cb608338d5338f24735201939dc91f
2024-12-18T01:35:33.1240122Z deleted: sha256:25133b63282efce5897b0e8fbf64d8b2a2625e15792dad95bdada277b78c777d
2024-12-18T01:35:33.1240730Z deleted: sha256:11fddfe713ea9ddd5acd8285319a606609b5c428111f15a019c8286e7164b9b4
2024-12-18T01:35:33.1241332Z deleted: sha256:b1464f3888e3b0a1694aa960a726584d01ae6b62df53fe8743d197d2d5051036
2024-12-18T01:35:33.1241922Z deleted: sha256:9627c2925d7a3936b439a6f519083e650d160bc2522d2a9244d6a1dfda02bc83
2024-12-18T01:35:33.1242527Z deleted: sha256:33880827ded8236cc91531c2f928d6de5c7cfb95a241da2653b7e8a8d1ea4881
2024-12-18T01:35:33.1243127Z deleted: sha256:6851b39610a99eb7ec6939a15e35657e144eb3e628a3856afa0a0adff49b8c2f
2024-12-18T01:35:33.1243771Z deleted: sha256:1a76c7f0af1e05689638a3792e8b9d4418648384aed1f56d3ccde5b0a9681d29
2024-12-18T01:35:33.1244376Z deleted: sha256:072f501c97aae0be55d1457426bc5f8238b0bb86eac9e92889b01924352d79a5
2024-12-18T01:35:33.1244967Z deleted: sha256:8277868848759cc86b200280dee8701d7e8328a811bf75330ed24aeaf490e69e
2024-12-18T01:35:33.1245561Z deleted: sha256:1da994d575d08f659512ea580648bfddd62020acc1aef2f7d4c6008d36a6f83f
2024-12-18T01:35:33.1246176Z deleted: sha256:ddf6d49d71559688ae65aeecccbbc58dd555887d4b3306d02a87d049114dd2f9
2024-12-18T01:35:33.1246785Z deleted: sha256:648d99cd03665c7b88aae7e3291f27daa65f496f7ab2c37f0ee62c72a4459151
2024-12-18T01:35:33.1247379Z deleted: sha256:6e57cd88598cddad4070d39467d8e1d4c387388f89d2970a55391653f9243e1a
2024-12-18T01:35:33.1247995Z deleted: sha256:9b0c5db353ac5d4e0995c2a70ae3c28390676afe60dbd11e4ade07c792eac4ce
2024-12-18T01:35:33.1248606Z deleted: sha256:8fa5311fe41b6caf697456e9678220a152bbf6a7645f4b7ff89ea9317ae04444
2024-12-18T01:35:33.1249210Z deleted: sha256:dc438d6465050b81bff4a8c5925e70c2167712dbab174c38f79b6a5d7af407af
2024-12-18T01:35:33.1249805Z deleted: sha256:29b8f0a42a1f3b944f2c54348348d173d322522b578e3d485428d67996f6adde
2024-12-18T01:35:33.1250389Z deleted: sha256:26231024334c7a704f9acba86430547a013e959014bfd7a31e26c6ed78521479
2024-12-18T01:35:33.1250987Z deleted: sha256:4166e9dd7564ae2f46ea2c3dcb8a8ef6bef9e06f4faf1742f4c6f21861967193
2024-12-18T01:35:33.1251702Z deleted: sha256:80a3113f256f146f0a2528ca69aa9244a1f9ba1ecdf54c3ef8ba0fb21507cbc4
2024-12-18T01:35:33.1252315Z deleted: sha256:e90bbba5d7c08e01c5bedb0a855cc5cdd371688e23994155674704963591296f
2024-12-18T01:35:33.1252928Z deleted: sha256:d3b3df431f230849965cc370c2f1853a81d78dd6eb3f7ac3bc3b4ca64aa9818a
2024-12-18T01:35:33.1253535Z deleted: sha256:7a33f9ca143b08c9f5d8c53974b2d6fe840739b5d2cd30694ba1c86db9393908
2024-12-18T01:35:33.1254324Z deleted: sha256:bad1d7dae968956b4a81369dedfbcf539fd5b61dd3051e2eccb5457174fe788c
2024-12-18T01:35:33.1254943Z deleted: sha256:08a420245ca6eed4aa5662d0fec04e7892165cdb3fd2f5b99428adbb4d6199c0
2024-12-18T01:35:33.1255556Z deleted: sha256:c3fda6b9a4d08a345824508dc5625f52e4d23163843d0caf337c60feda68a830
2024-12-18T01:35:33.1256177Z deleted: sha256:c74a8a3ca5e31670eed41e4bbb4a89f25c7c05dacabdf93cbd03984ead0330e2
2024-12-18T01:35:33.1256790Z deleted: sha256:4171d4c54045bd3035546906a7dd3b63c9ca748e0a34eeba27fe918f52cdcf85
2024-12-18T01:35:33.1257394Z deleted: sha256:f6a1d112657a4aeaca96ac2792a10ea09463dc3d633cb8572314814d5abf1ddb
2024-12-18T01:35:33.1258013Z deleted: sha256:af5d54d84b9c302254498eed8b0bbce1cdd705351bf5cc210d3f11c0deb269b3
2024-12-18T01:35:33.1258622Z deleted: sha256:45ae227353c6db8230b86fc1c4b8c50def31bd64895b879d91ea6e600669045d
2024-12-18T01:35:33.1259222Z deleted: sha256:6032ea27ad90071e3b75290e03fd0763a74155f513f356d21e56663a878b0ed9
2024-12-18T01:35:33.1259827Z deleted: sha256:d5aef8ace96787c48070530c09b68e4bcb23bc92a9fb0978220204f64bfbd97f
2024-12-18T01:35:33.1260433Z deleted: sha256:e3116153021ece2a3d29eccdce1f7e0f092316e889c1d534257789f6b5743d51
2024-12-18T01:35:33.1261059Z deleted: sha256:fb0f11460e5fa2e4895230ae84c97fe0552e9509dfbae60abf94e1b8a6563fb9
2024-12-18T01:35:33.1261674Z deleted: sha256:e262415e0abb361b284df9e05790ae27cc3a1ea7b8d4618e7eb3eca17cf1d18e
2024-12-18T01:35:33.1262284Z deleted: sha256:7f810eab57a6121f1a2aae664f7a5940d12f789661445ce3500ecd3b3688b46b
2024-12-18T01:35:33.1262887Z deleted: sha256:6ed71f55ff78520154c43468df2f05a92eaa28ee2aa2e28b8ea860b9ffb258ef
2024-12-18T01:35:33.1263498Z deleted: sha256:92a10d8d42a56a8516d0e9e14fec53263302fe6e6438ced263a3845680456dd7
2024-12-18T01:35:33.1264106Z deleted: sha256:ec67f912ff53e9b7164f48250da3bb4c1ffc94c3eb2c1c56c534a94fe55e3845
2024-12-18T01:35:33.1264722Z deleted: sha256:c85033e4f60ace56eaad80a8151b1bdd1619865b10a9ccee5638878b0a027c23
2024-12-18T01:35:33.1265334Z deleted: sha256:0e8680489f0a37851beb2cb5e42f0df129247a68b755b08a6367ea42180b3e7a
2024-12-18T01:35:33.1265934Z deleted: sha256:3a767fb38f6b1124e5fe9a230fdd383ccb188c38fe1d6ec252260676efe255c8
2024-12-18T01:35:33.1266550Z deleted: sha256:2cce59d2a43773cb7c6b83d239dab3a53659767f96bc0d47a70ee0d51cfa096f
2024-12-18T01:35:33.1267162Z deleted: sha256:51e50e75789ab5b4e40a66bb2f284e2da947e37c0e757c74d49fdeea5da9cbec
2024-12-18T01:35:33.1267770Z deleted: sha256:29e55e4feccba0660950ed92941d309527fe5f41bea609662d0e703b9c76b3a1
2024-12-18T01:35:33.1268390Z deleted: sha256:d15bee8fc455fcb45d16b0abc67f18c99ce1cf2c9e998ef5a100bf8373f2f8ab
2024-12-18T01:35:33.1269005Z deleted: sha256:363936db6a8869d44b6f0bee97a786b168093848e9c08c8042830fc7fccd956b
2024-12-18T01:35:33.1269598Z deleted: sha256:0f54d1f89ceaf71ded488007635a618fac1528bf3603d9415a593fe4926f8b2a
2024-12-18T01:35:33.1270205Z deleted: sha256:2f5796e9879cab7dac02bdbb178aec5301ac00029b38be8613333ac639c4dc3a
2024-12-18T01:35:33.1270811Z deleted: sha256:e68525d57f9da8f6c74b456d92869c4e60e451b88cb403addbc4ea917d8ab3ac
2024-12-18T01:35:33.1271419Z deleted: sha256:3386e25983944ce3b2ebdde10d840832d581a7dfafd1df66568822a18979e268
2024-12-18T01:35:33.1272021Z deleted: sha256:d3a036e6bd36f014b3a1540d5b848655ecdf72a20dc81c0da5f80430700cfa9a
2024-12-18T01:35:33.1272627Z deleted: sha256:96d0556708ff4ec0d5d727d5947d6cb29a9d6c16e1a1ee76a92dc85aa3ea8f25
2024-12-18T01:35:33.1273235Z deleted: sha256:f3c2fb3abccb3a33dbf204d3f0d8857537348e6c1d3cc87a431afe866435d41b
2024-12-18T01:35:33.1273848Z deleted: sha256:45cd6c9a41f12991ee60a89e775abb604b0892ef3936526c18184bbbf4de57da
2024-12-18T01:35:33.1274458Z deleted: sha256:b3cacd0508bbfd2fd668f30c539033e7c6b606bf669258c8e8a3cbf80bf050c6
2024-12-18T01:35:33.1275160Z deleted: sha256:de72ce930361a96c151a8a4725f8360f33df6c812b6ed6d3dd5b6e6710a5d2c0
2024-12-18T01:35:33.1275771Z deleted: sha256:2e5d223d2bfa0f47e628b631a540d20c284dabb380ad9ee4caec65bf8d0ecde2
2024-12-18T01:35:33.1276376Z deleted: sha256:14a85e604a0401093323d356b3bb7f75a8473abd712b4bf1a2ef01fa432b344c
2024-12-18T01:35:33.1277059Z deleted: sha256:15a95a1e8f47d7c18dcbf9a6e2fc530cddcb161c0093c663181f3e73dc9f599e
2024-12-18T01:35:33.1277674Z deleted: sha256:106e8431b412f51ccd75ea46a2d5cb4343b23273cbcf50188377cb93aa9a6d82
2024-12-18T01:35:33.1278035Z 
2024-12-18T01:35:33.1278148Z Total reclaimed space: 47.91GB
2024-12-18T01:35:33.1364475Z Post job cleanup.
2024-12-18T01:35:33.1414652Z Post job cleanup.
2024-12-18T01:35:33.2244957Z [command]/usr/bin/git version
2024-12-18T01:35:33.2304821Z git version 2.40.1
2024-12-18T01:35:33.2344254Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/2723079b-1e93-4788-8666-9f35f14aa6a7' before making global git config changes
2024-12-18T01:35:33.2345190Z Adding repository directory to the temporary git global config as a safe directory
2024-12-18T01:35:33.2348420Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-12-18T01:35:33.2381141Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2024-12-18T01:35:33.2411028Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2024-12-18T01:35:33.2781629Z Entering 'android/libs/fbjni'
2024-12-18T01:35:33.2851335Z Entering 'third_party/FP16'
2024-12-18T01:35:33.2920537Z Entering 'third_party/FXdiv'
2024-12-18T01:35:33.2989479Z Entering 'third_party/NNPACK'
2024-12-18T01:35:33.3059410Z Entering 'third_party/NVTX'
2024-12-18T01:35:33.3130559Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T01:35:33.3200731Z Entering 'third_party/XNNPACK'
2024-12-18T01:35:33.3286313Z Entering 'third_party/benchmark'
2024-12-18T01:35:33.3355707Z Entering 'third_party/composable_kernel'
2024-12-18T01:35:33.3431635Z Entering 'third_party/cpp-httplib'
2024-12-18T01:35:33.3500592Z Entering 'third_party/cpuinfo'
2024-12-18T01:35:33.3571177Z Entering 'third_party/cudnn_frontend'
2024-12-18T01:35:33.3639897Z Entering 'third_party/cutlass'
2024-12-18T01:35:33.3716881Z Entering 'third_party/eigen'
2024-12-18T01:35:33.3788850Z Entering 'third_party/fbgemm'
2024-12-18T01:35:33.3856112Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T01:35:33.3924237Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T01:35:33.3989977Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T01:35:33.4062917Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T01:35:33.4133529Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T01:35:33.4205660Z Entering 'third_party/flatbuffers'
2024-12-18T01:35:33.4278989Z Entering 'third_party/fmt'
2024-12-18T01:35:33.4348535Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T01:35:33.4418929Z Entering 'third_party/gloo'
2024-12-18T01:35:33.4488527Z Entering 'third_party/googletest'
2024-12-18T01:35:33.4561647Z Entering 'third_party/ideep'
2024-12-18T01:35:33.4628716Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T01:35:33.4703682Z Entering 'third_party/ittapi'
2024-12-18T01:35:33.4774267Z Entering 'third_party/kineto'
2024-12-18T01:35:33.4842742Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T01:35:33.4908176Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T01:35:33.4977392Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T01:35:33.5048025Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T01:35:33.5119420Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T01:35:33.5186343Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T01:35:33.5259560Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T01:35:33.5327514Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T01:35:33.5396559Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T01:35:33.5473346Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T01:35:33.5544630Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T01:35:33.5610490Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T01:35:33.5682437Z Entering 'third_party/mimalloc'
2024-12-18T01:35:33.5753569Z Entering 'third_party/nccl/nccl'
2024-12-18T01:35:33.5824544Z Entering 'third_party/nlohmann'
2024-12-18T01:35:33.5896105Z Entering 'third_party/onnx'
2024-12-18T01:35:33.5978923Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T01:35:33.6054450Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T01:35:33.6125755Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T01:35:33.6192343Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T01:35:33.6259687Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T01:35:33.6328101Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T01:35:33.6395272Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T01:35:33.6462283Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T01:35:33.6528629Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T01:35:33.6594118Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T01:35:33.6662996Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T01:35:33.6732991Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T01:35:33.6823089Z Entering 'third_party/pocketfft'
2024-12-18T01:35:33.6893080Z Entering 'third_party/protobuf'
2024-12-18T01:35:33.6964246Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T01:35:33.7031625Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T01:35:33.7101382Z Entering 'third_party/psimd'
2024-12-18T01:35:33.7170111Z Entering 'third_party/pthreadpool'
2024-12-18T01:35:33.7241432Z Entering 'third_party/pybind11'
2024-12-18T01:35:33.7310311Z Entering 'third_party/python-peachpy'
2024-12-18T01:35:33.7379580Z Entering 'third_party/sleef'
2024-12-18T01:35:33.7448364Z Entering 'third_party/tensorpipe'
2024-12-18T01:35:33.7517673Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T01:35:33.7583469Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T01:35:33.7655115Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T01:35:33.7721023Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T01:35:33.7787427Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T01:35:33.7881707Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2024-12-18T01:35:33.7903342Z http.https://github.com/.extraheader
2024-12-18T01:35:33.7913185Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2024-12-18T01:35:33.7944973Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2024-12-18T01:35:33.8298150Z Entering 'android/libs/fbjni'
2024-12-18T01:35:33.8343207Z http.https://github.com/.extraheader
2024-12-18T01:35:33.8388045Z Entering 'third_party/FP16'
2024-12-18T01:35:33.8433418Z http.https://github.com/.extraheader
2024-12-18T01:35:33.8477989Z Entering 'third_party/FXdiv'
2024-12-18T01:35:33.8527059Z http.https://github.com/.extraheader
2024-12-18T01:35:33.8570265Z Entering 'third_party/NNPACK'
2024-12-18T01:35:33.8617336Z http.https://github.com/.extraheader
2024-12-18T01:35:33.8660417Z Entering 'third_party/NVTX'
2024-12-18T01:35:33.8706238Z http.https://github.com/.extraheader
2024-12-18T01:35:33.8750926Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T01:35:33.8795201Z http.https://github.com/.extraheader
2024-12-18T01:35:33.8839845Z Entering 'third_party/XNNPACK'
2024-12-18T01:35:33.8884370Z http.https://github.com/.extraheader
2024-12-18T01:35:33.8944335Z Entering 'third_party/benchmark'
2024-12-18T01:35:33.8989268Z http.https://github.com/.extraheader
2024-12-18T01:35:33.9036042Z Entering 'third_party/composable_kernel'
2024-12-18T01:35:33.9080941Z http.https://github.com/.extraheader
2024-12-18T01:35:33.9135827Z Entering 'third_party/cpp-httplib'
2024-12-18T01:35:33.9180826Z http.https://github.com/.extraheader
2024-12-18T01:35:33.9226242Z Entering 'third_party/cpuinfo'
2024-12-18T01:35:33.9270900Z http.https://github.com/.extraheader
2024-12-18T01:35:33.9319095Z Entering 'third_party/cudnn_frontend'
2024-12-18T01:35:33.9363784Z http.https://github.com/.extraheader
2024-12-18T01:35:33.9408681Z Entering 'third_party/cutlass'
2024-12-18T01:35:33.9452837Z http.https://github.com/.extraheader
2024-12-18T01:35:33.9505244Z Entering 'third_party/eigen'
2024-12-18T01:35:33.9549368Z http.https://github.com/.extraheader
2024-12-18T01:35:33.9595930Z Entering 'third_party/fbgemm'
2024-12-18T01:35:33.9640085Z http.https://github.com/.extraheader
2024-12-18T01:35:33.9684221Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T01:35:33.9728439Z http.https://github.com/.extraheader
2024-12-18T01:35:33.9771226Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T01:35:33.9814732Z http.https://github.com/.extraheader
2024-12-18T01:35:33.9857134Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T01:35:33.9899858Z http.https://github.com/.extraheader
2024-12-18T01:35:33.9949134Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T01:35:33.9991206Z http.https://github.com/.extraheader
2024-12-18T01:35:34.0035288Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T01:35:34.0077767Z http.https://github.com/.extraheader
2024-12-18T01:35:34.0123853Z Entering 'third_party/flatbuffers'
2024-12-18T01:35:34.0169723Z http.https://github.com/.extraheader
2024-12-18T01:35:34.0218848Z Entering 'third_party/fmt'
2024-12-18T01:35:34.0264566Z http.https://github.com/.extraheader
2024-12-18T01:35:34.0309232Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T01:35:34.0354094Z http.https://github.com/.extraheader
2024-12-18T01:35:34.0398431Z Entering 'third_party/gloo'
2024-12-18T01:35:34.0444076Z http.https://github.com/.extraheader
2024-12-18T01:35:34.0488248Z Entering 'third_party/googletest'
2024-12-18T01:35:34.0534041Z http.https://github.com/.extraheader
2024-12-18T01:35:34.0579064Z Entering 'third_party/ideep'
2024-12-18T01:35:34.0623748Z http.https://github.com/.extraheader
2024-12-18T01:35:34.0665755Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T01:35:34.0709123Z http.https://github.com/.extraheader
2024-12-18T01:35:34.0761042Z Entering 'third_party/ittapi'
2024-12-18T01:35:34.0806230Z http.https://github.com/.extraheader
2024-12-18T01:35:34.0850585Z Entering 'third_party/kineto'
2024-12-18T01:35:34.0895394Z http.https://github.com/.extraheader
2024-12-18T01:35:34.0940065Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T01:35:34.0983915Z http.https://github.com/.extraheader
2024-12-18T01:35:34.1029802Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T01:35:34.1072636Z http.https://github.com/.extraheader
2024-12-18T01:35:34.1118300Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T01:35:34.1161074Z http.https://github.com/.extraheader
2024-12-18T01:35:34.1205350Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T01:35:34.1247981Z http.https://github.com/.extraheader
2024-12-18T01:35:34.1292688Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T01:35:34.1337541Z http.https://github.com/.extraheader
2024-12-18T01:35:34.1379796Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T01:35:34.1423327Z http.https://github.com/.extraheader
2024-12-18T01:35:34.1471612Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T01:35:34.1516520Z http.https://github.com/.extraheader
2024-12-18T01:35:34.1561536Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T01:35:34.1604386Z http.https://github.com/.extraheader
2024-12-18T01:35:34.1648792Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T01:35:34.1691814Z http.https://github.com/.extraheader
2024-12-18T01:35:34.1737632Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T01:35:34.1779985Z http.https://github.com/.extraheader
2024-12-18T01:35:34.1827634Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T01:35:34.1873892Z http.https://github.com/.extraheader
2024-12-18T01:35:34.1917479Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T01:35:34.1960872Z http.https://github.com/.extraheader
2024-12-18T01:35:34.2009165Z Entering 'third_party/mimalloc'
2024-12-18T01:35:34.2053502Z http.https://github.com/.extraheader
2024-12-18T01:35:34.2098194Z Entering 'third_party/nccl/nccl'
2024-12-18T01:35:34.2142191Z http.https://github.com/.extraheader
2024-12-18T01:35:34.2186324Z Entering 'third_party/nlohmann'
2024-12-18T01:35:34.2230982Z http.https://github.com/.extraheader
2024-12-18T01:35:34.2277628Z Entering 'third_party/onnx'
2024-12-18T01:35:34.2320868Z http.https://github.com/.extraheader
2024-12-18T01:35:34.2379275Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T01:35:34.2423434Z http.https://github.com/.extraheader
2024-12-18T01:35:34.2472041Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T01:35:34.2517107Z http.https://github.com/.extraheader
2024-12-18T01:35:34.2561976Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T01:35:34.2604776Z http.https://github.com/.extraheader
2024-12-18T01:35:34.2647579Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T01:35:34.2690582Z http.https://github.com/.extraheader
2024-12-18T01:35:34.2735211Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T01:35:34.2777626Z http.https://github.com/.extraheader
2024-12-18T01:35:34.2820625Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T01:35:34.2862559Z http.https://github.com/.extraheader
2024-12-18T01:35:34.2906871Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T01:35:34.2953088Z http.https://github.com/.extraheader
2024-12-18T01:35:34.2999971Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T01:35:34.3042764Z http.https://github.com/.extraheader
2024-12-18T01:35:34.3086510Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T01:35:34.3129628Z http.https://github.com/.extraheader
2024-12-18T01:35:34.3171076Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T01:35:34.3214563Z http.https://github.com/.extraheader
2024-12-18T01:35:34.3260197Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T01:35:34.3302825Z http.https://github.com/.extraheader
2024-12-18T01:35:34.3349663Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T01:35:34.3393700Z http.https://github.com/.extraheader
2024-12-18T01:35:34.3462213Z Entering 'third_party/pocketfft'
2024-12-18T01:35:34.3506096Z http.https://github.com/.extraheader
2024-12-18T01:35:34.3549263Z Entering 'third_party/protobuf'
2024-12-18T01:35:34.3592772Z http.https://github.com/.extraheader
2024-12-18T01:35:34.3638505Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T01:35:34.3680528Z http.https://github.com/.extraheader
2024-12-18T01:35:34.3723592Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T01:35:34.3765080Z http.https://github.com/.extraheader
2024-12-18T01:35:34.3811656Z Entering 'third_party/psimd'
2024-12-18T01:35:34.3856896Z http.https://github.com/.extraheader
2024-12-18T01:35:34.3899452Z Entering 'third_party/pthreadpool'
2024-12-18T01:35:34.3942857Z http.https://github.com/.extraheader
2024-12-18T01:35:34.3988945Z Entering 'third_party/pybind11'
2024-12-18T01:35:34.4034008Z http.https://github.com/.extraheader
2024-12-18T01:35:34.4077522Z Entering 'third_party/python-peachpy'
2024-12-18T01:35:34.4122869Z http.https://github.com/.extraheader
2024-12-18T01:35:34.4165034Z Entering 'third_party/sleef'
2024-12-18T01:35:34.4209799Z http.https://github.com/.extraheader
2024-12-18T01:35:34.4252526Z Entering 'third_party/tensorpipe'
2024-12-18T01:35:34.4296938Z http.https://github.com/.extraheader
2024-12-18T01:35:34.4340205Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T01:35:34.4381352Z http.https://github.com/.extraheader
2024-12-18T01:35:34.4423946Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T01:35:34.4464880Z http.https://github.com/.extraheader
2024-12-18T01:35:34.4506070Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T01:35:34.4547315Z http.https://github.com/.extraheader
2024-12-18T01:35:34.4588538Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T01:35:34.4630552Z http.https://github.com/.extraheader
2024-12-18T01:35:34.4670373Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T01:35:34.4712452Z http.https://github.com/.extraheader
2024-12-18T01:35:34.4875239Z A job completed hook has been configured by the self-hosted runner administrator
2024-12-18T01:35:34.4911677Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh'
2024-12-18T01:35:34.4921238Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:35:34.4921777Z ##[endgroup]
2024-12-18T01:35:34.5042495Z [!ALERT!] Swap in detected! [!ALERT!]
2024-12-18T01:35:46.2053312Z [!ALERT!] Swap out detected [!ALERT!]
2024-12-18T01:36:05.0252902Z Cleaning up orphan processes