2024-12-18T00:18:00.3733651Z Current runner version: '2.321.0'
2024-12-18T00:18:00.3740493Z Runner name: 'i-0ad7d688021a5725d'
2024-12-18T00:18:00.3741553Z Runner group name: 'Default'
2024-12-18T00:18:00.3742665Z Machine name: 'ip-10-0-64-118'
2024-12-18T00:18:00.3748015Z ##[group]GITHUB_TOKEN Permissions
2024-12-18T00:18:00.3750712Z Actions: read
2024-12-18T00:18:00.3751545Z Attestations: read
2024-12-18T00:18:00.3752300Z Checks: read
2024-12-18T00:18:00.3753209Z Contents: read
2024-12-18T00:18:00.3754044Z Deployments: read
2024-12-18T00:18:00.3754769Z Discussions: read
2024-12-18T00:18:00.3755557Z Issues: read
2024-12-18T00:18:00.3756262Z Metadata: read
2024-12-18T00:18:00.3756952Z Packages: read
2024-12-18T00:18:00.3757722Z Pages: read
2024-12-18T00:18:00.3758433Z PullRequests: read
2024-12-18T00:18:00.3759203Z RepositoryProjects: read
2024-12-18T00:18:00.3760058Z SecurityEvents: read
2024-12-18T00:18:00.3760763Z Statuses: read
2024-12-18T00:18:00.3761482Z ##[endgroup]
2024-12-18T00:18:00.3764673Z Secret source: Actions
2024-12-18T00:18:00.3765628Z Prepare workflow directory
2024-12-18T00:18:00.4241820Z Prepare all required actions
2024-12-18T00:18:00.4278761Z Getting action download info
2024-12-18T00:18:00.6353522Z Download action repository 'pytorch/test-infra@release/2.6' (SHA:eb0adf5a84668865394af69e26428b32c8105c1c)
2024-12-18T00:18:02.1706826Z Download action repository 'pytorch/pytorch@release/2.6' (SHA:0cdf8b1d09254cfda66191d1bd01e3041c3c76f7)
2024-12-18T00:18:16.3469066Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0)
2024-12-18T00:18:16.5628997Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2024-12-18T00:18:16.8378044Z Getting action download info
2024-12-18T00:18:16.9949580Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c)
2024-12-18T00:18:17.2798922Z Getting action download info
2024-12-18T00:18:17.4079676Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e)
2024-12-18T00:18:17.5962137Z Getting action download info
2024-12-18T00:18:17.7035785Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482)
2024-12-18T00:18:17.8840554Z Getting action download info
2024-12-18T00:18:18.0001858Z Download action repository 'pytorch/test-infra@main' (SHA:a07505a74641a4ff5123d635defac481ef28ef1e)
2024-12-18T00:18:19.6001814Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/release/2.6 (0cdf8b1d09254cfda66191d1bd01e3041c3c76f7)
2024-12-18T00:18:19.6003714Z ##[group] Inputs
2024-12-18T00:18:19.6004117Z   build-environment: linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-12-18T00:18:19.6006052Z   test-matrix: {"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}
2024-12-18T00:18:19.6008476Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:18:19.6009365Z   sync-tag: 
2024-12-18T00:18:19.6010114Z   timeout-minutes: 240
2024-12-18T00:18:19.6010396Z   use-gha: 
2024-12-18T00:18:19.6010637Z   dashboard-tag: 
2024-12-18T00:18:19.6010903Z   s3-bucket: gha-artifacts
2024-12-18T00:18:19.6011195Z   aws-role-to-assume: 
2024-12-18T00:18:19.6011737Z   disable-monitor: false
2024-12-18T00:18:19.6012031Z ##[endgroup]
2024-12-18T00:18:19.6012790Z Complete job name: cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:18:19.6556513Z A job started hook has been configured by the self-hosted runner administrator
2024-12-18T00:18:19.6659826Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh'
2024-12-18T00:18:19.6671272Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:18:19.6671896Z ##[endgroup]
2024-12-18T00:18:20.8084655Z Runner Type: linux.g5.4xlarge.nvidia.gpu
2024-12-18T00:18:20.8085154Z Instance Type: g5.4xlarge
2024-12-18T00:18:20.8085434Z AMI Name: unknown
2024-12-18T00:18:20.8127221Z AMI ID: ami-0fff1b9a61dec8a5f
2024-12-18T00:18:26.5835242Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@release/2.6
2024-12-18T00:18:26.5835823Z with:
2024-12-18T00:18:26.5836576Z   github-secret: ***
2024-12-18T00:18:26.5837449Z   instructions: All testing is done inside the container, to start an interactive session run:
  docker exec -it $(docker container ps --format '{{.ID}}') bash

2024-12-18T00:18:26.5838317Z   activate-with-label: false
2024-12-18T00:18:26.5838802Z   label: with-ssh
2024-12-18T00:18:26.5839227Z   remove-existing-keys: true
2024-12-18T00:18:26.5839637Z   fail-silently: true
2024-12-18T00:18:26.5839978Z env:
2024-12-18T00:18:26.5840335Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:18:26.5840741Z ##[endgroup]
2024-12-18T00:18:26.7032118Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info.
2024-12-18T00:18:26.7033468Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys
2024-12-18T00:18:26.7191931Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@release/2.6
2024-12-18T00:18:26.7192529Z with:
2024-12-18T00:18:26.7192903Z   no-sudo: true
2024-12-18T00:18:26.7193304Z   submodules: recursive
2024-12-18T00:18:26.7193670Z   fetch-depth: 0
2024-12-18T00:18:26.7194006Z env:
2024-12-18T00:18:26.7194368Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:18:26.7194740Z ##[endgroup]
2024-12-18T00:18:26.7280394Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T00:18:26.7281503Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T00:18:26.7293799Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:18:26.7294383Z env:
2024-12-18T00:18:26.7294704Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:18:26.7295079Z ##[endgroup]
2024-12-18T00:18:26.7431563Z ##[group]Run retry () {
2024-12-18T00:18:26.7431890Z retry () {
2024-12-18T00:18:26.7432276Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2024-12-18T00:18:26.7432723Z }
2024-12-18T00:18:26.7432991Z echo "${GITHUB_WORKSPACE}"
2024-12-18T00:18:26.7433332Z if [ -z "${NO_SUDO}" ]; then
2024-12-18T00:18:26.7433765Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2024-12-18T00:18:26.7434134Z else
2024-12-18T00:18:26.7434432Z   retry rm -rf "${GITHUB_WORKSPACE}"
2024-12-18T00:18:26.7434763Z fi
2024-12-18T00:18:26.7435032Z mkdir "${GITHUB_WORKSPACE}"
2024-12-18T00:18:26.7443816Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:18:26.7444208Z env:
2024-12-18T00:18:26.7444448Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:18:26.7444723Z   NO_SUDO: true
2024-12-18T00:18:26.7444971Z ##[endgroup]
2024-12-18T00:18:26.7472452Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-12-18T00:18:26.7568271Z ##[group]Run malfet/checkout@silent-checkout
2024-12-18T00:18:26.7568625Z with:
2024-12-18T00:18:26.7568898Z   ref: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:18:26.7569257Z   fetch-depth: 0
2024-12-18T00:18:26.7569543Z   submodules: recursive
2024-12-18T00:18:26.7569820Z   quiet-checkout: true
2024-12-18T00:18:26.7570120Z   repository: pytorch/pytorch
2024-12-18T00:18:26.7570524Z   token: ***
2024-12-18T00:18:26.7570772Z   ssh-strict: true
2024-12-18T00:18:26.7571247Z   persist-credentials: true
2024-12-18T00:18:26.7571533Z   clean: true
2024-12-18T00:18:26.7571791Z   sparse-checkout-cone-mode: true
2024-12-18T00:18:26.7572101Z   lfs: false
2024-12-18T00:18:26.7572353Z   set-safe-directory: true
2024-12-18T00:18:26.7572638Z env:
2024-12-18T00:18:26.7572887Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:18:26.7573178Z ##[endgroup]
2024-12-18T00:18:26.8461127Z Syncing repository: pytorch/pytorch
2024-12-18T00:18:26.8462364Z ##[group]Getting Git version info
2024-12-18T00:18:26.8462853Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2024-12-18T00:18:26.8463527Z [command]/usr/bin/git version
2024-12-18T00:18:26.8463820Z git version 2.40.1
2024-12-18T00:18:26.8478209Z ##[endgroup]
2024-12-18T00:18:26.8491640Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/9e848544-31d1-457e-b11b-08fc4e5d2332' before making global git config changes
2024-12-18T00:18:26.8492600Z Adding repository directory to the temporary git global config as a safe directory
2024-12-18T00:18:26.8495440Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-12-18T00:18:26.8530331Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2024-12-18T00:18:26.8533170Z ##[group]Initializing the repository
2024-12-18T00:18:26.8535575Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-12-18T00:18:26.8567998Z hint: Using 'master' as the name for the initial branch. This default branch name
2024-12-18T00:18:26.8568690Z hint: is subject to change. To configure the initial branch name to use in all
2024-12-18T00:18:26.8569312Z hint: of your new repositories, which will suppress this warning, call:
2024-12-18T00:18:26.8569747Z hint: 
2024-12-18T00:18:26.8570077Z hint: 	git config --global init.defaultBranch <name>
2024-12-18T00:18:26.8570449Z hint: 
2024-12-18T00:18:26.8570804Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2024-12-18T00:18:26.8571391Z hint: 'development'. The just-created branch can be renamed via this command:
2024-12-18T00:18:26.8571841Z hint: 
2024-12-18T00:18:26.8572089Z hint: 	git branch -m <name>
2024-12-18T00:18:26.8573052Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2024-12-18T00:18:26.8581892Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2024-12-18T00:18:26.8610563Z ##[endgroup]
2024-12-18T00:18:26.8611073Z ##[group]Disabling automatic garbage collection
2024-12-18T00:18:26.8613478Z [command]/usr/bin/git config --local gc.auto 0
2024-12-18T00:18:26.8642799Z ##[endgroup]
2024-12-18T00:18:26.8643240Z ##[group]Setting up auth
2024-12-18T00:18:26.8647052Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2024-12-18T00:18:26.8674130Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2024-12-18T00:18:26.9037202Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2024-12-18T00:18:26.9063598Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2024-12-18T00:18:26.9400834Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2024-12-18T00:18:26.9446396Z ##[endgroup]
2024-12-18T00:18:26.9446906Z ##[group]Fetching the repository
2024-12-18T00:18:26.9452082Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2024-12-18T00:18:29.5068729Z remote: Enumerating objects: 1056563        
2024-12-18T00:18:29.5069261Z remote: Enumerating objects: 1057277, done.        
2024-12-18T00:18:29.5069667Z remote: Counting objects:   0% (1/714)        
2024-12-18T00:18:29.5070526Z remote: Counting objects:   1% (8/714)        
2024-12-18T00:18:29.5070912Z remote: Counting objects:   2% (15/714)        
2024-12-18T00:18:29.5071276Z remote: Counting objects:   3% (22/714)        
2024-12-18T00:18:29.5071657Z remote: Counting objects:   4% (29/714)        
2024-12-18T00:18:29.5072037Z remote: Counting objects:   5% (36/714)        
2024-12-18T00:18:29.5072555Z remote: Counting objects:   6% (43/714)        
2024-12-18T00:18:29.5073012Z remote: Counting objects:   7% (50/714)        
2024-12-18T00:18:29.5073383Z remote: Counting objects:   8% (58/714)        
2024-12-18T00:18:29.5073761Z remote: Counting objects:   9% (65/714)        
2024-12-18T00:18:29.5074134Z remote: Counting objects:  10% (72/714)        
2024-12-18T00:18:29.5074502Z remote: Counting objects:  11% (79/714)        
2024-12-18T00:18:29.5074871Z remote: Counting objects:  12% (86/714)        
2024-12-18T00:18:29.5075236Z remote: Counting objects:  13% (93/714)        
2024-12-18T00:18:29.5075609Z remote: Counting objects:  14% (100/714)        
2024-12-18T00:18:29.5075984Z remote: Counting objects:  15% (108/714)        
2024-12-18T00:18:29.5076356Z remote: Counting objects:  16% (115/714)        
2024-12-18T00:18:29.5076734Z remote: Counting objects:  17% (122/714)        
2024-12-18T00:18:29.5077121Z remote: Counting objects:  18% (129/714)        
2024-12-18T00:18:29.5077491Z remote: Counting objects:  19% (136/714)        
2024-12-18T00:18:29.5078071Z remote: Counting objects:  20% (143/714)        
2024-12-18T00:18:29.5078441Z remote: Counting objects:  21% (150/714)        
2024-12-18T00:18:29.5078818Z remote: Counting objects:  22% (158/714)        
2024-12-18T00:18:29.5079188Z remote: Counting objects:  23% (165/714)        
2024-12-18T00:18:29.5079588Z remote: Counting objects:  24% (172/714)        
2024-12-18T00:18:29.5080011Z remote: Counting objects:  25% (179/714)        
2024-12-18T00:18:29.5080376Z remote: Counting objects:  26% (186/714)        
2024-12-18T00:18:29.5080811Z remote: Counting objects:  27% (193/714)        
2024-12-18T00:18:29.5081184Z remote: Counting objects:  28% (200/714)        
2024-12-18T00:18:29.5081560Z remote: Counting objects:  29% (208/714)        
2024-12-18T00:18:29.5081980Z remote: Counting objects:  30% (215/714)        
2024-12-18T00:18:29.5082351Z remote: Counting objects:  31% (222/714)        
2024-12-18T00:18:29.5082726Z remote: Counting objects:  32% (229/714)        
2024-12-18T00:18:29.5083098Z remote: Counting objects:  33% (236/714)        
2024-12-18T00:18:29.5083470Z remote: Counting objects:  34% (243/714)        
2024-12-18T00:18:29.5083841Z remote: Counting objects:  35% (250/714)        
2024-12-18T00:18:29.5084215Z remote: Counting objects:  36% (258/714)        
2024-12-18T00:18:29.5084583Z remote: Counting objects:  37% (265/714)        
2024-12-18T00:18:29.5084952Z remote: Counting objects:  38% (272/714)        
2024-12-18T00:18:29.5085315Z remote: Counting objects:  39% (279/714)        
2024-12-18T00:18:29.5085692Z remote: Counting objects:  40% (286/714)        
2024-12-18T00:18:29.5086065Z remote: Counting objects:  41% (293/714)        
2024-12-18T00:18:29.5086440Z remote: Counting objects:  42% (300/714)        
2024-12-18T00:18:29.5086812Z remote: Counting objects:  43% (308/714)        
2024-12-18T00:18:29.5087174Z remote: Counting objects:  44% (315/714)        
2024-12-18T00:18:29.5087542Z remote: Counting objects:  45% (322/714)        
2024-12-18T00:18:29.5087915Z remote: Counting objects:  46% (329/714)        
2024-12-18T00:18:29.5088285Z remote: Counting objects:  47% (336/714)        
2024-12-18T00:18:29.5088655Z remote: Counting objects:  48% (343/714)        
2024-12-18T00:18:29.5089018Z remote: Counting objects:  49% (350/714)        
2024-12-18T00:18:29.5089423Z remote: Counting objects:  50% (357/714)        
2024-12-18T00:18:29.5089814Z remote: Counting objects:  51% (365/714)        
2024-12-18T00:18:29.5090184Z remote: Counting objects:  52% (372/714)        
2024-12-18T00:18:29.5090658Z remote: Counting objects:  53% (379/714)        
2024-12-18T00:18:29.5091021Z remote: Counting objects:  54% (386/714)        
2024-12-18T00:18:29.5091391Z remote: Counting objects:  55% (393/714)        
2024-12-18T00:18:29.5091762Z remote: Counting objects:  56% (400/714)        
2024-12-18T00:18:29.5092130Z remote: Counting objects:  57% (407/714)        
2024-12-18T00:18:29.5092502Z remote: Counting objects:  58% (415/714)        
2024-12-18T00:18:29.5092874Z remote: Counting objects:  59% (422/714)        
2024-12-18T00:18:29.5093242Z remote: Counting objects:  60% (429/714)        
2024-12-18T00:18:29.5093620Z remote: Counting objects:  61% (436/714)        
2024-12-18T00:18:29.5093990Z remote: Counting objects:  62% (443/714)        
2024-12-18T00:18:29.5094392Z remote: Counting objects:  63% (450/714)        
2024-12-18T00:18:29.5094765Z remote: Counting objects:  64% (457/714)        
2024-12-18T00:18:29.5095139Z remote: Counting objects:  65% (465/714)        
2024-12-18T00:18:29.5095500Z remote: Counting objects:  66% (472/714)        
2024-12-18T00:18:29.5095881Z remote: Counting objects:  67% (479/714)        
2024-12-18T00:18:29.5096257Z remote: Counting objects:  68% (486/714)        
2024-12-18T00:18:29.5096626Z remote: Counting objects:  69% (493/714)        
2024-12-18T00:18:29.5096999Z remote: Counting objects:  70% (500/714)        
2024-12-18T00:18:29.5097371Z remote: Counting objects:  71% (507/714)        
2024-12-18T00:18:29.5097734Z remote: Counting objects:  72% (515/714)        
2024-12-18T00:18:29.5098333Z remote: Counting objects:  73% (522/714)        
2024-12-18T00:18:29.5098753Z remote: Counting objects:  74% (529/714)        
2024-12-18T00:18:29.5099124Z remote: Counting objects:  75% (536/714)        
2024-12-18T00:18:29.5099538Z remote: Counting objects:  76% (543/714)        
2024-12-18T00:18:29.5099904Z remote: Counting objects:  77% (550/714)        
2024-12-18T00:18:29.5100276Z remote: Counting objects:  78% (557/714)        
2024-12-18T00:18:29.5100642Z remote: Counting objects:  79% (565/714)        
2024-12-18T00:18:29.5101020Z remote: Counting objects:  80% (572/714)        
2024-12-18T00:18:29.5101387Z remote: Counting objects:  81% (579/714)        
2024-12-18T00:18:29.5101753Z remote: Counting objects:  82% (586/714)        
2024-12-18T00:18:29.5102122Z remote: Counting objects:  83% (593/714)        
2024-12-18T00:18:29.5102489Z remote: Counting objects:  84% (600/714)        
2024-12-18T00:18:29.5102853Z remote: Counting objects:  85% (607/714)        
2024-12-18T00:18:29.5103225Z remote: Counting objects:  86% (615/714)        
2024-12-18T00:18:29.5103593Z remote: Counting objects:  87% (622/714)        
2024-12-18T00:18:29.5103954Z remote: Counting objects:  88% (629/714)        
2024-12-18T00:18:29.5104320Z remote: Counting objects:  89% (636/714)        
2024-12-18T00:18:29.5104692Z remote: Counting objects:  90% (643/714)        
2024-12-18T00:18:29.5105063Z remote: Counting objects:  91% (650/714)        
2024-12-18T00:18:29.5105436Z remote: Counting objects:  92% (657/714)        
2024-12-18T00:18:29.5105803Z remote: Counting objects:  93% (665/714)        
2024-12-18T00:18:29.5106171Z remote: Counting objects:  94% (672/714)        
2024-12-18T00:18:29.5106542Z remote: Counting objects:  95% (679/714)        
2024-12-18T00:18:29.5106908Z remote: Counting objects:  96% (686/714)        
2024-12-18T00:18:29.5107275Z remote: Counting objects:  97% (693/714)        
2024-12-18T00:18:29.5107638Z remote: Counting objects:  98% (700/714)        
2024-12-18T00:18:29.5108011Z remote: Counting objects:  99% (707/714)        
2024-12-18T00:18:29.5108382Z remote: Counting objects: 100% (714/714)        
2024-12-18T00:18:29.5108777Z remote: Counting objects: 100% (714/714), done.        
2024-12-18T00:18:29.5220317Z remote: Compressing objects:   0% (1/311)        
2024-12-18T00:18:29.5306344Z remote: Compressing objects:   1% (4/311)        
2024-12-18T00:18:29.5347879Z remote: Compressing objects:   2% (7/311)        
2024-12-18T00:18:29.5425085Z remote: Compressing objects:   3% (10/311)        
2024-12-18T00:18:29.5596418Z remote: Compressing objects:   4% (13/311)        
2024-12-18T00:18:29.6016236Z remote: Compressing objects:   5% (16/311)        
2024-12-18T00:18:29.6444033Z remote: Compressing objects:   6% (19/311)        
2024-12-18T00:18:29.7180601Z remote: Compressing objects:   7% (22/311)        
2024-12-18T00:18:29.7603907Z remote: Compressing objects:   8% (25/311)        
2024-12-18T00:18:29.8020511Z remote: Compressing objects:   9% (28/311)        
2024-12-18T00:18:29.8153310Z remote: Compressing objects:  10% (32/311)        
2024-12-18T00:18:29.8222935Z remote: Compressing objects:  11% (35/311)        
2024-12-18T00:18:29.8224007Z remote: Compressing objects:  12% (38/311)        
2024-12-18T00:18:29.8229820Z remote: Compressing objects:  13% (41/311)        
2024-12-18T00:18:29.8231604Z remote: Compressing objects:  14% (44/311)        
2024-12-18T00:18:29.8232768Z remote: Compressing objects:  15% (47/311)        
2024-12-18T00:18:29.8239410Z remote: Compressing objects:  16% (50/311)        
2024-12-18T00:18:29.8241757Z remote: Compressing objects:  17% (53/311)        
2024-12-18T00:18:29.8244230Z remote: Compressing objects:  18% (56/311)        
2024-12-18T00:18:29.8246664Z remote: Compressing objects:  19% (60/311)        
2024-12-18T00:18:29.8249885Z remote: Compressing objects:  20% (63/311)        
2024-12-18T00:18:29.8254676Z remote: Compressing objects:  21% (66/311)        
2024-12-18T00:18:29.8258183Z remote: Compressing objects:  22% (69/311)        
2024-12-18T00:18:29.8265187Z remote: Compressing objects:  23% (72/311)        
2024-12-18T00:18:29.8267190Z remote: Compressing objects:  24% (75/311)        
2024-12-18T00:18:29.8274990Z remote: Compressing objects:  25% (78/311)        
2024-12-18T00:18:29.8277392Z remote: Compressing objects:  26% (81/311)        
2024-12-18T00:18:29.8280054Z remote: Compressing objects:  27% (84/311)        
2024-12-18T00:18:29.8283842Z remote: Compressing objects:  28% (88/311)        
2024-12-18T00:18:29.8291596Z remote: Compressing objects:  29% (91/311)        
2024-12-18T00:18:29.8294193Z remote: Compressing objects:  30% (94/311)        
2024-12-18T00:18:29.8296342Z remote: Compressing objects:  31% (97/311)        
2024-12-18T00:18:29.8302769Z remote: Compressing objects:  32% (100/311)        
2024-12-18T00:18:29.8304939Z remote: Compressing objects:  33% (103/311)        
2024-12-18T00:18:29.8310082Z remote: Compressing objects:  34% (106/311)        
2024-12-18T00:18:29.8316058Z remote: Compressing objects:  35% (109/311)        
2024-12-18T00:18:29.8320031Z remote: Compressing objects:  36% (112/311)        
2024-12-18T00:18:29.8322701Z remote: Compressing objects:  37% (116/311)        
2024-12-18T00:18:29.8324941Z remote: Compressing objects:  38% (119/311)        
2024-12-18T00:18:29.8328633Z remote: Compressing objects:  39% (122/311)        
2024-12-18T00:18:29.8330355Z remote: Compressing objects:  40% (125/311)        
2024-12-18T00:18:29.8336092Z remote: Compressing objects:  41% (128/311)        
2024-12-18T00:18:29.8339615Z remote: Compressing objects:  42% (131/311)        
2024-12-18T00:18:29.8341112Z remote: Compressing objects:  43% (134/311)        
2024-12-18T00:18:29.8342125Z remote: Compressing objects:  44% (137/311)        
2024-12-18T00:18:29.8348433Z remote: Compressing objects:  45% (140/311)        
2024-12-18T00:18:29.8352662Z remote: Compressing objects:  46% (144/311)        
2024-12-18T00:18:29.8353527Z remote: Compressing objects:  47% (147/311)        
2024-12-18T00:18:29.8357171Z remote: Compressing objects:  48% (150/311)        
2024-12-18T00:18:29.8359403Z remote: Compressing objects:  49% (153/311)        
2024-12-18T00:18:29.8361390Z remote: Compressing objects:  50% (156/311)        
2024-12-18T00:18:29.8368092Z remote: Compressing objects:  51% (159/311)        
2024-12-18T00:18:29.8371159Z remote: Compressing objects:  52% (162/311)        
2024-12-18T00:18:29.8373309Z remote: Compressing objects:  53% (165/311)        
2024-12-18T00:18:29.8375734Z remote: Compressing objects:  54% (168/311)        
2024-12-18T00:18:29.8377841Z remote: Compressing objects:  55% (172/311)        
2024-12-18T00:18:29.8379612Z remote: Compressing objects:  56% (175/311)        
2024-12-18T00:18:29.8381801Z remote: Compressing objects:  57% (178/311)        
2024-12-18T00:18:29.8383656Z remote: Compressing objects:  58% (181/311)        
2024-12-18T00:18:29.8387094Z remote: Compressing objects:  59% (184/311)        
2024-12-18T00:18:29.8387522Z remote: Compressing objects:  60% (187/311)        
2024-12-18T00:18:29.8389002Z remote: Compressing objects:  61% (190/311)        
2024-12-18T00:18:29.8390494Z remote: Compressing objects:  62% (193/311)        
2024-12-18T00:18:29.8393668Z remote: Compressing objects:  63% (196/311)        
2024-12-18T00:18:29.8395256Z remote: Compressing objects:  64% (200/311)        
2024-12-18T00:18:29.8400559Z remote: Compressing objects:  65% (203/311)        
2024-12-18T00:18:29.8402500Z remote: Compressing objects:  66% (206/311)        
2024-12-18T00:18:29.8408514Z remote: Compressing objects:  67% (209/311)        
2024-12-18T00:18:29.8409530Z remote: Compressing objects:  68% (212/311)        
2024-12-18T00:18:29.8413322Z remote: Compressing objects:  69% (215/311)        
2024-12-18T00:18:29.8414052Z remote: Compressing objects:  70% (218/311)        
2024-12-18T00:18:29.8417350Z remote: Compressing objects:  71% (221/311)        
2024-12-18T00:18:29.8421886Z remote: Compressing objects:  72% (224/311)        
2024-12-18T00:18:29.8424108Z remote: Compressing objects:  73% (228/311)        
2024-12-18T00:18:29.8427984Z remote: Compressing objects:  74% (231/311)        
2024-12-18T00:18:29.8428547Z remote: Compressing objects:  75% (234/311)        
2024-12-18T00:18:29.8430385Z remote: Compressing objects:  76% (237/311)        
2024-12-18T00:18:29.8430796Z remote: Compressing objects:  77% (240/311)        
2024-12-18T00:18:29.8431238Z remote: Compressing objects:  78% (243/311)        
2024-12-18T00:18:29.8432001Z remote: Compressing objects:  79% (246/311)        
2024-12-18T00:18:29.8432649Z remote: Compressing objects:  80% (249/311)        
2024-12-18T00:18:29.8434834Z remote: Compressing objects:  81% (252/311)        
2024-12-18T00:18:29.8436964Z remote: Compressing objects:  82% (256/311)        
2024-12-18T00:18:29.8439630Z remote: Compressing objects:  83% (259/311)        
2024-12-18T00:18:29.8441921Z remote: Compressing objects:  84% (262/311)        
2024-12-18T00:18:29.8444413Z remote: Compressing objects:  85% (265/311)        
2024-12-18T00:18:29.8445731Z remote: Compressing objects:  86% (268/311)        
2024-12-18T00:18:29.8447507Z remote: Compressing objects:  87% (271/311)        
2024-12-18T00:18:29.8448460Z remote: Compressing objects:  88% (274/311)        
2024-12-18T00:18:29.8450145Z remote: Compressing objects:  89% (277/311)        
2024-12-18T00:18:29.8451648Z remote: Compressing objects:  90% (280/311)        
2024-12-18T00:18:29.8453784Z remote: Compressing objects:  91% (284/311)        
2024-12-18T00:18:29.8454355Z remote: Compressing objects:  92% (287/311)        
2024-12-18T00:18:29.8455304Z remote: Compressing objects:  93% (290/311)        
2024-12-18T00:18:29.8457722Z remote: Compressing objects:  94% (293/311)        
2024-12-18T00:18:29.8458337Z remote: Compressing objects:  95% (296/311)        
2024-12-18T00:18:29.8459499Z remote: Compressing objects:  96% (299/311)        
2024-12-18T00:18:29.8460897Z remote: Compressing objects:  97% (302/311)        
2024-12-18T00:18:29.8461302Z remote: Compressing objects:  98% (305/311)        
2024-12-18T00:18:29.8462599Z remote: Compressing objects:  99% (308/311)        
2024-12-18T00:18:29.8463045Z remote: Compressing objects: 100% (311/311)        
2024-12-18T00:18:29.8463462Z remote: Compressing objects: 100% (311/311), done.        
2024-12-18T00:18:54.3357035Z remote: Total 1057277 (delta 573), reused 407 (delta 403), pack-reused 1056563 (from 3)        
2024-12-18T00:19:16.7941454Z [command]/usr/bin/git rev-parse --verify --quiet 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7^{object}
2024-12-18T00:19:16.7968725Z 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:19:16.7973811Z ##[endgroup]
2024-12-18T00:19:16.7974277Z ##[group]Determining the checkout info
2024-12-18T00:19:16.7974801Z ##[endgroup]
2024-12-18T00:19:16.7975227Z ##[group]Checking out the ref
2024-12-18T00:19:16.7978737Z [command]/usr/bin/git checkout --quiet --force 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:19:18.5642614Z ##[endgroup]
2024-12-18T00:19:18.5643525Z ##[group]Setting up auth for fetching submodules
2024-12-18T00:19:18.5646059Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2024-12-18T00:19:18.5693029Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2024-12-18T00:19:18.5720891Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2024-12-18T00:19:18.5750064Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2024-12-18T00:19:18.5782937Z ##[endgroup]
2024-12-18T00:19:18.5783383Z ##[group]Fetching submodules
2024-12-18T00:19:18.5785383Z [command]/usr/bin/git submodule sync --recursive
2024-12-18T00:19:18.6155695Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2024-12-18T00:19:18.6506712Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2024-12-18T00:19:18.6509394Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2024-12-18T00:19:18.6512279Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2024-12-18T00:19:18.6516179Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2024-12-18T00:19:18.6519646Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX'
2024-12-18T00:19:18.6523467Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2024-12-18T00:19:18.6526756Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2024-12-18T00:19:18.6530694Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2024-12-18T00:19:18.6534447Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel'
2024-12-18T00:19:18.6538238Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib'
2024-12-18T00:19:18.6542169Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2024-12-18T00:19:18.6546046Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2024-12-18T00:19:18.6549963Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2024-12-18T00:19:18.6554172Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2024-12-18T00:19:18.6558222Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2024-12-18T00:19:18.6563723Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2024-12-18T00:19:18.6571580Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2024-12-18T00:19:18.6576513Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2024-12-18T00:19:18.6580915Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2024-12-18T00:19:18.6585685Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2024-12-18T00:19:18.6590133Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2024-12-18T00:19:18.6594742Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2024-12-18T00:19:18.6599373Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2024-12-18T00:19:18.6604181Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc'
2024-12-18T00:19:18.6608921Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2024-12-18T00:19:18.6613751Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2024-12-18T00:19:18.6619505Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2024-12-18T00:19:18.6624750Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp'
2024-12-18T00:19:18.6629668Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2024-12-18T00:19:18.6635188Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2024-12-18T00:19:18.6640202Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2024-12-18T00:19:18.6645606Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2024-12-18T00:19:18.6650864Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2024-12-18T00:19:18.6659993Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2024-12-18T00:19:18.6665594Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2024-12-18T00:19:18.6671281Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2024-12-18T00:19:18.6706438Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2024-12-18T00:19:18.9389046Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2024-12-18T00:19:19.1279133Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2024-12-18T00:19:19.3575762Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2024-12-18T00:19:19.6086921Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'...
2024-12-18T00:19:19.9352242Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2024-12-18T00:19:21.9457839Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2024-12-18T00:19:33.5723801Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2024-12-18T00:19:34.0395269Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'...
2024-12-18T00:19:36.2532336Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'...
2024-12-18T00:19:36.7773317Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2024-12-18T00:19:37.4264978Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2024-12-18T00:19:38.6435001Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2024-12-18T00:19:40.8311287Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2024-12-18T00:19:47.1084557Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2024-12-18T00:19:48.8269558Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2024-12-18T00:19:50.1528487Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2024-12-18T00:19:51.4947290Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2024-12-18T00:19:51.9812150Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2024-12-18T00:19:52.3548823Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2024-12-18T00:19:53.6260177Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2024-12-18T00:19:53.9933966Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2024-12-18T00:19:54.2556533Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2024-12-18T00:19:55.9198838Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'...
2024-12-18T00:19:56.8019790Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2024-12-18T00:19:57.2013930Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2024-12-18T00:20:04.5446542Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2024-12-18T00:20:06.8333155Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'...
2024-12-18T00:20:11.8588172Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2024-12-18T00:20:12.1174369Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2024-12-18T00:20:21.5079056Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2024-12-18T00:20:21.6952148Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2024-12-18T00:20:21.9174752Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2024-12-18T00:20:24.2508629Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2024-12-18T00:20:24.5701611Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2024-12-18T00:20:25.2052138Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2024-12-18T00:20:25.6612661Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2024-12-18T00:20:25.6759210Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2024-12-18T00:20:25.6895729Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2024-12-18T00:20:25.7192822Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2024-12-18T00:20:25.7639771Z Submodule path 'third_party/NVTX': checked out 'e170594ac7cf1dac584da473d4ca9301087090c1'
2024-12-18T00:20:25.8096155Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2024-12-18T00:20:26.7245354Z Submodule path 'third_party/XNNPACK': checked out '4ea82e595b36106653175dcb04b2aa532660d0d8'
2024-12-18T00:20:26.7533435Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2024-12-18T00:20:27.0522429Z Submodule path 'third_party/composable_kernel': checked out '50ee4267e27b875d149e642f4cebd47be1dc3b57'
2024-12-18T00:20:27.1083292Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006'
2024-12-18T00:20:27.2215771Z Submodule path 'third_party/cpuinfo': checked out '1e83a2fdd3102f65c6f1fb602c1b320486218a99'
2024-12-18T00:20:27.2623434Z Submodule path 'third_party/cudnn_frontend': checked out '936021bfed8c91dc416af1588b2c4eca631a9e45'
2024-12-18T00:20:27.8807178Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49'
2024-12-18T00:20:28.1674214Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2024-12-18T00:20:28.2744060Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e'
2024-12-18T00:20:28.2765224Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2024-12-18T00:20:28.2768442Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T00:20:28.2771854Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass'
2024-12-18T00:20:28.2774336Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2024-12-18T00:20:28.2778513Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T00:20:28.2810622Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2024-12-18T00:20:29.1006646Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2024-12-18T00:20:29.7293768Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'...
2024-12-18T00:20:31.8759572Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2024-12-18T00:20:32.9428372Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2024-12-18T00:20:33.3157066Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2024-12-18T00:20:33.4286093Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2024-12-18T00:20:33.9353000Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6'
2024-12-18T00:20:34.0063723Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2024-12-18T00:20:34.0216519Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1'
2024-12-18T00:20:34.1872367Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b'
2024-12-18T00:20:34.2321004Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592'
2024-12-18T00:20:34.2781008Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2024-12-18T00:20:34.3111184Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509'
2024-12-18T00:20:34.3620642Z Submodule path 'third_party/googletest': checked out 'b514bdc898e2951020cbdca1304b75f5950d1f59'
2024-12-18T00:20:34.3792214Z Submodule path 'third_party/ideep': checked out 'c7ccd5bdbe5434ba156f4e856dcef0601637334b'
2024-12-18T00:20:34.3811017Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2024-12-18T00:20:34.3841281Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2024-12-18T00:20:48.4737052Z Submodule path 'third_party/ideep/mkl-dnn': checked out '66f0cb9eb66affd2da3bf5f8d897376f04aae6af'
2024-12-18T00:20:48.4962945Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2024-12-18T00:20:48.5956149Z Submodule path 'third_party/kineto': checked out '338140f58a28d599da3434ced4fd2d75dd1a213d'
2024-12-18T00:20:48.5975903Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T00:20:48.5979167Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T00:20:48.5982670Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T00:20:48.6014921Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'...
2024-12-18T00:20:49.4463248Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2024-12-18T00:20:50.6442981Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2024-12-18T00:20:51.7572495Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e'
2024-12-18T00:20:51.7591426Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T00:20:51.7595113Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T00:20:51.7598447Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T00:20:51.7602114Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T00:20:51.7605807Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T00:20:51.7609696Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T00:20:51.7613578Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T00:20:51.7617708Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T00:20:51.7651536Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'...
2024-12-18T00:20:52.6987304Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'...
2024-12-18T00:20:53.1060723Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'...
2024-12-18T00:20:54.3349568Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'...
2024-12-18T00:20:54.6383551Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'...
2024-12-18T00:20:55.1281287Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'...
2024-12-18T00:20:56.1732689Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'...
2024-12-18T00:21:03.2620006Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'...
2024-12-18T00:21:03.6981924Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9'
2024-12-18T00:21:03.7215137Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400'
2024-12-18T00:21:03.7649716Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2024-12-18T00:21:03.7823766Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067'
2024-12-18T00:21:03.7843376Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T00:21:03.7880074Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'...
2024-12-18T00:21:04.1197521Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4'
2024-12-18T00:21:04.1426493Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446'
2024-12-18T00:21:04.1910676Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850'
2024-12-18T00:21:04.3183414Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5'
2024-12-18T00:21:04.3382773Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150'
2024-12-18T00:21:04.3841250Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164'
2024-12-18T00:21:04.4508689Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2024-12-18T00:21:04.4972162Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78'
2024-12-18T00:21:04.5324033Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290'
2024-12-18T00:21:04.6653495Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2024-12-18T00:21:05.1816059Z Submodule path 'third_party/onnx': checked out 'b8baa8446686496da4cc8fda09f2b6fe65c2a02c'
2024-12-18T00:21:05.1852478Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2024-12-18T00:21:05.1885179Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2024-12-18T00:21:06.3354758Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b'
2024-12-18T00:21:06.4259293Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878'
2024-12-18T00:21:06.4280561Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T00:21:06.4283907Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T00:21:06.4287237Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T00:21:06.4290920Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T00:21:06.4294595Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T00:21:06.4298297Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T00:21:06.4302087Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T00:21:06.4305810Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T00:21:06.4338656Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'...
2024-12-18T00:21:06.9052305Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'...
2024-12-18T00:21:07.9511716Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'...
2024-12-18T00:21:08.3343653Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'...
2024-12-18T00:21:15.4348712Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'...
2024-12-18T00:21:15.7332660Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'...
2024-12-18T00:21:15.9450800Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'...
2024-12-18T00:21:16.2871964Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'...
2024-12-18T00:21:23.4743641Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2'
2024-12-18T00:21:23.5220095Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1'
2024-12-18T00:21:23.5408950Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa'
2024-12-18T00:21:23.6679353Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d'
2024-12-18T00:21:23.6846188Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce'
2024-12-18T00:21:23.7029190Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5'
2024-12-18T00:21:23.7227800Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d'
2024-12-18T00:21:23.7247729Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T00:21:23.7250891Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T00:21:23.7282730Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'...
2024-12-18T00:21:25.5647094Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'...
2024-12-18T00:21:26.8870281Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4'
2024-12-18T00:21:26.9408444Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2024-12-18T00:21:27.5993597Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50'
2024-12-18T00:21:27.6138577Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe'
2024-12-18T00:21:27.9432293Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2024-12-18T00:21:27.9456530Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2024-12-18T00:21:27.9459915Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2024-12-18T00:21:27.9492851Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2024-12-18T00:21:28.3972852Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2024-12-18T00:21:29.4696590Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2024-12-18T00:21:29.5516933Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2024-12-18T00:21:29.5639297Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2024-12-18T00:21:29.5794080Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8'
2024-12-18T00:21:29.6293168Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4'
2024-12-18T00:21:29.6640652Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2024-12-18T00:21:29.7137854Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b'
2024-12-18T00:21:29.7476594Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2024-12-18T00:21:29.7498201Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2024-12-18T00:21:29.7501531Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2024-12-18T00:21:29.7504905Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2024-12-18T00:21:29.7508502Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T00:21:29.7539466Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2024-12-18T00:21:30.7921693Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2024-12-18T00:21:31.0408293Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2024-12-18T00:21:33.4291007Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2024-12-18T00:21:34.6167675Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2024-12-18T00:21:34.6356416Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2024-12-18T00:21:34.7078370Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2024-12-18T00:21:34.7429372Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2024-12-18T00:21:34.7449256Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T00:21:34.7481435Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2024-12-18T00:21:34.9775471Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2024-12-18T00:21:34.9822894Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2024-12-18T00:21:35.0179250Z Entering 'android/libs/fbjni'
2024-12-18T00:21:35.0232256Z Entering 'third_party/FP16'
2024-12-18T00:21:35.0285426Z Entering 'third_party/FXdiv'
2024-12-18T00:21:35.0339020Z Entering 'third_party/NNPACK'
2024-12-18T00:21:35.0395584Z Entering 'third_party/NVTX'
2024-12-18T00:21:35.0447808Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T00:21:35.0499926Z Entering 'third_party/XNNPACK'
2024-12-18T00:21:35.0569670Z Entering 'third_party/benchmark'
2024-12-18T00:21:35.0622987Z Entering 'third_party/composable_kernel'
2024-12-18T00:21:35.0681794Z Entering 'third_party/cpp-httplib'
2024-12-18T00:21:35.0740256Z Entering 'third_party/cpuinfo'
2024-12-18T00:21:35.0791948Z Entering 'third_party/cudnn_frontend'
2024-12-18T00:21:35.0845200Z Entering 'third_party/cutlass'
2024-12-18T00:21:35.0905676Z Entering 'third_party/eigen'
2024-12-18T00:21:35.0960802Z Entering 'third_party/fbgemm'
2024-12-18T00:21:35.1012320Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T00:21:35.1064323Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T00:21:35.1119718Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T00:21:35.1176926Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T00:21:35.1227768Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T00:21:35.1280421Z Entering 'third_party/flatbuffers'
2024-12-18T00:21:35.1334921Z Entering 'third_party/fmt'
2024-12-18T00:21:35.1386192Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T00:21:35.1438412Z Entering 'third_party/gloo'
2024-12-18T00:21:35.1489772Z Entering 'third_party/googletest'
2024-12-18T00:21:35.1541591Z Entering 'third_party/ideep'
2024-12-18T00:21:35.1590790Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T00:21:35.1650316Z Entering 'third_party/ittapi'
2024-12-18T00:21:35.1701769Z Entering 'third_party/kineto'
2024-12-18T00:21:35.1752244Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T00:21:35.1806790Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T00:21:35.1859580Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T00:21:35.1909765Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T00:21:35.1960208Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T00:21:35.2008397Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T00:21:35.2063074Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T00:21:35.2113219Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T00:21:35.2163359Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T00:21:35.2221379Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T00:21:35.2276602Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T00:21:35.2325960Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T00:21:35.2382126Z Entering 'third_party/mimalloc'
2024-12-18T00:21:35.2435014Z Entering 'third_party/nccl/nccl'
2024-12-18T00:21:35.2487202Z Entering 'third_party/nlohmann'
2024-12-18T00:21:35.2541661Z Entering 'third_party/onnx'
2024-12-18T00:21:35.2607263Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T00:21:35.2665372Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T00:21:35.2718286Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T00:21:35.2767860Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T00:21:35.2819034Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T00:21:35.2868607Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T00:21:35.2920863Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T00:21:35.2971208Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T00:21:35.3026399Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T00:21:35.3074713Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T00:21:35.3127198Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T00:21:35.3180511Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T00:21:35.3253157Z Entering 'third_party/pocketfft'
2024-12-18T00:21:35.3305385Z Entering 'third_party/protobuf'
2024-12-18T00:21:35.3359753Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T00:21:35.3413867Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T00:21:35.3472744Z Entering 'third_party/psimd'
2024-12-18T00:21:35.3527791Z Entering 'third_party/pthreadpool'
2024-12-18T00:21:35.3579577Z Entering 'third_party/pybind11'
2024-12-18T00:21:35.3631545Z Entering 'third_party/python-peachpy'
2024-12-18T00:21:35.3683090Z Entering 'third_party/sleef'
2024-12-18T00:21:35.3734931Z Entering 'third_party/tensorpipe'
2024-12-18T00:21:35.3784902Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T00:21:35.3836375Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T00:21:35.3886657Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T00:21:35.3936622Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T00:21:35.3984677Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T00:21:35.4059436Z ##[endgroup]
2024-12-18T00:21:35.4060848Z ##[group]Persisting credentials for submodules
2024-12-18T00:21:35.4065011Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :"
2024-12-18T00:21:35.4413503Z Entering 'android/libs/fbjni'
2024-12-18T00:21:35.4482095Z Entering 'third_party/FP16'
2024-12-18T00:21:35.4554243Z Entering 'third_party/FXdiv'
2024-12-18T00:21:35.4621717Z Entering 'third_party/NNPACK'
2024-12-18T00:21:35.4687966Z Entering 'third_party/NVTX'
2024-12-18T00:21:35.4756190Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T00:21:35.4823633Z Entering 'third_party/XNNPACK'
2024-12-18T00:21:35.4904379Z Entering 'third_party/benchmark'
2024-12-18T00:21:35.4972293Z Entering 'third_party/composable_kernel'
2024-12-18T00:21:35.5051024Z Entering 'third_party/cpp-httplib'
2024-12-18T00:21:35.5121677Z Entering 'third_party/cpuinfo'
2024-12-18T00:21:35.5188510Z Entering 'third_party/cudnn_frontend'
2024-12-18T00:21:35.5256514Z Entering 'third_party/cutlass'
2024-12-18T00:21:35.5329775Z Entering 'third_party/eigen'
2024-12-18T00:21:35.5399076Z Entering 'third_party/fbgemm'
2024-12-18T00:21:35.5465112Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T00:21:35.5533800Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T00:21:35.5600669Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T00:21:35.5674483Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T00:21:35.5742131Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T00:21:35.5811021Z Entering 'third_party/flatbuffers'
2024-12-18T00:21:35.5881685Z Entering 'third_party/fmt'
2024-12-18T00:21:35.5953013Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T00:21:35.6019177Z Entering 'third_party/gloo'
2024-12-18T00:21:35.6086186Z Entering 'third_party/googletest'
2024-12-18T00:21:35.6153826Z Entering 'third_party/ideep'
2024-12-18T00:21:35.6218661Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T00:21:35.6294186Z Entering 'third_party/ittapi'
2024-12-18T00:21:35.6360274Z Entering 'third_party/kineto'
2024-12-18T00:21:35.6425248Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T00:21:35.6491111Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T00:21:35.6564566Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T00:21:35.6635106Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T00:21:35.6701046Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T00:21:35.6765678Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T00:21:35.6837353Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T00:21:35.6904522Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T00:21:35.6972262Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T00:21:35.7040249Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T00:21:35.7109623Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T00:21:35.7176774Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T00:21:35.7248987Z Entering 'third_party/mimalloc'
2024-12-18T00:21:35.7314314Z Entering 'third_party/nccl/nccl'
2024-12-18T00:21:35.7384520Z Entering 'third_party/nlohmann'
2024-12-18T00:21:35.7451998Z Entering 'third_party/onnx'
2024-12-18T00:21:35.7530968Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T00:21:35.7609011Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T00:21:35.7677024Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T00:21:35.7743451Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T00:21:35.7808635Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T00:21:35.7873742Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T00:21:35.7941070Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T00:21:35.8006918Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T00:21:35.8072470Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T00:21:35.8135495Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T00:21:35.8207805Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T00:21:35.8278338Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T00:21:35.8366540Z Entering 'third_party/pocketfft'
2024-12-18T00:21:35.8439322Z Entering 'third_party/protobuf'
2024-12-18T00:21:35.8509961Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T00:21:35.8577386Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T00:21:35.8647717Z Entering 'third_party/psimd'
2024-12-18T00:21:35.8714016Z Entering 'third_party/pthreadpool'
2024-12-18T00:21:35.8782948Z Entering 'third_party/pybind11'
2024-12-18T00:21:35.8848588Z Entering 'third_party/python-peachpy'
2024-12-18T00:21:35.8917178Z Entering 'third_party/sleef'
2024-12-18T00:21:35.8984207Z Entering 'third_party/tensorpipe'
2024-12-18T00:21:35.9051525Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T00:21:35.9118007Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T00:21:35.9183272Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T00:21:35.9249718Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T00:21:35.9313154Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T00:21:35.9405472Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url"
2024-12-18T00:21:35.9755599Z Entering 'android/libs/fbjni'
2024-12-18T00:21:35.9819258Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2024-12-18T00:21:35.9841356Z Entering 'third_party/FP16'
2024-12-18T00:21:35.9904654Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2024-12-18T00:21:35.9928076Z Entering 'third_party/FXdiv'
2024-12-18T00:21:35.9990878Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2024-12-18T00:21:36.0012553Z Entering 'third_party/NNPACK'
2024-12-18T00:21:36.0075256Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2024-12-18T00:21:36.0097392Z Entering 'third_party/NVTX'
2024-12-18T00:21:36.0160181Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config	remote.origin.url
2024-12-18T00:21:36.0183024Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T00:21:36.0246276Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2024-12-18T00:21:36.0268684Z Entering 'third_party/XNNPACK'
2024-12-18T00:21:36.0332643Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2024-12-18T00:21:36.0370349Z Entering 'third_party/benchmark'
2024-12-18T00:21:36.0432098Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2024-12-18T00:21:36.0454589Z Entering 'third_party/composable_kernel'
2024-12-18T00:21:36.0519118Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config	remote.origin.url
2024-12-18T00:21:36.0547232Z Entering 'third_party/cpp-httplib'
2024-12-18T00:21:36.0609501Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config	remote.origin.url
2024-12-18T00:21:36.0631762Z Entering 'third_party/cpuinfo'
2024-12-18T00:21:36.0693449Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2024-12-18T00:21:36.0718154Z Entering 'third_party/cudnn_frontend'
2024-12-18T00:21:36.0780750Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2024-12-18T00:21:36.0802033Z Entering 'third_party/cutlass'
2024-12-18T00:21:36.0864860Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2024-12-18T00:21:36.0893930Z Entering 'third_party/eigen'
2024-12-18T00:21:36.0955641Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2024-12-18T00:21:36.0979271Z Entering 'third_party/fbgemm'
2024-12-18T00:21:36.1043614Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2024-12-18T00:21:36.1064389Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T00:21:36.1129629Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2024-12-18T00:21:36.1150596Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T00:21:36.1210699Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2024-12-18T00:21:36.1232532Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T00:21:36.1292459Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config	remote.origin.url
2024-12-18T00:21:36.1320211Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T00:21:36.1381414Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:36.1403159Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T00:21:36.1468228Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2024-12-18T00:21:36.1491936Z Entering 'third_party/flatbuffers'
2024-12-18T00:21:36.1556157Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2024-12-18T00:21:36.1581419Z Entering 'third_party/fmt'
2024-12-18T00:21:36.1646697Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2024-12-18T00:21:36.1667828Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T00:21:36.1731024Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2024-12-18T00:21:36.1753329Z Entering 'third_party/gloo'
2024-12-18T00:21:36.1816800Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2024-12-18T00:21:36.1838127Z Entering 'third_party/googletest'
2024-12-18T00:21:36.1899554Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:36.1921930Z Entering 'third_party/ideep'
2024-12-18T00:21:36.1984932Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2024-12-18T00:21:36.2004598Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T00:21:36.2070490Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2024-12-18T00:21:36.2109109Z Entering 'third_party/ittapi'
2024-12-18T00:21:36.2169579Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2024-12-18T00:21:36.2191157Z Entering 'third_party/kineto'
2024-12-18T00:21:36.2255043Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2024-12-18T00:21:36.2275677Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T00:21:36.2338117Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config	remote.origin.url
2024-12-18T00:21:36.2357398Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T00:21:36.2419456Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config	remote.origin.url
2024-12-18T00:21:36.2441602Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T00:21:36.2503899Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config	remote.origin.url
2024-12-18T00:21:36.2524793Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T00:21:36.2586134Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config	remote.origin.url
2024-12-18T00:21:36.2607641Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T00:21:36.2669291Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config	remote.origin.url
2024-12-18T00:21:36.2687864Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T00:21:36.2751396Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config	remote.origin.url
2024-12-18T00:21:36.2774809Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T00:21:36.2835947Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config	remote.origin.url
2024-12-18T00:21:36.2856846Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T00:21:36.2918333Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:36.2939292Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T00:21:36.3001638Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config	remote.origin.url
2024-12-18T00:21:36.3025339Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T00:21:36.3089039Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config	remote.origin.url
2024-12-18T00:21:36.3112509Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T00:21:36.3179740Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2024-12-18T00:21:36.3200276Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T00:21:36.3260694Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:36.3285171Z Entering 'third_party/mimalloc'
2024-12-18T00:21:36.3347435Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config	remote.origin.url
2024-12-18T00:21:36.3369162Z Entering 'third_party/nccl/nccl'
2024-12-18T00:21:36.3432694Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2024-12-18T00:21:36.3455287Z Entering 'third_party/nlohmann'
2024-12-18T00:21:36.3518791Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2024-12-18T00:21:36.3542850Z Entering 'third_party/onnx'
2024-12-18T00:21:36.3606640Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2024-12-18T00:21:36.3641469Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T00:21:36.3707967Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2024-12-18T00:21:36.3733886Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T00:21:36.3797864Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config	remote.origin.url
2024-12-18T00:21:36.3819536Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T00:21:36.3880611Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config	remote.origin.url
2024-12-18T00:21:36.3901493Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T00:21:36.3962190Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:36.3983009Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T00:21:36.4043541Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config	remote.origin.url
2024-12-18T00:21:36.4064222Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T00:21:36.4128514Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config	remote.origin.url
2024-12-18T00:21:36.4150439Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T00:21:36.4211281Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config	remote.origin.url
2024-12-18T00:21:36.4232473Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T00:21:36.4291891Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config	remote.origin.url
2024-12-18T00:21:36.4312277Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T00:21:36.4374135Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config	remote.origin.url
2024-12-18T00:21:36.4392817Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T00:21:36.4458569Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2024-12-18T00:21:36.4481494Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T00:21:36.4543955Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2024-12-18T00:21:36.4567974Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T00:21:36.4628363Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config	remote.origin.url
2024-12-18T00:21:36.4670335Z Entering 'third_party/pocketfft'
2024-12-18T00:21:36.4733656Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2024-12-18T00:21:36.4755253Z Entering 'third_party/protobuf'
2024-12-18T00:21:36.4821620Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2024-12-18T00:21:36.4843967Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T00:21:36.4908291Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2024-12-18T00:21:36.4929197Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T00:21:36.4989595Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:36.5014009Z Entering 'third_party/psimd'
2024-12-18T00:21:36.5077544Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2024-12-18T00:21:36.5098996Z Entering 'third_party/pthreadpool'
2024-12-18T00:21:36.5160694Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2024-12-18T00:21:36.5182916Z Entering 'third_party/pybind11'
2024-12-18T00:21:36.5245371Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2024-12-18T00:21:36.5267673Z Entering 'third_party/python-peachpy'
2024-12-18T00:21:36.5329496Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2024-12-18T00:21:36.5351599Z Entering 'third_party/sleef'
2024-12-18T00:21:36.5413044Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2024-12-18T00:21:36.5436211Z Entering 'third_party/tensorpipe'
2024-12-18T00:21:36.5499098Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2024-12-18T00:21:36.5519575Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T00:21:36.5580506Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:36.5601264Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T00:21:36.5667177Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2024-12-18T00:21:36.5687490Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T00:21:36.5748331Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2024-12-18T00:21:36.5769288Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T00:21:36.5831065Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2024-12-18T00:21:36.5849661Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T00:21:36.5910670Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2024-12-18T00:21:36.6575189Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2024-12-18T00:21:36.6935025Z Entering 'android/libs/fbjni'
2024-12-18T00:21:36.6988762Z Entering 'third_party/FP16'
2024-12-18T00:21:36.7041889Z Entering 'third_party/FXdiv'
2024-12-18T00:21:36.7094246Z Entering 'third_party/NNPACK'
2024-12-18T00:21:36.7147945Z Entering 'third_party/NVTX'
2024-12-18T00:21:36.7200495Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T00:21:36.7254617Z Entering 'third_party/XNNPACK'
2024-12-18T00:21:36.7320879Z Entering 'third_party/benchmark'
2024-12-18T00:21:36.7373485Z Entering 'third_party/composable_kernel'
2024-12-18T00:21:36.7432192Z Entering 'third_party/cpp-httplib'
2024-12-18T00:21:36.7484620Z Entering 'third_party/cpuinfo'
2024-12-18T00:21:36.7538118Z Entering 'third_party/cudnn_frontend'
2024-12-18T00:21:36.7590711Z Entering 'third_party/cutlass'
2024-12-18T00:21:36.7651616Z Entering 'third_party/eigen'
2024-12-18T00:21:36.7704876Z Entering 'third_party/fbgemm'
2024-12-18T00:21:36.7757071Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T00:21:36.7808204Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T00:21:36.7861734Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T00:21:36.7918225Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T00:21:36.7969416Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T00:21:36.8024578Z Entering 'third_party/flatbuffers'
2024-12-18T00:21:36.8081520Z Entering 'third_party/fmt'
2024-12-18T00:21:36.8134606Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T00:21:36.8186783Z Entering 'third_party/gloo'
2024-12-18T00:21:36.8244027Z Entering 'third_party/googletest'
2024-12-18T00:21:36.8295915Z Entering 'third_party/ideep'
2024-12-18T00:21:36.8346782Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T00:21:36.8404750Z Entering 'third_party/ittapi'
2024-12-18T00:21:36.8458127Z Entering 'third_party/kineto'
2024-12-18T00:21:36.8508475Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T00:21:36.8558960Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T00:21:36.8611534Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T00:21:36.8663269Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T00:21:36.8714100Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T00:21:36.8763841Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T00:21:36.8824038Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T00:21:36.8874949Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T00:21:36.8929254Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T00:21:36.8980080Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T00:21:36.9033230Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T00:21:36.9084692Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T00:21:36.9141055Z Entering 'third_party/mimalloc'
2024-12-18T00:21:36.9193737Z Entering 'third_party/nccl/nccl'
2024-12-18T00:21:36.9247276Z Entering 'third_party/nlohmann'
2024-12-18T00:21:36.9300384Z Entering 'third_party/onnx'
2024-12-18T00:21:36.9366658Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T00:21:36.9425724Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T00:21:36.9478710Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T00:21:36.9529137Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T00:21:36.9579123Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T00:21:36.9629587Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T00:21:36.9682782Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T00:21:36.9733020Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T00:21:36.9790700Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T00:21:36.9840166Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T00:21:36.9893155Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T00:21:36.9948381Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T00:21:37.0021542Z Entering 'third_party/pocketfft'
2024-12-18T00:21:37.0074503Z Entering 'third_party/protobuf'
2024-12-18T00:21:37.0128881Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T00:21:37.0179567Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T00:21:37.0233626Z Entering 'third_party/psimd'
2024-12-18T00:21:37.0285983Z Entering 'third_party/pthreadpool'
2024-12-18T00:21:37.0338076Z Entering 'third_party/pybind11'
2024-12-18T00:21:37.0390588Z Entering 'third_party/python-peachpy'
2024-12-18T00:21:37.0444472Z Entering 'third_party/sleef'
2024-12-18T00:21:37.0496688Z Entering 'third_party/tensorpipe'
2024-12-18T00:21:37.0549039Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T00:21:37.0598506Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T00:21:37.0649317Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T00:21:37.0701206Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T00:21:37.0750090Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T00:21:37.0825729Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2024-12-18T00:21:37.1182947Z Entering 'android/libs/fbjni'
2024-12-18T00:21:37.1239026Z Entering 'third_party/FP16'
2024-12-18T00:21:37.1291206Z Entering 'third_party/FXdiv'
2024-12-18T00:21:37.1343800Z Entering 'third_party/NNPACK'
2024-12-18T00:21:37.1396251Z Entering 'third_party/NVTX'
2024-12-18T00:21:37.1449065Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T00:21:37.1502095Z Entering 'third_party/XNNPACK'
2024-12-18T00:21:37.1572397Z Entering 'third_party/benchmark'
2024-12-18T00:21:37.1629410Z Entering 'third_party/composable_kernel'
2024-12-18T00:21:37.1688936Z Entering 'third_party/cpp-httplib'
2024-12-18T00:21:37.1741786Z Entering 'third_party/cpuinfo'
2024-12-18T00:21:37.1797547Z Entering 'third_party/cudnn_frontend'
2024-12-18T00:21:37.1849449Z Entering 'third_party/cutlass'
2024-12-18T00:21:37.1912991Z Entering 'third_party/eigen'
2024-12-18T00:21:37.1971329Z Entering 'third_party/fbgemm'
2024-12-18T00:21:37.2028984Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T00:21:37.2079696Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T00:21:37.2131552Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T00:21:37.2188684Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T00:21:37.2239772Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T00:21:37.2292687Z Entering 'third_party/flatbuffers'
2024-12-18T00:21:37.2348258Z Entering 'third_party/fmt'
2024-12-18T00:21:37.2401709Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T00:21:37.2456088Z Entering 'third_party/gloo'
2024-12-18T00:21:37.2508459Z Entering 'third_party/googletest'
2024-12-18T00:21:37.2561427Z Entering 'third_party/ideep'
2024-12-18T00:21:37.2631049Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T00:21:37.2690939Z Entering 'third_party/ittapi'
2024-12-18T00:21:37.2743373Z Entering 'third_party/kineto'
2024-12-18T00:21:37.2798505Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T00:21:37.2849478Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T00:21:37.2901556Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T00:21:37.2955150Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T00:21:37.3005454Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T00:21:37.3055506Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T00:21:37.3111211Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T00:21:37.3168580Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T00:21:37.3219185Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T00:21:37.3271707Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T00:21:37.3326502Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T00:21:37.3377247Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T00:21:37.3430797Z Entering 'third_party/mimalloc'
2024-12-18T00:21:37.3483923Z Entering 'third_party/nccl/nccl'
2024-12-18T00:21:37.3535589Z Entering 'third_party/nlohmann'
2024-12-18T00:21:37.3589330Z Entering 'third_party/onnx'
2024-12-18T00:21:37.3654846Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T00:21:37.3711236Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T00:21:37.3764955Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T00:21:37.3814823Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T00:21:37.3864303Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T00:21:37.3913507Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T00:21:37.3967462Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T00:21:37.4015975Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T00:21:37.4064870Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T00:21:37.4112500Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T00:21:37.4165984Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T00:21:37.4224453Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T00:21:37.4294558Z Entering 'third_party/pocketfft'
2024-12-18T00:21:37.4347300Z Entering 'third_party/protobuf'
2024-12-18T00:21:37.4401283Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T00:21:37.4451168Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T00:21:37.4504202Z Entering 'third_party/psimd'
2024-12-18T00:21:37.4556466Z Entering 'third_party/pthreadpool'
2024-12-18T00:21:37.4607330Z Entering 'third_party/pybind11'
2024-12-18T00:21:37.4663355Z Entering 'third_party/python-peachpy'
2024-12-18T00:21:37.4715207Z Entering 'third_party/sleef'
2024-12-18T00:21:37.4766729Z Entering 'third_party/tensorpipe'
2024-12-18T00:21:37.4818701Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T00:21:37.4869276Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T00:21:37.4919529Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T00:21:37.4969638Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T00:21:37.5018474Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T00:21:37.5105964Z ##[endgroup]
2024-12-18T00:21:37.5128722Z [command]/usr/bin/git log -1 --format='%H'
2024-12-18T00:21:37.5152644Z '0cdf8b1d09254cfda66191d1bd01e3041c3c76f7'
2024-12-18T00:21:37.5343267Z Prepare all required actions
2024-12-18T00:21:37.5343796Z Getting action download info
2024-12-18T00:21:37.6886249Z ##[group]Run ./.github/actions/setup-linux
2024-12-18T00:21:37.6886587Z env:
2024-12-18T00:21:37.6886821Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:37.6887100Z ##[endgroup]
2024-12-18T00:21:37.6941984Z ##[group]Run set -euo pipefail
2024-12-18T00:21:37.6942344Z set -euo pipefail
2024-12-18T00:21:37.6942725Z function get_ec2_metadata() {
2024-12-18T00:21:37.6943184Z   # Pulled from instance metadata endpoint for EC2
2024-12-18T00:21:37.6943819Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2024-12-18T00:21:37.6944411Z   category=$1
2024-12-18T00:21:37.6944786Z   # If it is GCP runner (runner name contains gcp), do not run this
2024-12-18T00:21:37.6945227Z   runner_name_str=i-0ad7d688021a5725d
2024-12-18T00:21:37.6945616Z   if [[ -f /.inarc ]]; then
2024-12-18T00:21:37.6945979Z     echo "ARC Runner, no info on ec2 metadata"
2024-12-18T00:21:37.6946370Z   elif [[ $runner_name_str == *"gcp"* ]]; then
2024-12-18T00:21:37.6946843Z     echo "Runner is from Google Cloud Platform, No info on ec2 metadata"
2024-12-18T00:21:37.6947415Z   else
2024-12-18T00:21:37.6948341Z     curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2024-12-18T00:21:37.6949259Z   fi
2024-12-18T00:21:37.6949498Z }
2024-12-18T00:21:37.6949780Z echo "ami-id: $(get_ec2_metadata ami-id)"
2024-12-18T00:21:37.6950220Z echo "instance-id: $(get_ec2_metadata instance-id)"
2024-12-18T00:21:37.6950863Z echo "instance-type: $(get_ec2_metadata instance-type)"
2024-12-18T00:21:37.6951314Z echo "system info $(uname -a)"
2024-12-18T00:21:37.6965662Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:37.6966047Z env:
2024-12-18T00:21:37.6966275Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:37.6966553Z ##[endgroup]
2024-12-18T00:21:37.7128841Z ami-id: ami-0fff1b9a61dec8a5f
2024-12-18T00:21:37.7256661Z instance-id: i-0ad7d688021a5725d
2024-12-18T00:21:37.7368437Z instance-type: g5.4xlarge
2024-12-18T00:21:37.7380859Z system info Linux ip-10-0-64-118.ec2.internal 6.1.109-118.189.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 10 08:59:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
2024-12-18T00:21:37.7408926Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T00:21:37.7409856Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T00:21:37.7420255Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:37.7420650Z env:
2024-12-18T00:21:37.7420874Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:37.7421149Z ##[endgroup]
2024-12-18T00:21:37.7483983Z ##[group]Run if systemctl is-active --quiet docker; then
2024-12-18T00:21:37.7484443Z if systemctl is-active --quiet docker; then
2024-12-18T00:21:37.7484843Z     echo "Docker daemon is running...";
2024-12-18T00:21:37.7485180Z else
2024-12-18T00:21:37.7485550Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2024-12-18T00:21:37.7485987Z fi
2024-12-18T00:21:37.7494742Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:37.7495318Z env:
2024-12-18T00:21:37.7495555Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:37.7495834Z ##[endgroup]
2024-12-18T00:21:37.7584934Z Docker daemon is running...
2024-12-18T00:21:37.7631327Z ##[group]Run nick-fields/retry@v3.0.0
2024-12-18T00:21:37.7631647Z with:
2024-12-18T00:21:37.7631872Z   shell: bash
2024-12-18T00:21:37.7632310Z   timeout_minutes: 5
2024-12-18T00:21:37.7632573Z   max_attempts: 3
2024-12-18T00:21:37.7632826Z   retry_wait_seconds: 30
2024-12-18T00:21:37.7635070Z   command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \
    --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"

# For LF Runners we need to make sure we also login to Meta's ECR docker registry too.
META_AWS_ACCOUNT_ID=308535385114
if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then
    aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \
        --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
fi

2024-12-18T00:21:37.7637421Z   polling_interval_seconds: 1
2024-12-18T00:21:37.7637722Z   warning_on_retry: true
2024-12-18T00:21:37.7637996Z   continue_on_error: false
2024-12-18T00:21:37.7638268Z env:
2024-12-18T00:21:37.7638495Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:37.7638779Z   AWS_RETRY_MODE: standard
2024-12-18T00:21:37.7639051Z   AWS_MAX_ATTEMPTS: 5
2024-12-18T00:21:37.7639329Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T00:21:37.7639617Z ##[endgroup]
2024-12-18T00:21:39.0327394Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-12-18T00:21:39.0328217Z Configure a credential helper to remove this warning. See
2024-12-18T00:21:39.0328989Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-12-18T00:21:39.0329506Z 
2024-12-18T00:21:39.0329649Z Login Succeeded
2024-12-18T00:21:39.9287844Z Command completed after 1 attempt(s).
2024-12-18T00:21:39.9370705Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2024-12-18T00:21:39.9371231Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2024-12-18T00:21:39.9371696Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2024-12-18T00:21:39.9382596Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:39.9382974Z env:
2024-12-18T00:21:39.9383213Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:39.9383493Z ##[endgroup]
2024-12-18T00:21:39.9484567Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2024-12-18T00:21:39.9485130Z # ignore expansion of "docker ps -q" since it could be empty
2024-12-18T00:21:39.9485562Z # shellcheck disable=SC2046
2024-12-18T00:21:39.9485909Z docker stop $(docker ps -q) || true
2024-12-18T00:21:39.9486269Z # Prune all of the docker images
2024-12-18T00:21:39.9486612Z docker system prune -af
2024-12-18T00:21:39.9494786Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:39.9495185Z env:
2024-12-18T00:21:39.9495414Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:39.9495682Z ##[endgroup]
2024-12-18T00:21:39.9803950Z "docker stop" requires at least 1 argument.
2024-12-18T00:21:39.9804352Z See 'docker stop --help'.
2024-12-18T00:21:39.9804535Z 
2024-12-18T00:21:39.9804741Z Usage:  docker stop [OPTIONS] CONTAINER [CONTAINER...]
2024-12-18T00:21:39.9805020Z 
2024-12-18T00:21:39.9805143Z Stop one or more running containers
2024-12-18T00:21:40.0170737Z Total reclaimed space: 0B
2024-12-18T00:21:40.0224400Z ##[group]Run set +e
2024-12-18T00:21:40.0224744Z set +e
2024-12-18T00:21:40.0225013Z set -x
2024-12-18T00:21:40.0225262Z 
2024-12-18T00:21:40.0225530Z PT_DOMAIN=download.pytorch.org
2024-12-18T00:21:40.0226116Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400,
2024-12-18T00:21:40.0226874Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last
2024-12-18T00:21:40.0227590Z # one is returned at random
2024-12-18T00:21:40.0227995Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1)
2024-12-18T00:21:40.0228388Z 
2024-12-18T00:21:40.0228816Z if [ -z "${RESOLVED_IP}" ]; then
2024-12-18T00:21:40.0229260Z   echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..."
2024-12-18T00:21:40.0229787Z   RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1)
2024-12-18T00:21:40.0230195Z 
2024-12-18T00:21:40.0230463Z   if [ -z "${RESOLVED_IP}" ]; then
2024-12-18T00:21:40.0230862Z     echo "Couldn't resolve ${PT_DOMAIN}, exiting..."
2024-12-18T00:21:40.0231241Z     exit 1
2024-12-18T00:21:40.0231499Z   fi
2024-12-18T00:21:40.0231741Z fi
2024-12-18T00:21:40.0231980Z 
2024-12-18T00:21:40.0232271Z if grep -r "${PT_DOMAIN}" /etc/hosts; then
2024-12-18T00:21:40.0232669Z   # Clean up any old records first
2024-12-18T00:21:40.0233045Z   sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts
2024-12-18T00:21:40.0233395Z fi
2024-12-18T00:21:40.0233627Z 
2024-12-18T00:21:40.0233973Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts
2024-12-18T00:21:40.0234480Z cat /etc/hosts
2024-12-18T00:21:40.0243977Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:40.0244486Z env:
2024-12-18T00:21:40.0262712Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:40.0263014Z ##[endgroup]
2024-12-18T00:21:40.0291384Z + PT_DOMAIN=download.pytorch.org
2024-12-18T00:21:40.0297494Z ++ dig -4 +short download.pytorch.org
2024-12-18T00:21:40.0298082Z ++ tail -n1
2024-12-18T00:21:40.0771823Z + RESOLVED_IP=18.160.18.20
2024-12-18T00:21:40.0772156Z + '[' -z 18.160.18.20 ']'
2024-12-18T00:21:40.0772460Z + grep -r download.pytorch.org /etc/hosts
2024-12-18T00:21:40.0791301Z + echo '18.160.18.20 download.pytorch.org'
2024-12-18T00:21:40.0791726Z + sudo tee -a /etc/hosts
2024-12-18T00:21:40.3020324Z 18.160.18.20 download.pytorch.org
2024-12-18T00:21:40.3041667Z + cat /etc/hosts
2024-12-18T00:21:40.3051969Z 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
2024-12-18T00:21:40.3057411Z ::1         localhost6 localhost6.localdomain6
2024-12-18T00:21:40.3057787Z 18.160.18.20 download.pytorch.org
2024-12-18T00:21:40.3211868Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@release/2.6
2024-12-18T00:21:40.3212364Z with:
2024-12-18T00:21:40.3213204Z   docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:40.3214138Z   docker-build-dir: .ci/docker
2024-12-18T00:21:40.3214827Z   working-directory: .
2024-12-18T00:21:40.3215233Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:40.3215653Z   force-push: false
2024-12-18T00:21:40.3215885Z env:
2024-12-18T00:21:40.3216109Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:40.3216378Z ##[endgroup]
2024-12-18T00:21:40.3246009Z ##[group]Run set -ex
2024-12-18T00:21:40.3246319Z set -ex
2024-12-18T00:21:40.3246568Z 
2024-12-18T00:21:40.3246970Z # If the docker build directory or the build script doesn't exist, the action will
2024-12-18T00:21:40.3247673Z # gracefully return the docker image name as it is.  Pulling docker image in Linux
2024-12-18T00:21:40.3248243Z # job could then download the pre-built image as usual
2024-12-18T00:21:40.3248754Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then
2024-12-18T00:21:40.3249235Z   echo "skip=true" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:40.3249689Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:40.3250104Z 
2024-12-18T00:21:40.3250647Z   echo "There is no Docker build script in ${REPO_NAME} repo, skipping..."
2024-12-18T00:21:40.3251088Z   exit 0
2024-12-18T00:21:40.3251335Z else
2024-12-18T00:21:40.3251624Z   echo "skip=false" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:40.3251963Z fi
2024-12-18T00:21:40.3252203Z 
2024-12-18T00:21:40.3252568Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then
2024-12-18T00:21:40.3253147Z   # The docker image name already includes the ECR prefix and tag, so we can just
2024-12-18T00:21:40.3253688Z   # use it as it is, but first let's extract the tag
2024-12-18T00:21:40.3254180Z   DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}')
2024-12-18T00:21:40.3254701Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:40.3255239Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:40.3255661Z else
2024-12-18T00:21:40.3255983Z   DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}")
2024-12-18T00:21:40.3256448Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:40.3257074Z   echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:40.3257616Z fi
2024-12-18T00:21:40.3267580Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:40.3267971Z env:
2024-12-18T00:21:40.3268193Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:40.3268479Z   REPO_NAME: pytorch
2024-12-18T00:21:40.3269339Z   DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:40.3270242Z   DOCKER_BUILD_DIR: .ci/docker
2024-12-18T00:21:40.3270630Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:40.3271023Z ##[endgroup]
2024-12-18T00:21:40.3300309Z + [[ ! -d .ci/docker ]]
2024-12-18T00:21:40.3300737Z + [[ ! -f .ci/docker/build.sh ]]
2024-12-18T00:21:40.3301127Z + echo skip=false
2024-12-18T00:21:40.3302489Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]]
2024-12-18T00:21:40.3308669Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:40.3309556Z ++ awk -F '[:,]' '{print $2}'
2024-12-18T00:21:40.3335383Z + DOCKER_TAG=45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:40.3335818Z + echo docker-tag=45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:40.3336764Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:40.3374427Z ##[group]Run set +e
2024-12-18T00:21:40.3374771Z set +e
2024-12-18T00:21:40.3375020Z set -x
2024-12-18T00:21:40.3375266Z 
2024-12-18T00:21:40.3375491Z login() {
2024-12-18T00:21:40.3375992Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2024-12-18T00:21:40.3376519Z }
2024-12-18T00:21:40.3376752Z 
2024-12-18T00:21:40.3376985Z retry () {
2024-12-18T00:21:40.3377273Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2024-12-18T00:21:40.3377600Z }
2024-12-18T00:21:40.3377825Z 
2024-12-18T00:21:40.3378145Z retry login "${DOCKER_REGISTRY}"
2024-12-18T00:21:40.3378460Z 
2024-12-18T00:21:40.3378701Z START_TIME=$(date +%s)
2024-12-18T00:21:40.3379013Z # Wait up to 90 minutes
2024-12-18T00:21:40.3379566Z while [[ $(( $(date +%s) - 5400 )) -lt $START_TIME ]]; do
2024-12-18T00:21:40.3380071Z   # Check if image already exists, if it does then skip building it
2024-12-18T00:21:40.3380568Z   if docker manifest inspect "${DOCKER_IMAGE}"; then
2024-12-18T00:21:40.3380936Z     exit 0
2024-12-18T00:21:40.3381190Z   fi
2024-12-18T00:21:40.3381426Z 
2024-12-18T00:21:40.3381820Z   # NB: This flag is used by Docker build workflow to push the image to ECR, so we can
2024-12-18T00:21:40.3382471Z   # use this to differentiate between the Docker build and regular build jobs. For the
2024-12-18T00:21:40.3383118Z   # latter, it will wait for the Docker images to become available before continuing
2024-12-18T00:21:40.3383625Z   if [ "${DOCKER_PUSH:-false}" == "true" ]; then
2024-12-18T00:21:40.3384036Z     # It's a Docker build job, let's build the image
2024-12-18T00:21:40.3384438Z     break
2024-12-18T00:21:40.3384697Z   else
2024-12-18T00:21:40.3385049Z     # It's a regular build job, wait for the image to become available
2024-12-18T00:21:40.3385464Z     sleep 300
2024-12-18T00:21:40.3385720Z   fi
2024-12-18T00:21:40.3385954Z done
2024-12-18T00:21:40.3386184Z 
2024-12-18T00:21:40.3386560Z # NB: This part requires a full checkout. Otherwise, the merge base will
2024-12-18T00:21:40.3387131Z # be empty.  The default action would be to continue rebuild the image
2024-12-18T00:21:40.3387630Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then
2024-12-18T00:21:40.3388088Z   # if we're on the base branch then use the parent commit
2024-12-18T00:21:40.3388498Z   MERGE_BASE=$(git rev-parse HEAD~)
2024-12-18T00:21:40.3388823Z else
2024-12-18T00:21:40.3389169Z   # otherwise we're on a PR, so use the most recent base commit
2024-12-18T00:21:40.3389638Z   MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION")
2024-12-18T00:21:40.3390007Z fi
2024-12-18T00:21:40.3390235Z 
2024-12-18T00:21:40.3390483Z if [[ -z "${MERGE_BASE}" ]]; then
2024-12-18T00:21:40.3390848Z   echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:40.3391187Z 
2024-12-18T00:21:40.3391796Z   echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..."
2024-12-18T00:21:40.3392350Z   exit 0
2024-12-18T00:21:40.3392595Z fi
2024-12-18T00:21:40.3392836Z 
2024-12-18T00:21:40.3393160Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then
2024-12-18T00:21:40.3393825Z   echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit"
2024-12-18T00:21:40.3394405Z   exit 1
2024-12-18T00:21:40.3394658Z fi
2024-12-18T00:21:40.3394889Z 
2024-12-18T00:21:40.3395270Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}")
2024-12-18T00:21:40.3395927Z # If no image exists but the hash is the same as the previous hash then we should error out here
2024-12-18T00:21:40.3396513Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then
2024-12-18T00:21:40.3397188Z   echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch"
2024-12-18T00:21:40.3397942Z   echo "         Will re-build docker image to store in local cache, TTS may be longer"
2024-12-18T00:21:40.3398407Z fi
2024-12-18T00:21:40.3398644Z 
2024-12-18T00:21:40.3398923Z echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:40.3407793Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:40.3408171Z env:
2024-12-18T00:21:40.3408407Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:40.3408694Z   DOCKER_BUILD_DIR: .ci/docker
2024-12-18T00:21:40.3409140Z   BASE_REVISION: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:21:40.3410084Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:40.3411027Z   DOCKER_TAG: 45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:40.3411479Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:40.3411870Z   DOCKER_PUSH: 
2024-12-18T00:21:40.3412108Z ##[endgroup]
2024-12-18T00:21:40.3442448Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:40.3442901Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:40.3445751Z + aws ecr get-login-password --region us-east-1
2024-12-18T00:21:40.3446962Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:40.8908231Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-12-18T00:21:40.8908837Z Configure a credential helper to remove this warning. See
2024-12-18T00:21:40.8909403Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-12-18T00:21:40.8909795Z 
2024-12-18T00:21:40.8910188Z Login Succeeded
2024-12-18T00:21:40.8937354Z ++ date +%s
2024-12-18T00:21:40.8949922Z + START_TIME=1734481300
2024-12-18T00:21:40.8953951Z ++ date +%s
2024-12-18T00:21:40.8965457Z + [[ 1734475900 -lt 1734481300 ]]
2024-12-18T00:21:40.8966467Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:41.1374154Z {
2024-12-18T00:21:41.1374875Z 	"schemaVersion": 2,
2024-12-18T00:21:41.1375352Z 	"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
2024-12-18T00:21:41.1375834Z 	"config": {
2024-12-18T00:21:41.1376197Z 		"mediaType": "application/vnd.docker.container.image.v1+json",
2024-12-18T00:21:41.1376649Z 		"size": 52680,
2024-12-18T00:21:41.1377084Z 		"digest": "sha256:67e93a8badaf799ceb664d7c453077c6c6785d56dc10c720be9358edeb91759b"
2024-12-18T00:21:41.1377620Z 	},
2024-12-18T00:21:41.1377849Z 	"layers": [
2024-12-18T00:21:41.1378176Z 		{
2024-12-18T00:21:41.1378525Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1378945Z 			"size": 28584506,
2024-12-18T00:21:41.1379679Z 			"digest": "sha256:80888bc6716fcbb8874e75ac88898d3e38e6f1bc55678f0e97ca9d706b7f3733"
2024-12-18T00:21:41.1380169Z 		},
2024-12-18T00:21:41.1380385Z 		{
2024-12-18T00:21:41.1380728Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1381146Z 			"size": 7944698,
2024-12-18T00:21:41.1381581Z 			"digest": "sha256:fbcd35dc5bc3a7bda41926aadd083020f942b001ebac6f1d30480f0f065394c0"
2024-12-18T00:21:41.1382072Z 		},
2024-12-18T00:21:41.1382284Z 		{
2024-12-18T00:21:41.1382621Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1383041Z 			"size": 57593527,
2024-12-18T00:21:41.1383488Z 			"digest": "sha256:c7232af9ae05f7de83f8d6171bd0c35a4dd0a85ebafb15b950dbc08f89ea5fb5"
2024-12-18T00:21:41.1383975Z 		},
2024-12-18T00:21:41.1384184Z 		{
2024-12-18T00:21:41.1384524Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1384990Z 			"size": 185,
2024-12-18T00:21:41.1385420Z 			"digest": "sha256:db6cdef1932a0d9ca6ef9a539e08d491f66d1b1ed81926ae1525375bdd8100cc"
2024-12-18T00:21:41.1385902Z 		},
2024-12-18T00:21:41.1386114Z 		{
2024-12-18T00:21:41.1386453Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1386867Z 			"size": 6886,
2024-12-18T00:21:41.1387311Z 			"digest": "sha256:56dc8550293751a1604e97ac949cfae82ba20cb2a28e034737bafd7382559609"
2024-12-18T00:21:41.1387780Z 		},
2024-12-18T00:21:41.1387995Z 		{
2024-12-18T00:21:41.1388328Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1388757Z 			"size": 1374287670,
2024-12-18T00:21:41.1389343Z 			"digest": "sha256:30c0ea6140d07e2a8deb70d780f277c63cf61836ff33d66eef944728a4bef6bd"
2024-12-18T00:21:41.1389811Z 		},
2024-12-18T00:21:41.1390023Z 		{
2024-12-18T00:21:41.1390354Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1390776Z 			"size": 62641,
2024-12-18T00:21:41.1391207Z 			"digest": "sha256:71bdb1a72c2d6dc97bbdbca82383f0260c4ee87556701e8e606c08a6bb0f0da5"
2024-12-18T00:21:41.1391744Z 		},
2024-12-18T00:21:41.1391952Z 		{
2024-12-18T00:21:41.1392279Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1392705Z 			"size": 1685,
2024-12-18T00:21:41.1393121Z 			"digest": "sha256:4829486be7c30f19f4136fa56adbb3de206ed0bbf0705b59fb2147406778ce38"
2024-12-18T00:21:41.1393592Z 		},
2024-12-18T00:21:41.1393798Z 		{
2024-12-18T00:21:41.1394131Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1394548Z 			"size": 1523,
2024-12-18T00:21:41.1394960Z 			"digest": "sha256:1f4e68d7b5e4224ba1da78ef461ff7f01e8d59c09d39281277521384105a9441"
2024-12-18T00:21:41.1395432Z 		},
2024-12-18T00:21:41.1395641Z 		{
2024-12-18T00:21:41.1395976Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1396395Z 			"size": 2626741021,
2024-12-18T00:21:41.1396821Z 			"digest": "sha256:7c373e2d9b7e82a6878d4a31293dd857915a0fe47d07dce541cea03b043d57fc"
2024-12-18T00:21:41.1397285Z 		},
2024-12-18T00:21:41.1397498Z 		{
2024-12-18T00:21:41.1397827Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1398246Z 			"size": 86545,
2024-12-18T00:21:41.1398654Z 			"digest": "sha256:6223811417458a3c93b84ee3b65f8b08d9e2828b926f0aed863041610d7d95d4"
2024-12-18T00:21:41.1399111Z 		},
2024-12-18T00:21:41.1399325Z 		{
2024-12-18T00:21:41.1399652Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1400071Z 			"size": 1824,
2024-12-18T00:21:41.1400490Z 			"digest": "sha256:f003d065e31e5e8c0cea490c88a912dd331bddcbd2f5ff22ccc692a5343bcbeb"
2024-12-18T00:21:41.1400970Z 		},
2024-12-18T00:21:41.1401176Z 		{
2024-12-18T00:21:41.1401508Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1401931Z 			"size": 246420139,
2024-12-18T00:21:41.1402360Z 			"digest": "sha256:cd6ce71168153e485b99493ce07032bbd9fd5cefceb68ef33ce088244306a653"
2024-12-18T00:21:41.1402831Z 		},
2024-12-18T00:21:41.1403040Z 		{
2024-12-18T00:21:41.1403468Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1403891Z 			"size": 703,
2024-12-18T00:21:41.1404352Z 			"digest": "sha256:e843066a8f68042266cfe814f1729975619fb8f733f371be115e8b2fe81cbc9d"
2024-12-18T00:21:41.1404931Z 		},
2024-12-18T00:21:41.1405210Z 		{
2024-12-18T00:21:41.1405605Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1406044Z 			"size": 1256,
2024-12-18T00:21:41.1406479Z 			"digest": "sha256:bac50ca2930bc4b1afa8a2f1b4b0d1d499d63b7922b784af690c3b31ec0c2f70"
2024-12-18T00:21:41.1406979Z 		},
2024-12-18T00:21:41.1407192Z 		{
2024-12-18T00:21:41.1407526Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1407954Z 			"size": 484,
2024-12-18T00:21:41.1408380Z 			"digest": "sha256:57e52c6751f4d865ada4cae919859fe563fe5dcbb93ca82cb4746f98caccbd3a"
2024-12-18T00:21:41.1408863Z 		},
2024-12-18T00:21:41.1409078Z 		{
2024-12-18T00:21:41.1409417Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1409853Z 			"size": 91715265,
2024-12-18T00:21:41.1410278Z 			"digest": "sha256:a9c5c14d77c035c7501566b1247024dfd241bd6bc12fbe2bca28ec7d90b26069"
2024-12-18T00:21:41.1410762Z 		},
2024-12-18T00:21:41.1410985Z 		{
2024-12-18T00:21:41.1411322Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1411752Z 			"size": 3706,
2024-12-18T00:21:41.1412166Z 			"digest": "sha256:b86099b8619ad88e8fe6c8e693180d52e0f2fbeac5927f0b403c8a9ef76d7f6b"
2024-12-18T00:21:41.1412644Z 		},
2024-12-18T00:21:41.1412958Z 		{
2024-12-18T00:21:41.1413288Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1413709Z 			"size": 1860,
2024-12-18T00:21:41.1414145Z 			"digest": "sha256:3c8ea8a600fff6cce4c1b36acc5f3949ecd664e55b5eaf4a50fe5499351574a6"
2024-12-18T00:21:41.1414948Z 		},
2024-12-18T00:21:41.1415178Z 		{
2024-12-18T00:21:41.1415532Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1415966Z 			"size": 701,
2024-12-18T00:21:41.1416391Z 			"digest": "sha256:bebd539a1d598fe86afc0da77bf16ae4c7ebfb9f031f21ed7634150fbe23f996"
2024-12-18T00:21:41.1416878Z 		},
2024-12-18T00:21:41.1417083Z 		{
2024-12-18T00:21:41.1417408Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1417824Z 			"size": 478,
2024-12-18T00:21:41.1418280Z 			"digest": "sha256:292b18ec2dab9bd60d00192387c1269d39b0b4886b782585d2c879808218fc66"
2024-12-18T00:21:41.1418741Z 		},
2024-12-18T00:21:41.1418952Z 		{
2024-12-18T00:21:41.1419288Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1419709Z 			"size": 2941869535,
2024-12-18T00:21:41.1420139Z 			"digest": "sha256:76e3817b4b4cec0a5b005161867dc257681b792271ba9d23c9b79b3a6801dcd9"
2024-12-18T00:21:41.1420604Z 		},
2024-12-18T00:21:41.1420803Z 		{
2024-12-18T00:21:41.1421144Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1421568Z 			"size": 380,
2024-12-18T00:21:41.1421988Z 			"digest": "sha256:50b1f245e9f5be8670d8ba5790e3b96450eee3428dcac7060367db48ae00234e"
2024-12-18T00:21:41.1422469Z 		},
2024-12-18T00:21:41.1422675Z 		{
2024-12-18T00:21:41.1423018Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1423448Z 			"size": 12901,
2024-12-18T00:21:41.1423882Z 			"digest": "sha256:ad93cf5205f24ada86b5f7eab5b63ed97ae7858368f6fdbce373343d2568d66e"
2024-12-18T00:21:41.1424358Z 		},
2024-12-18T00:21:41.1424560Z 		{
2024-12-18T00:21:41.1424899Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1425334Z 			"size": 863,
2024-12-18T00:21:41.1425754Z 			"digest": "sha256:8d17c59eaf7d84d49e3660380debcc70734b2a178e6cab86f0d4fd3e7a8100d1"
2024-12-18T00:21:41.1426230Z 		},
2024-12-18T00:21:41.1426436Z 		{
2024-12-18T00:21:41.1426782Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1427208Z 			"size": 106,
2024-12-18T00:21:41.1427792Z 			"digest": "sha256:b36a8ccf5c8c8ef0bf3e168ec9f568a0ada36c49103c9cacfef6dd072ddd681a"
2024-12-18T00:21:41.1428279Z 		},
2024-12-18T00:21:41.1428478Z 		{
2024-12-18T00:21:41.1428820Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1429246Z 			"size": 504,
2024-12-18T00:21:41.1429666Z 			"digest": "sha256:aa0a3b87e9e34ab15903206018bfd9fb1eb54fd21f05a61ef254557703890732"
2024-12-18T00:21:41.1430133Z 		},
2024-12-18T00:21:41.1430335Z 		{
2024-12-18T00:21:41.1430623Z + exit 0
2024-12-18T00:21:41.1430966Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1431401Z 			"size": 121477314,
2024-12-18T00:21:41.1431838Z 			"digest": "sha256:72a632052c5be4900c60ce09e32f8a0fd3b637ab0b7685dc5b07bda4776ddd15"
2024-12-18T00:21:41.1432307Z 		},
2024-12-18T00:21:41.1432505Z 		{
2024-12-18T00:21:41.1432843Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1433257Z 			"size": 109,
2024-12-18T00:21:41.1433682Z 			"digest": "sha256:ceed6316bc249ffffee9563cb7505023b3e70fba306f187bd612fece63508263"
2024-12-18T00:21:41.1434151Z 		},
2024-12-18T00:21:41.1434357Z 		{
2024-12-18T00:21:41.1434686Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1435106Z 			"size": 489,
2024-12-18T00:21:41.1435524Z 			"digest": "sha256:d0ca0f491f809a9f1dedd391f5b25cc3f59d8e95e7f76299f18ec3c931e56ed1"
2024-12-18T00:21:41.1435996Z 		},
2024-12-18T00:21:41.1436203Z 		{
2024-12-18T00:21:41.1436536Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1437097Z 			"size": 296,
2024-12-18T00:21:41.1437509Z 			"digest": "sha256:cf02c706cd567c781eeb567c90960801dc1ef78872b73fbc047df898951ecbb8"
2024-12-18T00:21:41.1437964Z 		},
2024-12-18T00:21:41.1438174Z 		{
2024-12-18T00:21:41.1438512Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1438941Z 			"size": 103,
2024-12-18T00:21:41.1439367Z 			"digest": "sha256:16073d3f107dab837d7edb5e6044449660fec7e2ded263ca81d5dcfea5beb486"
2024-12-18T00:21:41.1439832Z 		},
2024-12-18T00:21:41.1440042Z 		{
2024-12-18T00:21:41.1440388Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1440815Z 			"size": 1473,
2024-12-18T00:21:41.1441237Z 			"digest": "sha256:d234faf2f1bfb6cb9d4ba14103226d20d540e568b47690b67c2a2b2c5a49645f"
2024-12-18T00:21:41.1441707Z 		},
2024-12-18T00:21:41.1441922Z 		{
2024-12-18T00:21:41.1442263Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1442693Z 			"size": 430258758,
2024-12-18T00:21:41.1443141Z 			"digest": "sha256:3f0ade86af1a5bc72c9088735bb1a7ab9819fa20fdd96437fc080b2c711dfd3d"
2024-12-18T00:21:41.1443614Z 		},
2024-12-18T00:21:41.1443819Z 		{
2024-12-18T00:21:41.1444163Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1444633Z 			"size": 164,
2024-12-18T00:21:41.1445081Z 			"digest": "sha256:034b6a03f5f751ed12f7505175ce3443eb3ed7b34d02d1bb25de0ee52d93a44b"
2024-12-18T00:21:41.1445564Z 		},
2024-12-18T00:21:41.1445787Z 		{
2024-12-18T00:21:41.1446144Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1446574Z 			"size": 564,
2024-12-18T00:21:41.1447009Z 			"digest": "sha256:21f8f066852dd484ccbf79470452fe8005cde08773dd4fedc07fed783a0b9bf8"
2024-12-18T00:21:41.1447479Z 		},
2024-12-18T00:21:41.1447696Z 		{
2024-12-18T00:21:41.1448044Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1448480Z 			"size": 35865656,
2024-12-18T00:21:41.1448908Z 			"digest": "sha256:9dc1f315a979ec860a882d7df33584e02c1d18f86293521839ce42e21082e639"
2024-12-18T00:21:41.1449380Z 		},
2024-12-18T00:21:41.1449593Z 		{
2024-12-18T00:21:41.1449943Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1450373Z 			"size": 104,
2024-12-18T00:21:41.1450793Z 			"digest": "sha256:cd6a6063178b945c150fe21a31df6e33203295dd814802e0658835e810fb2bf9"
2024-12-18T00:21:41.1451257Z 		},
2024-12-18T00:21:41.1451597Z 		{
2024-12-18T00:21:41.1451955Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1452389Z 			"size": 425,
2024-12-18T00:21:41.1452814Z 			"digest": "sha256:2116738f4ea9299766a55629d39b881c4d33fc4fc0ce915965be1a850fcd7aaa"
2024-12-18T00:21:41.1453285Z 		},
2024-12-18T00:21:41.1453502Z 		{
2024-12-18T00:21:41.1453847Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1454283Z 			"size": 20262160,
2024-12-18T00:21:41.1454725Z 			"digest": "sha256:d7fa134316d91d2ec39bceb121619d44cdfdf1c7ac30c72768a9fd48f9763bd1"
2024-12-18T00:21:41.1455208Z 		},
2024-12-18T00:21:41.1455453Z 		{
2024-12-18T00:21:41.1455834Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1456277Z 			"size": 643,
2024-12-18T00:21:41.1456711Z 			"digest": "sha256:af03dc3fd15ea8f5f1fa41d514400f6a1d963985cf046b09d789f7f0359b2aba"
2024-12-18T00:21:41.1457193Z 		},
2024-12-18T00:21:41.1457418Z 		{
2024-12-18T00:21:41.1457772Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1458256Z 			"size": 701,
2024-12-18T00:21:41.1458692Z 			"digest": "sha256:bebd539a1d598fe86afc0da77bf16ae4c7ebfb9f031f21ed7634150fbe23f996"
2024-12-18T00:21:41.1459175Z 		},
2024-12-18T00:21:41.1459388Z 		{
2024-12-18T00:21:41.1459729Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1460153Z 			"size": 142,
2024-12-18T00:21:41.1460570Z 			"digest": "sha256:fd9de3dc425828e7b8984d4ced05f4102762137b81a5dc67104fc6b0805c570d"
2024-12-18T00:21:41.1461123Z 		},
2024-12-18T00:21:41.1461336Z 		{
2024-12-18T00:21:41.1461676Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1462104Z 			"size": 135,
2024-12-18T00:21:41.1462526Z 			"digest": "sha256:b85093661f7768ac691f4b3ee5f553b9ab14cb2a6877c8edc6133d74caa88f69"
2024-12-18T00:21:41.1463054Z 		},
2024-12-18T00:21:41.1463267Z 		{
2024-12-18T00:21:41.1463610Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1464240Z 			"size": 4995482131,
2024-12-18T00:21:41.1464778Z 			"digest": "sha256:ab6ef6c2ac4d23af7a3249cee1d4761f63a89513796ba720da7cc425c8afa6d2"
2024-12-18T00:21:41.1465500Z 		},
2024-12-18T00:21:41.1465909Z 		{
2024-12-18T00:21:41.1466354Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1466826Z 			"size": 196,
2024-12-18T00:21:41.1467378Z 			"digest": "sha256:56c896077120aaa4cbcc2203662d13b4ea14cf998e9657e91b252535ceae5348"
2024-12-18T00:21:41.1467946Z 		},
2024-12-18T00:21:41.1468229Z 		{
2024-12-18T00:21:41.1468740Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1469247Z 			"size": 564,
2024-12-18T00:21:41.1469701Z 			"digest": "sha256:d88e9c5e7c70fe36bd1bd2e94d43d4192d94a1f28e323234a73a75a2d1da48f1"
2024-12-18T00:21:41.1470345Z 		},
2024-12-18T00:21:41.1470637Z 		{
2024-12-18T00:21:41.1471072Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1491051Z 			"size": 43164612,
2024-12-18T00:21:41.1491532Z 			"digest": "sha256:53617ee83302d7f97eb56f6756d78d18f8dc41bf252bf2d04cb643198f400f6f"
2024-12-18T00:21:41.1492011Z 		},
2024-12-18T00:21:41.1492232Z 		{
2024-12-18T00:21:41.1492573Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1493005Z 			"size": 106,
2024-12-18T00:21:41.1493439Z 			"digest": "sha256:3edbde10d4043bf2ebf32fddfa55c8d0923435465637df7dc8f4addcc03a0b84"
2024-12-18T00:21:41.1493924Z 		},
2024-12-18T00:21:41.1494146Z 		{
2024-12-18T00:21:41.1494560Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1495103Z 			"size": 1402,
2024-12-18T00:21:41.1495624Z 			"digest": "sha256:f4038f7524c3ed705e7560db6dd6efe76f548363f16bc7a93cfe360d5bc6dcaa"
2024-12-18T00:21:41.1496211Z 		},
2024-12-18T00:21:41.1496446Z 		{
2024-12-18T00:21:41.1496778Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1497203Z 			"size": 701,
2024-12-18T00:21:41.1497761Z 			"digest": "sha256:bebd539a1d598fe86afc0da77bf16ae4c7ebfb9f031f21ed7634150fbe23f996"
2024-12-18T00:21:41.1498346Z 		},
2024-12-18T00:21:41.1498555Z 		{
2024-12-18T00:21:41.1498884Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1499305Z 			"size": 137,
2024-12-18T00:21:41.1499719Z 			"digest": "sha256:4a51173f77576ab68fba2f862dd4d12b9ff1f250145c090520b05f88f5c81391"
2024-12-18T00:21:41.1500185Z 		},
2024-12-18T00:21:41.1500398Z 		{
2024-12-18T00:21:41.1500729Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1501155Z 			"size": 120,
2024-12-18T00:21:41.1501566Z 			"digest": "sha256:ed6a71d41274e5b1eea4025d24a4851d6880b902b08f7d4dd0618ea81154f575"
2024-12-18T00:21:41.1502037Z 		},
2024-12-18T00:21:41.1502245Z 		{
2024-12-18T00:21:41.1502584Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1503011Z 			"size": 4886226234,
2024-12-18T00:21:41.1503450Z 			"digest": "sha256:389dc939e2a414ef26b642b1e8657a710358afa6a6be3ebc4380b01ffacbbacc"
2024-12-18T00:21:41.1503928Z 		},
2024-12-18T00:21:41.1504137Z 		{
2024-12-18T00:21:41.1504476Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1504946Z 			"size": 174,
2024-12-18T00:21:41.1505373Z 			"digest": "sha256:0341eaeed33df462f8bd28003bb9b09739b0786abba58ccc17ba4cec535ee4ef"
2024-12-18T00:21:41.1505849Z 		},
2024-12-18T00:21:41.1506059Z 		{
2024-12-18T00:21:41.1506394Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1506912Z 			"size": 908,
2024-12-18T00:21:41.1507327Z 			"digest": "sha256:537cdb2d31685c0dbaf612d24633539a141cfcb69d90c5ce932923a6e4ab4474"
2024-12-18T00:21:41.1507807Z 		},
2024-12-18T00:21:41.1508026Z 		{
2024-12-18T00:21:41.1508357Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1508785Z 			"size": 701,
2024-12-18T00:21:41.1509227Z 			"digest": "sha256:bebd539a1d598fe86afc0da77bf16ae4c7ebfb9f031f21ed7634150fbe23f996"
2024-12-18T00:21:41.1509718Z 		},
2024-12-18T00:21:41.1509928Z 		{
2024-12-18T00:21:41.1510269Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1510701Z 			"size": 135,
2024-12-18T00:21:41.1511115Z 			"digest": "sha256:76d4a07024975dd727ff7b1c8873c513043bc9a5d310b469e51c3b110fdb6084"
2024-12-18T00:21:41.1511590Z 		},
2024-12-18T00:21:41.1511803Z 		{
2024-12-18T00:21:41.1512151Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1512577Z 			"size": 32,
2024-12-18T00:21:41.1513007Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-12-18T00:21:41.1513486Z 		},
2024-12-18T00:21:41.1513692Z 		{
2024-12-18T00:21:41.1514033Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1514691Z 			"size": 159,
2024-12-18T00:21:41.1515146Z 			"digest": "sha256:3287a539d61e3913b93941ecd49e55887b5122d63403e0fe9e7a3a63b6ed03c6"
2024-12-18T00:21:41.1515606Z 		},
2024-12-18T00:21:41.1515805Z 		{
2024-12-18T00:21:41.1516140Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1516553Z 			"size": 1932,
2024-12-18T00:21:41.1516952Z 			"digest": "sha256:9b763e24a7d42c731eb337815675c9478832146e73305559d0db1c88a0731492"
2024-12-18T00:21:41.1517402Z 		},
2024-12-18T00:21:41.1517600Z 		{
2024-12-18T00:21:41.1517938Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1518359Z 			"size": 196810750,
2024-12-18T00:21:41.1518779Z 			"digest": "sha256:f6ef3d4c814e330a230273579dc978970563774df75f73fba78181ddcf6ab98e"
2024-12-18T00:21:41.1519242Z 		},
2024-12-18T00:21:41.1519444Z 		{
2024-12-18T00:21:41.1519779Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1520201Z 			"size": 165,
2024-12-18T00:21:41.1520607Z 			"digest": "sha256:06914b00eb49b8005245cba4a631c34d0a2aaf8bf6291bc6477705206de11a06"
2024-12-18T00:21:41.1521064Z 		},
2024-12-18T00:21:41.1522018Z 		{
2024-12-18T00:21:41.1522357Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1522779Z 			"size": 7943,
2024-12-18T00:21:41.1523186Z 			"digest": "sha256:724553524224508e03de9422ca47b1a1c047e53cd305c2a1344147bb0d82d1d0"
2024-12-18T00:21:41.1523644Z 		},
2024-12-18T00:21:41.1523840Z 		{
2024-12-18T00:21:41.1524177Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1524595Z 			"size": 8070,
2024-12-18T00:21:41.1525019Z 			"digest": "sha256:47d9fb6dae26902b0fdb744916689adbd42add22a476d1c5ec89ae4e30b91e39"
2024-12-18T00:21:41.1525504Z 		},
2024-12-18T00:21:41.1525706Z 		{
2024-12-18T00:21:41.1526038Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1526461Z 			"size": 303,
2024-12-18T00:21:41.1526890Z 			"digest": "sha256:acce7b690dcb2fde77ba3ce65df70452fc7c98de73a42fe72652e477f86fc62d"
2024-12-18T00:21:41.1527371Z 		},
2024-12-18T00:21:41.1527574Z 		{
2024-12-18T00:21:41.1527918Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1528338Z 			"size": 7633763,
2024-12-18T00:21:41.1528774Z 			"digest": "sha256:58dc9f64def75bcb2abecdf39b798c3800e837d37da971e766a05731ea5537c2"
2024-12-18T00:21:41.1529258Z 		},
2024-12-18T00:21:41.1529457Z 		{
2024-12-18T00:21:41.1529793Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1530216Z 			"size": 108,
2024-12-18T00:21:41.1530636Z 			"digest": "sha256:94dfe5ef1b7e91ebd6a1a5acc1d7732e051334d5102d8a69c509c1d2f6e95598"
2024-12-18T00:21:41.1531240Z 		},
2024-12-18T00:21:41.1531449Z 		{
2024-12-18T00:21:41.1531789Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1532212Z 			"size": 54145665,
2024-12-18T00:21:41.1532629Z 			"digest": "sha256:40a2c9135b0683a7e2432748a9a6ff19926f6a2a49722d844c07e307320f7362"
2024-12-18T00:21:41.1533088Z 		},
2024-12-18T00:21:41.1533294Z 		{
2024-12-18T00:21:41.1533639Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1534061Z 			"size": 484,
2024-12-18T00:21:41.1534472Z 			"digest": "sha256:8839d6b0c8218eccb426367e5bfe066b172b8c6059bae8473e915e11c4e582a9"
2024-12-18T00:21:41.1534940Z 		},
2024-12-18T00:21:41.1535149Z 		{
2024-12-18T00:21:41.1535489Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1535911Z 			"size": 1374859271,
2024-12-18T00:21:41.1536335Z 			"digest": "sha256:1c588f0a9e97f3e192252a179d987d1ba1e5201ba4b6e74c429074e9e34df1b2"
2024-12-18T00:21:41.1536800Z 		},
2024-12-18T00:21:41.1537014Z 		{
2024-12-18T00:21:41.1537356Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1537779Z 			"size": 106,
2024-12-18T00:21:41.1538246Z 			"digest": "sha256:6727e7e279360b8b66f0137a15b357efede06e69bfd0800ebad6ed7c8ed71df4"
2024-12-18T00:21:41.1538721Z 		},
2024-12-18T00:21:41.1538927Z 		{
2024-12-18T00:21:41.1539265Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1539702Z 			"size": 568,
2024-12-18T00:21:41.1540112Z 			"digest": "sha256:3a55100d1957b103c43e314deb55697c459aa6fa225f91e7b1c69bc4c01dca4e"
2024-12-18T00:21:41.1540578Z 		},
2024-12-18T00:21:41.1540782Z 		{
2024-12-18T00:21:41.1541129Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1541559Z 			"size": 303706071,
2024-12-18T00:21:41.1541990Z 			"digest": "sha256:3d6f160d335161c5321e13ce2823ec68379a6a3401b07bd7378602a69c85b80c"
2024-12-18T00:21:41.1542456Z 		},
2024-12-18T00:21:41.1542669Z 		{
2024-12-18T00:21:41.1543011Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1543446Z 			"size": 111,
2024-12-18T00:21:41.1543878Z 			"digest": "sha256:b2a8bb947a87562144fe67a39c280d4dee50cbe9d8ff349140f5db6c377517e6"
2024-12-18T00:21:41.1544361Z 		},
2024-12-18T00:21:41.1544568Z 		{
2024-12-18T00:21:41.1544934Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1545400Z 			"size": 529,
2024-12-18T00:21:41.1545906Z 			"digest": "sha256:69a74b2360f4ede95654274ce7ef6cc1597a7234cc17e86d5cdf4c98286d97c3"
2024-12-18T00:21:41.1546375Z 		},
2024-12-18T00:21:41.1546593Z 		{
2024-12-18T00:21:41.1546940Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1547367Z 			"size": 48289052,
2024-12-18T00:21:41.1547797Z 			"digest": "sha256:f9e12a253d89c16863b9d4d90138ffa673447c9b7f27afcba54a2a679e58b5ab"
2024-12-18T00:21:41.1548276Z 		},
2024-12-18T00:21:41.1548491Z 		{
2024-12-18T00:21:41.1548835Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1549267Z 			"size": 106,
2024-12-18T00:21:41.1549678Z 			"digest": "sha256:8fdb06d51b96d538045b9566a381de3dc25e59d1adb9d0a4e567afcb3213f676"
2024-12-18T00:21:41.1550158Z 		},
2024-12-18T00:21:41.1550373Z 		{
2024-12-18T00:21:41.1550713Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1551133Z 			"size": 32,
2024-12-18T00:21:41.1551549Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-12-18T00:21:41.1552025Z 		},
2024-12-18T00:21:41.1552238Z 		{
2024-12-18T00:21:41.1552584Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1553008Z 			"size": 32,
2024-12-18T00:21:41.1553424Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-12-18T00:21:41.1553907Z 		},
2024-12-18T00:21:41.1554123Z 		{
2024-12-18T00:21:41.1554473Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1554978Z 			"size": 32,
2024-12-18T00:21:41.1555394Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-12-18T00:21:41.1555881Z 		},
2024-12-18T00:21:41.1556100Z 		{
2024-12-18T00:21:41.1556451Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:41.1556876Z 			"size": 32,
2024-12-18T00:21:41.1557303Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-12-18T00:21:41.1557793Z 		}
2024-12-18T00:21:41.1558009Z 	]
2024-12-18T00:21:41.1558229Z }
2024-12-18T00:21:41.1597238Z ##[group]Run set -eux
2024-12-18T00:21:41.1597536Z set -eux
2024-12-18T00:21:41.1598372Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin
2024-12-18T00:21:41.1608458Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:41.1608849Z env:
2024-12-18T00:21:41.1609096Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:41.1609375Z ##[endgroup]
2024-12-18T00:21:41.1641711Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token
2024-12-18T00:21:41.1642703Z + jq --raw-output .SecretString
2024-12-18T00:21:41.1643759Z + jq -r .docker_hub_readonly_token
2024-12-18T00:21:41.1645865Z + docker login --username pytorchbot --password-stdin
2024-12-18T00:21:41.7887200Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-12-18T00:21:41.7888555Z Configure a credential helper to remove this warning. See
2024-12-18T00:21:41.7889704Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-12-18T00:21:41.7890705Z 
2024-12-18T00:21:41.7890991Z Login Succeeded
2024-12-18T00:21:41.7989786Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/}
2024-12-18T00:21:41.7990161Z tag=${ECR_DOCKER_IMAGE##*/}
2024-12-18T00:21:41.7990571Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}"
2024-12-18T00:21:41.7999479Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:41.7999867Z env:
2024-12-18T00:21:41.8000108Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:41.8000977Z   ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:41.8001863Z ##[endgroup]
2024-12-18T00:21:41.8033000Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks-45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:41.8084028Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@release/2.6
2024-12-18T00:21:41.8084514Z with:
2024-12-18T00:21:41.8085387Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:41.8086362Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:41.8086774Z env:
2024-12-18T00:21:41.8087013Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:41.8087292Z ##[endgroup]
2024-12-18T00:21:41.8112456Z ##[group]Run set -x
2024-12-18T00:21:41.8112743Z set -x
2024-12-18T00:21:41.8112985Z set +e
2024-12-18T00:21:41.8113216Z 
2024-12-18T00:21:41.8113448Z login() {
2024-12-18T00:21:41.8113936Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2024-12-18T00:21:41.8114826Z }
2024-12-18T00:21:41.8115055Z 
2024-12-18T00:21:41.8115316Z retry () {
2024-12-18T00:21:41.8115606Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2024-12-18T00:21:41.8115924Z }
2024-12-18T00:21:41.8116172Z 
2024-12-18T00:21:41.8116427Z retry login "${DOCKER_REGISTRY}"
2024-12-18T00:21:41.8116741Z 
2024-12-18T00:21:41.8116976Z set -e
2024-12-18T00:21:41.8117333Z # ignore output since only exit code is used for conditional
2024-12-18T00:21:41.8118004Z # only pull docker image if it's not available locally
2024-12-18T00:21:41.8118549Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2024-12-18T00:21:41.8119049Z   retry docker pull "${DOCKER_IMAGE}"
2024-12-18T00:21:41.8119375Z fi
2024-12-18T00:21:41.8127382Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:41.8127777Z env:
2024-12-18T00:21:41.8128014Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:41.8128880Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:41.8129845Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:41.8130230Z ##[endgroup]
2024-12-18T00:21:41.8156303Z + set +e
2024-12-18T00:21:41.8156912Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:41.8157378Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:41.8160248Z + aws ecr get-login-password --region us-east-1
2024-12-18T00:21:41.8161549Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:42.3511108Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-12-18T00:21:42.3511720Z Configure a credential helper to remove this warning. See
2024-12-18T00:21:42.3512307Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-12-18T00:21:42.3512718Z 
2024-12-18T00:21:42.3512862Z Login Succeeded
2024-12-18T00:21:42.3538207Z + set -e
2024-12-18T00:21:42.3539391Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:42.3699242Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:42.3700920Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:42.6034653Z 45e1356b47a284893081276eff3000b7b534f3b1: Pulling from pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks
2024-12-18T00:21:42.6035774Z 80888bc6716f: Pulling fs layer
2024-12-18T00:21:42.6036204Z fbcd35dc5bc3: Pulling fs layer
2024-12-18T00:21:42.6037069Z c7232af9ae05: Pulling fs layer
2024-12-18T00:21:42.6037489Z db6cdef1932a: Pulling fs layer
2024-12-18T00:21:42.6037899Z 56dc85502937: Pulling fs layer
2024-12-18T00:21:42.6038310Z 30c0ea6140d0: Pulling fs layer
2024-12-18T00:21:42.6038738Z 71bdb1a72c2d: Pulling fs layer
2024-12-18T00:21:42.6039163Z 4829486be7c3: Pulling fs layer
2024-12-18T00:21:42.6039569Z 1f4e68d7b5e4: Pulling fs layer
2024-12-18T00:21:42.6039986Z 7c373e2d9b7e: Pulling fs layer
2024-12-18T00:21:42.6040396Z 622381141745: Pulling fs layer
2024-12-18T00:21:42.6040807Z f003d065e31e: Pulling fs layer
2024-12-18T00:21:42.6041228Z cd6ce7116815: Pulling fs layer
2024-12-18T00:21:42.6041631Z e843066a8f68: Pulling fs layer
2024-12-18T00:21:42.6042019Z bac50ca2930b: Pulling fs layer
2024-12-18T00:21:42.6042416Z 57e52c6751f4: Pulling fs layer
2024-12-18T00:21:42.6042808Z a9c5c14d77c0: Pulling fs layer
2024-12-18T00:21:42.6043191Z 71bdb1a72c2d: Waiting
2024-12-18T00:21:42.6043547Z b86099b8619a: Pulling fs layer
2024-12-18T00:21:42.6043923Z 7c373e2d9b7e: Waiting
2024-12-18T00:21:42.6044281Z 3c8ea8a600ff: Pulling fs layer
2024-12-18T00:21:42.6044683Z bebd539a1d59: Pulling fs layer
2024-12-18T00:21:42.6045115Z 292b18ec2dab: Pulling fs layer
2024-12-18T00:21:42.6045495Z 4829486be7c3: Waiting
2024-12-18T00:21:42.6045832Z 1f4e68d7b5e4: Waiting
2024-12-18T00:21:42.6046175Z 76e3817b4b4c: Pulling fs layer
2024-12-18T00:21:42.6046570Z 50b1f245e9f5: Pulling fs layer
2024-12-18T00:21:42.6047182Z ad93cf5205f2: Pulling fs layer
2024-12-18T00:21:42.6047473Z 8d17c59eaf7d: Pulling fs layer
2024-12-18T00:21:42.6047759Z b36a8ccf5c8c: Pulling fs layer
2024-12-18T00:21:42.6048040Z aa0a3b87e9e3: Pulling fs layer
2024-12-18T00:21:42.6048323Z 72a632052c5b: Pulling fs layer
2024-12-18T00:21:42.6048609Z ceed6316bc24: Pulling fs layer
2024-12-18T00:21:42.6048894Z d0ca0f491f80: Pulling fs layer
2024-12-18T00:21:42.6049179Z cf02c706cd56: Pulling fs layer
2024-12-18T00:21:42.6049453Z 16073d3f107d: Pulling fs layer
2024-12-18T00:21:42.6049742Z d234faf2f1bf: Pulling fs layer
2024-12-18T00:21:42.6050019Z 622381141745: Waiting
2024-12-18T00:21:42.6050274Z 3f0ade86af1a: Pulling fs layer
2024-12-18T00:21:42.6050556Z 56dc85502937: Waiting
2024-12-18T00:21:42.6050812Z 034b6a03f5f7: Pulling fs layer
2024-12-18T00:21:42.6051087Z 30c0ea6140d0: Waiting
2024-12-18T00:21:42.6051339Z 76e3817b4b4c: Waiting
2024-12-18T00:21:42.6051597Z 21f8f066852d: Pulling fs layer
2024-12-18T00:21:42.6051878Z 9dc1f315a979: Pulling fs layer
2024-12-18T00:21:42.6052173Z cd6a6063178b: Pulling fs layer
2024-12-18T00:21:42.6052462Z 2116738f4ea9: Pulling fs layer
2024-12-18T00:21:42.6052740Z cd6ce7116815: Waiting
2024-12-18T00:21:42.6052992Z bac50ca2930b: Waiting
2024-12-18T00:21:42.6053234Z f003d065e31e: Waiting
2024-12-18T00:21:42.6053478Z 57e52c6751f4: Waiting
2024-12-18T00:21:42.6053734Z d7fa134316d9: Pulling fs layer
2024-12-18T00:21:42.6054026Z af03dc3fd15e: Pulling fs layer
2024-12-18T00:21:42.6054304Z e843066a8f68: Waiting
2024-12-18T00:21:42.6054547Z 292b18ec2dab: Waiting
2024-12-18T00:21:42.6054810Z db6cdef1932a: Waiting
2024-12-18T00:21:42.6055070Z fd9de3dc4258: Pulling fs layer
2024-12-18T00:21:42.6055349Z ceed6316bc24: Waiting
2024-12-18T00:21:42.6055608Z b85093661f77: Pulling fs layer
2024-12-18T00:21:42.6055876Z 50b1f245e9f5: Waiting
2024-12-18T00:21:42.6056141Z ab6ef6c2ac4d: Pulling fs layer
2024-12-18T00:21:42.6056432Z 56c896077120: Pulling fs layer
2024-12-18T00:21:42.6056707Z b86099b8619a: Waiting
2024-12-18T00:21:42.6056969Z d88e9c5e7c70: Pulling fs layer
2024-12-18T00:21:42.6057242Z b36a8ccf5c8c: Waiting
2024-12-18T00:21:42.6057492Z 16073d3f107d: Waiting
2024-12-18T00:21:42.6057750Z 3c8ea8a600ff: Waiting
2024-12-18T00:21:42.6058146Z 3f0ade86af1a: Waiting
2024-12-18T00:21:42.6058396Z d0ca0f491f80: Waiting
2024-12-18T00:21:42.6058647Z 53617ee83302: Pulling fs layer
2024-12-18T00:21:42.6058944Z 3edbde10d404: Pulling fs layer
2024-12-18T00:21:42.6059226Z 72a632052c5b: Waiting
2024-12-18T00:21:42.6059485Z f4038f7524c3: Pulling fs layer
2024-12-18T00:21:42.6059762Z 034b6a03f5f7: Waiting
2024-12-18T00:21:42.6060135Z cf02c706cd56: Waiting
2024-12-18T00:21:42.6060390Z 8d17c59eaf7d: Waiting
2024-12-18T00:21:42.6060634Z b85093661f77: Waiting
2024-12-18T00:21:42.6060887Z 4a51173f7757: Pulling fs layer
2024-12-18T00:21:42.6061153Z 56c896077120: Waiting
2024-12-18T00:21:42.6061408Z ed6a71d41274: Pulling fs layer
2024-12-18T00:21:42.6061695Z 389dc939e2a4: Pulling fs layer
2024-12-18T00:21:42.6061990Z 0341eaeed33d: Pulling fs layer
2024-12-18T00:21:42.6062283Z 537cdb2d3168: Pulling fs layer
2024-12-18T00:21:42.6062552Z aa0a3b87e9e3: Waiting
2024-12-18T00:21:42.6062806Z bebd539a1d59: Waiting
2024-12-18T00:21:42.6063067Z 76d4a0702497: Pulling fs layer
2024-12-18T00:21:42.6063352Z 4f4fb700ef54: Pulling fs layer
2024-12-18T00:21:42.6063635Z 3287a539d61e: Pulling fs layer
2024-12-18T00:21:42.6063914Z 9b763e24a7d4: Pulling fs layer
2024-12-18T00:21:42.6064190Z 3edbde10d404: Waiting
2024-12-18T00:21:42.6064455Z f6ef3d4c814e: Pulling fs layer
2024-12-18T00:21:42.6064738Z f4038f7524c3: Waiting
2024-12-18T00:21:42.6064998Z 06914b00eb49: Pulling fs layer
2024-12-18T00:21:42.6065268Z 4a51173f7757: Waiting
2024-12-18T00:21:42.6065523Z 724553524224: Pulling fs layer
2024-12-18T00:21:42.6065811Z 47d9fb6dae26: Pulling fs layer
2024-12-18T00:21:42.6066092Z 0341eaeed33d: Waiting
2024-12-18T00:21:42.6066365Z 9dc1f315a979: Waiting
2024-12-18T00:21:42.6066625Z acce7b690dcb: Pulling fs layer
2024-12-18T00:21:42.6066909Z 58dc9f64def7: Pulling fs layer
2024-12-18T00:21:42.6067292Z 94dfe5ef1b7e: Pulling fs layer
2024-12-18T00:21:42.6067615Z 537cdb2d3168: Waiting
2024-12-18T00:21:42.6067997Z 40a2c9135b06: Pulling fs layer
2024-12-18T00:21:42.6068344Z 8839d6b0c821: Pulling fs layer
2024-12-18T00:21:42.6068905Z ab6ef6c2ac4d: Waiting
2024-12-18T00:21:42.6069297Z 1c588f0a9e97: Pulling fs layer
2024-12-18T00:21:42.6069729Z 76d4a0702497: Waiting
2024-12-18T00:21:42.6070150Z 6727e7e27936: Pulling fs layer
2024-12-18T00:21:42.6087606Z 3287a539d61e: Waiting
2024-12-18T00:21:42.6087921Z 3a55100d1957: Pulling fs layer
2024-12-18T00:21:42.6088235Z 3d6f160d3351: Pulling fs layer
2024-12-18T00:21:42.6088535Z b2a8bb947a87: Pulling fs layer
2024-12-18T00:21:42.6088831Z 21f8f066852d: Waiting
2024-12-18T00:21:42.6089102Z 69a74b2360f4: Pulling fs layer
2024-12-18T00:21:42.6089396Z f9e12a253d89: Pulling fs layer
2024-12-18T00:21:42.6089700Z 8fdb06d51b96: Pulling fs layer
2024-12-18T00:21:42.6089992Z 47d9fb6dae26: Waiting
2024-12-18T00:21:42.6090255Z acce7b690dcb: Waiting
2024-12-18T00:21:42.6090520Z ad93cf5205f2: Waiting
2024-12-18T00:21:42.6090774Z f6ef3d4c814e: Waiting
2024-12-18T00:21:42.6091030Z 53617ee83302: Waiting
2024-12-18T00:21:42.6091285Z 58dc9f64def7: Waiting
2024-12-18T00:21:42.6091548Z 1c588f0a9e97: Waiting
2024-12-18T00:21:42.6091791Z 6727e7e27936: Waiting
2024-12-18T00:21:42.6092042Z f9e12a253d89: Waiting
2024-12-18T00:21:42.6092300Z ed6a71d41274: Waiting
2024-12-18T00:21:42.6092556Z 3a55100d1957: Waiting
2024-12-18T00:21:42.6092809Z 3d6f160d3351: Waiting
2024-12-18T00:21:42.6093056Z b2a8bb947a87: Waiting
2024-12-18T00:21:42.6093316Z 69a74b2360f4: Waiting
2024-12-18T00:21:42.6093566Z 06914b00eb49: Waiting
2024-12-18T00:21:42.6093828Z d7fa134316d9: Waiting
2024-12-18T00:21:42.6094083Z 40a2c9135b06: Waiting
2024-12-18T00:21:42.6094328Z 8839d6b0c821: Waiting
2024-12-18T00:21:42.6094585Z 9b763e24a7d4: Waiting
2024-12-18T00:21:42.6094848Z fd9de3dc4258: Waiting
2024-12-18T00:21:42.6095096Z 94dfe5ef1b7e: Waiting
2024-12-18T00:21:42.6095337Z 389dc939e2a4: Waiting
2024-12-18T00:21:42.6095577Z 724553524224: Waiting
2024-12-18T00:21:42.6095810Z cd6a6063178b: Waiting
2024-12-18T00:21:42.7542030Z fbcd35dc5bc3: Verifying Checksum
2024-12-18T00:21:42.7542401Z fbcd35dc5bc3: Download complete
2024-12-18T00:21:42.8411279Z db6cdef1932a: Verifying Checksum
2024-12-18T00:21:42.8411623Z db6cdef1932a: Download complete
2024-12-18T00:21:42.9376887Z 56dc85502937: Verifying Checksum
2024-12-18T00:21:42.9377248Z 56dc85502937: Download complete
2024-12-18T00:21:43.0047187Z 80888bc6716f: Download complete
2024-12-18T00:21:43.1102927Z 71bdb1a72c2d: Verifying Checksum
2024-12-18T00:21:43.1103573Z 71bdb1a72c2d: Download complete
2024-12-18T00:21:43.1946721Z 4829486be7c3: Verifying Checksum
2024-12-18T00:21:43.1947081Z 4829486be7c3: Download complete
2024-12-18T00:21:43.2801625Z 1f4e68d7b5e4: Verifying Checksum
2024-12-18T00:21:43.2802284Z 1f4e68d7b5e4: Download complete
2024-12-18T00:21:43.2909606Z c7232af9ae05: Verifying Checksum
2024-12-18T00:21:43.2910209Z c7232af9ae05: Download complete
2024-12-18T00:21:43.4193623Z 622381141745: Verifying Checksum
2024-12-18T00:21:43.4194181Z 622381141745: Download complete
2024-12-18T00:21:43.5328950Z f003d065e31e: Verifying Checksum
2024-12-18T00:21:43.5329319Z f003d065e31e: Download complete
2024-12-18T00:21:44.0778369Z 80888bc6716f: Pull complete
2024-12-18T00:21:44.3694424Z fbcd35dc5bc3: Pull complete
2024-12-18T00:21:45.0956717Z c7232af9ae05: Pull complete
2024-12-18T00:21:45.1175643Z db6cdef1932a: Pull complete
2024-12-18T00:21:45.1389203Z 56dc85502937: Pull complete
2024-12-18T00:21:46.0727307Z cd6ce7116815: Verifying Checksum
2024-12-18T00:21:46.0727816Z cd6ce7116815: Download complete
2024-12-18T00:21:46.1917465Z e843066a8f68: Verifying Checksum
2024-12-18T00:21:46.1917811Z e843066a8f68: Download complete
2024-12-18T00:21:46.2810479Z bac50ca2930b: Verifying Checksum
2024-12-18T00:21:46.2810815Z bac50ca2930b: Download complete
2024-12-18T00:21:46.3727550Z 57e52c6751f4: Verifying Checksum
2024-12-18T00:21:46.3728771Z 57e52c6751f4: Download complete
2024-12-18T00:21:47.4378616Z a9c5c14d77c0: Verifying Checksum
2024-12-18T00:21:47.4378987Z a9c5c14d77c0: Download complete
2024-12-18T00:21:47.5330028Z b86099b8619a: Verifying Checksum
2024-12-18T00:21:47.5330387Z b86099b8619a: Download complete
2024-12-18T00:21:47.6213318Z 3c8ea8a600ff: Verifying Checksum
2024-12-18T00:21:47.6213899Z 3c8ea8a600ff: Download complete
2024-12-18T00:21:47.7307703Z bebd539a1d59: Verifying Checksum
2024-12-18T00:21:47.7308206Z bebd539a1d59: Download complete
2024-12-18T00:21:47.8302997Z 292b18ec2dab: Download complete
2024-12-18T00:21:56.7988192Z 30c0ea6140d0: Verifying Checksum
2024-12-18T00:21:56.7988710Z 30c0ea6140d0: Download complete
2024-12-18T00:21:56.9198548Z 50b1f245e9f5: Verifying Checksum
2024-12-18T00:21:56.9198917Z 50b1f245e9f5: Download complete
2024-12-18T00:21:57.0175767Z ad93cf5205f2: Download complete
2024-12-18T00:21:57.1738945Z 8d17c59eaf7d: Verifying Checksum
2024-12-18T00:21:57.1739327Z 8d17c59eaf7d: Download complete
2024-12-18T00:21:57.2842804Z b36a8ccf5c8c: Download complete
2024-12-18T00:21:57.3790691Z aa0a3b87e9e3: Download complete
2024-12-18T00:21:58.6614884Z 72a632052c5b: Verifying Checksum
2024-12-18T00:21:58.6615298Z 72a632052c5b: Download complete
2024-12-18T00:21:58.7494627Z ceed6316bc24: Verifying Checksum
2024-12-18T00:21:58.7495104Z ceed6316bc24: Download complete
2024-12-18T00:21:58.8456010Z d0ca0f491f80: Verifying Checksum
2024-12-18T00:21:58.8456421Z d0ca0f491f80: Download complete
2024-12-18T00:21:58.9694546Z cf02c706cd56: Download complete
2024-12-18T00:21:59.0562755Z 16073d3f107d: Download complete
2024-12-18T00:21:59.1483761Z d234faf2f1bf: Download complete
2024-12-18T00:22:03.5365689Z 3f0ade86af1a: Verifying Checksum
2024-12-18T00:22:03.5366195Z 3f0ade86af1a: Download complete
2024-12-18T00:22:03.6755784Z 034b6a03f5f7: Verifying Checksum
2024-12-18T00:22:03.6756146Z 034b6a03f5f7: Download complete
2024-12-18T00:22:03.7891552Z 21f8f066852d: Verifying Checksum
2024-12-18T00:22:03.7892021Z 21f8f066852d: Download complete
2024-12-18T00:22:04.2038477Z 9dc1f315a979: Verifying Checksum
2024-12-18T00:22:04.2039088Z 9dc1f315a979: Download complete
2024-12-18T00:22:04.2905992Z cd6a6063178b: Download complete
2024-12-18T00:22:04.3740259Z 2116738f4ea9: Verifying Checksum
2024-12-18T00:22:04.3740703Z 2116738f4ea9: Download complete
2024-12-18T00:22:04.6593694Z d7fa134316d9: Verifying Checksum
2024-12-18T00:22:04.6594110Z d7fa134316d9: Download complete
2024-12-18T00:22:04.7480101Z af03dc3fd15e: Verifying Checksum
2024-12-18T00:22:04.7480573Z af03dc3fd15e: Download complete
2024-12-18T00:22:04.8301124Z fd9de3dc4258: Verifying Checksum
2024-12-18T00:22:04.8301851Z fd9de3dc4258: Download complete
2024-12-18T00:22:04.9072974Z b85093661f77: Verifying Checksum
2024-12-18T00:22:04.9073364Z b85093661f77: Download complete
2024-12-18T00:22:08.8474356Z 30c0ea6140d0: Pull complete
2024-12-18T00:22:08.9737248Z 71bdb1a72c2d: Pull complete
2024-12-18T00:22:09.1973154Z 4829486be7c3: Pull complete
2024-12-18T00:22:09.4424099Z 1f4e68d7b5e4: Pull complete
2024-12-18T00:22:09.6306237Z 7c373e2d9b7e: Verifying Checksum
2024-12-18T00:22:09.6306698Z 7c373e2d9b7e: Download complete
2024-12-18T00:22:09.7626816Z 56c896077120: Verifying Checksum
2024-12-18T00:22:09.7627296Z 56c896077120: Download complete
2024-12-18T00:22:09.8373447Z d88e9c5e7c70: Verifying Checksum
2024-12-18T00:22:09.8373934Z d88e9c5e7c70: Download complete
2024-12-18T00:22:10.3672165Z 53617ee83302: Verifying Checksum
2024-12-18T00:22:10.3672585Z 53617ee83302: Download complete
2024-12-18T00:22:10.4678662Z 3edbde10d404: Verifying Checksum
2024-12-18T00:22:10.4679045Z 3edbde10d404: Download complete
2024-12-18T00:22:10.5496374Z f4038f7524c3: Verifying Checksum
2024-12-18T00:22:10.5496830Z f4038f7524c3: Download complete
2024-12-18T00:22:10.6343827Z 4a51173f7757: Verifying Checksum
2024-12-18T00:22:10.6344184Z 4a51173f7757: Download complete
2024-12-18T00:22:10.7645878Z ed6a71d41274: Download complete
2024-12-18T00:22:17.3238153Z 76e3817b4b4c: Verifying Checksum
2024-12-18T00:22:17.3238628Z 76e3817b4b4c: Download complete
2024-12-18T00:22:17.4078250Z 0341eaeed33d: Download complete
2024-12-18T00:22:17.5005937Z 537cdb2d3168: Download complete
2024-12-18T00:22:17.5637101Z 76d4a0702497: Download complete
2024-12-18T00:22:17.5802081Z 4f4fb700ef54: Download complete
2024-12-18T00:22:17.6604550Z 3287a539d61e: Download complete
2024-12-18T00:22:17.7192523Z 9b763e24a7d4: Verifying Checksum
2024-12-18T00:22:17.7192933Z 9b763e24a7d4: Download complete
2024-12-18T00:22:20.0801177Z f6ef3d4c814e: Verifying Checksum
2024-12-18T00:22:20.0801642Z f6ef3d4c814e: Download complete
2024-12-18T00:22:20.1975277Z 06914b00eb49: Verifying Checksum
2024-12-18T00:22:20.1975688Z 06914b00eb49: Download complete
2024-12-18T00:22:20.3340278Z 724553524224: Verifying Checksum
2024-12-18T00:22:20.3340643Z 724553524224: Download complete
2024-12-18T00:22:20.4495290Z 47d9fb6dae26: Download complete
2024-12-18T00:22:20.5356134Z acce7b690dcb: Verifying Checksum
2024-12-18T00:22:20.5356584Z acce7b690dcb: Download complete
2024-12-18T00:22:20.7063861Z 58dc9f64def7: Verifying Checksum
2024-12-18T00:22:20.7064270Z 58dc9f64def7: Download complete
2024-12-18T00:22:20.7819032Z 94dfe5ef1b7e: Verifying Checksum
2024-12-18T00:22:20.7819406Z 94dfe5ef1b7e: Download complete
2024-12-18T00:22:21.3869440Z 40a2c9135b06: Download complete
2024-12-18T00:22:21.4573253Z 8839d6b0c821: Download complete
2024-12-18T00:22:57.0029632Z 1c588f0a9e97: Verifying Checksum
2024-12-18T00:22:57.0029993Z 1c588f0a9e97: Download complete
2024-12-18T00:22:57.0951731Z 6727e7e27936: Verifying Checksum
2024-12-18T00:22:57.0952081Z 6727e7e27936: Download complete
2024-12-18T00:22:57.1754672Z 3a55100d1957: Download complete
2024-12-18T00:23:05.7561487Z 3d6f160d3351: Verifying Checksum
2024-12-18T00:23:05.7561835Z 3d6f160d3351: Download complete
2024-12-18T00:23:05.9167780Z b2a8bb947a87: Download complete
2024-12-18T00:23:06.0244204Z 69a74b2360f4: Download complete
2024-12-18T00:23:07.6717873Z f9e12a253d89: Verifying Checksum
2024-12-18T00:23:07.6718371Z f9e12a253d89: Download complete
2024-12-18T00:23:07.7659722Z 8fdb06d51b96: Verifying Checksum
2024-12-18T00:23:07.7660095Z 8fdb06d51b96: Download complete
2024-12-18T00:23:36.0824820Z ab6ef6c2ac4d: Verifying Checksum
2024-12-18T00:23:36.0825322Z ab6ef6c2ac4d: Download complete
2024-12-18T00:23:40.4104249Z 389dc939e2a4: Verifying Checksum
2024-12-18T00:23:40.4104639Z 389dc939e2a4: Download complete
2024-12-18T00:23:52.3908802Z 7c373e2d9b7e: Pull complete
2024-12-18T00:23:52.6246820Z 622381141745: Pull complete
2024-12-18T00:23:52.8476869Z f003d065e31e: Pull complete
2024-12-18T00:24:00.4366823Z cd6ce7116815: Pull complete
2024-12-18T00:24:00.4680238Z e843066a8f68: Pull complete
2024-12-18T00:24:00.5232074Z bac50ca2930b: Pull complete
2024-12-18T00:24:00.5931723Z 57e52c6751f4: Pull complete
2024-12-18T00:24:02.8426357Z a9c5c14d77c0: Pull complete
2024-12-18T00:24:03.0321234Z b86099b8619a: Pull complete
2024-12-18T00:24:03.2402990Z 3c8ea8a600ff: Pull complete
2024-12-18T00:24:03.4171459Z bebd539a1d59: Pull complete
2024-12-18T00:24:03.6071714Z 292b18ec2dab: Pull complete
2024-12-18T00:24:56.6101090Z 76e3817b4b4c: Pull complete
2024-12-18T00:24:56.8411874Z 50b1f245e9f5: Pull complete
2024-12-18T00:24:57.0849518Z ad93cf5205f2: Pull complete
2024-12-18T00:24:57.2709948Z 8d17c59eaf7d: Pull complete
2024-12-18T00:24:57.4282665Z b36a8ccf5c8c: Pull complete
2024-12-18T00:24:57.4926588Z aa0a3b87e9e3: Pull complete
2024-12-18T00:24:59.9419242Z 72a632052c5b: Pull complete
2024-12-18T00:25:00.1022124Z ceed6316bc24: Pull complete
2024-12-18T00:25:00.3332478Z d0ca0f491f80: Pull complete
2024-12-18T00:25:00.5216605Z cf02c706cd56: Pull complete
2024-12-18T00:25:00.7006099Z 16073d3f107d: Pull complete
2024-12-18T00:25:00.8511356Z d234faf2f1bf: Pull complete
2024-12-18T00:25:09.1642404Z 3f0ade86af1a: Pull complete
2024-12-18T00:25:09.3277589Z 034b6a03f5f7: Pull complete
2024-12-18T00:25:09.4764927Z 21f8f066852d: Pull complete
2024-12-18T00:25:10.2289823Z 9dc1f315a979: Pull complete
2024-12-18T00:25:10.3477040Z cd6a6063178b: Pull complete
2024-12-18T00:25:10.4942893Z 2116738f4ea9: Pull complete
2024-12-18T00:25:10.8956555Z d7fa134316d9: Pull complete
2024-12-18T00:25:11.0853900Z af03dc3fd15e: Pull complete
2024-12-18T00:25:11.4909096Z fd9de3dc4258: Pull complete
2024-12-18T00:25:11.7000217Z b85093661f77: Pull complete
2024-12-18T00:25:43.2679928Z ab6ef6c2ac4d: Pull complete
2024-12-18T00:25:43.4058853Z 56c896077120: Pull complete
2024-12-18T00:25:43.4911170Z d88e9c5e7c70: Pull complete
2024-12-18T00:25:46.0445994Z 53617ee83302: Pull complete
2024-12-18T00:25:46.2715161Z 3edbde10d404: Pull complete
2024-12-18T00:25:46.5012952Z f4038f7524c3: Pull complete
2024-12-18T00:25:46.9283856Z 4a51173f7757: Pull complete
2024-12-18T00:25:47.1460685Z ed6a71d41274: Pull complete
2024-12-18T00:27:18.0902868Z 389dc939e2a4: Pull complete
2024-12-18T00:27:18.3313509Z 0341eaeed33d: Pull complete
2024-12-18T00:27:18.5662812Z 537cdb2d3168: Pull complete
2024-12-18T00:27:19.0168553Z 76d4a0702497: Pull complete
2024-12-18T00:27:19.2368765Z 4f4fb700ef54: Pull complete
2024-12-18T00:27:19.4604328Z 3287a539d61e: Pull complete
2024-12-18T00:27:19.6854802Z 9b763e24a7d4: Pull complete
2024-12-18T00:27:26.7229969Z f6ef3d4c814e: Pull complete
2024-12-18T00:27:26.9561998Z 06914b00eb49: Pull complete
2024-12-18T00:27:27.1915916Z 724553524224: Pull complete
2024-12-18T00:27:27.4267720Z 47d9fb6dae26: Pull complete
2024-12-18T00:27:27.6581298Z acce7b690dcb: Pull complete
2024-12-18T00:27:29.0601492Z 58dc9f64def7: Pull complete
2024-12-18T00:27:29.2841577Z 94dfe5ef1b7e: Pull complete
2024-12-18T00:27:30.9988854Z 40a2c9135b06: Pull complete
2024-12-18T00:27:31.2461104Z 8839d6b0c821: Pull complete
2024-12-18T00:27:43.6919587Z 1c588f0a9e97: Pull complete
2024-12-18T00:27:43.8400142Z 6727e7e27936: Pull complete
2024-12-18T00:27:43.9889945Z 3a55100d1957: Pull complete
2024-12-18T00:27:46.8458015Z 3d6f160d3351: Pull complete
2024-12-18T00:27:47.0867830Z b2a8bb947a87: Pull complete
2024-12-18T00:27:47.3142429Z 69a74b2360f4: Pull complete
2024-12-18T00:27:47.9548367Z f9e12a253d89: Pull complete
2024-12-18T00:27:48.1877446Z 8fdb06d51b96: Pull complete
2024-12-18T00:27:49.2464403Z Digest: sha256:ff80d4c93d18e8ab7a2501f0e62b7c40cf97aa83fc9702e88b9c2d02cdcbecdb
2024-12-18T00:27:49.2847116Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:27:49.3086106Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:27:49.3148265Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T00:27:49.3149197Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T00:27:49.3162134Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:27:49.3162521Z env:
2024-12-18T00:27:49.3162757Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:27:49.3163053Z ##[endgroup]
2024-12-18T00:27:49.3403631Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@release/2.6
2024-12-18T00:27:49.3404243Z with:
2024-12-18T00:27:49.3404479Z   driver-version: 550.54.15
2024-12-18T00:27:49.3404752Z env:
2024-12-18T00:27:49.3404977Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:27:49.3405241Z ##[endgroup]
2024-12-18T00:27:49.3542873Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2024-12-18T00:27:49.3543293Z with:
2024-12-18T00:27:49.3543523Z   timeout_minutes: 10
2024-12-18T00:27:49.3543800Z   max_attempts: 3
2024-12-18T00:27:49.3568645Z   command: # Is it disgusting to have a full shell script here in this github action? Sure
# But is it the best way to make it so that this action relies on nothing else? Absolutely
set -eou pipefail

DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID)
DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run"

install_nvidia_docker2_amzn2() {
    (
        set -x
        # Needed for yum-config-manager
        sudo yum install -y yum-utils
        if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then
          YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo"
        else
          # Amazon Linux 2
          YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo"
        fi

        sudo yum-config-manager --add-repo "${YUM_REPO_URL}"
        sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2
        sudo systemctl restart docker
    )
}

install_nvidia_docker2_ubuntu20() {
    (
        set -x
        # Install nvidia-driver package if not installed
        status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)"
        if [ ! $? = 0 ] || [ ! "$status" = installed ]; then
          sudo apt-get install -y nvidia-docker2 nvidia-container-toolkit-1.16.2
          sudo systemctl restart docker
        fi
    )
}

pre_install_nvidia_driver_amzn2() {
    (
        # Purge any nvidia driver installed from RHEL repo
        sudo yum remove -y nvidia-driver-latest-dkms
    )
}

install_nvidia_driver_common() {
    (
        # Try to gather more information about the runner and its existing NVIDIA driver if any
        echo "Before installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        HAS_NVIDIA_DRIVER=0
        # Check if NVIDIA driver has already been installed
        if [ -x "$(command -v nvidia-smi)" ]; then
            set +e
            # The driver exists, check its version next. Also check only the first GPU if there are more than one of them
            # so that the same driver version is not print over multiple lines
            INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
            NVIDIA_SMI_STATUS=$?

            if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing"
            elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing"
            else
                HAS_NVIDIA_DRIVER=1
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation"
            fi
            set -e
        fi

        if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then
            # CAUTION: this may need to be updated in future
            if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then
                  sudo yum groupinstall -y "Development Tools"
                  # ensure our kernel install is the same as our underlying kernel,
                  # groupinstall "Development Tools" has a habit of mismatching kernel headers
                  sudo yum install -y "kernel-devel-uname-r == $(uname -r)"
                  sudo modprobe backlight
            fi
            sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN"

            set +e
            sudo /bin/bash /tmp/nvidia_driver -s --no-drm
            NVIDIA_INSTALLATION_STATUS=$?

            RESET_GPU=0
            if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then
                sudo cat /var/log/nvidia-installer.log
                # Fail to install NVIDIA driver, try to reset the GPU
                RESET_GPU=1
            elif [ -x "$(command -v nvidia-smi)" ]; then
                # Check again if nvidia-smi works even if the driver installation completes successfully
                INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
                NVIDIA_SMI_STATUS=$?

                if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                    RESET_GPU=1
                fi
            fi

            if [ "$RESET_GPU" -eq 1 ]; then
                NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1)
                # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this
                # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388
                for PCI_ID in $NVIDIA_DEVICES; do
                    DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable)

                    echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)"
                    # This requires sudo permission of course
                    echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset
                    sleep 1
                done
            fi

            sudo rm -fv /tmp/nvidia_driver
            set -e
        fi
    )
}

post_install_nvidia_driver_common() {
    (
        sudo modprobe nvidia || true
        echo "After installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        (
            set +e

            nvidia-smi
            # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in
            # the case where the driver has already crashed as it still can get the driver version
            # and some basic information like the bus ID.  However, the rest of the information
            # would be missing (ERR!), for example:
            #
            # +-----------------------------------------------------------------------------+
            # | NVIDIA-SMI 525.89.02    Driver Version: 525.89.02    CUDA Version: 12.0     |
            # |-------------------------------+----------------------+----------------------+
            # | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
            # | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
            # |                               |                      |               MIG M. |
            # |===============================+======================+======================|
            # |   0  ERR!                Off  | 00000000:00:1E.0 Off |                 ERR! |
            # |ERR!  ERR! ERR!    ERR! / ERR! |   4184MiB / 23028MiB |    ERR!      Default |
            # |                               |                      |                 ERR! |
            # +-------------------------------+----------------------+----------------------+
            #
            # +-----------------------------------------------------------------------------+
            # | Processes:                                                                  |
            # |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
            # |        ID   ID                                                   Usage      |
            # |=============================================================================|
            # +-----------------------------------------------------------------------------+
            #
            # This should be reported as a failure instead as it will guarantee to fail when
            # Docker tries to run with --gpus all
            #
            # So, the correct check here is to query one of the missing piece of info like
            # GPU name, so that the command can fail accordingly
            nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0
            NVIDIA_SMI_STATUS=$?

            # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285
            if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then
                echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}"
            else
                echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}"
                exit ${NVIDIA_SMI_STATUS}
            fi
            set -e
        )
    )
}

install_nvidia_driver_amzn2() {
    (
        set -x
        pre_install_nvidia_driver_amzn2
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

install_nvidia_driver_ubuntu20() {
    (
        set -x
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

echo "== Installing nvidia driver ${DRIVER_FN} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_driver_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_driver_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac

# Install container toolkit based on distribution
echo "== Installing nvidia container toolkit for ${DISTRIBUTION} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_docker2_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_docker2_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac
echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}"

# Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with
# more than one GPUs. This just needs to be run once. The command fails
# on subsequent runs and complains that the mode is already on, but that's
# ok
sudo nvidia-persistenced || true
# This should show persistence mode ON
nvidia-smi

2024-12-18T00:27:49.3593456Z   retry_wait_seconds: 10
2024-12-18T00:27:49.3593751Z   polling_interval_seconds: 1
2024-12-18T00:27:49.3594044Z   warning_on_retry: true
2024-12-18T00:27:49.3594322Z   continue_on_error: false
2024-12-18T00:27:49.3594585Z env:
2024-12-18T00:27:49.3594809Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:27:49.3595092Z   DRIVER_VERSION: 550.54.15
2024-12-18T00:27:49.3595364Z ##[endgroup]
2024-12-18T00:27:49.4456690Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run ==
2024-12-18T00:27:49.4458292Z + pre_install_nvidia_driver_amzn2
2024-12-18T00:27:49.4458693Z + sudo yum remove -y nvidia-driver-latest-dkms
2024-12-18T00:27:49.9113775Z No match for argument: nvidia-driver-latest-dkms
2024-12-18T00:27:49.9114907Z No packages marked for removal.
2024-12-18T00:27:49.9178859Z Dependencies resolved.
2024-12-18T00:27:49.9188020Z Nothing to do.
2024-12-18T00:27:49.9188491Z Complete!
2024-12-18T00:27:50.0022049Z + install_nvidia_driver_common
2024-12-18T00:27:50.0037001Z + echo 'Before installing NVIDIA driver'
2024-12-18T00:27:50.0037536Z Before installing NVIDIA driver
2024-12-18T00:27:50.0037940Z + lspci
2024-12-18T00:27:50.1150487Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma]
2024-12-18T00:27:50.1151039Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2024-12-18T00:27:50.1151648Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
2024-12-18T00:27:50.1152216Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111
2024-12-18T00:27:50.1152740Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller
2024-12-18T00:27:50.1153313Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2024-12-18T00:27:50.1153836Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1)
2024-12-18T00:27:50.1154358Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller
2024-12-18T00:27:50.1155103Z + lsmod
2024-12-18T00:27:50.1196262Z Module                  Size  Used by
2024-12-18T00:27:50.1196617Z xt_conntrack           16384  1
2024-12-18T00:27:50.1196915Z nft_chain_nat          16384  3
2024-12-18T00:27:50.1197203Z xt_MASQUERADE          20480  1
2024-12-18T00:27:50.1197538Z nf_nat                 57344  2 nft_chain_nat,xt_MASQUERADE
2024-12-18T00:27:50.1197904Z nf_conntrack_netlink    57344  0
2024-12-18T00:27:50.1198335Z nf_conntrack          184320  4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE
2024-12-18T00:27:50.1198803Z nf_defrag_ipv6         24576  1 nf_conntrack
2024-12-18T00:27:50.1199148Z nf_defrag_ipv4         16384  1 nf_conntrack
2024-12-18T00:27:50.1199469Z xfrm_user              57344  1
2024-12-18T00:27:50.1199768Z xfrm_algo              16384  1 xfrm_user
2024-12-18T00:27:50.1200082Z xt_addrtype            16384  2
2024-12-18T00:27:50.1200368Z nft_compat             20480  4
2024-12-18T00:27:50.1200714Z nf_tables             311296  57 nft_compat,nft_chain_nat
2024-12-18T00:27:50.1201329Z nfnetlink              20480  4 nft_compat,nf_conntrack_netlink,nf_tables
2024-12-18T00:27:50.1201790Z br_netfilter           36864  0
2024-12-18T00:27:50.1202097Z bridge                323584  1 br_netfilter
2024-12-18T00:27:50.1202424Z stp                    16384  1 bridge
2024-12-18T00:27:50.1202740Z llc                    16384  2 bridge,stp
2024-12-18T00:27:50.1203046Z overlay               167936  0
2024-12-18T00:27:50.1203325Z tls                   114688  0
2024-12-18T00:27:50.1203606Z nls_ascii              16384  1
2024-12-18T00:27:50.1203890Z nls_cp437              20480  1
2024-12-18T00:27:50.1204170Z sunrpc                692224  1
2024-12-18T00:27:50.1204441Z vfat                   24576  1
2024-12-18T00:27:50.1204721Z fat                    86016  1 vfat
2024-12-18T00:27:50.1205013Z ena                   167936  0
2024-12-18T00:27:50.1205286Z i8042                  45056  0
2024-12-18T00:27:50.1205569Z serio                  28672  3 i8042
2024-12-18T00:27:50.1205880Z ghash_clmulni_intel    16384  0
2024-12-18T00:27:50.1206170Z button                 24576  0
2024-12-18T00:27:50.1206458Z sch_fq_codel           20480  17
2024-12-18T00:27:50.1206746Z dm_mod                188416  0
2024-12-18T00:27:50.1207025Z fuse                  163840  1
2024-12-18T00:27:50.1207295Z loop                   36864  0
2024-12-18T00:27:50.1207585Z configfs               57344  1
2024-12-18T00:27:50.1207877Z dax                    45056  1 dm_mod
2024-12-18T00:27:50.1208179Z dmi_sysfs              20480  0
2024-12-18T00:27:50.1208462Z crc32_pclmul           16384  0
2024-12-18T00:27:50.1208750Z crc32c_intel           24576  0
2024-12-18T00:27:50.1209043Z efivarfs               24576  1
2024-12-18T00:27:50.1209331Z + modinfo nvidia
2024-12-18T00:27:50.1215841Z filename:       /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko
2024-12-18T00:27:50.1216488Z alias:          char-major-195-*
2024-12-18T00:27:50.1216804Z version:        550.54.15
2024-12-18T00:27:50.1217078Z supported:      external
2024-12-18T00:27:50.1217398Z license:        NVIDIA
2024-12-18T00:27:50.1217751Z firmware:       nvidia/550.54.15/gsp_tu10x.bin
2024-12-18T00:27:50.1218215Z firmware:       nvidia/550.54.15/gsp_ga10x.bin
2024-12-18T00:27:50.1218575Z srcversion:     833721318DA517F0C2FEC97
2024-12-18T00:27:50.1218923Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2024-12-18T00:27:50.1219297Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2024-12-18T00:27:50.1219676Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2024-12-18T00:27:50.1220025Z depends:        i2c-core,drm
2024-12-18T00:27:50.1220312Z retpoline:      Y
2024-12-18T00:27:50.1220555Z name:           nvidia
2024-12-18T00:27:50.1220947Z vermagic:       6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 
2024-12-18T00:27:50.1221456Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2024-12-18T00:27:50.1221985Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2024-12-18T00:27:50.1222620Z parm:           NVreg_ResmanDebugLevel:int
2024-12-18T00:27:50.1222962Z parm:           NVreg_RmLogonRC:int
2024-12-18T00:27:50.1223301Z parm:           NVreg_ModifyDeviceFiles:int
2024-12-18T00:27:50.1223648Z parm:           NVreg_DeviceFileUID:int
2024-12-18T00:27:50.1223986Z parm:           NVreg_DeviceFileGID:int
2024-12-18T00:27:50.1224323Z parm:           NVreg_DeviceFileMode:int
2024-12-18T00:27:50.1224708Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2024-12-18T00:27:50.1225124Z parm:           NVreg_UsePageAttributeTable:int
2024-12-18T00:27:50.1225489Z parm:           NVreg_EnablePCIeGen3:int
2024-12-18T00:27:50.1225821Z parm:           NVreg_EnableMSI:int
2024-12-18T00:27:50.1226138Z parm:           NVreg_TCEBypassMode:int
2024-12-18T00:27:50.1226479Z parm:           NVreg_EnableStreamMemOPs:int
2024-12-18T00:27:50.1226877Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2024-12-18T00:27:50.1227303Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2024-12-18T00:27:50.1227851Z parm:           NVreg_EnableS0ixPowerManagement:int
2024-12-18T00:27:50.1228295Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2024-12-18T00:27:50.1228731Z parm:           NVreg_DynamicPowerManagement:int
2024-12-18T00:27:50.1229188Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2024-12-18T00:27:50.1229642Z parm:           NVreg_EnableGpuFirmware:int
2024-12-18T00:27:50.1230011Z parm:           NVreg_EnableGpuFirmwareLogs:int
2024-12-18T00:27:50.1230415Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2024-12-18T00:27:50.1230828Z parm:           NVreg_EnableUserNUMAManagement:int
2024-12-18T00:27:50.1231197Z parm:           NVreg_MemoryPoolSize:int
2024-12-18T00:27:50.1231584Z parm:           NVreg_KMallocHeapMaxSize:int
2024-12-18T00:27:50.1231967Z parm:           NVreg_VMallocHeapMaxSize:int
2024-12-18T00:27:50.1232328Z parm:           NVreg_IgnoreMMIOCheck:int
2024-12-18T00:27:50.1232669Z parm:           NVreg_NvLinkDisable:int
2024-12-18T00:27:50.1233055Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2024-12-18T00:27:50.1233451Z parm:           NVreg_RegisterPCIDriver:int
2024-12-18T00:27:50.1233816Z parm:           NVreg_EnableResizableBar:int
2024-12-18T00:27:50.1234188Z parm:           NVreg_EnableDbgBreakpoint:int
2024-12-18T00:27:50.1234565Z parm:           NVreg_EnableNonblockingOpen:int
2024-12-18T00:27:50.1234933Z parm:           NVreg_RegistryDwords:charp
2024-12-18T00:27:50.1235303Z parm:           NVreg_RegistryDwordsPerDevice:charp
2024-12-18T00:27:50.1235670Z parm:           NVreg_RmMsg:charp
2024-12-18T00:27:50.1235986Z parm:           NVreg_GpuBlacklist:charp
2024-12-18T00:27:50.1236345Z parm:           NVreg_TemporaryFilePath:charp
2024-12-18T00:27:50.1236707Z parm:           NVreg_ExcludedGpus:charp
2024-12-18T00:27:50.1237050Z parm:           NVreg_DmaRemapPeerMmio:int
2024-12-18T00:27:50.1237417Z parm:           NVreg_RmNvlinkBandwidth:charp
2024-12-18T00:27:50.1237784Z parm:           NVreg_ImexChannelCount:int
2024-12-18T00:27:50.1238132Z parm:           rm_firmware_active:charp
2024-12-18T00:27:50.1238459Z + HAS_NVIDIA_DRIVER=0
2024-12-18T00:27:50.1238727Z ++ command -v nvidia-smi
2024-12-18T00:27:50.1239016Z + '[' -x /usr/bin/nvidia-smi ']'
2024-12-18T00:27:50.1239301Z + set +e
2024-12-18T00:27:50.1239643Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0
2024-12-18T00:27:52.3757124Z + INSTALLED_DRIVER_VERSION=550.54.15
2024-12-18T00:27:52.3757490Z + NVIDIA_SMI_STATUS=0
2024-12-18T00:27:52.3758320Z + '[' 0 -ne 0 ']'
2024-12-18T00:27:52.3758719Z + '[' 550.54.15 '!=' 550.54.15 ']'
2024-12-18T00:27:52.3759138Z + HAS_NVIDIA_DRIVER=1
2024-12-18T00:27:52.3759790Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation'
2024-12-18T00:27:52.3760310Z + set -e
2024-12-18T00:27:52.3760534Z + '[' 1 -eq 0 ']'
2024-12-18T00:27:52.3760960Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation
2024-12-18T00:27:52.3761970Z + post_install_nvidia_driver_common
2024-12-18T00:27:52.3762543Z + sudo modprobe nvidia
2024-12-18T00:27:52.4911445Z + echo 'After installing NVIDIA driver'
2024-12-18T00:27:52.4911790Z + lspci
2024-12-18T00:27:52.4912066Z After installing NVIDIA driver
2024-12-18T00:27:52.5027184Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma]
2024-12-18T00:27:52.5027722Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2024-12-18T00:27:52.5028314Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
2024-12-18T00:27:52.5028916Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111
2024-12-18T00:27:52.5029709Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller
2024-12-18T00:27:52.5030286Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2024-12-18T00:27:52.5030812Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1)
2024-12-18T00:27:52.5031631Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller
2024-12-18T00:27:52.5032074Z + lsmod
2024-12-18T00:27:52.5060561Z Module                  Size  Used by
2024-12-18T00:27:52.5060896Z nvidia_uvm           4706304  0
2024-12-18T00:27:52.5061197Z nvidia              54071296  1 nvidia_uvm
2024-12-18T00:27:52.5061512Z drm                   602112  1 nvidia
2024-12-18T00:27:52.5061842Z drm_panel_orientation_quirks    32768  1 drm
2024-12-18T00:27:52.5062322Z backlight              24576  1 drm
2024-12-18T00:27:52.5062812Z i2c_core              106496  2 nvidia,drm
2024-12-18T00:27:52.5063159Z xt_conntrack           16384  1
2024-12-18T00:27:52.5063449Z nft_chain_nat          16384  3
2024-12-18T00:27:52.5063732Z xt_MASQUERADE          20480  1
2024-12-18T00:27:52.5064058Z nf_nat                 57344  2 nft_chain_nat,xt_MASQUERADE
2024-12-18T00:27:52.5064408Z nf_conntrack_netlink    57344  0
2024-12-18T00:27:52.5064830Z nf_conntrack          184320  4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE
2024-12-18T00:27:52.5065305Z nf_defrag_ipv6         24576  1 nf_conntrack
2024-12-18T00:27:52.5065645Z nf_defrag_ipv4         16384  1 nf_conntrack
2024-12-18T00:27:52.5065964Z xfrm_user              57344  1
2024-12-18T00:27:52.5066247Z xfrm_algo              16384  1 xfrm_user
2024-12-18T00:27:52.5066559Z xt_addrtype            16384  2
2024-12-18T00:27:52.5066838Z nft_compat             20480  4
2024-12-18T00:27:52.5067174Z nf_tables             311296  57 nft_compat,nft_chain_nat
2024-12-18T00:27:52.5067611Z nfnetlink              20480  4 nft_compat,nf_conntrack_netlink,nf_tables
2024-12-18T00:27:52.5068012Z br_netfilter           36864  0
2024-12-18T00:27:52.5068317Z bridge                323584  1 br_netfilter
2024-12-18T00:27:52.5068638Z stp                    16384  1 bridge
2024-12-18T00:27:52.5068950Z llc                    16384  2 bridge,stp
2024-12-18T00:27:52.5069263Z overlay               167936  0
2024-12-18T00:27:52.5069533Z tls                   114688  0
2024-12-18T00:27:52.5069823Z nls_ascii              16384  1
2024-12-18T00:27:52.5070110Z nls_cp437              20480  1
2024-12-18T00:27:52.5070395Z sunrpc                692224  1
2024-12-18T00:27:52.5070667Z vfat                   24576  1
2024-12-18T00:27:52.5070937Z fat                    86016  1 vfat
2024-12-18T00:27:52.5071228Z ena                   167936  0
2024-12-18T00:27:52.5071496Z i8042                  45056  0
2024-12-18T00:27:52.5071779Z serio                  28672  3 i8042
2024-12-18T00:27:52.5072096Z ghash_clmulni_intel    16384  0
2024-12-18T00:27:52.5072415Z button                 24576  0
2024-12-18T00:27:52.5072694Z sch_fq_codel           20480  17
2024-12-18T00:27:52.5072981Z dm_mod                188416  0
2024-12-18T00:27:52.5073250Z fuse                  163840  1
2024-12-18T00:27:52.5073521Z loop                   36864  0
2024-12-18T00:27:52.5073791Z configfs               57344  1
2024-12-18T00:27:52.5074070Z dax                    45056  1 dm_mod
2024-12-18T00:27:52.5074372Z dmi_sysfs              20480  0
2024-12-18T00:27:52.5074858Z crc32_pclmul           16384  0
2024-12-18T00:27:52.5075138Z crc32c_intel           24576  0
2024-12-18T00:27:52.5075416Z efivarfs               24576  1
2024-12-18T00:27:52.5075700Z + modinfo nvidia
2024-12-18T00:27:52.5079071Z filename:       /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko
2024-12-18T00:27:52.5079584Z alias:          char-major-195-*
2024-12-18T00:27:52.5079885Z version:        550.54.15
2024-12-18T00:27:52.5080156Z supported:      external
2024-12-18T00:27:52.5080422Z license:        NVIDIA
2024-12-18T00:27:52.5080716Z firmware:       nvidia/550.54.15/gsp_tu10x.bin
2024-12-18T00:27:52.5081082Z firmware:       nvidia/550.54.15/gsp_ga10x.bin
2024-12-18T00:27:52.5081426Z srcversion:     833721318DA517F0C2FEC97
2024-12-18T00:27:52.5081772Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2024-12-18T00:27:52.5082137Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2024-12-18T00:27:52.5082502Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2024-12-18T00:27:52.5082970Z depends:        i2c-core,drm
2024-12-18T00:27:52.5083377Z retpoline:      Y
2024-12-18T00:27:52.5083744Z name:           nvidia
2024-12-18T00:27:52.5084202Z vermagic:       6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 
2024-12-18T00:27:52.5084708Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2024-12-18T00:27:52.5085176Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2024-12-18T00:27:52.5085620Z parm:           NVreg_ResmanDebugLevel:int
2024-12-18T00:27:52.5085951Z parm:           NVreg_RmLogonRC:int
2024-12-18T00:27:52.5086268Z parm:           NVreg_ModifyDeviceFiles:int
2024-12-18T00:27:52.5086606Z parm:           NVreg_DeviceFileUID:int
2024-12-18T00:27:52.5086935Z parm:           NVreg_DeviceFileGID:int
2024-12-18T00:27:52.5087267Z parm:           NVreg_DeviceFileMode:int
2024-12-18T00:27:52.5087650Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2024-12-18T00:27:52.5088069Z parm:           NVreg_UsePageAttributeTable:int
2024-12-18T00:27:52.5088429Z parm:           NVreg_EnablePCIeGen3:int
2024-12-18T00:27:52.5088750Z parm:           NVreg_EnableMSI:int
2024-12-18T00:27:52.5089062Z parm:           NVreg_TCEBypassMode:int
2024-12-18T00:27:52.5089406Z parm:           NVreg_EnableStreamMemOPs:int
2024-12-18T00:27:52.5089792Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2024-12-18T00:27:52.5090214Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2024-12-18T00:27:52.5090621Z parm:           NVreg_EnableS0ixPowerManagement:int
2024-12-18T00:27:52.5091053Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2024-12-18T00:27:52.5091490Z parm:           NVreg_DynamicPowerManagement:int
2024-12-18T00:27:52.5091932Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2024-12-18T00:27:52.5092367Z parm:           NVreg_EnableGpuFirmware:int
2024-12-18T00:27:52.5092728Z parm:           NVreg_EnableGpuFirmwareLogs:int
2024-12-18T00:27:52.5093140Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2024-12-18T00:27:52.5093533Z parm:           NVreg_EnableUserNUMAManagement:int
2024-12-18T00:27:52.5093901Z parm:           NVreg_MemoryPoolSize:int
2024-12-18T00:27:52.5094250Z parm:           NVreg_KMallocHeapMaxSize:int
2024-12-18T00:27:52.5094604Z parm:           NVreg_VMallocHeapMaxSize:int
2024-12-18T00:27:52.5094957Z parm:           NVreg_IgnoreMMIOCheck:int
2024-12-18T00:27:52.5095286Z parm:           NVreg_NvLinkDisable:int
2024-12-18T00:27:52.5095654Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2024-12-18T00:27:52.5096048Z parm:           NVreg_RegisterPCIDriver:int
2024-12-18T00:27:52.5096405Z parm:           NVreg_EnableResizableBar:int
2024-12-18T00:27:52.5096769Z parm:           NVreg_EnableDbgBreakpoint:int
2024-12-18T00:27:52.5097135Z parm:           NVreg_EnableNonblockingOpen:int
2024-12-18T00:27:52.5097498Z parm:           NVreg_RegistryDwords:charp
2024-12-18T00:27:52.5097870Z parm:           NVreg_RegistryDwordsPerDevice:charp
2024-12-18T00:27:52.5098477Z parm:           NVreg_RmMsg:charp
2024-12-18T00:27:52.5098798Z parm:           NVreg_GpuBlacklist:charp
2024-12-18T00:27:52.5099143Z parm:           NVreg_TemporaryFilePath:charp
2024-12-18T00:27:52.5099501Z parm:           NVreg_ExcludedGpus:charp
2024-12-18T00:27:52.5099843Z parm:           NVreg_DmaRemapPeerMmio:int
2024-12-18T00:27:52.5100198Z parm:           NVreg_RmNvlinkBandwidth:charp
2024-12-18T00:27:52.5100567Z parm:           NVreg_ImexChannelCount:int
2024-12-18T00:27:52.5100898Z parm:           rm_firmware_active:charp
2024-12-18T00:27:52.5101210Z + set +e
2024-12-18T00:27:52.5101433Z + nvidia-smi
2024-12-18T00:27:54.0810589Z Wed Dec 18 00:27:54 2024       
2024-12-18T00:27:54.0811024Z +-----------------------------------------------------------------------------------------+
2024-12-18T00:27:54.0811566Z | NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
2024-12-18T00:27:54.0812108Z |-----------------------------------------+------------------------+----------------------+
2024-12-18T00:27:54.0813757Z | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
2024-12-18T00:27:54.0814340Z | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
2024-12-18T00:27:54.0814980Z |                                         |                        |               MIG M. |
2024-12-18T00:27:54.0815341Z |=========================================+========================+======================|
2024-12-18T00:27:54.0910177Z |   0  NVIDIA A10G                    Off |   00000000:00:1E.0 Off |                    0 |
2024-12-18T00:27:54.0910667Z |  0%   25C    P0             55W /  300W |       0MiB /  23028MiB |      5%      Default |
2024-12-18T00:27:54.0911121Z |                                         |                        |                  N/A |
2024-12-18T00:27:54.0911747Z +-----------------------------------------+------------------------+----------------------+
2024-12-18T00:27:54.0912515Z                                                                                          
2024-12-18T00:27:54.0912973Z +-----------------------------------------------------------------------------------------+
2024-12-18T00:27:54.0913453Z | Processes:                                                                              |
2024-12-18T00:27:54.0913953Z |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
2024-12-18T00:27:54.0914589Z |        ID   ID                                                               Usage      |
2024-12-18T00:27:54.0914974Z |=========================================================================================|
2024-12-18T00:27:54.0915576Z |  No running processes found                                                             |
2024-12-18T00:27:54.0916084Z +-----------------------------------------------------------------------------------------+
2024-12-18T00:27:54.7099671Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0
2024-12-18T00:27:56.2726376Z NVIDIA A10G
2024-12-18T00:27:56.7346174Z + NVIDIA_SMI_STATUS=0
2024-12-18T00:27:56.7346486Z + '[' 0 -eq 0 ']'
2024-12-18T00:27:56.7346764Z + echo 'INFO: Ignoring allowed status 0'
2024-12-18T00:27:56.7347093Z + set -e
2024-12-18T00:27:56.7347323Z INFO: Ignoring allowed status 0
2024-12-18T00:27:56.7354722Z == Installing nvidia container toolkit for amzn2023 ==
2024-12-18T00:27:56.7358454Z + sudo yum install -y yum-utils
2024-12-18T00:27:57.1613997Z Last metadata expiration check: 0:31:41 ago on Tue Dec 17 23:56:16 2024.
2024-12-18T00:27:57.1839895Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed.
2024-12-18T00:27:57.2174201Z Dependencies resolved.
2024-12-18T00:27:57.2314130Z Nothing to do.
2024-12-18T00:27:57.2314641Z Complete!
2024-12-18T00:27:57.3349451Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]]
2024-12-18T00:27:57.3350069Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
2024-12-18T00:27:57.3351362Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
2024-12-18T00:27:57.5799626Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
2024-12-18T00:27:57.6535002Z + sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2
2024-12-18T00:27:58.1357188Z nvidia-container-toolkit                         10 kB/s | 833  B     00:00    
2024-12-18T00:27:58.1588631Z Package nvidia-docker2-2.14.0-1.noarch is already installed.
2024-12-18T00:27:58.1923624Z Dependencies resolved.
2024-12-18T00:27:58.2060450Z ================================================================================
2024-12-18T00:27:58.2061101Z  Package                       Arch   Version    Repository                Size
2024-12-18T00:27:58.2061682Z ================================================================================
2024-12-18T00:27:58.2062200Z Downgrading:
2024-12-18T00:27:58.2063196Z  nvidia-container-toolkit      x86_64 1.16.2-1   nvidia-container-toolkit 1.2 M
2024-12-18T00:27:58.2064202Z  nvidia-container-toolkit-base x86_64 1.16.2-1   nvidia-container-toolkit 5.6 M
2024-12-18T00:27:58.2064812Z 
2024-12-18T00:27:58.2064977Z Transaction Summary
2024-12-18T00:27:58.2065398Z ================================================================================
2024-12-18T00:27:58.2065914Z Downgrade  2 Packages
2024-12-18T00:27:58.2066166Z 
2024-12-18T00:27:58.2066325Z Total download size: 6.8 M
2024-12-18T00:27:58.2066698Z Downloading Packages:
2024-12-18T00:27:58.2909587Z (1/2): nvidia-container-toolkit-1.16.2-1.x86_64  15 MB/s | 1.2 MB     00:00    
2024-12-18T00:27:58.4169492Z (2/2): nvidia-container-toolkit-base-1.16.2-1.x  27 MB/s | 5.6 MB     00:00    
2024-12-18T00:27:58.4180338Z --------------------------------------------------------------------------------
2024-12-18T00:27:58.4183709Z Total                                            32 MB/s | 6.8 MB     00:00     
2024-12-18T00:27:58.4186435Z Running transaction check
2024-12-18T00:27:58.4282690Z Transaction check succeeded.
2024-12-18T00:27:58.4283324Z Running transaction test
2024-12-18T00:27:58.4576439Z Transaction test succeeded.
2024-12-18T00:27:58.4579713Z Running transaction
2024-12-18T00:27:59.1649860Z   Preparing        :                                                        1/1 
2024-12-18T00:27:59.3178649Z   Downgrading      : nvidia-container-toolkit-base-1.16.2-1.x86_64          1/4 
2024-12-18T00:27:59.3455348Z   Downgrading      : nvidia-container-toolkit-1.16.2-1.x86_64               2/4 
2024-12-18T00:27:59.3738383Z   Running scriptlet: nvidia-container-toolkit-1.16.2-1.x86_64               2/4 
2024-12-18T00:27:59.3739157Z   Cleanup          : nvidia-container-toolkit-1.17.3-1.x86_64               3/4 
2024-12-18T00:27:59.4085712Z   Running scriptlet: nvidia-container-toolkit-1.17.3-1.x86_64               3/4 
2024-12-18T00:27:59.4430475Z   Cleanup          : nvidia-container-toolkit-base-1.17.3-1.x86_64          4/4 
2024-12-18T00:28:49.4247657Z   Running scriptlet: nvidia-container-toolkit-1.16.2-1.x86_64               4/4 
2024-12-18T00:28:49.4250346Z   Verifying        : nvidia-container-toolkit-1.16.2-1.x86_64               1/4 
2024-12-18T00:28:49.4251169Z   Verifying        : nvidia-container-toolkit-1.17.3-1.x86_64               2/4 
2024-12-18T00:28:49.4251973Z   Verifying        : nvidia-container-toolkit-base-1.16.2-1.x86_64          3/4 
2024-12-18T00:28:49.5622304Z   Verifying        : nvidia-container-toolkit-base-1.17.3-1.x86_64          4/4================================================================================
2024-12-18T00:28:49.5623083Z WARNING:
2024-12-18T00:28:49.5623437Z   A newer release of "Amazon Linux" is available.
2024-12-18T00:28:49.5623782Z 
2024-12-18T00:28:49.5623931Z   Available Versions:
2024-12-18T00:28:49.5624191Z 
2024-12-18T00:28:49.5624340Z   Version 2023.6.20241010:
2024-12-18T00:28:49.5624813Z     Run the following command to upgrade to 2023.6.20241010:
2024-12-18T00:28:49.5625206Z 
2024-12-18T00:28:49.5625716Z       dnf upgrade --releasever=2023.6.20241010
2024-12-18T00:28:49.5626056Z 
2024-12-18T00:28:49.5626194Z     Release notes:
2024-12-18T00:28:49.5626825Z      https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241010.html
2024-12-18T00:28:49.5627443Z 
2024-12-18T00:28:49.5627598Z   Version 2023.6.20241028:
2024-12-18T00:28:49.5628104Z     Run the following command to upgrade to 2023.6.20241028:
2024-12-18T00:28:49.5628563Z 
2024-12-18T00:28:49.5628770Z       dnf upgrade --releasever=2023.6.20241028
2024-12-18T00:28:49.5629123Z 
2024-12-18T00:28:49.5629272Z     Release notes:
2024-12-18T00:28:49.5629955Z      https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241028.html
2024-12-18T00:28:49.5630591Z 
2024-12-18T00:28:49.5630755Z   Version 2023.6.20241031:
2024-12-18T00:28:49.5631260Z     Run the following command to upgrade to 2023.6.20241031:
2024-12-18T00:28:49.5631677Z 
2024-12-18T00:28:49.5631868Z       dnf upgrade --releasever=2023.6.20241031
2024-12-18T00:28:49.5632547Z 
2024-12-18T00:28:49.5632692Z     Release notes:
2024-12-18T00:28:49.5633344Z      https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241031.html
2024-12-18T00:28:49.5633946Z 
2024-12-18T00:28:49.5634114Z   Version 2023.6.20241111:
2024-12-18T00:28:49.5634611Z     Run the following command to upgrade to 2023.6.20241111:
2024-12-18T00:28:49.5635024Z 
2024-12-18T00:28:49.5635227Z       dnf upgrade --releasever=2023.6.20241111
2024-12-18T00:28:49.5635581Z 
2024-12-18T00:28:49.5635720Z     Release notes:
2024-12-18T00:28:49.5636372Z      https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241111.html
2024-12-18T00:28:49.5636980Z 
2024-12-18T00:28:49.5637133Z   Version 2023.6.20241121:
2024-12-18T00:28:49.5637648Z     Run the following command to upgrade to 2023.6.20241121:
2024-12-18T00:28:49.5638076Z 
2024-12-18T00:28:49.5638311Z       dnf upgrade --releasever=2023.6.20241121
2024-12-18T00:28:49.5638678Z 
2024-12-18T00:28:49.5638836Z     Release notes:
2024-12-18T00:28:49.5639498Z      https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241121.html
2024-12-18T00:28:49.5640116Z 
2024-12-18T00:28:49.5640276Z   Version 2023.6.20241212:
2024-12-18T00:28:49.5640783Z     Run the following command to upgrade to 2023.6.20241212:
2024-12-18T00:28:49.5641192Z 
2024-12-18T00:28:49.5641381Z       dnf upgrade --releasever=2023.6.20241212
2024-12-18T00:28:49.5641771Z 
2024-12-18T00:28:49.5641921Z     Release notes:
2024-12-18T00:28:49.5642364Z      https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241212.html
2024-12-18T00:28:49.5642759Z 
2024-12-18T00:28:49.5642885Z ================================================================================
2024-12-18T00:28:49.5890376Z  
2024-12-18T00:28:49.5890719Z 
2024-12-18T00:28:49.5890986Z Downgraded:
2024-12-18T00:28:49.5891381Z   nvidia-container-toolkit-1.16.2-1.x86_64                                      
2024-12-18T00:28:49.5892017Z   nvidia-container-toolkit-base-1.16.2-1.x86_64                                 
2024-12-18T00:28:49.5892393Z 
2024-12-18T00:28:49.5892496Z Complete!
2024-12-18T00:28:49.6322950Z + sudo systemctl restart docker
2024-12-18T00:28:54.8913445Z Wed Dec 18 00:28:54 2024       
2024-12-18T00:28:54.8914069Z +-----------------------------------------------------------------------------------------+
2024-12-18T00:28:54.8915096Z | NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
2024-12-18T00:28:54.8915856Z |-----------------------------------------+------------------------+----------------------+
2024-12-18T00:28:54.8916619Z | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
2024-12-18T00:28:54.8917421Z | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
2024-12-18T00:28:54.8918083Z |                                         |                        |               MIG M. |
2024-12-18T00:28:54.8918998Z |=========================================+========================+======================|
2024-12-18T00:28:54.9031074Z |   0  NVIDIA A10G                    On  |   00000000:00:1E.0 Off |                    0 |
2024-12-18T00:28:54.9031916Z |  0%   26C    P0             55W /  300W |       0MiB /  23028MiB |      5%      Default |
2024-12-18T00:28:54.9032508Z |                                         |                        |                  N/A |
2024-12-18T00:28:54.9033092Z +-----------------------------------------+------------------------+----------------------+
2024-12-18T00:28:54.9033689Z                                                                                          
2024-12-18T00:28:54.9034273Z +-----------------------------------------------------------------------------------------+
2024-12-18T00:28:54.9034926Z | Processes:                                                                              |
2024-12-18T00:28:54.9035610Z |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
2024-12-18T00:28:54.9036599Z |        ID   ID                                                               Usage      |
2024-12-18T00:28:54.9037115Z |=========================================================================================|
2024-12-18T00:28:54.9037743Z |  No running processes found                                                             |
2024-12-18T00:28:54.9038426Z +-----------------------------------------------------------------------------------------+
2024-12-18T00:28:55.4824268Z Command completed after 1 attempt(s).
2024-12-18T00:28:55.4918695Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84
2024-12-18T00:28:55.4919287Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84
2024-12-18T00:28:55.4919798Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2024-12-18T00:28:55.4920301Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}"
2024-12-18T00:28:55.4934059Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:28:55.4934455Z env:
2024-12-18T00:28:55.4934683Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:28:55.4935021Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:28:55.4935380Z ##[endgroup]
2024-12-18T00:28:55.7744019Z Defaulting to user installation because normal site-packages is not writeable
2024-12-18T00:28:56.1925175Z Collecting psutil==5.9.1
2024-12-18T00:28:56.2283380Z   Downloading psutil-5.9.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB)
2024-12-18T00:28:56.2768094Z Collecting nvidia-ml-py==11.525.84
2024-12-18T00:28:56.2821276Z   Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB)
2024-12-18T00:28:56.3619551Z Installing collected packages: psutil, nvidia-ml-py
2024-12-18T00:28:56.5163739Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1
2024-12-18T00:28:56.7103737Z Prepare all required actions
2024-12-18T00:28:56.7104660Z Getting action download info
2024-12-18T00:28:56.9975410Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6)
2024-12-18T00:28:57.2210678Z Download action repository 'actions/download-artifact@v4' (SHA:fa0a91b85d4f404e444e00e005971372dc801d16)
2024-12-18T00:28:57.5091527Z ##[group]Run ./.github/actions/download-build-artifacts
2024-12-18T00:28:57.5092699Z with:
2024-12-18T00:28:57.5093208Z   name: linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-12-18T00:28:57.5093562Z   s3-bucket: gha-artifacts
2024-12-18T00:28:57.5093839Z env:
2024-12-18T00:28:57.5094074Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:28:57.5094426Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:28:57.5094799Z ##[endgroup]
2024-12-18T00:28:57.5129018Z ##[group]Run seemethere/download-artifact-s3@v4
2024-12-18T00:28:57.5129370Z with:
2024-12-18T00:28:57.5129670Z   name: linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-12-18T00:28:57.5130028Z   s3-bucket: gha-artifacts
2024-12-18T00:28:57.5130308Z   region: us-east-1
2024-12-18T00:28:57.5130564Z env:
2024-12-18T00:28:57.5130817Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:28:57.5131185Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:28:57.5131554Z ##[endgroup]
2024-12-18T00:28:57.9725011Z (node:53461) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2024-12-18T00:28:57.9725505Z 
2024-12-18T00:28:57.9725702Z Please migrate your code to use AWS SDK for JavaScript (v3).
2024-12-18T00:28:57.9726238Z For more information, check the migration guide at https://a.co/7PzMCcy
2024-12-18T00:28:57.9726794Z (Use `node --trace-warnings ...` to show where the warning was created)
2024-12-18T00:28:58.1691923Z Found 1 objects with prefix pytorch/pytorch/12383255690/linux-focal-cuda12.4-py3.10-gcc9-sm86/
2024-12-18T00:28:58.1692710Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2024-12-18T00:29:07.7742891Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2024-12-18T00:29:07.7748045Z Artifact download has finished successfully
2024-12-18T00:29:07.8108936Z ##[group]Run unzip -o artifacts.zip
2024-12-18T00:29:07.8109283Z unzip -o artifacts.zip
2024-12-18T00:29:07.8119732Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:07.8120126Z env:
2024-12-18T00:29:07.8120365Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:07.8120718Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:07.8121092Z ##[endgroup]
2024-12-18T00:29:07.8185884Z Archive:  artifacts.zip
2024-12-18T00:29:07.8187548Z    creating: dist/
2024-12-18T00:29:09.9957272Z   inflating: dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl  
2024-12-18T00:29:09.9957793Z    creating: build/custom_test_artifacts/
2024-12-18T00:29:09.9958210Z    creating: build/custom_test_artifacts/custom-op-build/
2024-12-18T00:29:09.9958744Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2024-12-18T00:29:09.9959349Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/
2024-12-18T00:29:09.9967785Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml  
2024-12-18T00:29:09.9968450Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/
2024-12-18T00:29:09.9969130Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2024-12-18T00:29:09.9969836Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/
2024-12-18T00:29:09.9970535Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-12-18T00:29:09.9973461Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2024-12-18T00:29:09.9975880Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2024-12-18T00:29:09.9976611Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-12-18T00:29:09.9977321Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-12-18T00:29:09.9980512Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2024-12-18T00:29:09.9982251Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2024-12-18T00:29:09.9984539Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2024-12-18T00:29:09.9985376Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2024-12-18T00:29:09.9987731Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2024-12-18T00:29:09.9989154Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2024-12-18T00:29:09.9989896Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2024-12-18T00:29:09.9990614Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2024-12-18T00:29:10.0032661Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2024-12-18T00:29:10.0073815Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2024-12-18T00:29:10.0075528Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2024-12-18T00:29:10.0123242Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2024-12-18T00:29:10.0124901Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2024-12-18T00:29:10.0125901Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2024-12-18T00:29:10.0127131Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2024-12-18T00:29:10.0128135Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2024-12-18T00:29:10.0129121Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2024-12-18T00:29:10.0130110Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2024-12-18T00:29:10.0131101Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2024-12-18T00:29:10.0132376Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2024-12-18T00:29:10.0133458Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2024-12-18T00:29:10.0134483Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2024-12-18T00:29:10.0135681Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2024-12-18T00:29:10.0136995Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2024-12-18T00:29:10.0138453Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2024-12-18T00:29:10.0140618Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2024-12-18T00:29:10.0214254Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2024-12-18T00:29:10.0288796Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2024-12-18T00:29:10.0290571Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2024-12-18T00:29:10.0292004Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/
2024-12-18T00:29:10.0293231Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2024-12-18T00:29:10.0294331Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2024-12-18T00:29:10.0295019Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2024-12-18T00:29:10.0295746Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2024-12-18T00:29:10.0296586Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2024-12-18T00:29:10.0297382Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2024-12-18T00:29:10.0298186Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2024-12-18T00:29:10.0298945Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2024-12-18T00:29:10.0299715Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2024-12-18T00:29:10.0300482Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2024-12-18T00:29:10.0301245Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2024-12-18T00:29:10.0302005Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2024-12-18T00:29:10.0323062Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2024-12-18T00:29:10.0467052Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2024-12-18T00:29:10.0467917Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2024-12-18T00:29:10.0468691Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2024-12-18T00:29:10.0470027Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2024-12-18T00:29:10.0470897Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2024-12-18T00:29:10.0471675Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2024-12-18T00:29:10.0472497Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2024-12-18T00:29:10.0473921Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2024-12-18T00:29:10.0474955Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2024-12-18T00:29:10.0475916Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2024-12-18T00:29:10.0477111Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2024-12-18T00:29:10.0498707Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2024-12-18T00:29:10.0582339Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2024-12-18T00:29:10.0583413Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2024-12-18T00:29:10.0584346Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2024-12-18T00:29:10.0585585Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2024-12-18T00:29:10.0586623Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2024-12-18T00:29:10.0588562Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2024-12-18T00:29:10.0589206Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2024-12-18T00:29:10.0592380Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2024-12-18T00:29:10.0593539Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2024-12-18T00:29:10.0594621Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2024-12-18T00:29:10.0715429Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2024-12-18T00:29:10.0778292Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2024-12-18T00:29:10.0778952Z    creating: build/custom_test_artifacts/jit-hook-build/
2024-12-18T00:29:10.0779452Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2024-12-18T00:29:10.0780034Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/
2024-12-18T00:29:10.0787660Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml  
2024-12-18T00:29:10.0788326Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/
2024-12-18T00:29:10.0789051Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2024-12-18T00:29:10.0789742Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/
2024-12-18T00:29:10.0790414Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-12-18T00:29:10.0793199Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2024-12-18T00:29:10.0794681Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2024-12-18T00:29:10.0795379Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-12-18T00:29:10.0796291Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-12-18T00:29:10.0799094Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2024-12-18T00:29:10.0800974Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2024-12-18T00:29:10.0803071Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2024-12-18T00:29:10.0804014Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2024-12-18T00:29:10.0806295Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2024-12-18T00:29:10.0807698Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2024-12-18T00:29:10.0808435Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2024-12-18T00:29:10.0809144Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2024-12-18T00:29:10.0851411Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2024-12-18T00:29:10.0892718Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2024-12-18T00:29:10.0893853Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2024-12-18T00:29:10.0941460Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2024-12-18T00:29:10.0942490Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2024-12-18T00:29:10.0943979Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2024-12-18T00:29:10.0945074Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2024-12-18T00:29:10.0946074Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2024-12-18T00:29:10.0947251Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2024-12-18T00:29:10.0948417Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2024-12-18T00:29:10.0949757Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2024-12-18T00:29:10.0951391Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2024-12-18T00:29:10.0952458Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2024-12-18T00:29:10.0953465Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2024-12-18T00:29:10.0954642Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2024-12-18T00:29:10.0955963Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2024-12-18T00:29:10.0957298Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2024-12-18T00:29:10.0959575Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2024-12-18T00:29:10.1033665Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2024-12-18T00:29:10.1107643Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2024-12-18T00:29:10.1108668Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2024-12-18T00:29:10.1109363Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/
2024-12-18T00:29:10.1109962Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2024-12-18T00:29:10.1110866Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2024-12-18T00:29:10.1111529Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2024-12-18T00:29:10.1112581Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2024-12-18T00:29:10.1113612Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2024-12-18T00:29:10.1114646Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2024-12-18T00:29:10.1115835Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2024-12-18T00:29:10.1117715Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2024-12-18T00:29:10.1118884Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2024-12-18T00:29:10.1119939Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2024-12-18T00:29:10.1120932Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2024-12-18T00:29:10.1122135Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2024-12-18T00:29:10.1143782Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2024-12-18T00:29:10.1207666Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2024-12-18T00:29:10.1208531Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2024-12-18T00:29:10.1209525Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2024-12-18T00:29:10.1210387Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2024-12-18T00:29:10.1211624Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2024-12-18T00:29:10.1213461Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2024-12-18T00:29:10.1214287Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2024-12-18T00:29:10.1217713Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2024-12-18T00:29:10.1218787Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2024-12-18T00:29:10.1219844Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2024-12-18T00:29:10.1269727Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2024-12-18T00:29:10.1270332Z    creating: build/custom_test_artifacts/custom-backend-build/
2024-12-18T00:29:10.1270924Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2024-12-18T00:29:10.1271625Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/
2024-12-18T00:29:10.1279294Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml  
2024-12-18T00:29:10.1279999Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/
2024-12-18T00:29:10.1280841Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2024-12-18T00:29:10.1281601Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/
2024-12-18T00:29:10.1282339Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-12-18T00:29:10.1285552Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2024-12-18T00:29:10.1287587Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2024-12-18T00:29:10.1288333Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-12-18T00:29:10.1289076Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-12-18T00:29:10.1291683Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2024-12-18T00:29:10.1293666Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2024-12-18T00:29:10.1295972Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2024-12-18T00:29:10.1296956Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2024-12-18T00:29:10.1299327Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2024-12-18T00:29:10.1300802Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2024-12-18T00:29:10.1301586Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2024-12-18T00:29:10.1302341Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2024-12-18T00:29:10.1344312Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2024-12-18T00:29:10.1385696Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2024-12-18T00:29:10.1386968Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2024-12-18T00:29:10.1434429Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2024-12-18T00:29:10.1435975Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2024-12-18T00:29:10.1437081Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2024-12-18T00:29:10.1438186Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2024-12-18T00:29:10.1439259Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2024-12-18T00:29:10.1440490Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2024-12-18T00:29:10.1441582Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2024-12-18T00:29:10.1442847Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2024-12-18T00:29:10.1444469Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2024-12-18T00:29:10.1445562Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2024-12-18T00:29:10.1446506Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2024-12-18T00:29:10.1447758Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2024-12-18T00:29:10.1449063Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2024-12-18T00:29:10.1450372Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2024-12-18T00:29:10.1452453Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2024-12-18T00:29:10.1526680Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2024-12-18T00:29:10.1600152Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2024-12-18T00:29:10.1601074Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2024-12-18T00:29:10.1601846Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/
2024-12-18T00:29:10.1602508Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2024-12-18T00:29:10.1603205Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2024-12-18T00:29:10.1603936Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2024-12-18T00:29:10.1605007Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2024-12-18T00:29:10.1606069Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2024-12-18T00:29:10.1607089Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2024-12-18T00:29:10.1608152Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2024-12-18T00:29:10.1609130Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2024-12-18T00:29:10.1610757Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2024-12-18T00:29:10.1611633Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2024-12-18T00:29:10.1612609Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2024-12-18T00:29:10.1613776Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2024-12-18T00:29:10.1619165Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2024-12-18T00:29:10.1741399Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2024-12-18T00:29:10.1742242Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2024-12-18T00:29:10.1743088Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2024-12-18T00:29:10.1744059Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2024-12-18T00:29:10.1744997Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2024-12-18T00:29:10.1745867Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2024-12-18T00:29:10.1746865Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2024-12-18T00:29:10.1748205Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2024-12-18T00:29:10.1749212Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2024-12-18T00:29:10.1750214Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2024-12-18T00:29:10.1751399Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2024-12-18T00:29:10.1773004Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2024-12-18T00:29:10.1828893Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2024-12-18T00:29:10.1829932Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2024-12-18T00:29:10.1830921Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2024-12-18T00:29:10.1831786Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2024-12-18T00:29:10.1833113Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2024-12-18T00:29:10.1835106Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2024-12-18T00:29:10.1836373Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2024-12-18T00:29:10.1839647Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2024-12-18T00:29:10.1840810Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2024-12-18T00:29:10.1841909Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2024-12-18T00:29:10.1943400Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2024-12-18T00:29:10.1986427Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2024-12-18T00:29:10.1986914Z    creating: build/lib/
2024-12-18T00:29:10.2075887Z   inflating: build/lib/libprotobuf-lite.a  
2024-12-18T00:29:10.2528127Z   inflating: build/lib/libprotobuf.a  
2024-12-18T00:29:10.2538111Z   inflating: build/lib/libpthreadpool.a  
2024-12-18T00:29:10.2546690Z   inflating: build/lib/libcpuinfo.a  
2024-12-18T00:29:10.2554969Z   inflating: build/lib/libcpuinfo_internals.a  
2024-12-18T00:29:10.2555906Z   inflating: build/lib/libclog.a     
2024-12-18T00:29:10.2574972Z   inflating: build/lib/libnnpack.a   
2024-12-18T00:29:10.2577768Z   inflating: build/lib/libnnpack_reference_layers.a  
2024-12-18T00:29:10.2765193Z   inflating: build/lib/libmicrokernels-prod.a  
2024-12-18T00:29:10.2833696Z   inflating: build/lib/libgtest.a    
2024-12-18T00:29:10.2908179Z   inflating: build/lib/libbenchmark.a  
2024-12-18T00:29:10.2972005Z   inflating: build/lib/libasmjit.a   
2024-12-18T00:29:10.2979909Z   inflating: build/lib/libittnotify.a  
2024-12-18T00:29:10.3008323Z   inflating: build/lib/libtensorpipe_uv.a  
2024-12-18T00:29:10.3135564Z   inflating: build/lib/libgloo.a     
2024-12-18T00:29:10.3155969Z   inflating: build/lib/libfmt.a      
2024-12-18T00:29:10.3255130Z   inflating: build/lib/libc10.so     
2024-12-18T00:29:10.3257191Z   inflating: build/lib/libcaffe2_nvrtc.so  
2024-12-18T00:29:10.3259274Z   inflating: build/lib/libtorch_global_deps.so  
2024-12-18T00:29:10.3760023Z   inflating: build/lib/libprotoc.a   
2024-12-18T00:29:10.3779028Z   inflating: build/lib/libpytorch_qnnpack.a  
2024-12-18T00:29:10.3796567Z   inflating: build/lib/libgmock.a    
2024-12-18T00:29:10.3797512Z   inflating: build/lib/libgtest_main.a  
2024-12-18T00:29:10.3798913Z   inflating: build/lib/libbenchmark_main.a  
2024-12-18T00:29:10.4368639Z   inflating: build/lib/libtensorpipe.a  
2024-12-18T00:29:10.4754653Z   inflating: build/lib/libgloo_cuda.a  
2024-12-18T00:29:11.4967940Z   inflating: build/lib/libdnnl.a     
2024-12-18T00:29:11.5026473Z   inflating: build/lib/libc10_cuda.so  
2024-12-18T00:29:11.5027348Z   inflating: build/lib/libgmock_main.a  
2024-12-18T00:29:11.6276774Z   inflating: build/lib/libfbgemm.a   
2024-12-18T00:29:11.6530127Z   inflating: build/lib/libtensorpipe_cuda.a  
2024-12-18T00:29:11.7033616Z   inflating: build/lib/libkineto.a   
2024-12-18T00:29:11.7122021Z   inflating: build/lib/libXNNPACK.a  
2024-12-18T00:29:11.7164345Z   inflating: build/lib/libonnx_proto.a  
2024-12-18T00:29:11.7889649Z   inflating: build/lib/libonnx.a     
2024-12-18T00:29:11.8785999Z   inflating: build/lib/libmicrokernels-all.a  
2024-12-18T00:29:14.3907963Z   inflating: build/lib/libtorch_cpu.so  
2024-12-18T00:29:14.3912718Z   inflating: build/lib/libunbox_lib.a  
2024-12-18T00:29:14.3917025Z   inflating: build/lib/libshm.so     
2024-12-18T00:29:16.4803611Z   inflating: build/lib/libtorch_cuda.so  
2024-12-18T00:29:16.4804828Z   inflating: build/lib/libtorch.so   
2024-12-18T00:29:16.4807393Z   inflating: build/lib/libc10d_cuda_test.so  
2024-12-18T00:29:17.3634565Z   inflating: build/lib/libtorch_cuda_linalg.so  
2024-12-18T00:29:17.5706992Z   inflating: build/lib/libtorch_python.so  
2024-12-18T00:29:17.5777790Z   inflating: build/lib/libtorchbind_test.so  
2024-12-18T00:29:17.5798059Z   inflating: build/lib/libjitbackend_test.so  
2024-12-18T00:29:17.5823699Z   inflating: build/lib/libbackend_with_compiler.so  
2024-12-18T00:29:17.5849012Z   inflating: build/lib/libaoti_custom_ops.so  
2024-12-18T00:29:17.5883514Z   inflating: build/lib/libnnapi_backend.so  
2024-12-18T00:29:17.5884708Z    creating: build/bin/
2024-12-18T00:29:17.5937322Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2024-12-18T00:29:17.5992239Z   inflating: build/bin/c10_DeviceGuard_test  
2024-12-18T00:29:17.6047474Z   inflating: build/bin/c10_Device_test  
2024-12-18T00:29:17.6110332Z   inflating: build/bin/c10_DispatchKeySet_test  
2024-12-18T00:29:17.6167215Z   inflating: build/bin/c10_Scalar_test  
2024-12-18T00:29:17.6219427Z   inflating: build/bin/c10_StreamGuard_test  
2024-12-18T00:29:17.6274128Z   inflating: build/bin/c10_SymInt_test  
2024-12-18T00:29:17.6331676Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2024-12-18T00:29:17.6390833Z   inflating: build/bin/c10_InlineStreamGuard_test  
2024-12-18T00:29:17.6450400Z   inflating: build/bin/c10_SizesAndStrides_test  
2024-12-18T00:29:17.6524585Z   inflating: build/bin/c10_cow_test  
2024-12-18T00:29:17.6578671Z   inflating: build/bin/c10_ArrayRef_test  
2024-12-18T00:29:17.6633990Z   inflating: build/bin/c10_Bitset_test  
2024-12-18T00:29:17.6685806Z   inflating: build/bin/c10_ConstexprCrc_test  
2024-12-18T00:29:17.6739077Z   inflating: build/bin/c10_DeadlockDetection_test  
2024-12-18T00:29:17.6793027Z   inflating: build/bin/c10_Half_test  
2024-12-18T00:29:17.6852935Z   inflating: build/bin/c10_LeftRight_test  
2024-12-18T00:29:17.6911478Z   inflating: build/bin/c10_Metaprogramming_test  
2024-12-18T00:29:17.6968027Z   inflating: build/bin/c10_NetworkFlow_test  
2024-12-18T00:29:17.7021907Z   inflating: build/bin/c10_Synchronized_test  
2024-12-18T00:29:17.7080715Z   inflating: build/bin/c10_ThreadLocal_test  
2024-12-18T00:29:17.7136065Z   inflating: build/bin/c10_TypeIndex_test  
2024-12-18T00:29:17.7190957Z   inflating: build/bin/c10_TypeList_test  
2024-12-18T00:29:17.7243266Z   inflating: build/bin/c10_TypeTraits_test  
2024-12-18T00:29:17.7299357Z   inflating: build/bin/c10_accumulate_test  
2024-12-18T00:29:17.7358702Z   inflating: build/bin/c10_bfloat16_test  
2024-12-18T00:29:17.7412785Z   inflating: build/bin/c10_bit_cast_test  
2024-12-18T00:29:17.7472979Z   inflating: build/bin/c10_complex_math_test  
2024-12-18T00:29:17.7531743Z   inflating: build/bin/c10_complex_test  
2024-12-18T00:29:17.7585668Z   inflating: build/bin/c10_error_test  
2024-12-18T00:29:17.7642455Z   inflating: build/bin/c10_exception_test  
2024-12-18T00:29:17.7695874Z   inflating: build/bin/c10_flags_test  
2024-12-18T00:29:17.7750145Z   inflating: build/bin/c10_generic_math_test  
2024-12-18T00:29:17.7921814Z   inflating: build/bin/c10_intrusive_ptr_test  
2024-12-18T00:29:17.7975449Z   inflating: build/bin/c10_irange_test  
2024-12-18T00:29:17.8032992Z   inflating: build/bin/c10_lazy_test  
2024-12-18T00:29:17.8093929Z   inflating: build/bin/c10_logging_test  
2024-12-18T00:29:17.8173456Z   inflating: build/bin/c10_optional_test  
2024-12-18T00:29:17.8239826Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2024-12-18T00:29:17.8296686Z   inflating: build/bin/c10_registry_test  
2024-12-18T00:29:17.8457968Z   inflating: build/bin/c10_small_vector_test  
2024-12-18T00:29:17.8512487Z   inflating: build/bin/c10_ssize_test  
2024-12-18T00:29:17.8568357Z   inflating: build/bin/c10_string_util_test  
2024-12-18T00:29:17.8631846Z   inflating: build/bin/c10_string_view_test  
2024-12-18T00:29:17.8685935Z   inflating: build/bin/c10_tempfile_test  
2024-12-18T00:29:17.8746302Z   inflating: build/bin/c10_typeid_test  
2024-12-18T00:29:17.8794406Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2024-12-18T00:29:17.9244024Z   inflating: build/bin/protoc-3.13.0.0  
2024-12-18T00:29:17.9691238Z   inflating: build/bin/protoc        
2024-12-18T00:29:17.9747156Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test  
2024-12-18T00:29:17.9804366Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream  
2024-12-18T00:29:17.9859640Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device  
2024-12-18T00:29:17.9915157Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes  
2024-12-18T00:29:17.9971297Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads  
2024-12-18T00:29:18.0027771Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks  
2024-12-18T00:29:18.0084269Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block  
2024-12-18T00:29:18.0137640Z   inflating: build/bin/c10_cuda_CUDATest  
2024-12-18T00:29:18.0537579Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2024-12-18T00:29:18.0947117Z   inflating: build/bin/vec_test_all_types_AVX512  
2024-12-18T00:29:18.1372907Z   inflating: build/bin/vec_test_all_types_AVX2  
2024-12-18T00:29:18.1429095Z   inflating: build/bin/BackoffTest   
2024-12-18T00:29:18.1485358Z   inflating: build/bin/FileStoreTest  
2024-12-18T00:29:18.1544609Z   inflating: build/bin/TCPStoreTest  
2024-12-18T00:29:18.1601375Z   inflating: build/bin/HashStoreTest  
2024-12-18T00:29:18.1615385Z   inflating: build/bin/ProcessGroupMPITest  
2024-12-18T00:29:18.1673301Z   inflating: build/bin/test_edge_op_registration  
2024-12-18T00:29:18.1677922Z   inflating: build/bin/torch_shm_manager  
2024-12-18T00:29:18.1681129Z   inflating: build/bin/example_allreduce  
2024-12-18T00:29:18.1740703Z   inflating: build/bin/test_dist_autograd  
2024-12-18T00:29:18.1813140Z   inflating: build/bin/test_cpp_rpc  
2024-12-18T00:29:18.1816176Z   inflating: build/bin/parallel_benchmark  
2024-12-18T00:29:18.1886000Z   inflating: build/bin/test_mobile_nnc  
2024-12-18T00:29:18.1895536Z   inflating: build/bin/aot_model_compiler_test  
2024-12-18T00:29:18.2249086Z   inflating: build/bin/test_lazy     
2024-12-18T00:29:18.3434505Z   inflating: build/bin/test_api      
2024-12-18T00:29:18.3512118Z   inflating: build/bin/Dict_test     
2024-12-18T00:29:18.3567871Z   inflating: build/bin/Dimname_test  
2024-12-18T00:29:18.3636426Z   inflating: build/bin/MaybeOwned_test  
2024-12-18T00:29:18.3697793Z   inflating: build/bin/NamedTensor_test  
2024-12-18T00:29:18.3760670Z   inflating: build/bin/apply_utils_test  
2024-12-18T00:29:18.3822911Z   inflating: build/bin/atest         
2024-12-18T00:29:18.3889250Z   inflating: build/bin/basic         
2024-12-18T00:29:18.3947835Z   inflating: build/bin/broadcast_test  
2024-12-18T00:29:18.4002910Z   inflating: build/bin/cpu_allocator_test  
2024-12-18T00:29:18.4065025Z   inflating: build/bin/cpu_generator_test  
2024-12-18T00:29:18.4122061Z   inflating: build/bin/cpu_profiling_allocator_test  
2024-12-18T00:29:18.4219917Z   inflating: build/bin/cpu_rng_test  
2024-12-18T00:29:18.4273403Z   inflating: build/bin/dispatch_key_set_test  
2024-12-18T00:29:18.4327614Z   inflating: build/bin/dlconvertor_test  
2024-12-18T00:29:18.4389301Z   inflating: build/bin/extension_backend_test  
2024-12-18T00:29:18.4447960Z   inflating: build/bin/half_test     
2024-12-18T00:29:18.4547791Z   inflating: build/bin/ivalue_test   
2024-12-18T00:29:18.4601517Z   inflating: build/bin/lazy_tensor_test  
2024-12-18T00:29:18.4660385Z   inflating: build/bin/math_kernel_test  
2024-12-18T00:29:18.4717496Z   inflating: build/bin/memory_format_test  
2024-12-18T00:29:18.4773588Z   inflating: build/bin/memory_overlapping_test  
2024-12-18T00:29:18.4829987Z   inflating: build/bin/mobile_memory_cleanup  
2024-12-18T00:29:18.4889960Z   inflating: build/bin/native_test   
2024-12-18T00:29:18.4944125Z   inflating: build/bin/operator_name_test  
2024-12-18T00:29:18.4998635Z   inflating: build/bin/operators_test  
2024-12-18T00:29:18.5054260Z   inflating: build/bin/packedtensoraccessor_test  
2024-12-18T00:29:18.5124930Z   inflating: build/bin/pow_test      
2024-12-18T00:29:18.5186288Z   inflating: build/bin/quantized_test  
2024-12-18T00:29:18.5240382Z   inflating: build/bin/reduce_ops_test  
2024-12-18T00:29:18.5294757Z   inflating: build/bin/reportMemoryUsage_test  
2024-12-18T00:29:18.5355424Z   inflating: build/bin/scalar_tensor_test  
2024-12-18T00:29:18.5418427Z   inflating: build/bin/scalar_test   
2024-12-18T00:29:18.5472197Z   inflating: build/bin/StorageUtils_test  
2024-12-18T00:29:18.5528383Z   inflating: build/bin/stride_properties_test  
2024-12-18T00:29:18.5611618Z   inflating: build/bin/tensor_iterator_test  
2024-12-18T00:29:18.5670204Z   inflating: build/bin/test_parallel  
2024-12-18T00:29:18.5673387Z   inflating: build/bin/thread_init_test  
2024-12-18T00:29:18.5732627Z   inflating: build/bin/type_ptr_test  
2024-12-18T00:29:18.5796547Z   inflating: build/bin/type_test     
2024-12-18T00:29:18.5853137Z   inflating: build/bin/undefined_tensor_test  
2024-12-18T00:29:18.5854304Z   inflating: build/bin/verify_api_visibility  
2024-12-18T00:29:18.5928309Z   inflating: build/bin/legacy_vmap_test  
2024-12-18T00:29:18.5985096Z   inflating: build/bin/weakref_test  
2024-12-18T00:29:18.6040263Z   inflating: build/bin/wrapdim_test  
2024-12-18T00:29:18.6096104Z   inflating: build/bin/xla_tensor_test  
2024-12-18T00:29:18.6159496Z   inflating: build/bin/IListRef_test  
2024-12-18T00:29:18.6271144Z   inflating: build/bin/List_test     
2024-12-18T00:29:18.6341610Z   inflating: build/bin/KernelFunction_test  
2024-12-18T00:29:18.6467885Z   inflating: build/bin/kernel_function_legacy_test  
2024-12-18T00:29:18.6570612Z   inflating: build/bin/kernel_function_test  
2024-12-18T00:29:18.6703083Z   inflating: build/bin/kernel_lambda_legacy_test  
2024-12-18T00:29:18.6809737Z   inflating: build/bin/kernel_lambda_test  
2024-12-18T00:29:18.6874195Z   inflating: build/bin/kernel_stackbased_test  
2024-12-18T00:29:18.6975081Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2024-12-18T00:29:18.7029610Z   inflating: build/bin/CppSignature_test  
2024-12-18T00:29:18.7088458Z   inflating: build/bin/backend_fallback_test  
2024-12-18T00:29:18.7140677Z   inflating: build/bin/op_allowlist_test  
2024-12-18T00:29:18.7446642Z   inflating: build/bin/op_registration_test  
2024-12-18T00:29:18.7513882Z   inflating: build/bin/inline_container_test  
2024-12-18T00:29:18.7570752Z   inflating: build/bin/cuda_apply_test  
2024-12-18T00:29:18.7627163Z   inflating: build/bin/cuda_allocator_test  
2024-12-18T00:29:18.7685557Z   inflating: build/bin/cuda_caching_host_allocator_test  
2024-12-18T00:29:18.7747775Z   inflating: build/bin/cuda_atomic_ops_test  
2024-12-18T00:29:18.7822616Z   inflating: build/bin/cuda_complex_math_test  
2024-12-18T00:29:18.7885343Z   inflating: build/bin/cuda_complex_test  
2024-12-18T00:29:18.7938354Z   inflating: build/bin/cuda_device_test  
2024-12-18T00:29:18.8000843Z   inflating: build/bin/cuda_cub_test  
2024-12-18T00:29:18.8054692Z   inflating: build/bin/cuda_dlconvertor_test  
2024-12-18T00:29:18.8124124Z   inflating: build/bin/cuda_distributions_test  
2024-12-18T00:29:18.8183490Z   inflating: build/bin/cuda_generator_test  
2024-12-18T00:29:18.8236948Z   inflating: build/bin/cuda_half_test  
2024-12-18T00:29:18.8291453Z   inflating: build/bin/cuda_integer_divider_test  
2024-12-18T00:29:18.8344225Z   inflating: build/bin/cuda_optional_test  
2024-12-18T00:29:18.8399354Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2024-12-18T00:29:18.8455948Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2024-12-18T00:29:18.8508975Z   inflating: build/bin/cuda_allocatorTraceTracker_test  
2024-12-18T00:29:18.8573291Z   inflating: build/bin/cuda_stream_test  
2024-12-18T00:29:18.8626152Z   inflating: build/bin/cuda_cudnn_test  
2024-12-18T00:29:18.8681427Z   inflating: build/bin/cuda_vectorized_test  
2024-12-18T00:29:18.8695885Z   inflating: build/bin/tutorial_tensorexpr  
2024-12-18T00:29:18.8765351Z   inflating: build/bin/ProcessGroupGlooTest  
2024-12-18T00:29:18.8826141Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2024-12-18T00:29:18.8894059Z   inflating: build/bin/ProcessGroupNCCLTest  
2024-12-18T00:29:18.8958818Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2024-12-18T00:29:18.9797445Z   inflating: build/bin/test_tensorexpr  
2024-12-18T00:29:19.0384606Z   inflating: build/bin/test_jit      
2024-12-18T00:29:19.0385054Z    creating: .additional_ci_files/
2024-12-18T00:29:19.0474649Z   inflating: .additional_ci_files/test-times.json  
2024-12-18T00:29:19.0830212Z   inflating: .additional_ci_files/test-class-times.json  
2024-12-18T00:29:19.0863930Z ##[group]Run rm artifacts.zip
2024-12-18T00:29:19.0864249Z rm artifacts.zip
2024-12-18T00:29:19.0873682Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:19.0874071Z env:
2024-12-18T00:29:19.0874303Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:19.0874649Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:19.0875009Z ##[endgroup]
2024-12-18T00:29:19.2558296Z ##[group]Run df -H
2024-12-18T00:29:19.2558570Z df -H
2024-12-18T00:29:19.2567276Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:19.2567664Z env:
2024-12-18T00:29:19.2567894Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:19.2568235Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:19.2568602Z ##[endgroup]
2024-12-18T00:29:19.2619677Z Filesystem        Size  Used Avail Use% Mounted on
2024-12-18T00:29:19.2620098Z devtmpfs          4.2M     0  4.2M   0% /dev
2024-12-18T00:29:19.2620465Z tmpfs              34G     0   34G   0% /dev/shm
2024-12-18T00:29:19.2620807Z tmpfs              14G  553k   14G   1% /run
2024-12-18T00:29:19.2621145Z /dev/nvme0n1p1    161G   59G  103G  37% /
2024-12-18T00:29:19.2621483Z tmpfs              34G   13k   34G   1% /tmp
2024-12-18T00:29:19.2621845Z /dev/nvme0n1p128   11M  1.4M  9.2M  13% /boot/efi
2024-12-18T00:29:19.2622216Z tmpfs             6.7G     0  6.7G   0% /run/user/0
2024-12-18T00:29:19.2655131Z Prepare all required actions
2024-12-18T00:29:19.2655506Z Getting action download info
2024-12-18T00:29:19.4395829Z ##[group]Run ./.github/actions/download-td-artifacts
2024-12-18T00:29:19.4396209Z with:
2024-12-18T00:29:19.4396433Z env:
2024-12-18T00:29:19.4396665Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:19.4397017Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:19.4397385Z ##[endgroup]
2024-12-18T00:29:19.4558030Z ##[group]Run seemethere/download-artifact-s3@v4
2024-12-18T00:29:19.4558393Z with:
2024-12-18T00:29:19.4558620Z   name: td_results
2024-12-18T00:29:19.4558881Z   s3-bucket: gha-artifacts
2024-12-18T00:29:19.4559154Z   region: us-east-1
2024-12-18T00:29:19.4559401Z env:
2024-12-18T00:29:19.4559632Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:19.4559975Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:19.4560345Z ##[endgroup]
2024-12-18T00:29:19.9038771Z (node:53482) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2024-12-18T00:29:19.9039429Z 
2024-12-18T00:29:19.9039703Z Please migrate your code to use AWS SDK for JavaScript (v3).
2024-12-18T00:29:19.9040292Z For more information, check the migration guide at https://a.co/7PzMCcy
2024-12-18T00:29:19.9040849Z (Use `node --trace-warnings ...` to show where the warning was created)
2024-12-18T00:29:19.9935370Z Found 0 objects with prefix pytorch/pytorch/12383255690/td_results/
2024-12-18T00:29:19.9941371Z Artifact download has finished successfully
2024-12-18T00:29:20.0427743Z ##[group]Run mkdir -p .additional_ci_files
2024-12-18T00:29:20.0428137Z mkdir -p .additional_ci_files
2024-12-18T00:29:20.0428583Z mv td_results.json .additional_ci_files/td_results.json || true
2024-12-18T00:29:20.0438407Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:20.0438785Z env:
2024-12-18T00:29:20.0439022Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:20.0439384Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:20.0439759Z ##[endgroup]
2024-12-18T00:29:20.0497089Z mv: cannot stat 'td_results.json': No such file or directory
2024-12-18T00:29:20.0632492Z ##[group]Run .github/scripts/parse_ref.py
2024-12-18T00:29:20.0632879Z .github/scripts/parse_ref.py
2024-12-18T00:29:20.0640935Z shell: /usr/bin/bash -e {0}
2024-12-18T00:29:20.0641222Z env:
2024-12-18T00:29:20.0641452Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:20.0641795Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:20.0642164Z ##[endgroup]
2024-12-18T00:29:20.0919296Z Prepare all required actions
2024-12-18T00:29:20.1356424Z ##[group]Run ./.github/actions/get-workflow-job-id
2024-12-18T00:29:20.1356886Z with:
2024-12-18T00:29:20.1357450Z   github-token: ***
2024-12-18T00:29:20.1357836Z env:
2024-12-18T00:29:20.1358210Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:20.1358656Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:20.1359117Z ##[endgroup]
2024-12-18T00:29:20.1550593Z ##[group]Run set -eux
2024-12-18T00:29:20.1550889Z set -eux
2024-12-18T00:29:20.1551382Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2024-12-18T00:29:20.1560726Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:20.1561105Z env:
2024-12-18T00:29:20.1561341Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:20.1561686Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:20.1562294Z   GITHUB_TOKEN: ***
2024-12-18T00:29:20.1562558Z ##[endgroup]
2024-12-18T00:29:20.1589426Z + python3 .github/scripts/get_workflow_job_id.py 12383255690 i-0ad7d688021a5725d
2024-12-18T00:29:21.1936992Z setting job-id=34567153040
2024-12-18T00:29:21.1937549Z setting job-name=cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:21.2110773Z Prepare all required actions
2024-12-18T00:29:21.2111149Z Getting action download info
2024-12-18T00:29:21.3668297Z ##[group]Run ./.github/actions/filter-test-configs
2024-12-18T00:29:21.3668667Z with:
2024-12-18T00:29:21.3669080Z   github-token: ***
2024-12-18T00:29:21.3671108Z   test-matrix: {"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}
2024-12-18T00:29:21.3673101Z   job-name: cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:21.3673622Z env:
2024-12-18T00:29:21.3674048Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:21.3674408Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:21.3674774Z ##[endgroup]
2024-12-18T00:29:21.3872657Z ##[group]Run nick-fields/retry@v3.0.0
2024-12-18T00:29:21.3872981Z with:
2024-12-18T00:29:21.3873208Z   shell: bash
2024-12-18T00:29:21.3873452Z   timeout_minutes: 10
2024-12-18T00:29:21.3873706Z   max_attempts: 5
2024-12-18T00:29:21.3873960Z   retry_wait_seconds: 30
2024-12-18T00:29:21.3874736Z   command: set -eux
# PyYAML 6.0 doesn't work with MacOS x86 anymore
# This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2
python3 -m pip install requests==2.27.1 pyyaml==6.0.1

2024-12-18T00:29:21.3875738Z   polling_interval_seconds: 1
2024-12-18T00:29:21.3876036Z   warning_on_retry: true
2024-12-18T00:29:21.3876312Z   continue_on_error: false
2024-12-18T00:29:21.3876582Z env:
2024-12-18T00:29:21.3876817Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:21.3877169Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:21.3877852Z   GITHUB_TOKEN: ***
2024-12-18T00:29:21.3878106Z ##[endgroup]
2024-12-18T00:29:21.4874077Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1
2024-12-18T00:29:21.7252303Z Defaulting to user installation because normal site-packages is not writeable
2024-12-18T00:29:21.8934747Z Collecting requests==2.27.1
2024-12-18T00:29:21.9307078Z   Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB)
2024-12-18T00:29:22.1770384Z Collecting pyyaml==6.0.1
2024-12-18T00:29:22.1814096Z   Downloading PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (738 kB)
2024-12-18T00:29:22.3137729Z Collecting certifi>=2017.4.17
2024-12-18T00:29:22.3180014Z   Downloading certifi-2024.12.14-py3-none-any.whl (164 kB)
2024-12-18T00:29:22.3506013Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10)
2024-12-18T00:29:22.7055358Z Collecting charset-normalizer~=2.0.0
2024-12-18T00:29:22.7098260Z   Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
2024-12-18T00:29:22.7391356Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10)
2024-12-18T00:29:22.8148396Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml
2024-12-18T00:29:23.1164498Z Successfully installed certifi-2024.12.14 charset-normalizer-2.0.12 pyyaml-6.0.1 requests-2.27.1
2024-12-18T00:29:23.4652690Z Command completed after 1 attempt(s).
2024-12-18T00:29:23.4776667Z ##[group]Run set -x
2024-12-18T00:29:23.4776952Z set -x
2024-12-18T00:29:23.4777200Z 
2024-12-18T00:29:23.4777596Z # Use relative path here as this could be checked out anywhere, not necessarily
2024-12-18T00:29:23.4778174Z # in runner workspace
2024-12-18T00:29:23.4778574Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py"
2024-12-18T00:29:23.4787876Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:23.4788495Z env:
2024-12-18T00:29:23.4788724Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:23.4789077Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:23.4789454Z ##[endgroup]
2024-12-18T00:29:23.4818617Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py
2024-12-18T00:29:23.5197014Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}"
2024-12-18T00:29:23.5197443Z echo "Workflow: ${GITHUB_WORKFLOW}"
2024-12-18T00:29:23.5197802Z echo "Job name: ${JOB_NAME}"
2024-12-18T00:29:23.5198105Z 
2024-12-18T00:29:23.5198549Z # Use relative path here as this could be checked out anywhere, not necessarily
2024-12-18T00:29:23.5199060Z # in runner workspace
2024-12-18T00:29:23.5199504Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \
2024-12-18T00:29:23.5199991Z   --workflow "${GITHUB_WORKFLOW}" \
2024-12-18T00:29:23.5200352Z   --job-name "${JOB_NAME}" \
2024-12-18T00:29:23.5202200Z   --test-matrix "{"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \
2024-12-18T00:29:23.5204058Z   --selected-test-configs "" \
2024-12-18T00:29:23.5204601Z   --pr-number "${PR_NUMBER}" \
2024-12-18T00:29:23.5204937Z   --tag "${TAG}" \
2024-12-18T00:29:23.5205249Z   --event-name "${EVENT_NAME}" \
2024-12-18T00:29:23.5205598Z   --schedule "${SCHEDULE}" \
2024-12-18T00:29:23.5205937Z   --branch "${HEAD_BRANCH}"
2024-12-18T00:29:23.5214901Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:23.5215293Z env:
2024-12-18T00:29:23.5215538Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:23.5215900Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:23.5216456Z   GITHUB_TOKEN: ***
2024-12-18T00:29:23.5216929Z   JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:23.5217450Z   PR_NUMBER: 
2024-12-18T00:29:23.5217693Z   TAG: 
2024-12-18T00:29:23.5218002Z   EVENT_NAME: push
2024-12-18T00:29:23.5218256Z   SCHEDULE: 
2024-12-18T00:29:23.5218500Z   HEAD_BRANCH: 
2024-12-18T00:29:23.5218739Z ##[endgroup]
2024-12-18T00:29:23.5245664Z Workflow: inductor
2024-12-18T00:29:23.5246251Z Job name: cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:23.7883350Z ##[group]Run echo "Filtered matrix:"
2024-12-18T00:29:23.7883702Z echo "Filtered matrix:"
2024-12-18T00:29:23.7885483Z echo "{"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}"
2024-12-18T00:29:23.7887243Z 
2024-12-18T00:29:23.7887464Z echo
2024-12-18T00:29:23.7887755Z echo "Is the current job unstable? False"
2024-12-18T00:29:23.7888112Z 
2024-12-18T00:29:23.7888342Z echo
2024-12-18T00:29:23.7888617Z echo "Is keep-going label set? False"
2024-12-18T00:29:23.7888944Z 
2024-12-18T00:29:23.7889165Z echo
2024-12-18T00:29:23.7889622Z echo "Renabled issues? "
2024-12-18T00:29:23.7898502Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:23.7898887Z env:
2024-12-18T00:29:23.7899111Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:23.7899454Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:23.7899829Z ##[endgroup]
2024-12-18T00:29:23.7928754Z Filtered matrix:
2024-12-18T00:29:23.7930558Z {include: [{config: inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}]}
2024-12-18T00:29:23.7932540Z 
2024-12-18T00:29:23.7932683Z Is the current job unstable? False
2024-12-18T00:29:23.7932895Z 
2024-12-18T00:29:23.7933020Z Is keep-going label set? False
2024-12-18T00:29:23.7933218Z 
2024-12-18T00:29:23.7933318Z Renabled issues? 
2024-12-18T00:29:23.8217962Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2024-12-18T00:29:23.8218500Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2024-12-18T00:29:23.8226883Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:23.8227263Z env:
2024-12-18T00:29:23.8227493Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:23.8227837Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:23.8228242Z   JOB_TIMEOUT: 240
2024-12-18T00:29:23.8228514Z ##[endgroup]
2024-12-18T00:29:23.8446182Z ##[group]Run set -x
2024-12-18T00:29:23.8446677Z set -x
2024-12-18T00:29:23.8446916Z 
2024-12-18T00:29:23.8447185Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2024-12-18T00:29:23.8447591Z   TEST_COMMAND=.ci/pytorch/multigpu-test.sh
2024-12-18T00:29:23.8447991Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2024-12-18T00:29:23.8448366Z   TEST_COMMAND=.ci/onnx/test.sh
2024-12-18T00:29:23.8448683Z else
2024-12-18T00:29:23.8448956Z   TEST_COMMAND=.ci/pytorch/test.sh
2024-12-18T00:29:23.8449277Z fi
2024-12-18T00:29:23.8449501Z 
2024-12-18T00:29:23.8449848Z # detached container should get cleaned up by teardown_ec2_linux
2024-12-18T00:29:23.8450386Z # TODO: Stop building test binaries as part of the build phase
2024-12-18T00:29:23.8450864Z # Used for GPU_FLAG since that doesn't play nice
2024-12-18T00:29:23.8451277Z # shellcheck disable=SC2086,SC2090
2024-12-18T00:29:23.8451628Z container_name=$(docker run \
2024-12-18T00:29:23.8451958Z   ${GPU_FLAG:-} \
2024-12-18T00:29:23.8452273Z   ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \
2024-12-18T00:29:23.8452629Z   -e BUILD_ENVIRONMENT \
2024-12-18T00:29:23.8452941Z   -e PR_NUMBER \
2024-12-18T00:29:23.8453220Z   -e GITHUB_ACTIONS \
2024-12-18T00:29:23.8453527Z   -e GITHUB_REPOSITORY \
2024-12-18T00:29:23.8453838Z   -e GITHUB_WORKFLOW \
2024-12-18T00:29:23.8454141Z   -e GITHUB_JOB \
2024-12-18T00:29:23.8454424Z   -e GITHUB_RUN_ID \
2024-12-18T00:29:23.8454712Z   -e GITHUB_RUN_NUMBER \
2024-12-18T00:29:23.8455025Z   -e GITHUB_RUN_ATTEMPT \
2024-12-18T00:29:23.8455330Z   -e JOB_ID \
2024-12-18T00:29:23.8455598Z   -e JOB_NAME \
2024-12-18T00:29:23.8455877Z   -e BASE_SHA \
2024-12-18T00:29:23.8456142Z   -e BRANCH \
2024-12-18T00:29:23.8456410Z   -e SHA1 \
2024-12-18T00:29:23.8456687Z   -e AWS_DEFAULT_REGION \
2024-12-18T00:29:23.8456998Z   -e IN_WHEEL_TEST \
2024-12-18T00:29:23.8457289Z   -e SHARD_NUMBER \
2024-12-18T00:29:23.8457573Z   -e TEST_CONFIG \
2024-12-18T00:29:23.8457948Z   -e NUM_TEST_SHARDS \
2024-12-18T00:29:23.8458253Z   -e REENABLED_ISSUES \
2024-12-18T00:29:23.8458573Z   -e CONTINUE_THROUGH_ERROR \
2024-12-18T00:29:23.8458898Z   -e VERBOSE_TEST_LOGS \
2024-12-18T00:29:23.8459198Z   -e TEST_SHOWLOCALS \
2024-12-18T00:29:23.8459504Z   -e NO_TEST_TIMEOUT \
2024-12-18T00:29:23.8459799Z   -e NO_TD \
2024-12-18T00:29:23.8460072Z   -e TD_DISTRIBUTED \
2024-12-18T00:29:23.8460485Z   -e PR_LABELS \
2024-12-18T00:29:23.8460836Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2024-12-18T00:29:23.8461176Z   -e SCCACHE_BUCKET \
2024-12-18T00:29:23.8461472Z   -e SCCACHE_REGION \
2024-12-18T00:29:23.8461769Z   -e SCCACHE_S3_KEY_PREFIX \
2024-12-18T00:29:23.8462088Z   -e XLA_CUDA \
2024-12-18T00:29:23.8462381Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2024-12-18T00:29:23.8462751Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2024-12-18T00:29:23.8463123Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2024-12-18T00:29:23.8463496Z   -e SKIP_SCCACHE_INITIALIZATION=1 \
2024-12-18T00:29:23.8463846Z   -e HUGGING_FACE_HUB_TOKEN \
2024-12-18T00:29:23.8464183Z   -e SCRIBE_GRAPHQL_ACCESS_TOKEN \
2024-12-18T00:29:23.8464508Z   -e DASHBOARD_TAG \
2024-12-18T00:29:23.8464797Z   -e IS_A100_RUNNER \
2024-12-18T00:29:23.8465098Z   -e ARTIFACTS_FILE_SUFFIX \
2024-12-18T00:29:23.8465472Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2024-12-18T00:29:23.8465881Z   --security-opt seccomp=unconfined \
2024-12-18T00:29:23.8466370Z   --cap-add=SYS_PTRACE \
2024-12-18T00:29:23.8466680Z   --ipc=host \
2024-12-18T00:29:23.8467045Z   --shm-size="${SHM_SIZE}" \
2024-12-18T00:29:23.8467349Z   --tty \
2024-12-18T00:29:23.8467600Z   --detach \
2024-12-18T00:29:23.8467871Z   --name="${container_name}" \
2024-12-18T00:29:23.8468204Z   --user jenkins \
2024-12-18T00:29:23.8468567Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2024-12-18T00:29:23.8468976Z   -w /var/lib/jenkins/workspace \
2024-12-18T00:29:23.8469298Z   "${DOCKER_IMAGE}"
2024-12-18T00:29:23.8469567Z )
2024-12-18T00:29:23.8469868Z # Propagate download.pytorch.org IP to container
2024-12-18T00:29:23.8470526Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts"
2024-12-18T00:29:23.8471227Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}"
2024-12-18T00:29:23.8472082Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"
2024-12-18T00:29:23.8480522Z shell: /usr/bin/bash -e {0}
2024-12-18T00:29:23.8480810Z env:
2024-12-18T00:29:23.8481037Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:23.8481387Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:23.8481843Z   BUILD_ENVIRONMENT: linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-12-18T00:29:23.8482226Z   PR_NUMBER: 
2024-12-18T00:29:23.8482490Z   GITHUB_REPOSITORY: pytorch/pytorch
2024-12-18T00:29:23.8482816Z   GITHUB_WORKFLOW: inductor
2024-12-18T00:29:23.8483103Z   GITHUB_JOB: test
2024-12-18T00:29:23.8483357Z   GITHUB_RUN_ID: 12383255690
2024-12-18T00:29:23.8483634Z   GITHUB_RUN_NUMBER: 109429
2024-12-18T00:29:23.8483920Z   GITHUB_RUN_ATTEMPT: 1
2024-12-18T00:29:23.8484177Z   JOB_ID: 34567153040
2024-12-18T00:29:23.8484637Z   JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:23.8485153Z   BRANCH: release/2.6
2024-12-18T00:29:23.8485453Z   SHA1: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:23.8485840Z   BASE_SHA: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:23.8486199Z   TEST_CONFIG: inductor_timm
2024-12-18T00:29:23.8486488Z   SHARD_NUMBER: 2
2024-12-18T00:29:23.8486733Z   NUM_TEST_SHARDS: 2
2024-12-18T00:29:23.8486988Z   REENABLED_ISSUES: 
2024-12-18T00:29:23.8487248Z   CONTINUE_THROUGH_ERROR: False
2024-12-18T00:29:23.8487548Z   VERBOSE_TEST_LOGS: False
2024-12-18T00:29:23.8487832Z   TEST_SHOWLOCALS: False
2024-12-18T00:29:23.8488103Z   NO_TEST_TIMEOUT: False
2024-12-18T00:29:23.8488359Z   NO_TD: False
2024-12-18T00:29:23.8488602Z   TD_DISTRIBUTED: False
2024-12-18T00:29:23.8488926Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2024-12-18T00:29:23.8489298Z   SCCACHE_REGION: us-east-1
2024-12-18T00:29:23.8489592Z   SCCACHE_S3_KEY_PREFIX: inductor
2024-12-18T00:29:23.8489887Z   SHM_SIZE: 2g
2024-12-18T00:29:23.8490703Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:29:23.8491576Z   XLA_CUDA: 
2024-12-18T00:29:23.8491945Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2024-12-18T00:29:23.8492401Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2024-12-18T00:29:23.8492735Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2024-12-18T00:29:23.8493044Z   DASHBOARD_TAG: 
2024-12-18T00:29:23.8493471Z   HUGGING_FACE_HUB_TOKEN: ***
2024-12-18T00:29:23.8493887Z   SCRIBE_GRAPHQL_ACCESS_TOKEN: ***
2024-12-18T00:29:23.8494190Z   IS_A100_RUNNER: 0
2024-12-18T00:29:23.8494624Z   ARTIFACTS_FILE_SUFFIX: test-inductor_timm-2-2-linux.g5.4xlarge.nvidia.gpu_34567153040
2024-12-18T00:29:23.8495117Z ##[endgroup]
2024-12-18T00:29:23.8522330Z + [[ inductor_timm == \m\u\l\t\i\g\p\u ]]
2024-12-18T00:29:23.8522735Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *onnx* ]]
2024-12-18T00:29:23.8523112Z + TEST_COMMAND=.ci/pytorch/test.sh
2024-12-18T00:29:23.8532303Z +++ nproc --ignore=2
2024-12-18T00:29:23.8759505Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_REGION -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e IS_A100_RUNNER -e ARTIFACTS_FILE_SUFFIX --env-file=/tmp/github_env_12383255690 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:29:30.1998542Z + container_name=6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T00:29:30.2001494Z + grep download.pytorch.org /etc/hosts
2024-12-18T00:29:30.2002548Z + docker exec -i 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684 sudo bash -c '/bin/cat >> /etc/hosts'
2024-12-18T00:29:30.3283870Z + echo DOCKER_CONTAINER_ID=6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T00:29:30.3288228Z ++ echo dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl
2024-12-18T00:29:30.3291446Z + docker exec -t 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684 sh -c 'python3 -m pip install dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh'
2024-12-18T00:29:30.7244530Z Processing ./dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl (from torch==2.6.0a0+git0cdf8b1)
2024-12-18T00:29:31.0544909Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.16.1)
2024-12-18T00:29:31.0547915Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (4.12.2)
2024-12-18T00:29:31.0550652Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (2.8.8)
2024-12-18T00:29:31.0553852Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.1.4)
2024-12-18T00:29:31.0556913Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (2024.10.0)
2024-12-18T00:29:31.0561824Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.13.1)
2024-12-18T00:29:31.0578055Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.3.0)
2024-12-18T00:29:31.0591355Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.3.0)
2024-12-18T00:29:31.0608539Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.22.4)
2024-12-18T00:29:31.0991623Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.0.2)
2024-12-18T00:29:31.6890643Z Installing collected packages: torch
2024-12-18T00:29:42.2162947Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
2024-12-18T00:29:42.2164214Z timm 1.0.9.dev0 requires torchvision, which is not installed.
2024-12-18T00:29:42.2164761Z Successfully installed torch-2.6.0a0+git0cdf8b1
2024-12-18T00:29:42.2982965Z + export TERM=vt100
2024-12-18T00:29:42.2983255Z + TERM=vt100
2024-12-18T00:29:42.2986493Z ++ dirname .ci/pytorch/test.sh
2024-12-18T00:29:42.2997399Z + source .ci/pytorch/common.sh
2024-12-18T00:29:42.3001335Z +++ dirname .ci/pytorch/common.sh
2024-12-18T00:29:42.3011683Z ++ source .ci/pytorch/common_utils.sh
2024-12-18T00:29:42.3013644Z +++ declare -f -t trap_add
2024-12-18T00:29:42.3020499Z ++ set -ex
2024-12-18T00:29:42.3021133Z ++ [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *rocm* ]]
2024-12-18T00:29:42.3021608Z ++ BUILD_TEST_LIBTORCH=0
2024-12-18T00:29:42.3021982Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *rocm* ]]
2024-12-18T00:29:42.3022360Z + [[ -d /var/lib/jenkins/workspace ]]
2024-12-18T00:29:42.3025761Z ++ stat -c %u /var/lib/jenkins/workspace
2024-12-18T00:29:42.3044013Z + WORKSPACE_ORIGINAL_OWNER_ID=1000
2024-12-18T00:29:42.3044345Z + trap_add cleanup_workspace EXIT
2024-12-18T00:29:42.3044671Z + trap_add_cmd=cleanup_workspace
2024-12-18T00:29:42.3045020Z + shift
2024-12-18T00:29:42.3045258Z + for trap_add_name in "$@"
2024-12-18T00:29:42.3053174Z +++ trap -p EXIT
2024-12-18T00:29:42.3056481Z ++ eval 'extract_trap_cmd '
2024-12-18T00:29:42.3056788Z +++ extract_trap_cmd
2024-12-18T00:29:42.3057049Z +++ printf '%s\n' ''
2024-12-18T00:29:42.3057316Z ++ printf '%s\n' cleanup_workspace
2024-12-18T00:29:42.3060488Z + trap -- '
2024-12-18T00:29:42.3060775Z cleanup_workspace' EXIT
2024-12-18T00:29:42.3061112Z + sudo chown -R jenkins /var/lib/jenkins/workspace
2024-12-18T00:29:43.0268582Z + git config --global --add safe.directory /var/lib/jenkins/workspace
2024-12-18T00:29:43.0293615Z + echo 'Environment variables:'
2024-12-18T00:29:43.0294102Z Environment variables:
2024-12-18T00:29:43.0294467Z + env
2024-12-18T00:29:43.0305355Z INSTALLED_DB=yes
2024-12-18T00:29:43.0305815Z NV_LIBCUBLAS_VERSION=12.4.5.8-1
2024-12-18T00:29:43.0306261Z NVIDIA_VISIBLE_DEVICES=all
2024-12-18T00:29:43.0306676Z NV_NVML_DEV_VERSION=12.4.127-1
2024-12-18T00:29:43.0307279Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-12-18T00:29:43.0307828Z CONTINUE_THROUGH_ERROR=False
2024-12-18T00:29:43.0308261Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.21.5-1+cuda12.4
2024-12-18T00:29:43.0308680Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.21.5-1
2024-12-18T00:29:43.0309079Z BUILD_ENVIRONMENT=linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-12-18T00:29:43.0309483Z HOSTNAME=6581d0cd7381
2024-12-18T00:29:43.0310059Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_24c60c98-1ac2-4da6-a269-1c1e3dd7ad15
2024-12-18T00:29:43.0310700Z GITHUB_ACTION=__self
2024-12-18T00:29:43.0310986Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2024-12-18T00:29:43.0315755Z NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536
2024-12-18T00:29:43.0320238Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-4=12.4.5.8-1
2024-12-18T00:29:43.0320615Z NV_NVTX_VERSION=12.4.127-1
2024-12-18T00:29:43.0320901Z GITHUB_RUN_NUMBER=109429
2024-12-18T00:29:43.0321182Z TEST_CONFIG=inductor_timm
2024-12-18T00:29:43.0321465Z GITHUB_REPOSITORY_OWNER_ID=21003710
2024-12-18T00:29:43.0321802Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2024-12-18T00:29:43.0322118Z IS_A100_RUNNER=0
2024-12-18T00:29:43.0322385Z NV_CUDA_CUDART_DEV_VERSION=12.4.127-1
2024-12-18T00:29:43.0322707Z NV_LIBCUSPARSE_VERSION=12.3.1.170-1
2024-12-18T00:29:43.0323238Z SCRIBE_GRAPHQL_ACCESS_TOKEN=***
2024-12-18T00:29:43.0323539Z NV_LIBNPP_VERSION=12.2.5.30-1
2024-12-18T00:29:43.0323842Z GITHUB_TRIGGERING_ACTOR=malfet
2024-12-18T00:29:43.0324180Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2024-12-18T00:29:43.0324537Z GITHUB_REF_TYPE=branch
2024-12-18T00:29:43.0324799Z TORCH_CUDA_ARCH_LIST=Maxwell
2024-12-18T00:29:43.0325083Z NCCL_VERSION=2.21.5-1
2024-12-18T00:29:43.0325387Z BASE_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:43.0325727Z XLA_CUDA=
2024-12-18T00:29:43.0326085Z HUGGING_FACE_HUB_TOKEN=***
2024-12-18T00:29:43.0328079Z ***
2024-12-18T00:29:43.0328313Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2024-12-18T00:29:43.0328626Z GITHUB_REPOSITORY_ID=65600975
2024-12-18T00:29:43.0328914Z GITHUB_ACTIONS=true
2024-12-18T00:29:43.0329179Z NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:43.0329515Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4=12.4.127-1
2024-12-18T00:29:43.0329883Z NV_LIBNPP_PACKAGE=libnpp-12-4=12.2.5.30-1
2024-12-18T00:29:43.0330250Z SHA1=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:43.0330607Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2024-12-18T00:29:43.0330981Z GITHUB_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:43.0331530Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/release/2.6
2024-12-18T00:29:43.0332021Z UCC_HOME=/usr
2024-12-18T00:29:43.0332275Z NV_LIBCUBLAS_DEV_VERSION=12.4.5.8-1
2024-12-18T00:29:43.0332582Z VERBOSE_TEST_LOGS=False
2024-12-18T00:29:43.0332861Z NVIDIA_PRODUCT_NAME=CUDA
2024-12-18T00:29:43.0333185Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-4
2024-12-18T00:29:43.0333541Z GITHUB_REF=refs/heads/release/2.6
2024-12-18T00:29:43.0333847Z NV_CUDA_CUDART_VERSION=12.4.127-1
2024-12-18T00:29:43.0334138Z SHARD_NUMBER=2
2024-12-18T00:29:43.0334391Z GITHUB_REF_PROTECTED=true
2024-12-18T00:29:43.0334668Z HOME=/var/lib/jenkins
2024-12-18T00:29:43.0334950Z GITHUB_API_URL=https://api.github.com
2024-12-18T00:29:43.0335295Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2024-12-18T00:29:43.0335656Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb
2024-12-18T00:29:43.0336034Z SCCACHE_S3_KEY_PREFIX=inductor
2024-12-18T00:29:43.0336321Z CUDA_VERSION=12.4.1
2024-12-18T00:29:43.0336600Z NV_LIBCUBLAS_PACKAGE=libcublas-12-4=12.4.5.8-1
2024-12-18T00:29:43.0336941Z NUM_TEST_SHARDS=2
2024-12-18T00:29:43.0337185Z UCX_HOME=/usr
2024-12-18T00:29:43.0337535Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-4=12.4.1-1
2024-12-18T00:29:43.0338396Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_24c60c98-1ac2-4da6-a269-1c1e3dd7ad15
2024-12-18T00:29:43.0339203Z JOB_NAME=cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:43.0339997Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_24c60c98-1ac2-4da6-a269-1c1e3dd7ad15
2024-12-18T00:29:43.0340779Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2024-12-18T00:29:43.0341270Z GITHUB_EVENT_NAME=push
2024-12-18T00:29:43.0341656Z DASHBOARD_TAG=
2024-12-18T00:29:43.0341906Z GITHUB_RUN_ID=12383255690
2024-12-18T00:29:43.0342295Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-4=12.2.5.30-1
2024-12-18T00:29:43.0342675Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-4
2024-12-18T00:29:43.0343343Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_24c60c98-1ac2-4da6-a269-1c1e3dd7ad15
2024-12-18T00:29:43.0343997Z GITHUB_ACTOR=malfet
2024-12-18T00:29:43.0344261Z NV_LIBNPP_DEV_VERSION=12.2.5.30-1
2024-12-18T00:29:43.0344551Z PR_NUMBER=
2024-12-18T00:29:43.0344786Z GITHUB_RUN_ATTEMPT=1
2024-12-18T00:29:43.0345062Z ANACONDA_PYTHON_VERSION=3.10
2024-12-18T00:29:43.0345405Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2024-12-18T00:29:43.0345755Z TERM=vt100
2024-12-18T00:29:43.0345998Z NV_LIBCUSPARSE_DEV_VERSION=12.3.1.170-1
2024-12-18T00:29:43.0346321Z INSTALLED_VISION=yes
2024-12-18T00:29:43.0346577Z BRANCH=release/2.6
2024-12-18T00:29:43.0346835Z SCCACHE_REGION=us-east-1
2024-12-18T00:29:43.0347117Z OPENSSL_ROOT_DIR=/opt/openssl
2024-12-18T00:29:43.0347432Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2024-12-18T00:29:43.0347759Z CUDA_PATH=/usr/local/cuda
2024-12-18T00:29:43.0348281Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2024-12-18T00:29:43.0348858Z GITHUB_SERVER_URL=https://github.com
2024-12-18T00:29:43.0349224Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b
2024-12-18T00:29:43.0349577Z REENABLED_ISSUES=
2024-12-18T00:29:43.0349819Z SHLVL=1
2024-12-18T00:29:43.0350036Z MAX_JOBS=14
2024-12-18T00:29:43.0350277Z NV_CUDA_LIB_VERSION=12.4.1-1
2024-12-18T00:29:43.0350551Z NVARCH=x86_64
2024-12-18T00:29:43.0350792Z GITHUB_ACTOR_ID=2453524
2024-12-18T00:29:43.0351154Z GITHUB_WORKFLOW_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:43.0351548Z GITHUB_REF_NAME=release/2.6
2024-12-18T00:29:43.0351850Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-4
2024-12-18T00:29:43.0352301Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2024-12-18T00:29:43.0352723Z GITHUB_JOB=test
2024-12-18T00:29:43.0353007Z NV_LIBNCCL_PACKAGE=libnccl2=2.21.5-1+cuda12.4
2024-12-18T00:29:43.0353418Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2024-12-18T00:29:43.0353807Z NO_TEST_TIMEOUT=False
2024-12-18T00:29:43.0354068Z TD_DISTRIBUTED=False
2024-12-18T00:29:43.0354348Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.1-1
2024-12-18T00:29:43.0354684Z GITHUB_REPOSITORY=pytorch/pytorch
2024-12-18T00:29:43.0354999Z NV_NVPROF_VERSION=12.4.127-1
2024-12-18T00:29:43.0355282Z GITHUB_RETENTION_DAYS=90
2024-12-18T00:29:43.0355553Z OPENSSL_DIR=/opt/openssl
2024-12-18T00:29:43.0355821Z GITHUB_ACTION_REPOSITORY=
2024-12-18T00:29:43.0356575Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-12-18T00:29:43.0357363Z GITHUB_BASE_REF=
2024-12-18T00:29:43.0357799Z ARTIFACTS_FILE_SUFFIX=test-inductor_timm-2-2-linux.g5.4xlarge.nvidia.gpu_34567153040
2024-12-18T00:29:43.0358311Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2024-12-18T00:29:43.0358599Z CI=true
2024-12-18T00:29:43.0358837Z NV_LIBNCCL_PACKAGE_VERSION=2.21.5-1
2024-12-18T00:29:43.0359155Z GITHUB_REPOSITORY_OWNER=pytorch
2024-12-18T00:29:43.0359445Z JOB_ID=34567153040
2024-12-18T00:29:43.0359693Z INSTALLED_PROTOBUF=yes
2024-12-18T00:29:43.0359948Z GITHUB_HEAD_REF=
2024-12-18T00:29:43.0360195Z GITHUB_ACTION_REF=
2024-12-18T00:29:43.0360495Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2024-12-18T00:29:43.0360853Z TEST_SHOWLOCALS=False
2024-12-18T00:29:43.0361119Z GITHUB_WORKFLOW=inductor
2024-12-18T00:29:43.0361393Z DEBIAN_FRONTEND=noninteractive
2024-12-18T00:29:43.0362001Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_24c60c98-1ac2-4da6-a269-1c1e3dd7ad15
2024-12-18T00:29:43.0362618Z NO_TD=False
2024-12-18T00:29:43.0362867Z SKIP_SCCACHE_INITIALIZATION=1
2024-12-18T00:29:43.0363160Z _=/usr/bin/env
2024-12-18T00:29:43.0363580Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2024-12-18T00:29:43.0541224Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch
2024-12-18T00:29:43.0542002Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2024-12-18T00:29:43.0542652Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib
2024-12-18T00:29:43.0543423Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test
2024-12-18T00:29:43.0543877Z + BUILD_DIR=build
2024-12-18T00:29:43.0544139Z + BUILD_RENAMED_DIR=build_renamed
2024-12-18T00:29:43.0544442Z + BUILD_BIN_DIR=build/bin
2024-12-18T00:29:43.0544705Z + SHARD_NUMBER=2
2024-12-18T00:29:43.0544950Z + NUM_TEST_SHARDS=2
2024-12-18T00:29:43.0545205Z + export VALGRIND=ON
2024-12-18T00:29:43.0545460Z + VALGRIND=ON
2024-12-18T00:29:43.0545777Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *clang9* ]]
2024-12-18T00:29:43.0546204Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *xpu* ]]
2024-12-18T00:29:43.0546565Z + [[ 0 == \1 ]]
2024-12-18T00:29:43.0546826Z + [[ False == \1 ]]
2024-12-18T00:29:43.0547209Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *bazel* ]]
2024-12-18T00:29:43.0549359Z ++ realpath build/custom_test_artifacts
2024-12-18T00:29:43.0561687Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2024-12-18T00:29:43.0562370Z + [[ -n '' ]]
2024-12-18T00:29:43.0562708Z + echo 'Environment variables'
2024-12-18T00:29:43.0563018Z Environment variables
2024-12-18T00:29:43.0563276Z + env
2024-12-18T00:29:43.0571980Z INSTALLED_DB=yes
2024-12-18T00:29:43.0572348Z NV_LIBCUBLAS_VERSION=12.4.5.8-1
2024-12-18T00:29:43.0572768Z NVIDIA_VISIBLE_DEVICES=all
2024-12-18T00:29:43.0573154Z NV_NVML_DEV_VERSION=12.4.127-1
2024-12-18T00:29:43.0573641Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-12-18T00:29:43.0574165Z CONTINUE_THROUGH_ERROR=False
2024-12-18T00:29:43.0574509Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.21.5-1+cuda12.4
2024-12-18T00:29:43.0574895Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.21.5-1
2024-12-18T00:29:43.0575305Z BUILD_ENVIRONMENT=linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-12-18T00:29:43.0575719Z HOSTNAME=6581d0cd7381
2024-12-18T00:29:43.0576383Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_24c60c98-1ac2-4da6-a269-1c1e3dd7ad15
2024-12-18T00:29:43.0577102Z GITHUB_ACTION=__self
2024-12-18T00:29:43.0577418Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2024-12-18T00:29:43.0581723Z NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536
2024-12-18T00:29:43.0586522Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-4=12.4.5.8-1
2024-12-18T00:29:43.0586890Z NV_NVTX_VERSION=12.4.127-1
2024-12-18T00:29:43.0587171Z GITHUB_RUN_NUMBER=109429
2024-12-18T00:29:43.0587447Z TEST_CONFIG=inductor_timm
2024-12-18T00:29:43.0587736Z GITHUB_REPOSITORY_OWNER_ID=21003710
2024-12-18T00:29:43.0588073Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2024-12-18T00:29:43.0588383Z IS_A100_RUNNER=0
2024-12-18T00:29:43.0588828Z NV_CUDA_CUDART_DEV_VERSION=12.4.127-1
2024-12-18T00:29:43.0589283Z NV_LIBCUSPARSE_VERSION=12.3.1.170-1
2024-12-18T00:29:43.0589777Z SCRIBE_GRAPHQL_ACCESS_TOKEN=***
2024-12-18T00:29:43.0590078Z NV_LIBNPP_VERSION=12.2.5.30-1
2024-12-18T00:29:43.0590377Z GITHUB_TRIGGERING_ACTOR=malfet
2024-12-18T00:29:43.0590719Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2024-12-18T00:29:43.0591079Z GITHUB_REF_TYPE=branch
2024-12-18T00:29:43.0591353Z TORCH_CUDA_ARCH_LIST=Maxwell
2024-12-18T00:29:43.0591629Z NCCL_VERSION=2.21.5-1
2024-12-18T00:29:43.0591943Z BASE_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:43.0592283Z XLA_CUDA=
2024-12-18T00:29:43.0592615Z HUGGING_FACE_HUB_TOKEN=***
2024-12-18T00:29:43.0593202Z ***
2024-12-18T00:29:43.0593442Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2024-12-18T00:29:43.0593749Z GITHUB_REPOSITORY_ID=65600975
2024-12-18T00:29:43.0594036Z GITHUB_ACTIONS=true
2024-12-18T00:29:43.0594303Z NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:43.0594654Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4=12.4.127-1
2024-12-18T00:29:43.0595024Z NV_LIBNPP_PACKAGE=libnpp-12-4=12.2.5.30-1
2024-12-18T00:29:43.0595454Z SHA1=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:43.0595811Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2024-12-18T00:29:43.0596181Z GITHUB_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:43.0596734Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/release/2.6
2024-12-18T00:29:43.0597240Z UCC_HOME=/usr
2024-12-18T00:29:43.0597493Z NV_LIBCUBLAS_DEV_VERSION=12.4.5.8-1
2024-12-18T00:29:43.0597798Z VERBOSE_TEST_LOGS=False
2024-12-18T00:29:43.0598069Z NVIDIA_PRODUCT_NAME=CUDA
2024-12-18T00:29:43.0598391Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-4
2024-12-18T00:29:43.0598755Z GITHUB_REF=refs/heads/release/2.6
2024-12-18T00:29:43.0599074Z NV_CUDA_CUDART_VERSION=12.4.127-1
2024-12-18T00:29:43.0599365Z SHARD_NUMBER=2
2024-12-18T00:29:43.0599613Z GITHUB_REF_PROTECTED=true
2024-12-18T00:29:43.0599892Z HOME=/var/lib/jenkins
2024-12-18T00:29:43.0600187Z GITHUB_API_URL=https://api.github.com
2024-12-18T00:29:43.0600533Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2024-12-18T00:29:43.0600884Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb
2024-12-18T00:29:43.0601253Z SCCACHE_S3_KEY_PREFIX=inductor
2024-12-18T00:29:43.0601544Z CUDA_VERSION=12.4.1
2024-12-18T00:29:43.0601834Z NV_LIBCUBLAS_PACKAGE=libcublas-12-4=12.4.5.8-1
2024-12-18T00:29:43.0602174Z NUM_TEST_SHARDS=2
2024-12-18T00:29:43.0602413Z UCX_HOME=/usr
2024-12-18T00:29:43.0602765Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-4=12.4.1-1
2024-12-18T00:29:43.0603508Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_24c60c98-1ac2-4da6-a269-1c1e3dd7ad15
2024-12-18T00:29:43.0604333Z JOB_NAME=cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:43.0605140Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_24c60c98-1ac2-4da6-a269-1c1e3dd7ad15
2024-12-18T00:29:43.0605930Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2024-12-18T00:29:43.0606426Z GITHUB_EVENT_NAME=push
2024-12-18T00:29:43.0606695Z DASHBOARD_TAG=
2024-12-18T00:29:43.0606943Z GITHUB_RUN_ID=12383255690
2024-12-18T00:29:43.0607268Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-4=12.2.5.30-1
2024-12-18T00:29:43.0607647Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-4
2024-12-18T00:29:43.0608316Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_24c60c98-1ac2-4da6-a269-1c1e3dd7ad15
2024-12-18T00:29:43.0608976Z GITHUB_ACTOR=malfet
2024-12-18T00:29:43.0609248Z NV_LIBNPP_DEV_VERSION=12.2.5.30-1
2024-12-18T00:29:43.0609544Z PR_NUMBER=
2024-12-18T00:29:43.0609787Z GITHUB_RUN_ATTEMPT=1
2024-12-18T00:29:43.0610035Z VALGRIND=ON
2024-12-18T00:29:43.0610281Z ANACONDA_PYTHON_VERSION=3.10
2024-12-18T00:29:43.0610637Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2024-12-18T00:29:43.0610994Z TERM=vt100
2024-12-18T00:29:43.0611343Z NV_LIBCUSPARSE_DEV_VERSION=12.3.1.170-1
2024-12-18T00:29:43.0611674Z INSTALLED_VISION=yes
2024-12-18T00:29:43.0612012Z BRANCH=release/2.6
2024-12-18T00:29:43.0612274Z SCCACHE_REGION=us-east-1
2024-12-18T00:29:43.0612565Z OPENSSL_ROOT_DIR=/opt/openssl
2024-12-18T00:29:43.0612873Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2024-12-18T00:29:43.0613205Z CUDA_PATH=/usr/local/cuda
2024-12-18T00:29:43.0613730Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2024-12-18T00:29:43.0614309Z GITHUB_SERVER_URL=https://github.com
2024-12-18T00:29:43.0614933Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b
2024-12-18T00:29:43.0615292Z REENABLED_ISSUES=
2024-12-18T00:29:43.0615536Z SHLVL=1
2024-12-18T00:29:43.0615749Z MAX_JOBS=14
2024-12-18T00:29:43.0615982Z NV_CUDA_LIB_VERSION=12.4.1-1
2024-12-18T00:29:43.0616259Z NVARCH=x86_64
2024-12-18T00:29:43.0616490Z GITHUB_ACTOR_ID=2453524
2024-12-18T00:29:43.0616837Z GITHUB_WORKFLOW_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:43.0617241Z GITHUB_REF_NAME=release/2.6
2024-12-18T00:29:43.0617558Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-4
2024-12-18T00:29:43.0618076Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2024-12-18T00:29:43.0618498Z GITHUB_JOB=test
2024-12-18T00:29:43.0618773Z NV_LIBNCCL_PACKAGE=libnccl2=2.21.5-1+cuda12.4
2024-12-18T00:29:43.0619185Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2024-12-18T00:29:43.0619568Z NO_TEST_TIMEOUT=False
2024-12-18T00:29:43.0619828Z TD_DISTRIBUTED=False
2024-12-18T00:29:43.0620101Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.1-1
2024-12-18T00:29:43.0620435Z GITHUB_REPOSITORY=pytorch/pytorch
2024-12-18T00:29:43.0620744Z NV_NVPROF_VERSION=12.4.127-1
2024-12-18T00:29:43.0621033Z GITHUB_RETENTION_DAYS=90
2024-12-18T00:29:43.0621312Z OPENSSL_DIR=/opt/openssl
2024-12-18T00:29:43.0621587Z GITHUB_ACTION_REPOSITORY=
2024-12-18T00:29:43.0622365Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-12-18T00:29:43.0623152Z GITHUB_BASE_REF=
2024-12-18T00:29:43.0623586Z ARTIFACTS_FILE_SUFFIX=test-inductor_timm-2-2-linux.g5.4xlarge.nvidia.gpu_34567153040
2024-12-18T00:29:43.0624102Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2024-12-18T00:29:43.0624382Z CI=true
2024-12-18T00:29:43.0624626Z NV_LIBNCCL_PACKAGE_VERSION=2.21.5-1
2024-12-18T00:29:43.0624950Z GITHUB_REPOSITORY_OWNER=pytorch
2024-12-18T00:29:43.0625243Z JOB_ID=34567153040
2024-12-18T00:29:43.0625498Z INSTALLED_PROTOBUF=yes
2024-12-18T00:29:43.0625754Z GITHUB_HEAD_REF=
2024-12-18T00:29:43.0625999Z GITHUB_ACTION_REF=
2024-12-18T00:29:43.0626303Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2024-12-18T00:29:43.0626667Z TEST_SHOWLOCALS=False
2024-12-18T00:29:43.0626943Z GITHUB_WORKFLOW=inductor
2024-12-18T00:29:43.0627224Z DEBIAN_FRONTEND=noninteractive
2024-12-18T00:29:43.0627851Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_24c60c98-1ac2-4da6-a269-1c1e3dd7ad15
2024-12-18T00:29:43.0628484Z NO_TD=False
2024-12-18T00:29:43.0628738Z SKIP_SCCACHE_INITIALIZATION=1
2024-12-18T00:29:43.0629026Z _=/usr/bin/env
2024-12-18T00:29:43.0629267Z + echo 'Testing pytorch'
2024-12-18T00:29:43.0629535Z Testing pytorch
2024-12-18T00:29:43.0629788Z + export LANG=C.UTF-8
2024-12-18T00:29:43.0630046Z + LANG=C.UTF-8
2024-12-18T00:29:43.0630281Z + PR_NUMBER=
2024-12-18T00:29:43.0630520Z + [[ inductor_timm == \d\e\f\a\u\l\t ]]
2024-12-18T00:29:43.0630857Z + [[ inductor_timm == \d\i\s\t\r\i\b\u\t\e\d ]]
2024-12-18T00:29:43.0631192Z + [[ inductor_timm == \s\l\o\w ]]
2024-12-18T00:29:43.0631594Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *slow-gradcheck* ]]
2024-12-18T00:29:43.0632070Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *cuda* ]]
2024-12-18T00:29:43.0632467Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2024-12-18T00:29:43.0632825Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2024-12-18T00:29:43.0633152Z + [[ inductor_timm == *crossref* ]]
2024-12-18T00:29:43.0633665Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *rocm* ]]
2024-12-18T00:29:43.0634093Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *xpu* ]]
2024-12-18T00:29:43.0634630Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]]
2024-12-18T00:29:43.0635031Z + pip_install --user ninja==1.10.2
2024-12-18T00:29:43.0635430Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T00:29:43.0635923Z + python3 -m pip install --progress-bar off --user ninja==1.10.2
2024-12-18T00:29:44.1975676Z Collecting ninja==1.10.2
2024-12-18T00:29:44.2152387Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB)
2024-12-18T00:29:44.2834741Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2024-12-18T00:29:44.8859609Z Installing collected packages: ninja
2024-12-18T00:29:44.8939418Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2024-12-18T00:29:44.8941362Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2024-12-18T00:29:44.9451268Z Successfully installed ninja-1.10.2
2024-12-18T00:29:45.0245607Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-12-18T00:29:45.0247236Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-12-18T00:29:45.0248219Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *aarch64* ]]
2024-12-18T00:29:45.0248604Z + install_tlparse
2024-12-18T00:29:45.0248877Z + pip_install --user tlparse==0.3.25
2024-12-18T00:29:45.0249278Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T00:29:45.0249779Z + python3 -m pip install --progress-bar off --user tlparse==0.3.25
2024-12-18T00:29:45.4627322Z Collecting tlparse==0.3.25
2024-12-18T00:29:45.4799046Z   Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB)
2024-12-18T00:29:45.5333483Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB)
2024-12-18T00:29:46.1844374Z Installing collected packages: tlparse
2024-12-18T00:29:46.2179243Z Successfully installed tlparse-0.3.25
2024-12-18T00:29:46.2968495Z ++ python -m site --user-base
2024-12-18T00:29:46.3214149Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-12-18T00:29:46.3215619Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *asan* ]]
2024-12-18T00:29:46.3216085Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *-debug* ]]
2024-12-18T00:29:46.3216613Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]]
2024-12-18T00:29:46.3217308Z + echo 'We are not in debug mode: linux-focal-cuda12.4-py3.10-gcc9-sm86. Expect the assertion to pass'
2024-12-18T00:29:46.3218226Z We are not in debug mode: linux-focal-cuda12.4-py3.10-gcc9-sm86. Expect the assertion to pass
2024-12-18T00:29:46.3219256Z + cd test
2024-12-18T00:29:46.3219613Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)'
2024-12-18T00:29:47.9948068Z + [[ inductor_timm == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2024-12-18T00:29:47.9948491Z + [[ inductor_timm == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2024-12-18T00:29:47.9953158Z + DYNAMO_BENCHMARK_FLAGS=()
2024-12-18T00:29:47.9953629Z + [[ inductor_timm == *pr_time_benchmarks* ]]
2024-12-18T00:29:47.9954078Z + [[ inductor_timm == *dynamo_eager* ]]
2024-12-18T00:29:47.9954497Z + [[ inductor_timm == *aot_eager* ]]
2024-12-18T00:29:47.9954856Z + [[ inductor_timm == *aot_inductor* ]]
2024-12-18T00:29:47.9955185Z + [[ inductor_timm == *inductor* ]]
2024-12-18T00:29:47.9955497Z + [[ inductor_timm != *perf* ]]
2024-12-18T00:29:47.9956230Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor)
2024-12-18T00:29:47.9956575Z + [[ inductor_timm == *dynamic* ]]
2024-12-18T00:29:47.9957040Z + [[ inductor_timm == *cpu* ]]
2024-12-18T00:29:47.9957349Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda)
2024-12-18T00:29:47.9985954Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *libtorch* ]]
2024-12-18T00:29:47.9986596Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *-bazel-* ]]
2024-12-18T00:29:47.9989534Z + cd test
2024-12-18T00:29:47.9990010Z + python -c 'import torch; print(torch.__config__.show())'
2024-12-18T00:29:49.4921914Z PyTorch built with:
2024-12-18T00:29:49.4922367Z   - GCC 9.4
2024-12-18T00:29:49.4922858Z   - C++ Version: 201703
2024-12-18T00:29:49.4923509Z   - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2024-12-18T00:29:49.4924308Z   - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af)
2024-12-18T00:29:49.4924932Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2024-12-18T00:29:49.4925424Z   - LAPACK is enabled (usually provided by MKL)
2024-12-18T00:29:49.4925911Z   - NNPACK is enabled
2024-12-18T00:29:49.4926280Z   - CPU capability usage: AVX2
2024-12-18T00:29:49.4926645Z   - CUDA Runtime 12.4
2024-12-18T00:29:49.4936478Z   - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86
2024-12-18T00:29:49.4936887Z   - CuDNN 90.1
2024-12-18T00:29:49.4937133Z   - Magma 2.6.1
2024-12-18T00:29:49.4942021Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7, CUDA_VERSION=12.4, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.6.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 
2024-12-18T00:29:49.4946913Z 
2024-12-18T00:29:49.8261732Z + cd test
2024-12-18T00:29:49.8262132Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2024-12-18T00:29:51.2046088Z ATen/Parallel:
2024-12-18T00:29:51.2046471Z 	at::get_num_threads() : 8
2024-12-18T00:29:51.2046788Z 	at::get_num_interop_threads() : 16
2024-12-18T00:29:51.2047154Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2024-12-18T00:29:51.2047477Z 	omp_get_max_threads() : 8
2024-12-18T00:29:51.2048041Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2024-12-18T00:29:51.2048622Z 	mkl_get_max_threads() : 8
2024-12-18T00:29:51.2049025Z Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af)
2024-12-18T00:29:51.2049483Z std::thread::hardware_concurrency() : 16
2024-12-18T00:29:51.2049817Z Environment variables:
2024-12-18T00:29:51.2050091Z 	OMP_NUM_THREADS : [not set]
2024-12-18T00:29:51.2050370Z 	MKL_NUM_THREADS : [not set]
2024-12-18T00:29:51.2050665Z ATen parallel backend: OpenMP
2024-12-18T00:29:51.2050860Z 
2024-12-18T00:29:51.4922503Z + [[ inductor_timm == *numpy_2* ]]
2024-12-18T00:29:51.4923027Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *aarch64* ]]
2024-12-18T00:29:51.4923507Z + [[ inductor_timm == *backward* ]]
2024-12-18T00:29:51.4923835Z + [[ inductor_timm == *xla* ]]
2024-12-18T00:29:51.4924582Z + [[ inductor_timm == *executorch* ]]
2024-12-18T00:29:51.4925111Z + [[ inductor_timm == \j\i\t\_\l\e\g\a\c\y ]]
2024-12-18T00:29:51.4925522Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *libtorch* ]]
2024-12-18T00:29:51.4925926Z + [[ inductor_timm == distributed ]]
2024-12-18T00:29:51.4926275Z + [[ inductor_timm == *inductor_distributed* ]]
2024-12-18T00:29:51.4926633Z + [[ inductor_timm == *inductor-halide* ]]
2024-12-18T00:29:51.4927002Z + [[ inductor_timm == *inductor-triton-cpu* ]]
2024-12-18T00:29:51.4927395Z + [[ inductor_timm == *inductor-micro-benchmark* ]]
2024-12-18T00:29:51.4927773Z + [[ inductor_timm == *huggingface* ]]
2024-12-18T00:29:51.4928097Z + [[ inductor_timm == *timm* ]]
2024-12-18T00:29:51.4928448Z + install_torchvision
2024-12-18T00:29:51.4928764Z + local orig_preload
2024-12-18T00:29:51.4929028Z + local commit
2024-12-18T00:29:51.4929345Z ++ get_pinned_commit vision
2024-12-18T00:29:51.4929669Z ++ cat .github/ci_commit_pins/vision.txt
2024-12-18T00:29:51.4947005Z + commit=d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:29:51.4947452Z + orig_preload=
2024-12-18T00:29:51.4947711Z + '[' -n '' ']'
2024-12-18T00:29:51.4948342Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:29:51.4949055Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T00:29:51.4949860Z + python3 -m pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:29:51.8300009Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:29:51.8303694Z   Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-a16xxw7b
2024-12-18T00:29:51.8336092Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-a16xxw7b
2024-12-18T00:29:53.2954061Z   Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f'
2024-12-18T00:29:53.2981462Z   Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:29:54.6685421Z   Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:29:55.0367054Z   Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:29:57.6529402Z   Preparing metadata (setup.py) ... [?25l- \ done
2024-12-18T00:29:57.6562666Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4)
2024-12-18T00:29:57.6565637Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.6.0a0+git0cdf8b1)
2024-12-18T00:29:57.6570114Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (11.0.0)
2024-12-18T00:29:57.6638190Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.16.1)
2024-12-18T00:29:57.6641993Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2)
2024-12-18T00:29:57.6645741Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8)
2024-12-18T00:29:57.6649096Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4)
2024-12-18T00:29:57.6652355Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.10.0)
2024-12-18T00:29:57.6657731Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.1)
2024-12-18T00:29:57.6671945Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0)
2024-12-18T00:29:57.7150462Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (3.0.2)
2024-12-18T00:29:57.7213400Z Building wheels for collected packages: torchvision
2024-12-18T00:31:11.3619898Z   Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done
2024-12-18T00:31:11.3654497Z [?25h  Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2028224 sha256=6ff1e50d2287793b00a39436ec3fe465cd06dd5383c6c88fad46a3804fe3eacc
2024-12-18T00:31:11.3656948Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96
2024-12-18T00:31:11.3690250Z Successfully built torchvision
2024-12-18T00:31:11.8649007Z Installing collected packages: torchvision
2024-12-18T00:31:12.2718560Z Successfully installed torchvision-0.19.0a0+d23a6e1
2024-12-18T00:31:12.4142075Z + '[' -n '' ']'
2024-12-18T00:31:12.4142370Z + id=1
2024-12-18T00:31:12.4142612Z + test_dynamo_benchmark timm_models 1
2024-12-18T00:31:12.4146306Z ++ pwd
2024-12-18T00:31:12.4149823Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2024-12-18T00:31:12.4150253Z + local suite=timm_models
2024-12-18T00:31:12.4150526Z + shift
2024-12-18T00:31:12.4150746Z + local shard_id=1
2024-12-18T00:31:12.4150981Z + shift
2024-12-18T00:31:12.4151221Z + [[ inductor_timm == *perf_compare* ]]
2024-12-18T00:31:12.4151552Z + [[ inductor_timm == *perf* ]]
2024-12-18T00:31:12.4151861Z + [[ inductor_timm == *cpu* ]]
2024-12-18T00:31:12.4152163Z + [[ inductor_timm == *aot_inductor* ]]
2024-12-18T00:31:12.4152624Z + test_single_dynamo_benchmark inference timm_models 1 --inference --bfloat16
2024-12-18T00:31:12.4155605Z ++ pwd
2024-12-18T00:31:12.4158437Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2024-12-18T00:31:12.4158910Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports
2024-12-18T00:31:12.4215326Z + local name=inference
2024-12-18T00:31:12.4215844Z + shift
2024-12-18T00:31:12.4216134Z + local suite=timm_models
2024-12-18T00:31:12.4216490Z + shift
2024-12-18T00:31:12.4216725Z + local shard_id=1
2024-12-18T00:31:12.4216966Z + shift
2024-12-18T00:31:12.4217233Z + partition_flags=()
2024-12-18T00:31:12.4217541Z + local partition_flags
2024-12-18T00:31:12.4217815Z + [[ -n 2 ]]
2024-12-18T00:31:12.4218107Z + [[ -n 1 ]]
2024-12-18T00:31:12.4218511Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id")
2024-12-18T00:31:12.4219031Z + [[ inductor_timm == *perf_compare* ]]
2024-12-18T00:31:12.4219367Z + [[ inductor_timm == *perf* ]]
2024-12-18T00:31:12.4219669Z + [[ inductor_timm == *_avx2* ]]
2024-12-18T00:31:12.4219960Z + [[ inductor_timm == *_avx512* ]]
2024-12-18T00:31:12.4221032Z + python benchmarks/dynamo/timm_models.py --ci --accuracy --timing --explain --inductor --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/inference_timm_models.csv
2024-12-18T00:31:19.4164156Z 
2024-12-18T00:31:19.5879916Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:31:19.5880166Z 
2024-12-18T00:31:19.7156254Z model.safetensors:   0% 0.00/29.6M [00:00<?, ?B/s]
2024-12-18T00:31:19.7156572Z 
2024-12-18T00:31:19.7209006Z model.safetensors: 100% 29.6M/29.6M [00:00<00:00, 232MB/s]
2024-12-18T00:31:19.7209483Z model.safetensors: 100% 29.6M/29.6M [00:00<00:00, 223MB/s]
2024-12-18T00:31:20.4792027Z 
2024-12-18T00:31:20.4792427Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:31:20.4895561Z cuda eval  mixnet_l                           
2024-12-18T00:31:55.1194405Z pass
2024-12-18T00:31:55.1208961Z TIMING: _recursive_pre_grad_passes:0.04402 _recursive_joint_graph_passes:0.24858 _recursive_post_grad_passes:0.3335 async_compile.wait:3.74855 code_gen:9.94762 inductor_compile:16.1261 backend_compile:23.46131 entire_frame_compile:32.00359 total_wall_time:32.00359
2024-12-18T00:31:55.1210547Z STATS: call_* op count: 1042 | FakeTensorMode.__torch_dispatch__:27008 | FakeTensor.__torch_dispatch__:4270 | ProxyTorchDispatchMode.__torch_dispatch__:11806
2024-12-18T00:31:55.1211395Z Dynamo produced 1 graphs covering 1042 ops with 0 graph breaks (0 unique)
2024-12-18T00:32:00.7410467Z 
2024-12-18T00:32:00.8517202Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:32:00.8517680Z 
2024-12-18T00:32:00.9346656Z model.safetensors:   0% 0.00/17.7M [00:00<?, ?B/s]
2024-12-18T00:32:00.9347107Z model.safetensors: 100% 17.7M/17.7M [00:00<00:00, 214MB/s]
2024-12-18T00:32:01.5134472Z 
2024-12-18T00:32:01.5135009Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:32:01.5195277Z cuda eval  mnasnet_100                        
2024-12-18T00:32:16.6700822Z pass
2024-12-18T00:32:16.6713245Z TIMING: _recursive_pre_grad_passes:0.017 _recursive_joint_graph_passes:0.19237 _recursive_post_grad_passes:0.21801 async_compile.wait:0.31617 code_gen:1.65067 inductor_compile:5.03142 backend_compile:9.04399 entire_frame_compile:13.33234 total_wall_time:13.33234
2024-12-18T00:32:16.6715123Z STATS: call_* op count: 304 | FakeTensorMode.__torch_dispatch__:14707 | ProxyTorchDispatchMode.__torch_dispatch__:6825 | FakeTensor.__torch_dispatch__:2626
2024-12-18T00:32:16.6715979Z Dynamo produced 1 graphs covering 304 ops with 0 graph breaks (0 unique)
2024-12-18T00:32:21.7062087Z 
2024-12-18T00:32:21.8164057Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:32:21.8164316Z 
2024-12-18T00:32:21.8917750Z model.safetensors:   0% 0.00/14.2M [00:00<?, ?B/s]
2024-12-18T00:32:21.8918208Z model.safetensors: 100% 14.2M/14.2M [00:00<00:00, 189MB/s]
2024-12-18T00:32:22.4715494Z 
2024-12-18T00:32:22.4716154Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:32:22.4780272Z cuda eval  mobilenetv2_100                    
2024-12-18T00:32:37.9946644Z pass
2024-12-18T00:32:37.9958795Z TIMING: _recursive_pre_grad_passes:0.01718 _recursive_joint_graph_passes:0.20373 _recursive_post_grad_passes:0.22621 async_compile.wait:0.28147 code_gen:1.58272 inductor_compile:5.01891 backend_compile:9.39293 entire_frame_compile:13.71125 total_wall_time:13.71125
2024-12-18T00:32:37.9960505Z STATS: call_* op count: 304 | FakeTensorMode.__torch_dispatch__:15178 | FakeTensor.__torch_dispatch__:2650 | ProxyTorchDispatchMode.__torch_dispatch__:7735
2024-12-18T00:32:37.9961347Z Dynamo produced 1 graphs covering 304 ops with 0 graph breaks (0 unique)
2024-12-18T00:32:43.0311769Z 
2024-12-18T00:32:43.1550230Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:32:43.1550476Z 
2024-12-18T00:32:43.2538866Z model.safetensors:   0% 0.00/22.1M [00:00<?, ?B/s]
2024-12-18T00:32:43.2539357Z model.safetensors: 100% 22.1M/22.1M [00:00<00:00, 224MB/s]
2024-12-18T00:32:43.8781752Z 
2024-12-18T00:32:43.8782321Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:32:43.8847352Z cuda eval  mobilenetv3_large_100              
2024-12-18T00:33:01.3754347Z pass
2024-12-18T00:33:01.3765701Z TIMING: _recursive_pre_grad_passes:0.01859 _recursive_joint_graph_passes:0.20056 _recursive_post_grad_passes:0.45326 async_compile.wait:0.53037 code_gen:2.6522 inductor_compile:6.54558 backend_compile:11.07027 entire_frame_compile:15.65969 total_wall_time:15.65969
2024-12-18T00:33:01.3767200Z STATS: call_* op count: 372 | FakeTensorMode.__torch_dispatch__:15942 | FakeTensor.__torch_dispatch__:2824 | ProxyTorchDispatchMode.__torch_dispatch__:7879
2024-12-18T00:33:01.3768033Z Dynamo produced 1 graphs covering 372 ops with 0 graph breaks (0 unique)
2024-12-18T00:33:06.4186963Z 
2024-12-18T00:33:06.5794360Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:33:06.5794620Z 
2024-12-18T00:33:06.6787356Z model.safetensors:   0% 0.00/22.4M [00:00<?, ?B/s]
2024-12-18T00:33:06.6787809Z model.safetensors: 100% 22.4M/22.4M [00:00<00:00, 226MB/s]
2024-12-18T00:33:07.2811054Z 
2024-12-18T00:33:07.2811247Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:33:07.2885349Z cuda eval  mobilevit_s                        
2024-12-18T00:33:36.5312771Z pass
2024-12-18T00:33:36.5324335Z TIMING: _recursive_pre_grad_passes:0.02329 _recursive_joint_graph_passes:0.55826 _recursive_post_grad_passes:0.44761 async_compile.wait:1.43423 code_gen:10.6164 inductor_compile:16.0336 backend_compile:21.46447 entire_frame_compile:26.85666 total_wall_time:26.85666
2024-12-18T00:33:36.5325815Z STATS: call_* op count: 652 | FakeTensorMode.__torch_dispatch__:23417 | FakeTensor.__torch_dispatch__:3029 | ProxyTorchDispatchMode.__torch_dispatch__:10832
2024-12-18T00:33:36.5326655Z Dynamo produced 1 graphs covering 652 ops with 0 graph breaks (0 unique)
2024-12-18T00:33:41.8197132Z 
2024-12-18T00:33:42.2651693Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:33:42.2652201Z 
2024-12-18T00:33:42.3968068Z model.safetensors:   0% 0.00/140M [00:00<?, ?B/s]
2024-12-18T00:33:42.3968417Z 
2024-12-18T00:33:42.5228216Z model.safetensors:  22% 31.5M/140M [00:00<00:00, 239MB/s]
2024-12-18T00:33:42.5228556Z 
2024-12-18T00:33:42.6497444Z model.safetensors:  45% 62.9M/140M [00:00<00:00, 245MB/s]
2024-12-18T00:33:42.6497751Z 
2024-12-18T00:33:42.7809189Z model.safetensors:  67% 94.4M/140M [00:00<00:00, 246MB/s]
2024-12-18T00:33:42.7809484Z 
2024-12-18T00:33:42.8446618Z model.safetensors:  90% 126M/140M [00:00<00:00, 244MB/s] 
2024-12-18T00:33:42.8447083Z model.safetensors: 100% 140M/140M [00:00<00:00, 242MB/s]
2024-12-18T00:33:43.4135798Z 
2024-12-18T00:33:43.4136588Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:33:43.4190125Z cuda eval  nfnet_l0                           
2024-12-18T00:34:06.2221212Z pass
2024-12-18T00:34:06.2232132Z TIMING: _recursive_pre_grad_passes:0.03216 _recursive_joint_graph_passes:0.32548 _recursive_post_grad_passes:0.27853 async_compile.wait:0.721 code_gen:4.85056 inductor_compile:8.90891 backend_compile:14.46658 entire_frame_compile:20.01397 total_wall_time:20.01397
2024-12-18T00:34:06.2233735Z STATS: call_* op count: 1042 | FakeTensorMode.__torch_dispatch__:22514 | FakeTensor.__torch_dispatch__:2929 | ProxyTorchDispatchMode.__torch_dispatch__:9916
2024-12-18T00:34:06.2234579Z Dynamo produced 1 graphs covering 1042 ops with 0 graph breaks (0 unique)
2024-12-18T00:34:11.5400210Z 
2024-12-18T00:34:12.1913349Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:34:12.1913593Z 
2024-12-18T00:34:12.3272366Z model.safetensors:   0% 0.00/295M [00:00<?, ?B/s]
2024-12-18T00:34:12.3272656Z 
2024-12-18T00:34:12.4529640Z model.safetensors:  11% 31.5M/295M [00:00<00:01, 232MB/s]
2024-12-18T00:34:12.4529996Z 
2024-12-18T00:34:12.5859823Z model.safetensors:  21% 62.9M/295M [00:00<00:00, 242MB/s]
2024-12-18T00:34:12.5860164Z 
2024-12-18T00:34:12.7132661Z model.safetensors:  32% 94.4M/295M [00:00<00:00, 240MB/s]
2024-12-18T00:34:12.7132972Z 
2024-12-18T00:34:12.8481174Z model.safetensors:  43% 126M/295M [00:00<00:00, 242MB/s] 
2024-12-18T00:34:12.8481506Z 
2024-12-18T00:34:12.9773819Z model.safetensors:  53% 157M/295M [00:00<00:00, 239MB/s]
2024-12-18T00:34:12.9774132Z 
2024-12-18T00:34:13.1092048Z model.safetensors:  64% 189M/295M [00:00<00:00, 241MB/s]
2024-12-18T00:34:13.1092370Z 
2024-12-18T00:34:13.2390566Z model.safetensors:  75% 220M/295M [00:00<00:00, 240MB/s]
2024-12-18T00:34:13.2390870Z 
2024-12-18T00:34:13.3816353Z model.safetensors:  85% 252M/295M [00:01<00:00, 241MB/s]
2024-12-18T00:34:13.3816659Z 
2024-12-18T00:34:13.4364848Z model.safetensors:  96% 283M/295M [00:01<00:00, 234MB/s]
2024-12-18T00:34:13.4365315Z model.safetensors: 100% 295M/295M [00:01<00:00, 237MB/s]
2024-12-18T00:34:14.0948475Z 
2024-12-18T00:34:14.0948993Z loading model: 0it [00:02, ?it/s]
2024-12-18T00:34:14.0993131Z cuda eval  pit_b_224                          
2024-12-18T00:34:30.1537891Z pass
2024-12-18T00:34:30.1579947Z TIMING: _recursive_pre_grad_passes:0.01849 _recursive_joint_graph_passes:0.3566 _recursive_post_grad_passes:0.45375 async_compile.wait:1.19523 code_gen:3.08071 inductor_compile:6.12183 backend_compile:9.84641 entire_frame_compile:13.05267 total_wall_time:13.05267
2024-12-18T00:34:30.1581640Z STATS: call_* op count: 612 | FakeTensorMode.__torch_dispatch__:18443 | FakeTensor.__torch_dispatch__:1929 | ProxyTorchDispatchMode.__torch_dispatch__:8361
2024-12-18T00:34:30.1582480Z Dynamo produced 1 graphs covering 612 ops with 0 graph breaks (0 unique)
2024-12-18T00:34:35.1561007Z 
2024-12-18T00:34:35.7027153Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:34:35.7027490Z 
2024-12-18T00:34:35.8379111Z model.safetensors:   0% 0.00/345M [00:00<?, ?B/s]
2024-12-18T00:34:35.8379408Z 
2024-12-18T00:34:35.9635483Z model.safetensors:   9% 31.5M/345M [00:00<00:01, 233MB/s]
2024-12-18T00:34:35.9635841Z 
2024-12-18T00:34:36.0894711Z model.safetensors:  18% 62.9M/345M [00:00<00:01, 243MB/s]
2024-12-18T00:34:36.0895035Z 
2024-12-18T00:34:36.2167682Z model.safetensors:  27% 94.4M/345M [00:00<00:01, 246MB/s]
2024-12-18T00:34:36.2167996Z 
2024-12-18T00:34:36.3558611Z model.safetensors:  36% 126M/345M [00:00<00:00, 246MB/s] 
2024-12-18T00:34:36.3558931Z 
2024-12-18T00:34:36.4866807Z model.safetensors:  46% 157M/345M [00:00<00:00, 239MB/s]
2024-12-18T00:34:36.4867123Z 
2024-12-18T00:34:36.6157155Z model.safetensors:  55% 189M/345M [00:00<00:00, 239MB/s]
2024-12-18T00:34:36.6157473Z 
2024-12-18T00:34:36.7437102Z model.safetensors:  64% 220M/345M [00:00<00:00, 241MB/s]
2024-12-18T00:34:36.7437417Z 
2024-12-18T00:34:36.8734666Z model.safetensors:  73% 252M/345M [00:01<00:00, 242MB/s]
2024-12-18T00:34:36.8734981Z 
2024-12-18T00:34:37.0042855Z model.safetensors:  82% 283M/345M [00:01<00:00, 242MB/s]
2024-12-18T00:34:37.0043201Z 
2024-12-18T00:34:37.1335639Z model.safetensors:  91% 315M/345M [00:01<00:00, 242MB/s]
2024-12-18T00:34:37.1335956Z 
2024-12-18T00:34:37.1394194Z model.safetensors: 100% 345M/345M [00:01<00:00, 240MB/s]
2024-12-18T00:34:37.1394684Z model.safetensors: 100% 345M/345M [00:01<00:00, 240MB/s]
2024-12-18T00:34:38.2072425Z 
2024-12-18T00:34:38.2072834Z loading model: 0it [00:03, ?it/s]
2024-12-18T00:34:38.2349793Z cuda eval  pnasnet5large                      
2024-12-18T00:35:51.3112420Z pass
2024-12-18T00:35:51.3113496Z TIMING: _recursive_pre_grad_passes:0.08392 _recursive_joint_graph_passes:0.42734 _recursive_post_grad_passes:0.88325 async_compile.wait:2.75125 code_gen:15.32914 inductor_compile:28.31334 backend_compile:46.66092 entire_frame_compile:67.65006 total_wall_time:67.65006
2024-12-18T00:35:51.3115327Z STATS: call_* op count: 1912 | FakeTensorMode.__torch_dispatch__:68290 | ProxyTorchDispatchMode.__torch_dispatch__:32084 | FakeTensor.__torch_dispatch__:11167
2024-12-18T00:35:51.3116182Z Dynamo produced 1 graphs covering 1912 ops with 0 graph breaks (0 unique)
2024-12-18T00:35:58.1081042Z 
2024-12-18T00:35:58.7516760Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:35:58.7517024Z 
2024-12-18T00:35:58.8832607Z model.safetensors:   0% 0.00/225M [00:00<?, ?B/s]
2024-12-18T00:35:58.8833021Z 
2024-12-18T00:35:59.0003212Z model.safetensors:   9% 21.0M/225M [00:00<00:01, 160MB/s]
2024-12-18T00:35:59.0003708Z 
2024-12-18T00:35:59.1305203Z model.safetensors:  19% 41.9M/225M [00:00<00:01, 170MB/s]
2024-12-18T00:35:59.1305655Z 
2024-12-18T00:35:59.2585590Z model.safetensors:  33% 73.4M/225M [00:00<00:00, 204MB/s]
2024-12-18T00:35:59.2586027Z 
2024-12-18T00:35:59.3854116Z model.safetensors:  47% 105M/225M [00:00<00:00, 221MB/s] 
2024-12-18T00:35:59.3854554Z 
2024-12-18T00:35:59.5124697Z model.safetensors:  61% 136M/225M [00:00<00:00, 231MB/s]
2024-12-18T00:35:59.5125125Z 
2024-12-18T00:35:59.6432484Z model.safetensors:  75% 168M/225M [00:00<00:00, 236MB/s]
2024-12-18T00:35:59.6432918Z 
2024-12-18T00:35:59.7535339Z model.safetensors:  89% 199M/225M [00:00<00:00, 238MB/s]
2024-12-18T00:35:59.7535802Z 
2024-12-18T00:35:59.7563778Z model.safetensors: 100% 225M/225M [00:01<00:00, 236MB/s]
2024-12-18T00:35:59.7564823Z model.safetensors: 100% 225M/225M [00:01<00:00, 224MB/s]
2024-12-18T00:36:00.4444635Z 
2024-12-18T00:36:00.4445208Z loading model: 0it [00:02, ?it/s]
2024-12-18T00:36:00.4543572Z cuda eval  poolformer_m36                     
2024-12-18T00:36:46.4577155Z pass
2024-12-18T00:36:46.4589245Z TIMING: _recursive_pre_grad_passes:0.03521 _recursive_joint_graph_passes:0.41148 _recursive_post_grad_passes:0.35517 async_compile.wait:0.732 code_gen:20.56797 inductor_compile:28.44296 backend_compile:35.69748 entire_frame_compile:42.87046 total_wall_time:42.87046
2024-12-18T00:36:46.4591284Z STATS: call_* op count: 1100 | FakeTensorMode.__torch_dispatch__:27124 | FakeTensor.__torch_dispatch__:4390 | ProxyTorchDispatchMode.__torch_dispatch__:13709
2024-12-18T00:36:46.4592139Z Dynamo produced 1 graphs covering 1100 ops with 0 graph breaks (0 unique)
2024-12-18T00:36:52.5877871Z 
2024-12-18T00:36:52.6956808Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:36:52.6957075Z 
2024-12-18T00:36:52.7447312Z model.safetensors:   0% 0.00/12.8M [00:00<?, ?B/s]
2024-12-18T00:36:52.7447786Z model.safetensors: 100% 12.8M/12.8M [00:00<00:00, 261MB/s]
2024-12-18T00:36:53.3555598Z 
2024-12-18T00:36:53.3556080Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:36:53.3623091Z cuda eval  regnety_002                        
2024-12-18T00:37:10.2744806Z pass
2024-12-18T00:37:10.2755932Z TIMING: _recursive_pre_grad_passes:0.02098 _recursive_joint_graph_passes:0.18817 _recursive_post_grad_passes:0.20301 async_compile.wait:0.40905 code_gen:2.41947 inductor_compile:6.12186 backend_compile:10.29005 entire_frame_compile:15.08466 total_wall_time:15.08466
2024-12-18T00:37:10.2757438Z STATS: call_* op count: 446 | FakeTensorMode.__torch_dispatch__:15236 | ProxyTorchDispatchMode.__torch_dispatch__:6925 | FakeTensor.__torch_dispatch__:2539
2024-12-18T00:37:10.2758287Z Dynamo produced 1 graphs covering 446 ops with 0 graph breaks (0 unique)
2024-12-18T00:37:15.3277643Z 
2024-12-18T00:37:15.6762091Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:37:15.6762344Z 
2024-12-18T00:37:15.8054180Z model.safetensors:   0% 0.00/113M [00:00<?, ?B/s]
2024-12-18T00:37:15.8054485Z 
2024-12-18T00:37:15.9252817Z model.safetensors:  28% 31.5M/113M [00:00<00:00, 244MB/s]
2024-12-18T00:37:15.9253128Z 
2024-12-18T00:37:16.0599110Z model.safetensors:  56% 62.9M/113M [00:00<00:00, 254MB/s]
2024-12-18T00:37:16.0599420Z 
2024-12-18T00:37:16.1454100Z model.safetensors:  83% 94.4M/113M [00:00<00:00, 245MB/s]
2024-12-18T00:37:16.1455004Z model.safetensors: 100% 113M/113M [00:00<00:00, 241MB/s] 
2024-12-18T00:37:16.7958388Z 
2024-12-18T00:37:16.7958791Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:37:16.8030104Z cuda eval  repvgg_a2                          
2024-12-18T00:37:33.7868660Z pass
2024-12-18T00:37:33.7880500Z TIMING: _recursive_pre_grad_passes:0.01814 _recursive_joint_graph_passes:0.20053 _recursive_post_grad_passes:0.25215 async_compile.wait:0.55569 code_gen:1.96311 inductor_compile:5.55572 backend_compile:9.77946 entire_frame_compile:14.51071 total_wall_time:14.51071
2024-12-18T00:37:33.7882016Z STATS: call_* op count: 340 | FakeTensorMode.__torch_dispatch__:15783 | ProxyTorchDispatchMode.__torch_dispatch__:7841 | FakeTensor.__torch_dispatch__:3033
2024-12-18T00:37:33.7882872Z Dynamo produced 1 graphs covering 340 ops with 0 graph breaks (0 unique)
2024-12-18T00:37:38.8001425Z 
2024-12-18T00:37:39.3452434Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:37:39.3452739Z 
2024-12-18T00:37:39.4845055Z model.safetensors:   0% 0.00/181M [00:00<?, ?B/s]
2024-12-18T00:37:39.4845583Z 
2024-12-18T00:37:39.6068676Z model.safetensors:  17% 31.5M/181M [00:00<00:00, 226MB/s]
2024-12-18T00:37:39.6068974Z 
2024-12-18T00:37:39.7330770Z model.safetensors:  35% 62.9M/181M [00:00<00:00, 243MB/s]
2024-12-18T00:37:39.7331057Z 
2024-12-18T00:37:39.8609883Z model.safetensors:  52% 94.4M/181M [00:00<00:00, 246MB/s]
2024-12-18T00:37:39.8610234Z 
2024-12-18T00:37:39.9937373Z model.safetensors:  69% 126M/181M [00:00<00:00, 246MB/s] 
2024-12-18T00:37:39.9937758Z 
2024-12-18T00:37:40.0926892Z model.safetensors:  87% 157M/181M [00:00<00:00, 243MB/s]
2024-12-18T00:37:40.0927539Z model.safetensors: 100% 181M/181M [00:00<00:00, 243MB/s]
2024-12-18T00:37:40.8865866Z 
2024-12-18T00:37:40.8866313Z loading model: 0it [00:02, ?it/s]
2024-12-18T00:37:40.9071089Z cuda eval  res2net101_26w_4s                  
2024-12-18T00:38:29.8799043Z pass
2024-12-18T00:38:29.8838171Z TIMING: _recursive_pre_grad_passes:0.06251 _recursive_joint_graph_passes:0.36844 _recursive_post_grad_passes:0.75556 async_compile.wait:0.55007 code_gen:7.99631 inductor_compile:17.49611 backend_compile:31.97672 entire_frame_compile:45.56517 total_wall_time:45.56517
2024-12-18T00:38:29.8840411Z STATS: call_* op count: 1606 | FakeTensorMode.__torch_dispatch__:50370 | ProxyTorchDispatchMode.__torch_dispatch__:24225 | FakeTensor.__torch_dispatch__:9572
2024-12-18T00:38:29.8841414Z Dynamo produced 1 graphs covering 1606 ops with 0 graph breaks (0 unique)
2024-12-18T00:38:36.0784946Z 
2024-12-18T00:38:36.4157102Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:38:36.4157383Z 
2024-12-18T00:38:36.5758586Z model.safetensors:   0% 0.00/101M [00:00<?, ?B/s]
2024-12-18T00:38:36.5758908Z 
2024-12-18T00:38:36.6771994Z model.safetensors:  21% 21.0M/101M [00:00<00:00, 131MB/s]
2024-12-18T00:38:36.6772306Z 
2024-12-18T00:38:36.8103788Z model.safetensors:  42% 41.9M/101M [00:00<00:00, 167MB/s]
2024-12-18T00:38:36.8104091Z 
2024-12-18T00:38:36.9243488Z model.safetensors:  73% 73.4M/101M [00:00<00:00, 200MB/s]
2024-12-18T00:38:36.9243798Z 
2024-12-18T00:38:36.9283148Z model.safetensors: 100% 101M/101M [00:00<00:00, 214MB/s] 
2024-12-18T00:38:36.9283612Z model.safetensors: 100% 101M/101M [00:00<00:00, 196MB/s]
2024-12-18T00:38:37.7083676Z 
2024-12-18T00:38:37.7084280Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:38:37.7247609Z cuda eval  res2net50_14w_8s                   
2024-12-18T00:39:22.5574124Z pass
2024-12-18T00:39:22.5590663Z TIMING: _recursive_pre_grad_passes:0.0546 _recursive_joint_graph_passes:0.33863 _recursive_post_grad_passes:0.65224 async_compile.wait:0.55 code_gen:8.27483 inductor_compile:16.98235 backend_compile:30.67944 entire_frame_compile:41.93763 total_wall_time:41.93763
2024-12-18T00:39:22.5592475Z STATS: call_* op count: 1398 | FakeTensorMode.__torch_dispatch__:45144 | ProxyTorchDispatchMode.__torch_dispatch__:21352 | FakeTensor.__torch_dispatch__:8937
2024-12-18T00:39:22.5593351Z Dynamo produced 1 graphs covering 1398 ops with 0 graph breaks (0 unique)
2024-12-18T00:39:28.6815629Z 
2024-12-18T00:39:28.9900646Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:39:28.9900948Z 
2024-12-18T00:39:29.0925825Z model.safetensors:   0% 0.00/99.0M [00:00<?, ?B/s]
2024-12-18T00:39:29.0926097Z 
2024-12-18T00:39:29.2178738Z model.safetensors:  21% 21.0M/99.0M [00:00<00:00, 205MB/s]
2024-12-18T00:39:29.2179030Z 
2024-12-18T00:39:29.3544441Z model.safetensors:  53% 52.4M/99.0M [00:00<00:00, 234MB/s]
2024-12-18T00:39:29.3544739Z 
2024-12-18T00:39:29.4207527Z model.safetensors:  85% 83.9M/99.0M [00:00<00:00, 232MB/s]
2024-12-18T00:39:29.4208007Z model.safetensors: 100% 99.0M/99.0M [00:00<00:00, 230MB/s]
2024-12-18T00:39:30.0842391Z 
2024-12-18T00:39:30.0842579Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:39:30.0939301Z cuda eval  res2next50                         
2024-12-18T00:39:57.2484723Z pass
2024-12-18T00:39:57.2502133Z TIMING: _recursive_pre_grad_passes:0.03255 _recursive_joint_graph_passes:0.24177 _recursive_post_grad_passes:0.37043 async_compile.wait:0.40533 code_gen:5.4842 inductor_compile:10.74908 backend_compile:18.01461 entire_frame_compile:24.74181 total_wall_time:24.74181
2024-12-18T00:39:57.2503944Z STATS: call_* op count: 790 | FakeTensorMode.__torch_dispatch__:25573 | ProxyTorchDispatchMode.__torch_dispatch__:12104 | FakeTensor.__torch_dispatch__:4761
2024-12-18T00:39:57.2504814Z Dynamo produced 1 graphs covering 790 ops with 0 graph breaks (0 unique)
2024-12-18T00:40:02.6843030Z 
2024-12-18T00:40:02.8898487Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:40:02.8898738Z 
2024-12-18T00:40:03.0801381Z model.safetensors:   0% 0.00/61.4M [00:00<?, ?B/s]
2024-12-18T00:40:03.0801672Z 
2024-12-18T00:40:03.2035607Z model.safetensors:  34% 21.0M/61.4M [00:00<00:00, 110MB/s]
2024-12-18T00:40:03.2036007Z 
2024-12-18T00:40:03.2500655Z model.safetensors:  85% 52.4M/61.4M [00:00<00:00, 180MB/s]
2024-12-18T00:40:03.2501171Z model.safetensors: 100% 61.4M/61.4M [00:00<00:00, 171MB/s]
2024-12-18T00:40:03.7454976Z 
2024-12-18T00:40:03.7455324Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:40:03.7491649Z cuda eval  resmlp_12_224                      
2024-12-18T00:40:14.3182311Z pass
2024-12-18T00:40:14.3192111Z TIMING: _recursive_pre_grad_passes:0.01192 _recursive_joint_graph_passes:1.11541 _recursive_post_grad_passes:0.31946 async_compile.wait:0.31006 code_gen:1.27065 inductor_compile:2.95247 backend_compile:6.3623 entire_frame_compile:8.74979 total_wall_time:8.74979
2024-12-18T00:40:14.3193631Z STATS: call_* op count: 350 | FakeTensorMode.__torch_dispatch__:11266 | ProxyTorchDispatchMode.__torch_dispatch__:5623 | FakeTensor.__torch_dispatch__:1507
2024-12-18T00:40:14.3194475Z Dynamo produced 1 graphs covering 350 ops with 0 graph breaks (0 unique)
2024-12-18T00:40:19.0846933Z 
2024-12-18T00:40:19.6604845Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:40:19.6605322Z 
2024-12-18T00:40:19.7619786Z model.safetensors:   0% 0.00/194M [00:00<?, ?B/s]
2024-12-18T00:40:19.7620482Z 
2024-12-18T00:40:19.8891859Z model.safetensors:  11% 21.0M/194M [00:00<00:00, 207MB/s]
2024-12-18T00:40:19.8892167Z 
2024-12-18T00:40:20.0217691Z model.safetensors:  27% 52.4M/194M [00:00<00:00, 233MB/s]
2024-12-18T00:40:20.0218050Z 
2024-12-18T00:40:20.1473099Z model.safetensors:  43% 83.9M/194M [00:00<00:00, 235MB/s]
2024-12-18T00:40:20.1473406Z 
2024-12-18T00:40:20.2858581Z model.safetensors:  60% 115M/194M [00:00<00:00, 241MB/s] 
2024-12-18T00:40:20.2858889Z 
2024-12-18T00:40:20.4098439Z model.safetensors:  76% 147M/194M [00:00<00:00, 236MB/s]
2024-12-18T00:40:20.4098750Z 
2024-12-18T00:40:20.4829623Z model.safetensors:  92% 178M/194M [00:00<00:00, 242MB/s]
2024-12-18T00:40:20.4830103Z model.safetensors: 100% 194M/194M [00:00<00:00, 236MB/s]
2024-12-18T00:40:21.3054982Z 
2024-12-18T00:40:21.3055168Z loading model: 0it [00:02, ?it/s]
2024-12-18T00:40:21.3234962Z cuda eval  resnest101e                        
2024-12-18T00:41:07.4007774Z pass
2024-12-18T00:41:07.4032533Z TIMING: _recursive_pre_grad_passes:0.06307 _recursive_joint_graph_passes:0.77483 _recursive_post_grad_passes:0.66801 async_compile.wait:0.53265 code_gen:7.14469 inductor_compile:15.89684 backend_compile:29.00355 entire_frame_compile:41.88173 total_wall_time:41.88173
2024-12-18T00:41:07.4034736Z STATS: call_* op count: 1770 | FakeTensorMode.__torch_dispatch__:48888 | ProxyTorchDispatchMode.__torch_dispatch__:23131 | FakeTensor.__torch_dispatch__:8281
2024-12-18T00:41:07.4035613Z Dynamo produced 1 graphs covering 1770 ops with 0 graph breaks (0 unique)
2024-12-18T00:41:13.3886896Z 
2024-12-18T00:41:13.5082664Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:41:13.5083079Z 
2024-12-18T00:41:13.6015336Z model.safetensors:   0% 0.00/19.4M [00:00<?, ?B/s]
2024-12-18T00:41:13.6015792Z model.safetensors: 100% 19.4M/19.4M [00:00<00:00, 208MB/s]
2024-12-18T00:41:14.3049542Z 
2024-12-18T00:41:14.3049995Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:41:14.3130570Z cuda eval  rexnet_100                         
2024-12-18T00:41:37.8557430Z pass
2024-12-18T00:41:37.8561566Z TIMING: _recursive_pre_grad_passes:0.02516 _recursive_joint_graph_passes:0.22396 _recursive_post_grad_passes:0.3032 async_compile.wait:0.63061 code_gen:4.36742 inductor_compile:9.18638 backend_compile:14.90318 entire_frame_compile:21.54702 total_wall_time:21.54702
2024-12-18T00:41:37.8563545Z STATS: call_* op count: 540 | FakeTensorMode.__torch_dispatch__:22550 | FakeTensor.__torch_dispatch__:3496 | ProxyTorchDispatchMode.__torch_dispatch__:10100
2024-12-18T00:41:37.8564477Z Dynamo produced 1 graphs covering 540 ops with 0 graph breaks (0 unique)
2024-12-18T00:41:43.1258814Z 
2024-12-18T00:41:43.3484905Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:41:43.3485187Z 
2024-12-18T00:41:43.4796134Z model.safetensors:   0% 0.00/55.0M [00:00<?, ?B/s]
2024-12-18T00:41:43.4796518Z 
2024-12-18T00:41:43.5815786Z model.safetensors:  57% 31.5M/55.0M [00:00<00:00, 240MB/s]
2024-12-18T00:41:43.5816936Z model.safetensors: 100% 55.0M/55.0M [00:00<00:00, 236MB/s]
2024-12-18T00:41:44.2312598Z 
2024-12-18T00:41:44.2313115Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:41:44.2369196Z cuda eval  sebotnet33ts_256                   
2024-12-18T00:42:09.3976429Z pass
2024-12-18T00:42:09.3987387Z TIMING: _recursive_pre_grad_passes:0.02364 _recursive_joint_graph_passes:1.61634 _recursive_post_grad_passes:0.44191 async_compile.wait:1.9214 code_gen:8.08525 inductor_compile:12.19601 backend_compile:18.03987 entire_frame_compile:22.58348 total_wall_time:22.58348
2024-12-18T00:42:09.3989188Z STATS: call_* op count: 658 | FakeTensorMode.__torch_dispatch__:19503 | FakeTensor.__torch_dispatch__:2555 | ProxyTorchDispatchMode.__torch_dispatch__:8795
2024-12-18T00:42:09.3990185Z Dynamo produced 1 graphs covering 658 ops with 0 graph breaks (0 unique)
2024-12-18T00:42:14.5977716Z 
2024-12-18T00:42:14.9865305Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:42:14.9865561Z 
2024-12-18T00:42:15.1392478Z model.safetensors:   0% 0.00/130M [00:00<?, ?B/s]
2024-12-18T00:42:15.1392777Z 
2024-12-18T00:42:15.2789287Z model.safetensors:  16% 21.0M/130M [00:00<00:00, 137MB/s]
2024-12-18T00:42:15.2789601Z 
2024-12-18T00:42:15.4147194Z model.safetensors:  40% 52.4M/130M [00:00<00:00, 187MB/s]
2024-12-18T00:42:15.4147503Z 
2024-12-18T00:42:15.5434694Z model.safetensors:  65% 83.9M/130M [00:00<00:00, 207MB/s]
2024-12-18T00:42:15.5435007Z 
2024-12-18T00:42:15.6050220Z model.safetensors:  89% 115M/130M [00:00<00:00, 221MB/s] 
2024-12-18T00:42:15.6050863Z model.safetensors: 100% 130M/130M [00:00<00:00, 210MB/s]
2024-12-18T00:42:16.2028964Z 
2024-12-18T00:42:16.2029448Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:42:16.2078858Z cuda eval  selecsls42b                        
2024-12-18T00:42:29.7624070Z pass
2024-12-18T00:42:29.7649936Z TIMING: _recursive_pre_grad_passes:0.01538 _recursive_joint_graph_passes:0.1767 _recursive_post_grad_passes:0.18186 async_compile.wait:0.51868 code_gen:2.43598 inductor_compile:5.10518 backend_compile:8.14098 entire_frame_compile:11.48357 total_wall_time:11.48357
2024-12-18T00:42:29.7651524Z STATS: call_* op count: 266 | FakeTensorMode.__torch_dispatch__:11710 | ProxyTorchDispatchMode.__torch_dispatch__:5444 | FakeTensor.__torch_dispatch__:2121
2024-12-18T00:42:29.7652508Z Dynamo produced 1 graphs covering 266 ops with 0 graph breaks (0 unique)
2024-12-18T00:42:34.6570167Z 
2024-12-18T00:42:34.7671990Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:42:34.7674044Z 
2024-12-18T00:42:34.8549637Z model.safetensors:   0% 0.00/17.9M [00:00<?, ?B/s]
2024-12-18T00:42:34.8550099Z model.safetensors: 100% 17.9M/17.9M [00:00<00:00, 204MB/s]
2024-12-18T00:42:35.4505714Z 
2024-12-18T00:42:35.4506099Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:42:35.4579995Z cuda eval  spnasnet_100                       
2024-12-18T00:42:53.0618096Z pass
2024-12-18T00:42:53.0630389Z TIMING: _recursive_pre_grad_passes:0.02035 _recursive_joint_graph_passes:0.20628 _recursive_post_grad_passes:0.26587 async_compile.wait:0.28554 code_gen:1.81527 inductor_compile:5.73622 backend_compile:10.57953 entire_frame_compile:15.74793 total_wall_time:15.74793
2024-12-18T00:42:53.0632169Z STATS: call_* op count: 376 | FakeTensorMode.__torch_dispatch__:17893 | ProxyTorchDispatchMode.__torch_dispatch__:8397 | FakeTensor.__torch_dispatch__:3234
2024-12-18T00:42:53.0633013Z Dynamo produced 1 graphs covering 376 ops with 0 graph breaks (0 unique)
2024-12-18T00:42:58.0499787Z 
2024-12-18T00:42:59.0052498Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:42:59.0052753Z 
2024-12-18T00:42:59.1097723Z model.safetensors:   0% 0.00/353M [00:00<?, ?B/s]
2024-12-18T00:42:59.1098046Z 
2024-12-18T00:42:59.2181184Z model.safetensors:   6% 21.0M/353M [00:00<00:01, 201MB/s]
2024-12-18T00:42:59.2181799Z 
2024-12-18T00:42:59.3470680Z model.safetensors:  12% 41.9M/353M [00:00<00:01, 196MB/s]
2024-12-18T00:42:59.3471650Z 
2024-12-18T00:42:59.4760009Z model.safetensors:  21% 73.4M/353M [00:00<00:01, 220MB/s]
2024-12-18T00:42:59.4760299Z 
2024-12-18T00:42:59.6062352Z model.safetensors:  30% 105M/353M [00:00<00:01, 230MB/s] 
2024-12-18T00:42:59.6062650Z 
2024-12-18T00:42:59.7318320Z model.safetensors:  39% 136M/353M [00:00<00:00, 235MB/s]
2024-12-18T00:42:59.7318620Z 
2024-12-18T00:42:59.8657420Z model.safetensors:  48% 168M/353M [00:00<00:00, 240MB/s]
2024-12-18T00:42:59.8657710Z 
2024-12-18T00:43:00.0826297Z model.safetensors:  56% 199M/353M [00:00<00:00, 238MB/s]
2024-12-18T00:43:00.0826595Z 
2024-12-18T00:43:00.2155033Z model.safetensors:  65% 231M/353M [00:01<00:00, 197MB/s]
2024-12-18T00:43:00.2155321Z 
2024-12-18T00:43:00.3436932Z model.safetensors:  74% 262M/353M [00:01<00:00, 208MB/s]
2024-12-18T00:43:00.3437218Z 
2024-12-18T00:43:00.4700773Z model.safetensors:  83% 294M/353M [00:01<00:00, 218MB/s]
2024-12-18T00:43:00.4701351Z 
2024-12-18T00:43:00.5873146Z model.safetensors:  92% 325M/353M [00:01<00:00, 227MB/s]
2024-12-18T00:43:00.5873449Z 
2024-12-18T00:43:00.5915144Z model.safetensors: 100% 353M/353M [00:01<00:00, 229MB/s]
2024-12-18T00:43:00.5915608Z model.safetensors: 100% 353M/353M [00:01<00:00, 222MB/s]
2024-12-18T00:43:01.3577877Z 
2024-12-18T00:43:01.3578316Z loading model: 0it [00:03, ?it/s]
2024-12-18T00:43:01.3666868Z cuda eval  swin_base_patch4_window7_224       
2024-12-18T00:43:48.0567737Z pass
2024-12-18T00:43:48.0589697Z TIMING: _recursive_pre_grad_passes:0.06191 _recursive_joint_graph_passes:3.23163 _recursive_post_grad_passes:1.73309 async_compile.wait:0.87897 code_gen:12.11305 inductor_compile:20.28027 backend_compile:34.45238 entire_frame_compile:43.01744 total_wall_time:43.01744
2024-12-18T00:43:48.0591765Z STATS: call_* op count: 2508 | FakeTensorMode.__torch_dispatch__:60713 | FakeTensor.__torch_dispatch__:5259 | ProxyTorchDispatchMode.__torch_dispatch__:26232
2024-12-18T00:43:48.0592730Z Dynamo produced 1 graphs covering 2508 ops with 0 graph breaks (0 unique)
2024-12-18T00:43:54.0066168Z 
2024-12-18T00:43:56.0297743Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:43:56.0298095Z 
2024-12-18T00:43:56.1703982Z model.safetensors:   0% 0.00/777M [00:00<?, ?B/s]
2024-12-18T00:43:56.1704256Z 
2024-12-18T00:43:56.2973656Z model.safetensors:   4% 31.5M/777M [00:00<00:03, 224MB/s]
2024-12-18T00:43:56.2974253Z 
2024-12-18T00:43:56.4247681Z model.safetensors:   8% 62.9M/777M [00:00<00:03, 237MB/s]
2024-12-18T00:43:56.4248273Z 
2024-12-18T00:43:56.5530896Z model.safetensors:  12% 94.4M/777M [00:00<00:02, 242MB/s]
2024-12-18T00:43:56.5533296Z 
2024-12-18T00:43:56.6803388Z model.safetensors:  16% 126M/777M [00:00<00:02, 243MB/s] 
2024-12-18T00:43:56.6803697Z 
2024-12-18T00:43:56.8093840Z model.safetensors:  20% 157M/777M [00:00<00:02, 244MB/s]
2024-12-18T00:43:56.8094141Z 
2024-12-18T00:43:57.0488526Z model.safetensors:  24% 189M/777M [00:00<00:02, 244MB/s]
2024-12-18T00:43:57.0488848Z 
2024-12-18T00:43:57.1750494Z model.safetensors:  28% 220M/777M [00:01<00:02, 191MB/s]
2024-12-18T00:43:57.1750892Z 
2024-12-18T00:43:57.3060641Z model.safetensors:  32% 252M/777M [00:01<00:02, 206MB/s]
2024-12-18T00:43:57.3060936Z 
2024-12-18T00:43:57.4323301Z model.safetensors:  36% 283M/777M [00:01<00:02, 216MB/s]
2024-12-18T00:43:57.4323596Z 
2024-12-18T00:43:57.5625913Z model.safetensors:  40% 315M/777M [00:01<00:02, 225MB/s]
2024-12-18T00:43:57.5626208Z 
2024-12-18T00:43:57.6844723Z model.safetensors:  45% 346M/777M [00:01<00:01, 230MB/s]
2024-12-18T00:43:57.6845015Z 
2024-12-18T00:43:57.8101271Z model.safetensors:  49% 377M/777M [00:01<00:01, 238MB/s]
2024-12-18T00:43:57.8101557Z 
2024-12-18T00:43:58.0896539Z model.safetensors:  53% 409M/777M [00:01<00:01, 241MB/s]
2024-12-18T00:43:58.0897122Z 
2024-12-18T00:43:58.2257049Z model.safetensors:  57% 440M/777M [00:02<00:01, 179MB/s]
2024-12-18T00:43:58.2257342Z 
2024-12-18T00:43:58.3538933Z model.safetensors:  61% 472M/777M [00:02<00:01, 192MB/s]
2024-12-18T00:43:58.3539809Z 
2024-12-18T00:43:58.4833530Z model.safetensors:  65% 503M/777M [00:02<00:01, 206MB/s]
2024-12-18T00:43:58.4833817Z 
2024-12-18T00:43:58.6096658Z model.safetensors:  69% 535M/777M [00:02<00:01, 216MB/s]
2024-12-18T00:43:58.6096946Z 
2024-12-18T00:43:58.7427840Z model.safetensors:  73% 566M/777M [00:02<00:00, 225MB/s]
2024-12-18T00:43:58.7428136Z 
2024-12-18T00:43:59.0015996Z model.safetensors:  77% 598M/777M [00:02<00:00, 228MB/s]
2024-12-18T00:43:59.0016289Z 
2024-12-18T00:43:59.1280771Z model.safetensors:  81% 629M/777M [00:02<00:00, 181MB/s]
2024-12-18T00:43:59.1281087Z 
2024-12-18T00:43:59.2571783Z model.safetensors:  85% 661M/777M [00:03<00:00, 197MB/s]
2024-12-18T00:43:59.2572092Z 
2024-12-18T00:43:59.3864945Z model.safetensors:  89% 692M/777M [00:03<00:00, 209MB/s]
2024-12-18T00:43:59.3865248Z 
2024-12-18T00:43:59.5135412Z model.safetensors:  93% 724M/777M [00:03<00:00, 218MB/s]
2024-12-18T00:43:59.5135739Z 
2024-12-18T00:43:59.6083431Z model.safetensors:  97% 755M/777M [00:03<00:00, 226MB/s]
2024-12-18T00:43:59.6093442Z model.safetensors: 100% 777M/777M [00:03<00:00, 217MB/s]
2024-12-18T00:44:00.4813087Z 
2024-12-18T00:44:00.4814092Z loading model: 0it [00:06, ?it/s]
2024-12-18T00:44:00.4934700Z cuda eval  swsl_resnext101_32x16d             
2024-12-18T00:44:30.6465988Z pass
2024-12-18T00:44:30.6497191Z TIMING: _recursive_pre_grad_passes:0.0325 _recursive_joint_graph_passes:0.25794 _recursive_post_grad_passes:0.43749 async_compile.wait:0.30189 code_gen:2.80343 inductor_compile:8.52004 backend_compile:16.33617 entire_frame_compile:24.4513 total_wall_time:24.4513
2024-12-18T00:44:30.6498752Z STATS: call_* op count: 690 | FakeTensorMode.__torch_dispatch__:27924 | ProxyTorchDispatchMode.__torch_dispatch__:13639 | FakeTensor.__torch_dispatch__:5401
2024-12-18T00:44:30.6499613Z Dynamo produced 1 graphs covering 690 ops with 0 graph breaks (0 unique)
2024-12-18T00:44:35.9825831Z 
2024-12-18T00:44:36.1193604Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:44:36.1193965Z 
2024-12-18T00:44:36.2116803Z model.safetensors:   0% 0.00/21.4M [00:00<?, ?B/s]
2024-12-18T00:44:36.2117365Z model.safetensors: 100% 21.4M/21.4M [00:00<00:00, 232MB/s]
2024-12-18T00:44:36.8318156Z 
2024-12-18T00:44:36.8319333Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:44:36.8391486Z cuda eval  tf_efficientnet_b0                 
2024-12-18T00:44:57.2619364Z pass
2024-12-18T00:44:57.2629757Z TIMING: _recursive_pre_grad_passes:0.0222 _recursive_joint_graph_passes:0.21082 _recursive_post_grad_passes:0.26864 async_compile.wait:0.47013 code_gen:3.76672 inductor_compile:8.0111 backend_compile:12.90011 entire_frame_compile:18.47976 total_wall_time:18.47976
2024-12-18T00:44:57.2632212Z STATS: call_* op count: 488 | FakeTensorMode.__torch_dispatch__:18088 | FakeTensor.__torch_dispatch__:2982 | ProxyTorchDispatchMode.__torch_dispatch__:8450
2024-12-18T00:44:57.2633559Z Dynamo produced 1 graphs covering 488 ops with 0 graph breaks (0 unique)
2024-12-18T00:45:02.4120960Z 
2024-12-18T00:45:02.5694485Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:45:02.5694745Z 
2024-12-18T00:45:02.6692895Z model.safetensors:   0% 0.00/29.6M [00:00<?, ?B/s]
2024-12-18T00:45:02.6693363Z model.safetensors: 100% 29.6M/29.6M [00:00<00:00, 297MB/s]
2024-12-18T00:45:03.3481230Z 
2024-12-18T00:45:03.3481793Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:45:03.3583111Z cuda eval  tf_mixnet_l                        
2024-12-18T00:45:35.5831614Z pass
2024-12-18T00:45:35.5845352Z TIMING: _recursive_pre_grad_passes:0.04007 _recursive_joint_graph_passes:0.24539 _recursive_post_grad_passes:0.32926 async_compile.wait:1.3796 code_gen:7.73675 inductor_compile:13.49345 backend_compile:20.87759 entire_frame_compile:29.60087 total_wall_time:29.60087
2024-12-18T00:45:35.5847859Z STATS: call_* op count: 1072 | FakeTensorMode.__torch_dispatch__:27353 | FakeTensor.__torch_dispatch__:4285 | ProxyTorchDispatchMode.__torch_dispatch__:11956
2024-12-18T00:45:35.5849532Z Dynamo produced 1 graphs covering 1072 ops with 0 graph breaks (0 unique)
2024-12-18T00:45:41.1155030Z 
2024-12-18T00:45:41.2617248Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:45:41.2617724Z 
2024-12-18T00:45:41.3686704Z model.safetensors:   0% 0.00/25.0M [00:00<?, ?B/s]
2024-12-18T00:45:41.3687216Z 
2024-12-18T00:45:41.3710326Z model.safetensors: 100% 25.0M/25.0M [00:00<00:00, 234MB/s]
2024-12-18T00:45:41.3710794Z model.safetensors: 100% 25.0M/25.0M [00:00<00:00, 229MB/s]
2024-12-18T00:45:41.9787933Z 
2024-12-18T00:45:41.9788921Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:45:41.9874160Z cuda eval  tinynet_a                          
2024-12-18T00:46:04.7165764Z pass
2024-12-18T00:46:04.7176568Z TIMING: _recursive_pre_grad_passes:0.02568 _recursive_joint_graph_passes:0.22784 _recursive_post_grad_passes:0.31222 async_compile.wait:0.46376 code_gen:3.76484 inductor_compile:8.34056 backend_compile:14.35696 entire_frame_compile:20.77634 total_wall_time:20.77634
2024-12-18T00:46:04.7178714Z STATS: call_* op count: 568 | FakeTensorMode.__torch_dispatch__:21009 | FakeTensor.__torch_dispatch__:3520 | ProxyTorchDispatchMode.__torch_dispatch__:9945
2024-12-18T00:46:04.7179775Z Dynamo produced 1 graphs covering 568 ops with 0 graph breaks (0 unique)
2024-12-18T00:46:09.9322734Z 
2024-12-18T00:46:10.1982419Z loading model: 0it [00:00, ?it/s]Downloading: "https://github.com/contrastive/pytorch-image-models/releases/download/TNT/tnt_s_patch16_224.pth.tar" to /var/lib/jenkins/.cache/torch/hub/checkpoints/tnt_s_patch16_224.pth.tar
2024-12-18T00:46:11.2628821Z 
2024-12-18T00:46:11.2629469Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:46:11.2720315Z cuda eval  tnt_s_patch16_224                  
2024-12-18T00:46:46.1637788Z pass
2024-12-18T00:46:46.1654529Z TIMING: _recursive_pre_grad_passes:0.04213 _recursive_joint_graph_passes:2.36858 _recursive_post_grad_passes:1.00066 async_compile.wait:0.73387 code_gen:7.68702 inductor_compile:14.10117 backend_compile:25.57467 entire_frame_compile:32.14066 total_wall_time:32.14066
2024-12-18T00:46:46.1656035Z STATS: call_* op count: 1550 | FakeTensorMode.__torch_dispatch__:48328 | FakeTensor.__torch_dispatch__:4886 | ProxyTorchDispatchMode.__torch_dispatch__:23090
2024-12-18T00:46:46.1656898Z Dynamo produced 1 graphs covering 1550 ops with 0 graph breaks (0 unique)
2024-12-18T00:46:52.0732926Z 
2024-12-18T00:46:52.6085141Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:46:52.6085398Z 
2024-12-18T00:46:52.7523969Z model.safetensors:   0% 0.00/175M [00:00<?, ?B/s]
2024-12-18T00:46:52.7524243Z 
2024-12-18T00:46:52.8824007Z model.safetensors:  18% 31.5M/175M [00:00<00:00, 219MB/s]
2024-12-18T00:46:52.8824300Z 
2024-12-18T00:46:53.0123118Z model.safetensors:  36% 62.9M/175M [00:00<00:00, 232MB/s]
2024-12-18T00:46:53.0123445Z 
2024-12-18T00:46:53.1355241Z model.safetensors:  54% 94.4M/175M [00:00<00:00, 236MB/s]
2024-12-18T00:46:53.1355541Z 
2024-12-18T00:46:53.2646767Z model.safetensors:  72% 126M/175M [00:00<00:00, 244MB/s] 
2024-12-18T00:46:53.2647062Z 
2024-12-18T00:46:53.3467289Z model.safetensors:  90% 157M/175M [00:00<00:00, 244MB/s]
2024-12-18T00:46:53.3467753Z model.safetensors: 100% 175M/175M [00:00<00:00, 238MB/s]
2024-12-18T00:46:53.9765847Z 
2024-12-18T00:46:53.9766081Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:46:53.9897758Z cuda eval  twins_pcpvt_base                   
2024-12-18T00:47:27.8574776Z pass
2024-12-18T00:47:27.8618945Z TIMING: _recursive_pre_grad_passes:0.04881 _recursive_joint_graph_passes:0.78501 _recursive_post_grad_passes:1.00617 async_compile.wait:0.43322 code_gen:4.85225 inductor_compile:11.70898 backend_compile:22.29121 entire_frame_compile:30.87564 total_wall_time:30.87564
2024-12-18T00:47:27.8620847Z STATS: call_* op count: 1702 | FakeTensorMode.__torch_dispatch__:49427 | FakeTensor.__torch_dispatch__:5386 | ProxyTorchDispatchMode.__torch_dispatch__:22625
2024-12-18T00:47:27.8621838Z Dynamo produced 1 graphs covering 1702 ops with 0 graph breaks (0 unique)
2024-12-18T00:47:33.5691452Z 
2024-12-18T00:47:34.0423591Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:47:34.0423923Z 
2024-12-18T00:47:34.1429422Z model.safetensors:   0% 0.00/161M [00:00<?, ?B/s]
2024-12-18T00:47:34.1429715Z 
2024-12-18T00:47:34.2696363Z model.safetensors:  13% 21.0M/161M [00:00<00:00, 209MB/s]
2024-12-18T00:47:34.2696671Z 
2024-12-18T00:47:34.4026073Z model.safetensors:  33% 52.4M/161M [00:00<00:00, 234MB/s]
2024-12-18T00:47:34.4026370Z 
2024-12-18T00:47:34.5318109Z model.safetensors:  52% 83.9M/161M [00:00<00:00, 235MB/s]
2024-12-18T00:47:34.5318415Z 
2024-12-18T00:47:34.6637204Z model.safetensors:  72% 115M/161M [00:00<00:00, 239MB/s] 
2024-12-18T00:47:34.6637508Z 
2024-12-18T00:47:34.7258285Z model.safetensors:  91% 147M/161M [00:00<00:00, 239MB/s]
2024-12-18T00:47:34.7259206Z model.safetensors: 100% 161M/161M [00:00<00:00, 236MB/s]
2024-12-18T00:47:35.3825982Z 
2024-12-18T00:47:35.3826449Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:47:35.3869969Z cuda eval  visformer_small                    
2024-12-18T00:47:52.4851425Z pass
2024-12-18T00:47:52.4865444Z TIMING: _recursive_pre_grad_passes:0.02021 _recursive_joint_graph_passes:1.63929 _recursive_post_grad_passes:0.20326 async_compile.wait:0.78742 code_gen:3.02227 inductor_compile:5.83138 backend_compile:11.01293 entire_frame_compile:14.75312 total_wall_time:14.75312
2024-12-18T00:47:52.4866970Z STATS: call_* op count: 582 | FakeTensorMode.__torch_dispatch__:16839 | ProxyTorchDispatchMode.__torch_dispatch__:7774 | FakeTensor.__torch_dispatch__:2279
2024-12-18T00:47:52.4867822Z Dynamo produced 1 graphs covering 582 ops with 0 graph breaks (0 unique)
2024-12-18T00:47:57.4142078Z 
2024-12-18T00:47:58.3473984Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:47:58.3474336Z 
2024-12-18T00:47:58.4869164Z model.safetensors:   0% 0.00/346M [00:00<?, ?B/s]
2024-12-18T00:47:58.4869458Z 
2024-12-18T00:47:58.6154566Z model.safetensors:   9% 31.5M/346M [00:00<00:01, 226MB/s]
2024-12-18T00:47:58.6154886Z 
2024-12-18T00:47:58.7452150Z model.safetensors:  18% 62.9M/346M [00:00<00:01, 237MB/s]
2024-12-18T00:47:58.7452485Z 
2024-12-18T00:47:58.8783295Z model.safetensors:  27% 94.4M/346M [00:00<00:01, 239MB/s]
2024-12-18T00:47:58.8783638Z 
2024-12-18T00:47:59.0440067Z model.safetensors:  36% 126M/346M [00:00<00:00, 238MB/s] 
2024-12-18T00:47:59.0440378Z 
2024-12-18T00:47:59.1736893Z model.safetensors:  45% 157M/346M [00:00<00:00, 218MB/s]
2024-12-18T00:47:59.1737209Z 
2024-12-18T00:47:59.3080868Z model.safetensors:  55% 189M/346M [00:00<00:00, 226MB/s]
2024-12-18T00:47:59.3081183Z 
2024-12-18T00:47:59.4399385Z model.safetensors:  64% 220M/346M [00:00<00:00, 228MB/s]
2024-12-18T00:47:59.4399688Z 
2024-12-18T00:47:59.5681120Z model.safetensors:  73% 252M/346M [00:01<00:00, 232MB/s]
2024-12-18T00:47:59.5681423Z 
2024-12-18T00:47:59.6932039Z model.safetensors:  82% 283M/346M [00:01<00:00, 236MB/s]
2024-12-18T00:47:59.6932342Z 
2024-12-18T00:47:59.8183745Z model.safetensors:  91% 315M/346M [00:01<00:00, 240MB/s]
2024-12-18T00:47:59.8184046Z 
2024-12-18T00:47:59.8290318Z model.safetensors: 100% 346M/346M [00:01<00:00, 244MB/s]
2024-12-18T00:47:59.8291243Z model.safetensors: 100% 346M/346M [00:01<00:00, 234MB/s]
2024-12-18T00:48:00.4188185Z 
2024-12-18T00:48:00.4188668Z loading model: 0it [00:03, ?it/s]
2024-12-18T00:48:00.4229397Z cuda eval  vit_base_patch16_224               
2024-12-18T00:48:13.7868532Z pass
2024-12-18T00:48:13.7888352Z TIMING: _recursive_pre_grad_passes:0.01652 _recursive_joint_graph_passes:0.32498 _recursive_post_grad_passes:0.37823 async_compile.wait:0.39882 code_gen:1.95145 inductor_compile:4.55159 backend_compile:7.76451 entire_frame_compile:10.33998 total_wall_time:10.33998
2024-12-18T00:48:13.7890031Z STATS: call_* op count: 526 | FakeTensorMode.__torch_dispatch__:15558 | FakeTensor.__torch_dispatch__:1710 | ProxyTorchDispatchMode.__torch_dispatch__:7371
2024-12-18T00:48:13.7890954Z Dynamo produced 1 graphs covering 526 ops with 0 graph breaks (0 unique)
2024-12-18T00:48:18.6796530Z 
2024-12-18T00:48:19.0201205Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:48:19.0201792Z 
2024-12-18T00:48:19.1600549Z model.safetensors:   0% 0.00/107M [00:00<?, ?B/s]
2024-12-18T00:48:19.1601477Z 
2024-12-18T00:48:19.2778871Z model.safetensors:  30% 31.5M/107M [00:00<00:00, 225MB/s]
2024-12-18T00:48:19.2779159Z 
2024-12-18T00:48:19.4081937Z model.safetensors:  59% 62.9M/107M [00:00<00:00, 248MB/s]
2024-12-18T00:48:19.4082224Z 
2024-12-18T00:48:19.4663117Z model.safetensors:  89% 94.4M/107M [00:00<00:00, 245MB/s]
2024-12-18T00:48:19.4663570Z model.safetensors: 100% 107M/107M [00:00<00:00, 239MB/s] 
2024-12-18T00:48:20.1241534Z 
2024-12-18T00:48:20.1241716Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:48:20.1309501Z cuda eval  volo_d1_224                        
2024-12-18T00:48:42.9433164Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T00:48:42.9433867Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2771, in run_n_iterations
2024-12-18T00:48:42.9434528Z     return self.model_iter_fn(mod, inputs, collect_outputs=True)
2024-12-18T00:48:42.9435158Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 399, in forward_pass
2024-12-18T00:48:42.9435716Z     return mod(*inputs)
2024-12-18T00:48:42.9436219Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/volo.py", line 822, in forward
2024-12-18T00:48:42.9436767Z     x = self.forward_features(x)
2024-12-18T00:48:42.9437331Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/volo.py", line 795, in forward_features
2024-12-18T00:48:42.9437921Z     x = self.forward_tokens(x)
2024-12-18T00:48:42.9438478Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/volo.py", line 642, in forward_tokens
2024-12-18T00:48:42.9439034Z     x = block(x)
2024-12-18T00:48:42.9439507Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/volo.py", line 135, in forward
2024-12-18T00:48:42.9440081Z     x = x + self.drop_path1(self.attn(self.norm1(x)))
2024-12-18T00:48:42.9440652Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/volo.py", line 90, in forward
2024-12-18T00:48:42.9441412Z     x = F.fold(x, output_size=(H, W), kernel_size=self.kernel_size, padding=self.padding, stride=self.stride)
2024-12-18T00:48:42.9441845Z 
2024-12-18T00:48:44.7975369Z pass
2024-12-18T00:48:44.8130889Z TIMING: _recursive_pre_grad_passes:0.02927 _recursive_joint_graph_passes:2.12579 _recursive_post_grad_passes:0.73435 async_compile.wait:1.27813 code_gen:5.39881 inductor_compile:9.49507 backend_compile:17.33972 entire_frame_compile:22.12111 total_wall_time:22.12111
2024-12-18T00:48:44.8132397Z STATS: call_* op count: 1010 | FakeTensorMode.__torch_dispatch__:30662 | FakeTensor.__torch_dispatch__:2893 | ProxyTorchDispatchMode.__torch_dispatch__:14768
2024-12-18T00:48:44.8133252Z Dynamo produced 1 graphs covering 1010 ops with 0 graph breaks (0 unique)
2024-12-18T00:48:50.1534067Z 
2024-12-18T00:48:52.1929606Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:48:52.1929871Z 
2024-12-18T00:48:52.2937177Z model.safetensors:   0% 0.00/756M [00:00<?, ?B/s]
2024-12-18T00:48:52.2937472Z 
2024-12-18T00:48:52.4131018Z model.safetensors:   3% 21.0M/756M [00:00<00:03, 208MB/s]
2024-12-18T00:48:52.4131325Z 
2024-12-18T00:48:52.5397776Z model.safetensors:   7% 52.4M/756M [00:00<00:02, 243MB/s]
2024-12-18T00:48:52.5398086Z 
2024-12-18T00:48:52.6677743Z model.safetensors:  11% 83.9M/756M [00:00<00:02, 246MB/s]
2024-12-18T00:48:52.6678345Z 
2024-12-18T00:48:52.7922482Z model.safetensors:  15% 115M/756M [00:00<00:02, 246MB/s] 
2024-12-18T00:48:52.7922786Z 
2024-12-18T00:48:52.9145665Z model.safetensors:  19% 147M/756M [00:00<00:02, 248MB/s]
2024-12-18T00:48:52.9145983Z 
2024-12-18T00:48:53.0462462Z model.safetensors:  24% 178M/756M [00:00<00:02, 251MB/s]
2024-12-18T00:48:53.0462762Z 
2024-12-18T00:48:53.1703066Z model.safetensors:  28% 210M/756M [00:00<00:02, 247MB/s]
2024-12-18T00:48:53.1703355Z 
2024-12-18T00:48:53.3001520Z model.safetensors:  32% 241M/756M [00:00<00:02, 249MB/s]
2024-12-18T00:48:53.3002237Z 
2024-12-18T00:48:53.4295500Z model.safetensors:  36% 273M/756M [00:01<00:01, 247MB/s]
2024-12-18T00:48:53.4296017Z 
2024-12-18T00:48:53.5539605Z model.safetensors:  40% 304M/756M [00:01<00:01, 246MB/s]
2024-12-18T00:48:53.5539902Z 
2024-12-18T00:48:53.6810593Z model.safetensors:  44% 336M/756M [00:01<00:01, 248MB/s]
2024-12-18T00:48:53.6810892Z 
2024-12-18T00:48:53.8063020Z model.safetensors:  49% 367M/756M [00:01<00:01, 248MB/s]
2024-12-18T00:48:53.8063313Z 
2024-12-18T00:48:54.0430038Z model.safetensors:  53% 398M/756M [00:01<00:01, 249MB/s]
2024-12-18T00:48:54.0430336Z 
2024-12-18T00:48:54.1715755Z model.safetensors:  57% 430M/756M [00:01<00:01, 197MB/s]
2024-12-18T00:48:54.1716332Z 
2024-12-18T00:48:54.3002328Z model.safetensors:  61% 461M/756M [00:01<00:01, 209MB/s]
2024-12-18T00:48:54.3002625Z 
2024-12-18T00:48:54.4302495Z model.safetensors:  65% 493M/756M [00:02<00:01, 219MB/s]
2024-12-18T00:48:54.4302880Z 
2024-12-18T00:48:54.5607174Z model.safetensors:  69% 524M/756M [00:02<00:01, 225MB/s]
2024-12-18T00:48:54.5607527Z 
2024-12-18T00:48:54.6917328Z model.safetensors:  74% 556M/756M [00:02<00:00, 230MB/s]
2024-12-18T00:48:54.6917652Z 
2024-12-18T00:48:54.8222528Z model.safetensors:  78% 587M/756M [00:02<00:00, 233MB/s]
2024-12-18T00:48:54.8222821Z 
2024-12-18T00:48:55.0871846Z model.safetensors:  82% 619M/756M [00:02<00:00, 235MB/s]
2024-12-18T00:48:55.0872141Z 
2024-12-18T00:48:55.2152164Z model.safetensors:  86% 650M/756M [00:02<00:00, 182MB/s]
2024-12-18T00:48:55.2152455Z 
2024-12-18T00:48:55.3465577Z model.safetensors:  90% 682M/756M [00:03<00:00, 197MB/s]
2024-12-18T00:48:55.3465869Z 
2024-12-18T00:48:55.4732816Z model.safetensors:  94% 713M/756M [00:03<00:00, 208MB/s]
2024-12-18T00:48:55.4733117Z 
2024-12-18T00:48:55.5278289Z model.safetensors:  98% 744M/756M [00:03<00:00, 219MB/s]
2024-12-18T00:48:55.5278749Z model.safetensors: 100% 756M/756M [00:03<00:00, 227MB/s]
2024-12-18T00:48:56.1160407Z WARNING:common:Model xcit_large_24_p8_224 does not support bfloat16, running with amp instead
2024-12-18T00:48:56.5593580Z 
2024-12-18T00:48:56.5593882Z loading model: 0it [00:06, ?it/s]
2024-12-18T00:48:56.5594389Z WARNING:common:Model xcit_large_24_p8_224 does not support bfloat16, running with amp instead
2024-12-18T00:48:56.5594928Z cuda eval  xcit_large_24_p8_224               
2024-12-18T00:49:07.7018481Z WARNING:common:Model xcit_large_24_p8_224 does not support bfloat16, running with amp instead
2024-12-18T00:49:58.9488176Z pass
2024-12-18T00:49:58.9726342Z TIMING: _recursive_pre_grad_passes:0.06345 _recursive_joint_graph_passes:1.4953 _recursive_post_grad_passes:2.14027 async_compile.wait:1.25469 code_gen:8.74814 inductor_compile:19.6955 backend_compile:36.46226 entire_frame_compile:48.40301 total_wall_time:48.40301
2024-12-18T00:49:58.9728706Z STATS: call_* op count: 2224 | FakeTensorMode.__torch_dispatch__:86824 | FakeTensor.__torch_dispatch__:9079 | ProxyTorchDispatchMode.__torch_dispatch__:38797
2024-12-18T00:49:58.9730020Z Dynamo produced 1 graphs covering 2224 ops with 0 graph breaks (0 unique)
2024-12-18T00:50:02.0078848Z accuracy            pass_rate=100.00%
2024-12-18T00:50:02.0079255Z calls_captured      gmean=756.22x mean=929.032x
2024-12-18T00:50:02.0083043Z unique_graphs       gmean=1.00x mean=1.000x
2024-12-18T00:50:02.0086406Z graph_breaks        gmean=0.00x mean=0.000x
2024-12-18T00:50:02.0089981Z unique_graph_breaks gmean=0.00x mean=0.000x
2024-12-18T00:50:02.0093610Z autograd_captures   gmean=0.00x mean=0.000x
2024-12-18T00:50:02.0097189Z autograd_compiles   gmean=0.00x mean=0.000x
2024-12-18T00:50:02.0100948Z cudagraph_skips     gmean=0.00x mean=0.032x
2024-12-18T00:50:03.3448486Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_timm_models.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_timm_inference.csv
2024-12-18T00:50:03.6080321Z mixnet_l                            PASS
2024-12-18T00:50:03.6085208Z mnasnet_100                         PASS
2024-12-18T00:50:03.6088921Z mobilenetv2_100                     PASS
2024-12-18T00:50:03.6092226Z mobilenetv3_large_100               PASS
2024-12-18T00:50:03.6096623Z mobilevit_s                         PASS
2024-12-18T00:50:03.6101313Z nfnet_l0                            PASS
2024-12-18T00:50:03.6105608Z pit_b_224                           PASS
2024-12-18T00:50:03.6110498Z pnasnet5large                       PASS
2024-12-18T00:50:03.6115424Z poolformer_m36                      PASS
2024-12-18T00:50:03.6120023Z regnety_002                         PASS
2024-12-18T00:50:03.6124348Z repvgg_a2                           PASS
2024-12-18T00:50:03.6128710Z res2net101_26w_4s                   PASS
2024-12-18T00:50:03.6133158Z res2net50_14w_8s                    PASS
2024-12-18T00:50:03.6137343Z res2next50                          PASS
2024-12-18T00:50:03.6142014Z resmlp_12_224                       PASS
2024-12-18T00:50:03.6146249Z resnest101e                         PASS
2024-12-18T00:50:03.6150615Z rexnet_100                          PASS
2024-12-18T00:50:03.6155098Z sebotnet33ts_256                    PASS
2024-12-18T00:50:03.6159438Z selecsls42b                         PASS
2024-12-18T00:50:03.6163730Z spnasnet_100                        PASS
2024-12-18T00:50:03.6168135Z swin_base_patch4_window7_224        PASS
2024-12-18T00:50:03.6172510Z swsl_resnext101_32x16d              PASS
2024-12-18T00:50:03.6176898Z tf_efficientnet_b0                  PASS
2024-12-18T00:50:03.6181465Z tf_mixnet_l                         PASS
2024-12-18T00:50:03.6185813Z tinynet_a                           PASS
2024-12-18T00:50:03.6190129Z tnt_s_patch16_224                   PASS
2024-12-18T00:50:03.6194534Z twins_pcpvt_base                    PASS
2024-12-18T00:50:03.6198861Z visformer_small                     PASS
2024-12-18T00:50:03.6203281Z vit_base_patch16_224                PASS
2024-12-18T00:50:03.6207714Z volo_d1_224                         PASS
2024-12-18T00:50:03.6212031Z xcit_large_24_p8_224                PASS
2024-12-18T00:50:03.6633967Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_timm_models.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_timm_inference.csv
2024-12-18T00:50:03.9272957Z mixnet_l                            PASS
2024-12-18T00:50:03.9276267Z mnasnet_100                         PASS
2024-12-18T00:50:03.9281006Z mobilenetv2_100                     PASS
2024-12-18T00:50:03.9286012Z mobilenetv3_large_100               PASS
2024-12-18T00:50:03.9291054Z mobilevit_s                         PASS
2024-12-18T00:50:03.9296173Z nfnet_l0                            PASS
2024-12-18T00:50:03.9301029Z pit_b_224                           PASS
2024-12-18T00:50:03.9305959Z pnasnet5large                       PASS
2024-12-18T00:50:03.9311192Z poolformer_m36                      PASS
2024-12-18T00:50:03.9316113Z regnety_002                         PASS
2024-12-18T00:50:03.9322164Z repvgg_a2                           PASS
2024-12-18T00:50:03.9327117Z res2net101_26w_4s                   PASS
2024-12-18T00:50:03.9332015Z res2net50_14w_8s                    PASS
2024-12-18T00:50:03.9336934Z res2next50                          PASS
2024-12-18T00:50:03.9342201Z resmlp_12_224                       PASS
2024-12-18T00:50:03.9347065Z resnest101e                         PASS
2024-12-18T00:50:03.9351986Z rexnet_100                          PASS
2024-12-18T00:50:03.9356906Z sebotnet33ts_256                    PASS
2024-12-18T00:50:03.9361850Z selecsls42b                         PASS
2024-12-18T00:50:03.9366750Z spnasnet_100                        PASS
2024-12-18T00:50:03.9371617Z swin_base_patch4_window7_224        PASS
2024-12-18T00:50:03.9377067Z swsl_resnext101_32x16d              PASS
2024-12-18T00:50:03.9381668Z tf_efficientnet_b0                  PASS
2024-12-18T00:50:03.9386368Z tf_mixnet_l                         PASS
2024-12-18T00:50:03.9391430Z tinynet_a                           PASS
2024-12-18T00:50:03.9396143Z tnt_s_patch16_224                   PASS
2024-12-18T00:50:03.9401291Z twins_pcpvt_base                    PASS
2024-12-18T00:50:03.9406036Z visformer_small                     PASS
2024-12-18T00:50:03.9410850Z vit_base_patch16_224                PASS
2024-12-18T00:50:03.9415727Z volo_d1_224                         PASS
2024-12-18T00:50:03.9421539Z xcit_large_24_p8_224                PASS
2024-12-18T00:50:03.9850517Z + test_single_dynamo_benchmark training timm_models 1 --training --amp
2024-12-18T00:50:03.9853950Z ++ pwd
2024-12-18T00:50:03.9857312Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2024-12-18T00:50:03.9857790Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports
2024-12-18T00:50:03.9880931Z + local name=training
2024-12-18T00:50:03.9881464Z + shift
2024-12-18T00:50:03.9881916Z + local suite=timm_models
2024-12-18T00:50:03.9882436Z + shift
2024-12-18T00:50:03.9882883Z + local shard_id=1
2024-12-18T00:50:03.9883352Z + shift
2024-12-18T00:50:03.9883792Z + partition_flags=()
2024-12-18T00:50:03.9884322Z + local partition_flags
2024-12-18T00:50:03.9884848Z + [[ -n 2 ]]
2024-12-18T00:50:03.9885315Z + [[ -n 1 ]]
2024-12-18T00:50:03.9886134Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id")
2024-12-18T00:50:03.9887131Z + [[ inductor_timm == *perf_compare* ]]
2024-12-18T00:50:03.9887592Z + [[ inductor_timm == *perf* ]]
2024-12-18T00:50:03.9887893Z + [[ inductor_timm == *_avx2* ]]
2024-12-18T00:50:03.9888198Z + [[ inductor_timm == *_avx512* ]]
2024-12-18T00:50:03.9889246Z + python benchmarks/dynamo/timm_models.py --ci --accuracy --timing --explain --inductor --device cuda --training --amp --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/training_timm_models.csv
2024-12-18T00:50:10.7966078Z 
2024-12-18T00:50:11.9692535Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:50:11.9693076Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:50:11.9693598Z cuda train mixnet_l                           
2024-12-18T00:51:33.3259527Z pass
2024-12-18T00:51:33.3356249Z TIMING: entire_frame_compile:51.56243 _recursive_pre_grad_passes:0.02597 _recursive_joint_graph_passes:0.53212 _recursive_post_grad_passes:1.20524 async_compile.wait:6.46335 code_gen:33.8017 inductor_compile:49.78822 backend_compile:39.21816 entire_backward_compile:23.94074 total_wall_time:75.50317
2024-12-18T00:51:33.3359437Z STATS: call_* op count: 583 | FakeTensorMode.__torch_dispatch__:46783 | FakeTensor.__torch_dispatch__:13988 | ProxyTorchDispatchMode.__torch_dispatch__:19678
2024-12-18T00:51:33.3361124Z Dynamo produced 2 graphs covering 583 ops with 6 graph breaks (5 unique)
2024-12-18T00:51:40.8192843Z 
2024-12-18T00:51:41.7488097Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:51:41.7488483Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:51:41.7488933Z cuda train mnasnet_100                        
2024-12-18T00:52:21.9888459Z W1218 00:52:21.987000 14464 site-packages/torch/_logging/_internal.py:1089] [7/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T00:52:23.7850127Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T00:53:00.9264405Z pass
2024-12-18T00:53:00.9311715Z TIMING: entire_frame_compile:62.10734 _recursive_pre_grad_passes:0.05166 _recursive_joint_graph_passes:0.50068 _recursive_post_grad_passes:0.76589 async_compile.wait:6.783 code_gen:26.04164 inductor_compile:41.54416 backend_compile:47.60075 entire_backward_compile:9.91325 total_wall_time:72.02059
2024-12-18T00:53:00.9313546Z STATS: call_* op count: 1018 | FakeTensorMode.__torch_dispatch__:43101 | FakeTensor.__torch_dispatch__:15166 | ProxyTorchDispatchMode.__torch_dispatch__:18960
2024-12-18T00:53:00.9314785Z Dynamo produced 4 graphs covering 1018 ops with 7 graph breaks (5 unique)
2024-12-18T00:53:08.2078786Z 
2024-12-18T00:53:09.1371112Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:53:09.1372145Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:53:09.1372794Z cuda train mobilenetv2_100                    
2024-12-18T00:53:50.7580229Z W1218 00:53:50.756000 15472 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T00:53:52.5373973Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T00:54:26.9519928Z pass
2024-12-18T00:54:26.9569074Z TIMING: entire_frame_compile:60.56764 _recursive_pre_grad_passes:0.05108 _recursive_joint_graph_passes:0.51554 _recursive_post_grad_passes:0.88899 async_compile.wait:4.65071 code_gen:24.9648 inductor_compile:40.97155 backend_compile:46.03491 entire_backward_compile:11.40969 total_wall_time:71.97733
2024-12-18T00:54:26.9572321Z STATS: call_* op count: 1017 | FakeTensorMode.__torch_dispatch__:43808 | FakeTensor.__torch_dispatch__:16128 | ProxyTorchDispatchMode.__torch_dispatch__:18976
2024-12-18T00:54:26.9574076Z Dynamo produced 3 graphs covering 1017 ops with 7 graph breaks (5 unique)
2024-12-18T00:54:34.2721407Z 
2024-12-18T00:54:35.2581968Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:54:35.2582334Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:54:35.2582665Z cuda train mobilenetv3_large_100              
2024-12-18T00:55:21.8234909Z W1218 00:55:21.822000 16284 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T00:55:23.8221503Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T00:56:03.2441262Z pass
2024-12-18T00:56:03.2513446Z TIMING: entire_frame_compile:68.14219 _recursive_pre_grad_passes:0.05519 _recursive_joint_graph_passes:0.54992 _recursive_post_grad_passes:0.99087 async_compile.wait:6.60608 code_gen:29.62811 inductor_compile:48.46722 backend_compile:52.77177 entire_backward_compile:13.58401 total_wall_time:81.7262
2024-12-18T00:56:03.2515303Z STATS: call_* op count: 1125 | FakeTensorMode.__torch_dispatch__:47926 | FakeTensor.__torch_dispatch__:17206 | ProxyTorchDispatchMode.__torch_dispatch__:20960
2024-12-18T00:56:03.2516151Z Dynamo produced 3 graphs covering 1125 ops with 7 graph breaks (5 unique)
2024-12-18T00:56:10.9300838Z 
2024-12-18T00:56:11.9131653Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:56:11.9132021Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:56:11.9132356Z cuda train mobilevit_s                        
2024-12-18T00:57:21.9249900Z pass
2024-12-18T00:57:21.9332580Z TIMING: entire_frame_compile:34.68452 _recursive_pre_grad_passes:0.01675 _recursive_joint_graph_passes:0.73808 _recursive_post_grad_passes:0.98871 async_compile.wait:2.49649 code_gen:30.84853 inductor_compile:46.22414 backend_compile:27.21647 entire_backward_compile:28.96244 total_wall_time:63.64696
2024-12-18T00:57:21.9334251Z STATS: call_* op count: 363 | FakeTensorMode.__torch_dispatch__:36954 | FakeTensor.__torch_dispatch__:10071 | ProxyTorchDispatchMode.__torch_dispatch__:16472
2024-12-18T00:57:21.9335116Z Dynamo produced 3 graphs covering 363 ops with 6 graph breaks (5 unique)
2024-12-18T00:57:28.4558922Z 
2024-12-18T00:57:29.8004064Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:57:29.8004430Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:57:29.8004764Z cuda train nfnet_l0                           
2024-12-18T00:58:13.0662214Z W1218 00:58:13.065000 18606 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T00:59:09.0423680Z pass
2024-12-18T00:59:09.0658525Z TIMING: entire_frame_compile:77.82657 _recursive_pre_grad_passes:0.07124 _recursive_joint_graph_passes:0.85555 _recursive_post_grad_passes:0.96489 async_compile.wait:6.02239 code_gen:28.42495 inductor_compile:47.00436 backend_compile:56.22876 entire_backward_compile:13.48774 total_wall_time:91.31432
2024-12-18T00:59:09.0660418Z STATS: call_* op count: 1649 | FakeTensorMode.__torch_dispatch__:61432 | FakeTensor.__torch_dispatch__:19342 | ProxyTorchDispatchMode.__torch_dispatch__:27720
2024-12-18T00:59:09.0661271Z Dynamo produced 3 graphs covering 1649 ops with 7 graph breaks (5 unique)
2024-12-18T00:59:16.9383281Z 
2024-12-18T00:59:18.1683962Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:59:18.1684327Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:59:18.1684654Z cuda train pit_b_224                          
2024-12-18T00:59:50.8178437Z pass
2024-12-18T00:59:50.8722416Z TIMING: entire_frame_compile:16.84633 _recursive_pre_grad_passes:0.01238 _recursive_joint_graph_passes:0.96051 _recursive_post_grad_passes:0.69842 async_compile.wait:1.48271 code_gen:9.21026 inductor_compile:16.09153 backend_compile:12.51744 entire_backward_compile:9.58585 total_wall_time:26.43218
2024-12-18T00:59:50.8725588Z STATS: call_* op count: 310 | FakeTensorMode.__torch_dispatch__:28105 | FakeTensor.__torch_dispatch__:5891 | ProxyTorchDispatchMode.__torch_dispatch__:12562
2024-12-18T00:59:50.8727197Z Dynamo produced 2 graphs covering 310 ops with 6 graph breaks (5 unique)
2024-12-18T00:59:56.4181867Z 
2024-12-18T00:59:58.4097721Z loading model: 0it [00:00, ?it/s]
2024-12-18T00:59:58.4098399Z loading model: 0it [00:01, ?it/s]
2024-12-18T00:59:58.4099059Z cuda train pnasnet5large                      
2024-12-18T01:04:27.4210269Z pass
2024-12-18T01:04:27.5646310Z TIMING: entire_frame_compile:123.54372 _recursive_pre_grad_passes:0.05811 _recursive_joint_graph_passes:1.25603 _recursive_post_grad_passes:2.78237 async_compile.wait:4.6896 code_gen:146.07909 inductor_compile:182.84767 backend_compile:92.46337 entire_backward_compile:124.97904 total_wall_time:248.52276
2024-12-18T01:04:27.5648052Z STATS: call_* op count: 1161 | FakeTensorMode.__torch_dispatch__:110409 | FakeTensor.__torch_dispatch__:34112 | ProxyTorchDispatchMode.__torch_dispatch__:49820
2024-12-18T01:04:27.5648930Z Dynamo produced 2 graphs covering 1161 ops with 5 graph breaks (4 unique)
2024-12-18T01:04:39.8185509Z 
2024-12-18T01:04:41.3152917Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:04:41.3153438Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:04:41.3163277Z cuda train poolformer_m36                     
2024-12-18T01:07:49.2583145Z pass
2024-12-18T01:07:49.2988222Z TIMING: entire_frame_compile:54.5128 _recursive_pre_grad_passes:0.02124 _recursive_joint_graph_passes:1.42472 _recursive_post_grad_passes:1.78344 async_compile.wait:2.86192 code_gen:135.07757 inductor_compile:153.59889 backend_compile:44.60364 entire_backward_compile:123.82748 total_wall_time:178.34028
2024-12-18T01:07:49.2989858Z STATS: call_* op count: 554 | FakeTensorMode.__torch_dispatch__:50172 | FakeTensor.__torch_dispatch__:15805 | ProxyTorchDispatchMode.__torch_dispatch__:22827
2024-12-18T01:07:49.2990731Z Dynamo produced 2 graphs covering 554 ops with 6 graph breaks (5 unique)
2024-12-18T01:07:57.4296096Z 
2024-12-18T01:07:58.4145010Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:07:58.4145401Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:07:58.4145751Z cuda train regnety_002                        
2024-12-18T01:08:37.5463071Z pass
2024-12-18T01:08:37.5499392Z TIMING: entire_frame_compile:25.55326 _recursive_pre_grad_passes:0.01478 _recursive_joint_graph_passes:0.38482 _recursive_post_grad_passes:0.70791 async_compile.wait:1.27593 code_gen:11.49885 inductor_compile:20.5311 backend_compile:18.43361 entire_backward_compile:10.03272 total_wall_time:35.58598
2024-12-18T01:08:37.5501196Z STATS: call_* op count: 271 | FakeTensorMode.__torch_dispatch__:26200 | FakeTensor.__torch_dispatch__:8417 | ProxyTorchDispatchMode.__torch_dispatch__:12209
2024-12-18T01:08:37.5502043Z Dynamo produced 2 graphs covering 271 ops with 6 graph breaks (5 unique)
2024-12-18T01:08:43.4664309Z 
2024-12-18T01:08:44.7205890Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:08:44.7206477Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:08:44.7206816Z cuda train repvgg_a2                          
2024-12-18T01:09:28.6405821Z W1218 01:09:28.639000 22682 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T01:09:30.3949219Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][160].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T01:10:06.9274449Z pass
2024-12-18T01:10:06.9441118Z TIMING: entire_frame_compile:63.65219 _recursive_pre_grad_passes:0.05467 _recursive_joint_graph_passes:0.53214 _recursive_post_grad_passes:0.83618 async_compile.wait:6.76284 code_gen:26.69308 inductor_compile:43.44591 backend_compile:49.18595 entire_backward_compile:11.20381 total_wall_time:74.85599
2024-12-18T01:10:06.9443771Z STATS: call_* op count: 1094 | FakeTensorMode.__torch_dispatch__:44480 | FakeTensor.__torch_dispatch__:16445 | ProxyTorchDispatchMode.__torch_dispatch__:20318
2024-12-18T01:10:06.9445165Z Dynamo produced 3 graphs covering 1094 ops with 7 graph breaks (5 unique)
2024-12-18T01:10:14.4246755Z 
2024-12-18T01:10:16.0188260Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:10:16.0188728Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:10:16.0189087Z cuda train res2net101_26w_4s                  
2024-12-18T01:12:36.2884479Z pass
2024-12-18T01:12:36.3224768Z TIMING: entire_frame_compile:89.65329 _recursive_pre_grad_passes:0.04445 _recursive_joint_graph_passes:0.97483 _recursive_post_grad_passes:3.28931 async_compile.wait:1.98022 code_gen:51.72127 inductor_compile:81.52502 backend_compile:68.91485 entire_backward_compile:39.32259 total_wall_time:128.97587
2024-12-18T01:12:36.3226449Z STATS: call_* op count: 977 | FakeTensorMode.__torch_dispatch__:81947 | FakeTensor.__torch_dispatch__:30388 | ProxyTorchDispatchMode.__torch_dispatch__:37834
2024-12-18T01:12:36.3227316Z Dynamo produced 2 graphs covering 977 ops with 6 graph breaks (5 unique)
2024-12-18T01:12:46.2096775Z 
2024-12-18T01:12:47.5739170Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:12:47.5739621Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:12:47.5739976Z cuda train res2net50_14w_8s                   
2024-12-18T01:14:53.6511620Z pass
2024-12-18T01:14:53.6715643Z TIMING: entire_frame_compile:78.56241 _recursive_pre_grad_passes:0.03932 _recursive_joint_graph_passes:0.8567 _recursive_post_grad_passes:2.23194 async_compile.wait:2.49227 code_gen:48.43812 inductor_compile:75.12764 backend_compile:61.42167 entire_backward_compile:37.92571 total_wall_time:116.48813
2024-12-18T01:14:53.6717280Z STATS: call_* op count: 852 | FakeTensorMode.__torch_dispatch__:72583 | FakeTensor.__torch_dispatch__:26949 | ProxyTorchDispatchMode.__torch_dispatch__:33247
2024-12-18T01:14:53.6718531Z Dynamo produced 2 graphs covering 852 ops with 6 graph breaks (5 unique)
2024-12-18T01:15:03.2202842Z 
2024-12-18T01:15:04.4365498Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:15:04.4365864Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:15:04.4366188Z cuda train res2next50                         
2024-12-18T01:16:13.2575501Z pass
2024-12-18T01:16:13.2754327Z TIMING: entire_frame_compile:40.75124 _recursive_pre_grad_passes:0.02405 _recursive_joint_graph_passes:0.54675 _recursive_post_grad_passes:1.22672 async_compile.wait:1.60474 code_gen:24.99658 inductor_compile:39.63312 backend_compile:30.91779 entire_backward_compile:22.08536 total_wall_time:62.8366
2024-12-18T01:16:13.2755935Z STATS: call_* op count: 484 | FakeTensorMode.__torch_dispatch__:40830 | FakeTensor.__torch_dispatch__:15131 | ProxyTorchDispatchMode.__torch_dispatch__:18879
2024-12-18T01:16:13.2756778Z Dynamo produced 2 graphs covering 484 ops with 6 graph breaks (5 unique)
2024-12-18T01:16:20.2404215Z 
2024-12-18T01:16:21.0106469Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:16:21.0106886Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:16:21.0107228Z cuda train resmlp_12_224                      
2024-12-18T01:16:41.8476209Z pass
2024-12-18T01:16:41.8603450Z TIMING: entire_frame_compile:11.92587 _recursive_pre_grad_passes:0.00881 _recursive_joint_graph_passes:1.61384 _recursive_post_grad_passes:0.35568 async_compile.wait:0.65282 code_gen:5.19691 inductor_compile:8.97578 backend_compile:8.69572 entire_backward_compile:5.66689 total_wall_time:17.59276
2024-12-18T01:16:41.8605060Z STATS: call_* op count: 179 | FakeTensorMode.__torch_dispatch__:17957 | ProxyTorchDispatchMode.__torch_dispatch__:8357 | FakeTensor.__torch_dispatch__:3865
2024-12-18T01:16:41.8605907Z Dynamo produced 2 graphs covering 179 ops with 6 graph breaks (5 unique)
2024-12-18T01:16:47.0370079Z 
2024-12-18T01:16:48.7202877Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:16:48.7203240Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:16:48.7203574Z cuda train resnest101e                        
2024-12-18T01:18:50.7035699Z pass
2024-12-18T01:18:50.7450388Z TIMING: entire_frame_compile:74.4602 _recursive_pre_grad_passes:0.04318 _recursive_joint_graph_passes:1.19542 _recursive_post_grad_passes:3.05922 async_compile.wait:1.5621 code_gen:39.86821 inductor_compile:66.14405 backend_compile:55.48644 entire_backward_compile:35.91325 total_wall_time:110.37345
2024-12-18T01:18:50.7453584Z STATS: call_* op count: 1028 | FakeTensorMode.__torch_dispatch__:82200 | FakeTensor.__torch_dispatch__:26617 | ProxyTorchDispatchMode.__torch_dispatch__:38715
2024-12-18T01:18:50.7455279Z Dynamo produced 2 graphs covering 1028 ops with 6 graph breaks (5 unique)
2024-12-18T01:18:59.8489752Z 
2024-12-18T01:19:01.0446383Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:19:01.0446931Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:19:01.0447445Z cuda train rexnet_100                         
2024-12-18T01:20:07.2773956Z W1218 01:20:07.276000 27237 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T01:20:09.8345539Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T01:21:01.5893692Z pass
2024-12-18T01:21:01.5977996Z TIMING: entire_frame_compile:91.48675 _recursive_pre_grad_passes:0.07298 _recursive_joint_graph_passes:0.69318 _recursive_post_grad_passes:1.25385 async_compile.wait:8.99614 code_gen:44.18458 inductor_compile:68.43706 backend_compile:70.31516 entire_backward_compile:20.87349 total_wall_time:112.36024
2024-12-18T01:21:01.5979615Z STATS: call_* op count: 1490 | FakeTensorMode.__torch_dispatch__:65627 | FakeTensor.__torch_dispatch__:22414 | ProxyTorchDispatchMode.__torch_dispatch__:27866
2024-12-18T01:21:01.5980469Z Dynamo produced 3 graphs covering 1490 ops with 7 graph breaks (5 unique)
2024-12-18T01:21:10.4536675Z 
2024-12-18T01:21:11.6375718Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:21:11.6376089Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:21:11.6376642Z cuda train sebotnet33ts_256                   
2024-12-18T01:22:13.1447402Z pass
2024-12-18T01:22:13.1574401Z TIMING: entire_frame_compile:35.87558 _recursive_pre_grad_passes:0.01697 _recursive_joint_graph_passes:3.59611 _recursive_post_grad_passes:0.85632 async_compile.wait:3.07153 code_gen:25.44783 inductor_compile:36.95153 backend_compile:29.297 entire_backward_compile:19.24176 total_wall_time:55.11734
2024-12-18T01:22:13.1576193Z STATS: call_* op count: 372 | FakeTensorMode.__torch_dispatch__:31239 | FakeTensor.__torch_dispatch__:8928 | ProxyTorchDispatchMode.__torch_dispatch__:13438
2024-12-18T01:22:13.1577032Z Dynamo produced 3 graphs covering 372 ops with 6 graph breaks (5 unique)
2024-12-18T01:22:19.4570482Z 
2024-12-18T01:22:20.7324892Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:22:20.7325347Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:22:20.7325722Z cuda train selecsls42b                        
2024-12-18T01:22:52.3172982Z pass
2024-12-18T01:22:52.3445060Z TIMING: entire_frame_compile:19.85176 _recursive_pre_grad_passes:0.01193 _recursive_joint_graph_passes:0.53428 _recursive_post_grad_passes:0.53148 async_compile.wait:1.0262 code_gen:9.55434 inductor_compile:16.72349 backend_compile:14.94736 entire_backward_compile:7.87489 total_wall_time:27.72665
2024-12-18T01:22:52.3446749Z STATS: call_* op count: 178 | FakeTensorMode.__torch_dispatch__:19446 | FakeTensor.__torch_dispatch__:6626 | ProxyTorchDispatchMode.__torch_dispatch__:8917
2024-12-18T01:22:52.3447591Z Dynamo produced 2 graphs covering 178 ops with 6 graph breaks (5 unique)
2024-12-18T01:22:57.9949290Z 
2024-12-18T01:22:59.0853772Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:22:59.0854456Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:22:59.0855098Z cuda train spnasnet_100                       
2024-12-18T01:23:46.2708714Z W1218 01:23:46.269000 31318 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T01:23:48.4526012Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T01:24:31.9326317Z pass
2024-12-18T01:24:31.9366139Z TIMING: entire_frame_compile:74.1602 _recursive_pre_grad_passes:0.06126 _recursive_joint_graph_passes:0.58995 _recursive_post_grad_passes:0.93606 async_compile.wait:5.77475 code_gen:29.73487 inductor_compile:49.26503 backend_compile:57.06977 entire_backward_compile:11.80373 total_wall_time:85.96392
2024-12-18T01:24:31.9367753Z STATS: call_* op count: 1245 | FakeTensorMode.__torch_dispatch__:52487 | FakeTensor.__torch_dispatch__:18582 | ProxyTorchDispatchMode.__torch_dispatch__:23261
2024-12-18T01:24:31.9368613Z Dynamo produced 3 graphs covering 1245 ops with 7 graph breaks (5 unique)
2024-12-18T01:24:39.9175585Z 
2024-12-18T01:24:42.1296047Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:24:42.1296883Z loading model: 0it [00:02, ?it/s]
2024-12-18T01:24:42.1297548Z cuda train swin_base_patch4_window7_224       
2024-12-18T01:26:14.1543360Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T01:26:14.1544229Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 405, in torch_dynamo_resume_in_forward_and_backward_pass_at_403
2024-12-18T01:26:14.1545093Z     pred = mod(*cloned_inputs)
2024-12-18T01:26:14.1545911Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/swin_transformer.py", line 838, in forward
2024-12-18T01:26:14.1546746Z     x = self.forward_features(x)
2024-12-18T01:26:14.1547372Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/swin_transformer.py", line 830, in forward_features
2024-12-18T01:26:14.1548013Z     x = self.layers(x)
2024-12-18T01:26:14.1548902Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/swin_transformer.py", line 559, in forward
2024-12-18T01:26:14.1549686Z     x = self.blocks(x)
2024-12-18T01:26:14.1550228Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/swin_transformer.py", line 406, in forward
2024-12-18T01:26:14.1550862Z     x = x + self.drop_path1(self._attn(self.norm1(x)))
2024-12-18T01:26:14.1551487Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/swin_transformer.py", line 390, in _attn
2024-12-18T01:26:14.1552249Z     attn_windows = self.attn(x_windows, mask=attn_mask)  # nW*B, window_size*window_size, C
2024-12-18T01:26:14.1553007Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/swin_transformer.py", line 194, in forward
2024-12-18T01:26:14.1553615Z     attn = attn + self._get_rel_pos_bias()
2024-12-18T01:26:14.1554261Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/swin_transformer.py", line 165, in _get_rel_pos_bias
2024-12-18T01:26:14.1555004Z     relative_position_bias = self.relative_position_bias_table[
2024-12-18T01:26:14.1555304Z 
2024-12-18T01:26:14.7668148Z W1218 01:26:14.765000 32165 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T01:26:17.6631473Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][163].grad", "L['self'].param_groups[0]['params'][164].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][166].grad", "L['self'].param_groups[0]['params'][167].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][169].grad", "L['self'].param_groups[0]['params'][170].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][172].grad", "L['self'].param_groups[0]['params'][173].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][175].grad", "L['self'].param_groups[0]['params'][176].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][178].grad", "L['self'].param_groups[0]['params'][179].grad", "L['self'].param_groups[0]['params'][180].grad", "L['self'].param_groups[0]['params'][181].grad", "L['self'].param_groups[0]['params'][182].grad", "L['self'].param_groups[0]['params'][183].grad", "L['self'].param_groups[0]['params'][184].grad", "L['self'].param_groups[0]['params'][185].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][187].grad", "L['self'].param_groups[0]['params'][188].grad", "L['self'].param_groups[0]['params'][189].grad", "L['self'].param_groups[0]['params'][190].grad", "L['self'].param_groups[0]['params'][191].grad", "L['self'].param_groups[0]['params'][192].grad", "L['self'].param_groups[0]['params'][193].grad", "L['self'].param_groups[0]['params'][194].grad", "L['self'].param_groups[0]['params'][195].grad", "L['self'].param_groups[0]['params'][196].grad", "L['self'].param_groups[0]['params'][197].grad", "L['self'].param_groups[0]['params'][198].grad", "L['self'].param_groups[0]['params'][199].grad", "L['self'].param_groups[0]['params'][200].grad", "L['self'].param_groups[0]['params'][201].grad", "L['self'].param_groups[0]['params'][202].grad", "L['self'].param_groups[0]['params'][203].grad", "L['self'].param_groups[0]['params'][204].grad", "L['self'].param_groups[0]['params'][205].grad", "L['self'].param_groups[0]['params'][206].grad", "L['self'].param_groups[0]['params'][207].grad", "L['self'].param_groups[0]['params'][208].grad", "L['self'].param_groups[0]['params'][209].grad", "L['self'].param_groups[0]['params'][210].grad", "L['self'].param_groups[0]['params'][211].grad", "L['self'].param_groups[0]['params'][212].grad", "L['self'].param_groups[0]['params'][213].grad", "L['self'].param_groups[0]['params'][214].grad", "L['self'].param_groups[0]['params'][215].grad", "L['self'].param_groups[0]['params'][216].grad", "L['self'].param_groups[0]['params'][217].grad", "L['self'].param_groups[0]['params'][218].grad", "L['self'].param_groups[0]['params'][219].grad", "L['self'].param_groups[0]['params'][220].grad", "L['self'].param_groups[0]['params'][221].grad", "L['self'].param_groups[0]['params'][222].grad", "L['self'].param_groups[0]['params'][223].grad", "L['self'].param_groups[0]['params'][224].grad", "L['self'].param_groups[0]['params'][225].grad", "L['self'].param_groups[0]['params'][226].grad", "L['self'].param_groups[0]['params'][227].grad", "L['self'].param_groups[0]['params'][228].grad", "L['self'].param_groups[0]['params'][229].grad", "L['self'].param_groups[0]['params'][230].grad", "L['self'].param_groups[0]['params'][231].grad", "L['self'].param_groups[0]['params'][232].grad", "L['self'].param_groups[0]['params'][233].grad", "L['self'].param_groups[0]['params'][234].grad", "L['self'].param_groups[0]['params'][235].grad", "L['self'].param_groups[0]['params'][236].grad", "L['self'].param_groups[0]['params'][237].grad", "L['self'].param_groups[0]['params'][238].grad", "L['self'].param_groups[0]['params'][239].grad", "L['self'].param_groups[0]['params'][240].grad", "L['self'].param_groups[0]['params'][241].grad", "L['self'].param_groups[0]['params'][242].grad", "L['self'].param_groups[0]['params'][243].grad", "L['self'].param_groups[0]['params'][244].grad", "L['self'].param_groups[0]['params'][245].grad", "L['self'].param_groups[0]['params'][246].grad", "L['self'].param_groups[0]['params'][247].grad", "L['self'].param_groups[0]['params'][248].grad", "L['self'].param_groups[0]['params'][249].grad", "L['self'].param_groups[0]['params'][250].grad", "L['self'].param_groups[0]['params'][251].grad", "L['self'].param_groups[0]['params'][252].grad", "L['self'].param_groups[0]['params'][253].grad", "L['self'].param_groups[0]['params'][254].grad", "L['self'].param_groups[0]['params'][255].grad", "L['self'].param_groups[0]['params'][256].grad", "L['self'].param_groups[0]['params'][257].grad", "L['self'].param_groups[0]['params'][258].grad", "L['self'].param_groups[0]['params'][259].grad", "L['self'].param_groups[0]['params'][260].grad", "L['self'].param_groups[0]['params'][261].grad", "L['self'].param_groups[0]['params'][262].grad", "L['self'].param_groups[0]['params'][263].grad", "L['self'].param_groups[0]['params'][264].grad", "L['self'].param_groups[0]['params'][265].grad", "L['self'].param_groups[0]['params'][266].grad", "L['self'].param_groups[0]['params'][267].grad", "L['self'].param_groups[0]['params'][268].grad", "L['self'].param_groups[0]['params'][269].grad", "L['self'].param_groups[0]['params'][270].grad", "L['self'].param_groups[0]['params'][271].grad", "L['self'].param_groups[0]['params'][272].grad", "L['self'].param_groups[0]['params'][273].grad", "L['self'].param_groups[0]['params'][274].grad", "L['self'].param_groups[0]['params'][275].grad", "L['self'].param_groups[0]['params'][276].grad", "L['self'].param_groups[0]['params'][277].grad", "L['self'].param_groups[0]['params'][278].grad", "L['self'].param_groups[0]['params'][279].grad", "L['self'].param_groups[0]['params'][280].grad", "L['self'].param_groups[0]['params'][281].grad", "L['self'].param_groups[0]['params'][282].grad", "L['self'].param_groups[0]['params'][283].grad", "L['self'].param_groups[0]['params'][284].grad", "L['self'].param_groups[0]['params'][285].grad", "L['self'].param_groups[0]['params'][286].grad", "L['self'].param_groups[0]['params'][287].grad", "L['self'].param_groups[0]['params'][288].grad", "L['self'].param_groups[0]['params'][289].grad", "L['self'].param_groups[0]['params'][290].grad", "L['self'].param_groups[0]['params'][291].grad", "L['self'].param_groups[0]['params'][292].grad", "L['self'].param_groups[0]['params'][293].grad", "L['self'].param_groups[0]['params'][294].grad", "L['self'].param_groups[0]['params'][295].grad", "L['self'].param_groups[0]['params'][296].grad", "L['self'].param_groups[0]['params'][297].grad", "L['self'].param_groups[0]['params'][298].grad", "L['self'].param_groups[0]['params'][299].grad", "L['self'].param_groups[0]['params'][300].grad", "L['self'].param_groups[0]['params'][301].grad", "L['self'].param_groups[0]['params'][302].grad", "L['self'].param_groups[0]['params'][303].grad", "L['self'].param_groups[0]['params'][304].grad", "L['self'].param_groups[0]['params'][305].grad", "L['self'].param_groups[0]['params'][306].grad", "L['self'].param_groups[0]['params'][307].grad", "L['self'].param_groups[0]['params'][308].grad", "L['self'].param_groups[0]['params'][309].grad", "L['self'].param_groups[0]['params'][310].grad", "L['self'].param_groups[0]['params'][311].grad", "L['self'].param_groups[0]['params'][312].grad", "L['self'].param_groups[0]['params'][313].grad", "L['self'].param_groups[0]['params'][314].grad", "L['self'].param_groups[0]['params'][315].grad", "L['self'].param_groups[0]['params'][316].grad", "L['self'].param_groups[0]['params'][317].grad", "L['self'].param_groups[0]['params'][318].grad", "L['self'].param_groups[0]['params'][319].grad", "L['self'].param_groups[0]['params'][320].grad", "L['self'].param_groups[0]['params'][321].grad", "L['self'].param_groups[0]['params'][322].grad", "L['self'].param_groups[0]['params'][323].grad", "L['self'].param_groups[0]['params'][324].grad", "L['self'].param_groups[0]['params'][325].grad", "L['self'].param_groups[0]['params'][326].grad", "L['self'].param_groups[0]['params'][327].grad", "L['self'].param_groups[0]['params'][328].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T01:27:24.3939466Z pass
2024-12-18T01:27:24.4378292Z TIMING: entire_frame_compile:121.2932 _recursive_pre_grad_passes:0.1201 _recursive_joint_graph_passes:8.25724 _recursive_post_grad_passes:2.51163 async_compile.wait:10.29279 code_gen:55.49559 inductor_compile:85.79445 backend_compile:97.01033 entire_backward_compile:28.53878 total_wall_time:149.83198
2024-12-18T01:27:24.4379923Z STATS: call_* op count: 3106 | FakeTensorMode.__torch_dispatch__:120636 | FakeTensor.__torch_dispatch__:28073 | ProxyTorchDispatchMode.__torch_dispatch__:53006
2024-12-18T01:27:24.4380782Z Dynamo produced 3 graphs covering 3106 ops with 7 graph breaks (5 unique)
2024-12-18T01:27:34.3675700Z 
2024-12-18T01:27:37.5867468Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:27:37.5867867Z loading model: 0it [00:03, ?it/s]
2024-12-18T01:27:37.5868242Z cuda train swsl_resnext101_32x16d             
2024-12-18T01:28:53.2903236Z pass
2024-12-18T01:28:53.4274135Z TIMING: entire_frame_compile:44.36982 _recursive_pre_grad_passes:0.02467 _recursive_joint_graph_passes:0.91656 _recursive_post_grad_passes:1.27994 async_compile.wait:1.18004 code_gen:18.09093 inductor_compile:33.65532 backend_compile:32.57306 entire_backward_compile:16.53619 total_wall_time:60.906
2024-12-18T01:28:53.4275749Z STATS: call_* op count: 453 | FakeTensorMode.__torch_dispatch__:45478 | FakeTensor.__torch_dispatch__:16154 | ProxyTorchDispatchMode.__torch_dispatch__:22001
2024-12-18T01:28:53.4276600Z Dynamo produced 2 graphs covering 453 ops with 6 graph breaks (5 unique)
2024-12-18T01:29:00.3262157Z 
2024-12-18T01:29:01.3212644Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:29:01.3213344Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:29:01.3213996Z cuda train tf_efficientnet_b0                 
2024-12-18T01:29:56.5205998Z pass
2024-12-18T01:29:56.5293180Z TIMING: entire_frame_compile:34.05149 _recursive_pre_grad_passes:0.01749 _recursive_joint_graph_passes:0.42659 _recursive_post_grad_passes:0.96902 async_compile.wait:1.9831 code_gen:20.47305 inductor_compile:32.87257 backend_compile:25.96897 entire_backward_compile:16.78367 total_wall_time:50.83516
2024-12-18T01:29:56.5296489Z STATS: call_* op count: 297 | FakeTensorMode.__torch_dispatch__:33061 | FakeTensor.__torch_dispatch__:11021 | ProxyTorchDispatchMode.__torch_dispatch__:14649
2024-12-18T01:29:56.5298158Z Dynamo produced 2 graphs covering 297 ops with 6 graph breaks (5 unique)
2024-12-18T01:30:03.0922882Z 
2024-12-18T01:30:04.2427580Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:30:04.2427949Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:30:04.2428291Z cuda train tf_mixnet_l                        
2024-12-18T01:31:23.0340529Z pass
2024-12-18T01:31:23.0433869Z TIMING: entire_frame_compile:49.57694 _recursive_pre_grad_passes:0.02681 _recursive_joint_graph_passes:0.53882 _recursive_post_grad_passes:1.66848 async_compile.wait:2.57939 code_gen:30.72937 inductor_compile:46.85695 backend_compile:36.98691 entire_backward_compile:23.28766 total_wall_time:72.8646
2024-12-18T01:31:23.0435674Z STATS: call_* op count: 598 | FakeTensorMode.__torch_dispatch__:47118 | FakeTensor.__torch_dispatch__:14017 | ProxyTorchDispatchMode.__torch_dispatch__:19823
2024-12-18T01:31:23.0436521Z Dynamo produced 2 graphs covering 598 ops with 6 graph breaks (5 unique)
2024-12-18T01:31:30.5731974Z 
2024-12-18T01:31:31.6251656Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:31:31.6252370Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:31:31.6253028Z cuda train tinynet_a                          
2024-12-18T01:32:32.1849060Z pass
2024-12-18T01:32:32.1932925Z TIMING: entire_frame_compile:38.0838 _recursive_pre_grad_passes:0.01867 _recursive_joint_graph_passes:0.48619 _recursive_post_grad_passes:1.16167 async_compile.wait:2.04667 code_gen:20.87477 inductor_compile:34.73435 backend_compile:28.83645 entire_backward_compile:17.6967 total_wall_time:55.7805
2024-12-18T01:32:32.1934542Z STATS: call_* op count: 346 | FakeTensorMode.__torch_dispatch__:38539 | FakeTensor.__torch_dispatch__:13019 | ProxyTorchDispatchMode.__torch_dispatch__:17283
2024-12-18T01:32:32.1935395Z Dynamo produced 2 graphs covering 346 ops with 6 graph breaks (5 unique)
2024-12-18T01:32:39.1032774Z 
2024-12-18T01:32:40.1648346Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:32:40.1648720Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:32:40.1649045Z cuda train tnt_s_patch16_224                  
2024-12-18T01:33:47.0935475Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T01:33:47.0936336Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 405, in torch_dynamo_resume_in_forward_and_backward_pass_at_403
2024-12-18T01:33:47.0937063Z     pred = mod(*cloned_inputs)
2024-12-18T01:33:47.0937636Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/tnt.py", line 335, in forward
2024-12-18T01:33:47.0938199Z     x = self.forward_features(x)
2024-12-18T01:33:47.0938801Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/tnt.py", line 311, in forward_features
2024-12-18T01:33:47.0939462Z     pixel_embed = self.pixel_embed(x, self.pixel_pos)
2024-12-18T01:33:47.0940056Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/tnt.py", line 183, in forward
2024-12-18T01:33:47.0940592Z     x = self.unfold(x)
2024-12-18T01:33:47.0940761Z 
2024-12-18T01:33:47.4558787Z W1218 01:33:47.455000 37936 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T01:33:51.3293346Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][163].grad", "L['self'].param_groups[0]['params'][164].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][166].grad", "L['self'].param_groups[0]['params'][167].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][169].grad", "L['self'].param_groups[0]['params'][170].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][172].grad", "L['self'].param_groups[0]['params'][173].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][175].grad", "L['self'].param_groups[0]['params'][176].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][178].grad", "L['self'].param_groups[0]['params'][179].grad", "L['self'].param_groups[0]['params'][180].grad", "L['self'].param_groups[0]['params'][181].grad", "L['self'].param_groups[0]['params'][182].grad", "L['self'].param_groups[0]['params'][183].grad", "L['self'].param_groups[0]['params'][184].grad", "L['self'].param_groups[0]['params'][185].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][187].grad", "L['self'].param_groups[0]['params'][188].grad", "L['self'].param_groups[0]['params'][189].grad", "L['self'].param_groups[0]['params'][190].grad", "L['self'].param_groups[0]['params'][191].grad", "L['self'].param_groups[0]['params'][192].grad", "L['self'].param_groups[0]['params'][193].grad", "L['self'].param_groups[0]['params'][194].grad", "L['self'].param_groups[0]['params'][195].grad", "L['self'].param_groups[0]['params'][196].grad", "L['self'].param_groups[0]['params'][197].grad", "L['self'].param_groups[0]['params'][198].grad", "L['self'].param_groups[0]['params'][199].grad", "L['self'].param_groups[0]['params'][200].grad", "L['self'].param_groups[0]['params'][201].grad", "L['self'].param_groups[0]['params'][202].grad", "L['self'].param_groups[0]['params'][203].grad", "L['self'].param_groups[0]['params'][204].grad", "L['self'].param_groups[0]['params'][205].grad", "L['self'].param_groups[0]['params'][206].grad", "L['self'].param_groups[0]['params'][207].grad", "L['self'].param_groups[0]['params'][208].grad", "L['self'].param_groups[0]['params'][209].grad", "L['self'].param_groups[0]['params'][210].grad", "L['self'].param_groups[0]['params'][211].grad", "L['self'].param_groups[0]['params'][212].grad", "L['self'].param_groups[0]['params'][213].grad", "L['self'].param_groups[0]['params'][214].grad", "L['self'].param_groups[0]['params'][215].grad", "L['self'].param_groups[0]['params'][216].grad", "L['self'].param_groups[0]['params'][217].grad", "L['self'].param_groups[0]['params'][218].grad", "L['self'].param_groups[0]['params'][219].grad", "L['self'].param_groups[0]['params'][220].grad", "L['self'].param_groups[0]['params'][221].grad", "L['self'].param_groups[0]['params'][222].grad", "L['self'].param_groups[0]['params'][223].grad", "L['self'].param_groups[0]['params'][224].grad", "L['self'].param_groups[0]['params'][225].grad", "L['self'].param_groups[0]['params'][226].grad", "L['self'].param_groups[0]['params'][227].grad", "L['self'].param_groups[0]['params'][228].grad", "L['self'].param_groups[0]['params'][229].grad", "L['self'].param_groups[0]['params'][230].grad", "L['self'].param_groups[0]['params'][231].grad", "L['self'].param_groups[0]['params'][232].grad", "L['self'].param_groups[0]['params'][233].grad", "L['self'].param_groups[0]['params'][234].grad", "L['self'].param_groups[0]['params'][235].grad", "L['self'].param_groups[0]['params'][236].grad", "L['self'].param_groups[0]['params'][237].grad", "L['self'].param_groups[0]['params'][238].grad", "L['self'].param_groups[0]['params'][239].grad", "L['self'].param_groups[0]['params'][240].grad", "L['self'].param_groups[0]['params'][241].grad", "L['self'].param_groups[0]['params'][242].grad", "L['self'].param_groups[0]['params'][243].grad", "L['self'].param_groups[0]['params'][244].grad", "L['self'].param_groups[0]['params'][245].grad", "L['self'].param_groups[0]['params'][246].grad", "L['self'].param_groups[0]['params'][247].grad", "L['self'].param_groups[0]['params'][248].grad", "L['self'].param_groups[0]['params'][249].grad", "L['self'].param_groups[0]['params'][250].grad", "L['self'].param_groups[0]['params'][251].grad", "L['self'].param_groups[0]['params'][252].grad", "L['self'].param_groups[0]['params'][253].grad", "L['self'].param_groups[0]['params'][254].grad", "L['self'].param_groups[0]['params'][255].grad", "L['self'].param_groups[0]['params'][256].grad", "L['self'].param_groups[0]['params'][257].grad", "L['self'].param_groups[0]['params'][258].grad", "L['self'].param_groups[0]['params'][259].grad", "L['self'].param_groups[0]['params'][260].grad", "L['self'].param_groups[0]['params'][261].grad", "L['self'].param_groups[0]['params'][262].grad", "L['self'].param_groups[0]['params'][263].grad", "L['self'].param_groups[0]['params'][264].grad", "L['self'].param_groups[0]['params'][265].grad", "L['self'].param_groups[0]['params'][266].grad", "L['self'].param_groups[0]['params'][267].grad", "L['self'].param_groups[0]['params'][268].grad", "L['self'].param_groups[0]['params'][269].grad", "L['self'].param_groups[0]['params'][270].grad", "L['self'].param_groups[0]['params'][271].grad", "L['self'].param_groups[0]['params'][272].grad", "L['self'].param_groups[0]['params'][273].grad", "L['self'].param_groups[0]['params'][274].grad", "L['self'].param_groups[0]['params'][275].grad", "L['self'].param_groups[0]['params'][276].grad", "L['self'].param_groups[0]['params'][277].grad", "L['self'].param_groups[0]['params'][278].grad", "L['self'].param_groups[0]['params'][279].grad", "L['self'].param_groups[0]['params'][280].grad", "L['self'].param_groups[0]['params'][281].grad", "L['self'].param_groups[0]['params'][282].grad", "L['self'].param_groups[0]['params'][283].grad", "L['self'].param_groups[0]['params'][284].grad", "L['self'].param_groups[0]['params'][285].grad", "L['self'].param_groups[0]['params'][286].grad", "L['self'].param_groups[0]['params'][287].grad", "L['self'].param_groups[0]['params'][288].grad", "L['self'].param_groups[0]['params'][289].grad", "L['self'].param_groups[0]['params'][290].grad", "L['self'].param_groups[0]['params'][291].grad", "L['self'].param_groups[0]['params'][292].grad", "L['self'].param_groups[0]['params'][293].grad", "L['self'].param_groups[0]['params'][294].grad", "L['self'].param_groups[0]['params'][295].grad", "L['self'].param_groups[0]['params'][296].grad", "L['self'].param_groups[0]['params'][297].grad", "L['self'].param_groups[0]['params'][298].grad", "L['self'].param_groups[0]['params'][299].grad", "L['self'].param_groups[0]['params'][300].grad", "L['self'].param_groups[0]['params'][301].grad", "L['self'].param_groups[0]['params'][302].grad", "L['self'].param_groups[0]['params'][303].grad", "L['self'].param_groups[0]['params'][304].grad", "L['self'].param_groups[0]['params'][305].grad", "L['self'].param_groups[0]['params'][306].grad", "L['self'].param_groups[0]['params'][307].grad", "L['self'].param_groups[0]['params'][308].grad", "L['self'].param_groups[0]['params'][309].grad", "L['self'].param_groups[0]['params'][310].grad", "L['self'].param_groups[0]['params'][311].grad", "L['self'].param_groups[0]['params'][312].grad", "L['self'].param_groups[0]['params'][313].grad", "L['self'].param_groups[0]['params'][314].grad", "L['self'].param_groups[0]['params'][315].grad", "L['self'].param_groups[0]['params'][316].grad", "L['self'].param_groups[0]['params'][317].grad", "L['self'].param_groups[0]['params'][318].grad", "L['self'].param_groups[0]['params'][319].grad", "L['self'].param_groups[0]['params'][320].grad", "L['self'].param_groups[0]['params'][321].grad", "L['self'].param_groups[0]['params'][322].grad", "L['self'].param_groups[0]['params'][323].grad", "L['self'].param_groups[0]['params'][324].grad", "L['self'].param_groups[0]['params'][325].grad", "L['self'].param_groups[0]['params'][326].grad", "L['self'].param_groups[0]['params'][327].grad", "L['self'].param_groups[0]['params'][328].grad", "L['self'].param_groups[0]['params'][329].grad", "L['self'].param_groups[0]['params'][330].grad", "L['self'].param_groups[0]['params'][331].grad", "L['self'].param_groups[0]['params'][332].grad", "L['self'].param_groups[0]['params'][333].grad", "L['self'].param_groups[0]['params'][334].grad", "L['self'].param_groups[0]['params'][335].grad", "L['self'].param_groups[0]['params'][336].grad", "L['self'].param_groups[0]['params'][337].grad", "L['self'].param_groups[0]['params'][338].grad", "L['self'].param_groups[0]['params'][339].grad", "L['self'].param_groups[0]['params'][340].grad", "L['self'].param_groups[0]['params'][341].grad", "L['self'].param_groups[0]['params'][342].grad", "L['self'].param_groups[0]['params'][343].grad", "L['self'].param_groups[0]['params'][344].grad", "L['self'].param_groups[0]['params'][345].grad", "L['self'].param_groups[0]['params'][346].grad", "L['self'].param_groups[0]['params'][347].grad", "L['self'].param_groups[0]['params'][348].grad", "L['self'].param_groups[0]['params'][349].grad", "L['self'].param_groups[0]['params'][350].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T01:35:03.1624944Z pass
2024-12-18T01:35:03.1874072Z TIMING: entire_frame_compile:114.16878 _recursive_pre_grad_passes:0.10566 _recursive_joint_graph_passes:5.66172 _recursive_post_grad_passes:2.21494 async_compile.wait:8.33534 code_gen:43.98777 inductor_compile:72.80823 backend_compile:89.15317 entire_backward_compile:19.43302 total_wall_time:133.6018
2024-12-18T01:35:03.1876056Z STATS: call_* op count: 2553 | FakeTensorMode.__torch_dispatch__:113180 | FakeTensor.__torch_dispatch__:27800 | ProxyTorchDispatchMode.__torch_dispatch__:52116
2024-12-18T01:35:03.1877103Z Dynamo produced 3 graphs covering 2553 ops with 7 graph breaks (5 unique)
2024-12-18T01:35:13.4329465Z 
2024-12-18T01:35:14.8671904Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:35:14.8681942Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:35:14.8682308Z cuda train twins_pcpvt_base                   
2024-12-18T01:36:33.5516017Z W1218 01:36:33.550000 38575 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T01:36:38.4506738Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][204].grad", "L['self'].param_groups[0]['params'][222].grad", "L['self'].param_groups[0]['params'][240].grad", "L['self'].param_groups[0]['params'][258].grad", "L['self'].param_groups[0]['params'][276].grad", "L['self'].param_groups[0]['params'][294].grad", "L['self'].param_groups[0]['params'][312].grad", "L['self'].param_groups[0]['params'][330].grad", "L['self'].param_groups[0]['params'][348].grad", "L['self'].param_groups[0]['params'][366].grad", "L['self'].param_groups[0]['params'][384].grad", "L['self'].param_groups[0]['params'][402].grad", "L['self'].param_groups[0]['params'][420].grad", "L['self'].param_groups[0]['params'][438].grad", "L['self'].param_groups[0]['params'][456].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T01:38:32.6136664Z pass
2024-12-18T01:38:32.6445633Z TIMING: entire_frame_compile:152.16371 _recursive_pre_grad_passes:0.15311 _recursive_joint_graph_passes:2.38978 _recursive_post_grad_passes:2.91113 async_compile.wait:10.50569 code_gen:62.64205 inductor_compile:102.82965 backend_compile:117.61922 entire_backward_compile:27.58725 total_wall_time:179.75095
2024-12-18T01:38:32.6447270Z STATS: call_* op count: 3474 | FakeTensorMode.__torch_dispatch__:142743 | FakeTensor.__torch_dispatch__:39921 | ProxyTorchDispatchMode.__torch_dispatch__:63230
2024-12-18T01:38:32.6448130Z Dynamo produced 3 graphs covering 3474 ops with 7 graph breaks (5 unique)
2024-12-18T01:38:44.5782944Z 
2024-12-18T01:38:45.9277202Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:38:45.9277566Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:38:45.9277916Z cuda train visformer_small                    
2024-12-18T01:39:19.9112965Z W1218 01:39:19.910000 39499 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T01:39:49.8524241Z pass
2024-12-18T01:39:49.8778208Z TIMING: entire_frame_compile:50.67024 _recursive_pre_grad_passes:0.04551 _recursive_joint_graph_passes:3.93577 _recursive_post_grad_passes:0.77409 async_compile.wait:4.01575 code_gen:17.32235 inductor_compile:29.4866 backend_compile:38.36339 entire_backward_compile:7.14482 total_wall_time:57.81506
2024-12-18T01:39:49.8779821Z STATS: call_* op count: 947 | FakeTensorMode.__torch_dispatch__:39095 | FakeTensor.__torch_dispatch__:12293 | ProxyTorchDispatchMode.__torch_dispatch__:16835
2024-12-18T01:39:49.8780662Z Dynamo produced 3 graphs covering 947 ops with 7 graph breaks (5 unique)
2024-12-18T01:39:56.4227689Z 
2024-12-18T01:39:58.0199308Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:39:58.0200316Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:39:58.0200915Z cuda train vit_base_patch16_224               
2024-12-18T01:40:24.8387675Z W1218 01:40:24.837000 40135 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T01:40:56.5324803Z pass
2024-12-18T01:40:56.5829468Z TIMING: entire_frame_compile:42.82172 _recursive_pre_grad_passes:0.04846 _recursive_joint_graph_passes:0.80988 _recursive_post_grad_passes:0.71316 async_compile.wait:4.28342 code_gen:16.36341 inductor_compile:28.13782 backend_compile:33.4932 entire_backward_compile:6.99425 total_wall_time:49.81598
2024-12-18T01:40:56.5831104Z STATS: call_* op count: 1046 | FakeTensorMode.__torch_dispatch__:41708 | FakeTensor.__torch_dispatch__:11714 | ProxyTorchDispatchMode.__torch_dispatch__:19022
2024-12-18T01:40:56.5831978Z Dynamo produced 3 graphs covering 1046 ops with 7 graph breaks (5 unique)
2024-12-18T01:41:02.9715101Z 
2024-12-18T01:41:04.3576465Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:41:04.3576829Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:41:04.3577174Z cuda train volo_d1_224                        
2024-12-18T01:41:37.3374038Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T01:41:37.3374909Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 405, in torch_dynamo_resume_in_forward_and_backward_pass_at_403
2024-12-18T01:41:37.3375634Z     pred = mod(*cloned_inputs)
2024-12-18T01:41:37.3376179Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/volo.py", line 822, in forward
2024-12-18T01:41:37.3376730Z     x = self.forward_features(x)
2024-12-18T01:41:37.3377309Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/volo.py", line 795, in forward_features
2024-12-18T01:41:37.3377911Z     x = self.forward_tokens(x)
2024-12-18T01:41:37.3378506Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/volo.py", line 642, in forward_tokens
2024-12-18T01:41:37.3379076Z     x = block(x)
2024-12-18T01:41:37.3379573Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/volo.py", line 135, in forward
2024-12-18T01:41:37.3380149Z     x = x + self.drop_path1(self.attn(self.norm1(x)))
2024-12-18T01:41:37.3380725Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/volo.py", line 90, in forward
2024-12-18T01:41:37.3381470Z     x = F.fold(x, output_size=(H, W), kernel_size=self.kernel_size, padding=self.padding, stride=self.stride)
2024-12-18T01:41:37.3381894Z 
2024-12-18T01:42:00.3483863Z W1218 01:42:00.347000 40459 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T01:42:02.5061625Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][163].grad", "L['self'].param_groups[0]['params'][164].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][166].grad", "L['self'].param_groups[0]['params'][167].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][169].grad", "L['self'].param_groups[0]['params'][170].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][172].grad", "L['self'].param_groups[0]['params'][173].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][175].grad", "L['self'].param_groups[0]['params'][176].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][178].grad", "L['self'].param_groups[0]['params'][179].grad", "L['self'].param_groups[0]['params'][180].grad", "L['self'].param_groups[0]['params'][181].grad", "L['self'].param_groups[0]['params'][182].grad", "L['self'].param_groups[0]['params'][183].grad", "L['self'].param_groups[0]['params'][184].grad", "L['self'].param_groups[0]['params'][185].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][187].grad", "L['self'].param_groups[0]['params'][188].grad", "L['self'].param_groups[0]['params'][189].grad", "L['self'].param_groups[0]['params'][190].grad", "L['self'].param_groups[0]['params'][191].grad", "L['self'].param_groups[0]['params'][192].grad", "L['self'].param_groups[0]['params'][193].grad", "L['self'].param_groups[0]['params'][194].grad", "L['self'].param_groups[0]['params'][195].grad", "L['self'].param_groups[0]['params'][196].grad", "L['self'].param_groups[0]['params'][197].grad", "L['self'].param_groups[0]['params'][198].grad", "L['self'].param_groups[0]['params'][199].grad", "L['self'].param_groups[0]['params'][200].grad", "L['self'].param_groups[0]['params'][201].grad", "L['self'].param_groups[0]['params'][202].grad", "L['self'].param_groups[0]['params'][203].grad", "L['self'].param_groups[0]['params'][204].grad", "L['self'].param_groups[0]['params'][205].grad", "L['self'].param_groups[0]['params'][206].grad", "L['self'].param_groups[0]['params'][207].grad", "L['self'].param_groups[0]['params'][208].grad", "L['self'].param_groups[0]['params'][209].grad", "L['self'].param_groups[0]['params'][210].grad", "L['self'].param_groups[0]['params'][211].grad", "L['self'].param_groups[0]['params'][212].grad", "L['self'].param_groups[0]['params'][213].grad", "L['self'].param_groups[0]['params'][214].grad", "L['self'].param_groups[0]['params'][215].grad", "L['self'].param_groups[0]['params'][216].grad", "L['self'].param_groups[0]['params'][217].grad", "L['self'].param_groups[0]['params'][218].grad", "L['self'].param_groups[0]['params'][219].grad", "L['self'].param_groups[0]['params'][220].grad", "L['self'].param_groups[0]['params'][221].grad", "L['self'].param_groups[0]['params'][222].grad", "L['self'].param_groups[0]['params'][223].grad", "L['self'].param_groups[0]['params'][224].grad", "L['self'].param_groups[0]['params'][225].grad", "L['self'].param_groups[0]['params'][226].grad", "L['self'].param_groups[0]['params'][227].grad", "L['self'].param_groups[0]['params'][228].grad", "L['self'].param_groups[0]['params'][229].grad", "L['self'].param_groups[0]['params'][230].grad", "L['self'].param_groups[0]['params'][231].grad", "L['self'].param_groups[0]['params'][232].grad", "L['self'].param_groups[0]['params'][233].grad", "L['self'].param_groups[0]['params'][234].grad", "L['self'].param_groups[0]['params'][235].grad", "L['self'].param_groups[0]['params'][236].grad", "L['self'].param_groups[0]['params'][237].grad", "L['self'].param_groups[0]['params'][238].grad", "L['self'].param_groups[0]['params'][239].grad", "L['self'].param_groups[0]['params'][240].grad", "L['self'].param_groups[0]['params'][241].grad", "L['self'].param_groups[0]['params'][242].grad", "L['self'].param_groups[0]['params'][243].grad", "L['self'].param_groups[0]['params'][244].grad", "L['self'].param_groups[0]['params'][245].grad", "L['self'].param_groups[0]['params'][246].grad", "L['self'].param_groups[0]['params'][247].grad", "L['self'].param_groups[0]['params'][248].grad", "L['self'].param_groups[0]['params'][249].grad", "L['self'].param_groups[0]['params'][250].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T01:42:51.4296517Z pass
2024-12-18T01:42:51.4431322Z TIMING: entire_frame_compile:79.26634 _recursive_pre_grad_passes:0.07646 _recursive_joint_graph_passes:4.52262 _recursive_post_grad_passes:1.37987 async_compile.wait:7.46568 code_gen:38.04787 inductor_compile:57.77969 backend_compile:62.36401 entire_backward_compile:20.76567 total_wall_time:100.03201
2024-12-18T01:42:51.4433867Z STATS: call_* op count: 1786 | FakeTensorMode.__torch_dispatch__:74997 | FakeTensor.__torch_dispatch__:19150 | ProxyTorchDispatchMode.__torch_dispatch__:34665
2024-12-18T01:42:51.4435224Z Dynamo produced 3 graphs covering 1786 ops with 7 graph breaks (5 unique)
2024-12-18T01:42:59.8698529Z 
2024-12-18T01:43:03.4616107Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:43:03.4616597Z loading model: 0it [00:03, ?it/s]
2024-12-18T01:43:03.4616944Z cuda train xcit_large_24_p8_224               
2024-12-18T01:43:07.7690486Z WARNING:common:fp64 golden ref were not generated for xcit_large_24_p8_224. Setting accuracy check to cosine
2024-12-18T01:44:55.9306848Z W1218 01:44:55.929000 41147 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T01:45:02.0151288Z ('Grad tensors ["L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][7].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T01:47:22.6612191Z pass_due_to_skip
2024-12-18T01:47:22.7043940Z TIMING: entire_frame_compile:196.58421 _recursive_pre_grad_passes:0.1888 _recursive_joint_graph_passes:3.23215 _recursive_post_grad_passes:2.81367 async_compile.wait:10.20006 code_gen:79.52742 inductor_compile:130.84023 backend_compile:150.96442 entire_backward_compile:35.41035 total_wall_time:231.99456
2024-12-18T01:47:22.7045635Z STATS: call_* op count: 4300 | FakeTensorMode.__torch_dispatch__:180235 | FakeTensor.__torch_dispatch__:53119 | ProxyTorchDispatchMode.__torch_dispatch__:83485
2024-12-18T01:47:22.7046501Z Dynamo produced 3 graphs covering 4300 ops with 7 graph breaks (5 unique)
2024-12-18T01:47:33.2423368Z accuracy            pass_rate=96.77%
2024-12-18T01:47:33.2423805Z calls_captured      gmean=806.58x mean=1124.387x
2024-12-18T01:47:33.2424304Z unique_graphs       gmean=2.49x mean=2.548x
2024-12-18T01:47:33.2427900Z graph_breaks        gmean=6.39x mean=6.419x
2024-12-18T01:47:33.2431232Z unique_graph_breaks gmean=4.96x mean=4.968x
2024-12-18T01:47:33.2434614Z autograd_captures   gmean=0.00x mean=0.000x
2024-12-18T01:47:33.2438087Z autograd_compiles   gmean=0.00x mean=0.000x
2024-12-18T01:47:33.2441562Z cudagraph_skips     gmean=0.00x mean=0.097x
2024-12-18T01:47:37.2378149Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_timm_models.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_timm_training.csv
2024-12-18T01:47:37.5023799Z mixnet_l                            PASS
2024-12-18T01:47:37.5027899Z mnasnet_100                         PASS
2024-12-18T01:47:37.5032535Z mobilenetv2_100                     PASS
2024-12-18T01:47:37.5036951Z mobilenetv3_large_100               PASS
2024-12-18T01:47:37.5041521Z mobilevit_s                         PASS
2024-12-18T01:47:37.5046172Z nfnet_l0                            PASS
2024-12-18T01:47:37.5051000Z pit_b_224                           PASS
2024-12-18T01:47:37.5055400Z pnasnet5large                       PASS
2024-12-18T01:47:37.5060146Z poolformer_m36                      PASS
2024-12-18T01:47:37.5064632Z regnety_002                         PASS
2024-12-18T01:47:37.5069247Z repvgg_a2                           PASS
2024-12-18T01:47:37.5073746Z res2net101_26w_4s                   PASS
2024-12-18T01:47:37.5078137Z res2net50_14w_8s                    PASS
2024-12-18T01:47:37.5082497Z res2next50                          PASS
2024-12-18T01:47:37.5086746Z resmlp_12_224                       PASS
2024-12-18T01:47:37.5091230Z resnest101e                         PASS
2024-12-18T01:47:37.5095665Z rexnet_100                          PASS
2024-12-18T01:47:37.5099974Z sebotnet33ts_256                    PASS
2024-12-18T01:47:37.5104292Z selecsls42b                         PASS
2024-12-18T01:47:37.5108886Z spnasnet_100                        PASS
2024-12-18T01:47:37.5113249Z swin_base_patch4_window7_224        PASS
2024-12-18T01:47:37.5118094Z swsl_resnext101_32x16d              PASS
2024-12-18T01:47:37.5122434Z tf_efficientnet_b0                  PASS
2024-12-18T01:47:37.5126885Z tf_mixnet_l                         PASS
2024-12-18T01:47:37.5131153Z tinynet_a                           PASS
2024-12-18T01:47:37.5135459Z tnt_s_patch16_224                   PASS
2024-12-18T01:47:37.5139880Z twins_pcpvt_base                    PASS
2024-12-18T01:47:37.5144375Z visformer_small                     PASS
2024-12-18T01:47:37.5149063Z vit_base_patch16_224                PASS
2024-12-18T01:47:37.5153361Z volo_d1_224                         PASS
2024-12-18T01:47:37.5157871Z xcit_large_24_p8_224                XFAIL
2024-12-18T01:47:37.5579399Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_timm_models.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_timm_training.csv
2024-12-18T01:47:37.8216079Z mixnet_l                            PASS
2024-12-18T01:47:37.8219398Z mnasnet_100                         PASS
2024-12-18T01:47:37.8223126Z mobilenetv2_100                     PASS
2024-12-18T01:47:37.8227842Z mobilenetv3_large_100               PASS
2024-12-18T01:47:37.8232111Z mobilevit_s                         PASS
2024-12-18T01:47:37.8236704Z nfnet_l0                            PASS
2024-12-18T01:47:37.8240965Z pit_b_224                           PASS
2024-12-18T01:47:37.8245412Z pnasnet5large                       PASS
2024-12-18T01:47:37.8250017Z poolformer_m36                      PASS
2024-12-18T01:47:37.8254404Z regnety_002                         PASS
2024-12-18T01:47:37.8258925Z repvgg_a2                           PASS
2024-12-18T01:47:37.8263191Z res2net101_26w_4s                   PASS
2024-12-18T01:47:37.8267687Z res2net50_14w_8s                    PASS
2024-12-18T01:47:37.8272256Z res2next50                          PASS
2024-12-18T01:47:37.8276545Z resmlp_12_224                       PASS
2024-12-18T01:47:37.8280993Z resnest101e                         PASS
2024-12-18T01:47:37.8285594Z rexnet_100                          PASS
2024-12-18T01:47:37.8289873Z sebotnet33ts_256                    PASS
2024-12-18T01:47:37.8294245Z selecsls42b                         PASS
2024-12-18T01:47:37.8298740Z spnasnet_100                        PASS
2024-12-18T01:47:37.8303127Z swin_base_patch4_window7_224        PASS
2024-12-18T01:47:37.8307758Z swsl_resnext101_32x16d              PASS
2024-12-18T01:47:37.8312318Z tf_efficientnet_b0                  PASS
2024-12-18T01:47:37.8317352Z tf_mixnet_l                         PASS
2024-12-18T01:47:37.8322305Z tinynet_a                           PASS
2024-12-18T01:47:37.8326831Z tnt_s_patch16_224                   PASS
2024-12-18T01:47:37.8331180Z twins_pcpvt_base                    PASS
2024-12-18T01:47:37.8335641Z visformer_small                     PASS
2024-12-18T01:47:37.8340109Z vit_base_patch16_224                PASS
2024-12-18T01:47:37.8344529Z volo_d1_224                         PASS
2024-12-18T01:47:37.8349213Z xcit_large_24_p8_224                PASS
2024-12-18T01:47:37.8800797Z + cleanup_workspace
2024-12-18T01:47:37.8801379Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.'
2024-12-18T01:47:37.8802222Z sudo may print the following warning message that can be ignored. The chown command will still run.
2024-12-18T01:47:37.8802868Z + echo '    sudo: setrlimit(RLIMIT_STACK): Operation not permitted'
2024-12-18T01:47:37.8803420Z     sudo: setrlimit(RLIMIT_STACK): Operation not permitted
2024-12-18T01:47:37.8807523Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42'
2024-12-18T01:47:37.8808164Z For more details refer to https://github.com/sudo-project/sudo/issues/42
2024-12-18T01:47:37.8808702Z + sudo chown -R 1000 /var/lib/jenkins/workspace
2024-12-18T01:47:38.6582746Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@release/2.6
2024-12-18T01:47:38.6583265Z with:
2024-12-18T01:47:38.6583530Z   benchmark-results-dir: test/test-reports
2024-12-18T01:47:38.6583866Z   dry-run: false
2024-12-18T01:47:38.6584110Z   schema-version: v3
2024-12-18T01:47:38.6584560Z   github-token: ***
2024-12-18T01:47:38.6584804Z env:
2024-12-18T01:47:38.6585035Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:47:38.6585375Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:47:38.6585913Z   DOCKER_CONTAINER_ID: 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T01:47:38.6586495Z ##[endgroup]
2024-12-18T01:47:38.6614059Z ##[group]Run set -eux
2024-12-18T01:47:38.6614344Z set -eux
2024-12-18T01:47:38.6614909Z python3 -mpip install boto3==1.35.33
2024-12-18T01:47:38.6628753Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:47:38.6629320Z env:
2024-12-18T01:47:38.6629578Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:47:38.6629932Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:47:38.6630474Z   DOCKER_CONTAINER_ID: 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T01:47:38.6630961Z ##[endgroup]
2024-12-18T01:47:38.6670124Z + python3 -mpip install boto3==1.35.33
2024-12-18T01:47:38.9344160Z Defaulting to user installation because normal site-packages is not writeable
2024-12-18T01:47:40.0645174Z Collecting boto3==1.35.33
2024-12-18T01:47:40.0990270Z   Downloading boto3-1.35.33-py3-none-any.whl (139 kB)
2024-12-18T01:47:41.3557430Z Collecting botocore<1.36.0,>=1.35.33
2024-12-18T01:47:41.3596650Z   Downloading botocore-1.35.83-py3-none-any.whl (13.3 MB)
2024-12-18T01:47:41.5142540Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.0)
2024-12-18T01:47:41.5931117Z Collecting s3transfer<0.11.0,>=0.10.0
2024-12-18T01:47:41.5971338Z   Downloading s3transfer-0.10.4-py3-none-any.whl (83 kB)
2024-12-18T01:47:41.6063813Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.8.1)
2024-12-18T01:47:41.6073883Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.25.10)
2024-12-18T01:47:41.7537256Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.15.0)
2024-12-18T01:47:41.8381104Z Installing collected packages: botocore, s3transfer, boto3
2024-12-18T01:47:42.4454994Z Successfully installed boto3-1.35.33 botocore-1.35.83 s3transfer-0.10.4
2024-12-18T01:47:42.5579857Z ##[group]Run set -eux
2024-12-18T01:47:42.5580146Z set -eux
2024-12-18T01:47:42.5580395Z 
2024-12-18T01:47:42.5580651Z if [[ -z "${GITHUB_TOKEN}" ]]; then
2024-12-18T01:47:42.5581045Z   echo "Missing github-token input"
2024-12-18T01:47:42.5581379Z   exit 1
2024-12-18T01:47:42.5581618Z fi
2024-12-18T01:47:42.5590415Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:47:42.5590790Z env:
2024-12-18T01:47:42.5591025Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:47:42.5591383Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:47:42.5591931Z   DOCKER_CONTAINER_ID: 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T01:47:42.5592766Z   GITHUB_TOKEN: ***
2024-12-18T01:47:42.5593020Z ##[endgroup]
2024-12-18T01:47:42.5622223Z + [[ -z *** ]]
2024-12-18T01:47:42.5682866Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main
2024-12-18T01:47:42.5683307Z with:
2024-12-18T01:47:42.5683661Z   github-token: ***
2024-12-18T01:47:42.5683911Z env:
2024-12-18T01:47:42.5684140Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:47:42.5684488Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:47:42.5685241Z   DOCKER_CONTAINER_ID: 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T01:47:42.5685731Z ##[endgroup]
2024-12-18T01:47:42.5705335Z ##[group]Run set -eux
2024-12-18T01:47:42.5705619Z set -eux
2024-12-18T01:47:42.5705868Z 
2024-12-18T01:47:42.5706433Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2024-12-18T01:47:42.5715241Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:47:42.5715631Z env:
2024-12-18T01:47:42.5715870Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:47:42.5716219Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:47:42.5716766Z   DOCKER_CONTAINER_ID: 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T01:47:42.5717510Z   GITHUB_TOKEN: ***
2024-12-18T01:47:42.5717764Z ##[endgroup]
2024-12-18T01:47:42.5745078Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 12383255690 i-0ad7d688021a5725d
2024-12-18T01:47:43.8128423Z setting job-id=34567153040
2024-12-18T01:47:43.8129150Z setting job-name=cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T01:47:43.8234672Z ##[group]Run set -eux
2024-12-18T01:47:43.8234969Z set -eux
2024-12-18T01:47:43.8235276Z 
2024-12-18T01:47:43.8235705Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \
2024-12-18T01:47:43.8236229Z   --schema-version "${SCHEMA_VERSION}" \
2024-12-18T01:47:43.8236588Z   --repo "${REPO}" \
2024-12-18T01:47:43.8236903Z   --head-branch "${HEAD_BRANCH}" \
2024-12-18T01:47:43.8237247Z   --head-sha "${HEAD_SHA}" \
2024-12-18T01:47:43.8237591Z   --workflow-id "${WORKFLOW_RUN_ID}" \
2024-12-18T01:47:43.8237957Z   --run-attempt "${RUN_ATTEMPT}" \
2024-12-18T01:47:43.8238322Z   --job-id "${JOB_ID}" \
2024-12-18T01:47:43.8238635Z   --job-name "${JOB_NAME}"
2024-12-18T01:47:43.8247535Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:47:43.8247913Z env:
2024-12-18T01:47:43.8248141Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:47:43.8248485Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:47:43.8249039Z   DOCKER_CONTAINER_ID: 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T01:47:43.8249529Z   SCHEMA_VERSION: v3
2024-12-18T01:47:43.8249794Z   REPO: pytorch/pytorch
2024-12-18T01:47:43.8250078Z   HEAD_BRANCH: refs/heads/release/2.6
2024-12-18T01:47:43.8250443Z   HEAD_SHA: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T01:47:43.8250802Z   WORKFLOW_RUN_ID: 12383255690
2024-12-18T01:47:43.8251084Z   RUN_ATTEMPT: 1
2024-12-18T01:47:43.8251380Z   JOB_ID: 34567153040
2024-12-18T01:47:43.8251839Z   JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T01:47:43.8252353Z ##[endgroup]
2024-12-18T01:47:43.8282459Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/release/2.6/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/heads/release/2.6 --head-sha 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 --workflow-id 12383255690 --run-attempt 1 --job-id 34567153040 --job-name 'cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu)'
2024-12-18T01:47:43.8615369Z ##[group]Run set -eux
2024-12-18T01:47:43.8615658Z set -eux
2024-12-18T01:47:43.8615899Z 
2024-12-18T01:47:43.8616174Z # TODO (huydhn): Implement this part
2024-12-18T01:47:43.8616561Z echo "runners=[]" >> "${GITHUB_OUTPUT}"
2024-12-18T01:47:43.8625512Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:47:43.8625905Z env:
2024-12-18T01:47:43.8626152Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:47:43.8626782Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:47:43.8627341Z   DOCKER_CONTAINER_ID: 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T01:47:43.8627833Z ##[endgroup]
2024-12-18T01:47:43.8654950Z + echo 'runners=[]'
2024-12-18T01:47:43.8681263Z ##[group]Run set -eux
2024-12-18T01:47:43.8681547Z set -eux
2024-12-18T01:47:43.8681785Z 
2024-12-18T01:47:43.8682051Z # TODO (huydhn): Implement this part
2024-12-18T01:47:43.8682445Z echo "dependencies={}" >> "${GITHUB_OUTPUT}"
2024-12-18T01:47:43.8690626Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:47:43.8691016Z env:
2024-12-18T01:47:43.8691282Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:47:43.8691630Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:47:43.8692212Z   DOCKER_CONTAINER_ID: 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T01:47:43.8692891Z ##[endgroup]
2024-12-18T01:47:43.8718597Z + echo 'dependencies={}'
2024-12-18T01:47:43.8745061Z ##[group]Run set -eux
2024-12-18T01:47:43.8745362Z set -eux
2024-12-18T01:47:43.8745604Z 
2024-12-18T01:47:43.8745881Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then
2024-12-18T01:47:43.8746401Z   echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping"
2024-12-18T01:47:43.8746886Z   # We don't want the job to fail if the directory doesn't exist
2024-12-18T01:47:43.8747280Z   exit 0
2024-12-18T01:47:43.8747517Z fi
2024-12-18T01:47:43.8747740Z 
2024-12-18T01:47:43.8747995Z if [[ "${DRY_RUN}" == "true" ]]; then
2024-12-18T01:47:43.8748467Z   python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \
2024-12-18T01:47:43.8749112Z     --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \
2024-12-18T01:47:43.8749541Z     --metadata "${BENCHMARK_METADATA}" \
2024-12-18T01:47:43.8749916Z     --runners "${RUNNER_INFO}" \
2024-12-18T01:47:43.8750282Z     --dependencies "${DEPENDENCIES}" \
2024-12-18T01:47:43.8750621Z     --dry-run
2024-12-18T01:47:43.8750874Z else
2024-12-18T01:47:43.8751259Z   python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \
2024-12-18T01:47:43.8751800Z     --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \
2024-12-18T01:47:43.8752226Z     --metadata "${BENCHMARK_METADATA}" \
2024-12-18T01:47:43.8752586Z     --runners "${RUNNER_INFO}" \
2024-12-18T01:47:43.8752936Z     --dependencies "${DEPENDENCIES}"
2024-12-18T01:47:43.8753253Z fi
2024-12-18T01:47:43.8761400Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:47:43.8761783Z env:
2024-12-18T01:47:43.8762015Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:47:43.8762354Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:47:43.8762904Z   DOCKER_CONTAINER_ID: 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T01:47:43.8763422Z   BENCHMARK_RESULTS_DIR: test/test-reports
2024-12-18T01:47:43.8763752Z   DRY_RUN: false
2024-12-18T01:47:43.8765038Z   BENCHMARK_METADATA: {"timestamp": 1734486463, "schema_version": "v3", "name": "cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/release/2.6", "head_sha": "0cdf8b1d09254cfda66191d1bd01e3041c3c76f7", "workflow_id": 12383255690, "run_attempt": 1, "job_id": 34567153040}
2024-12-18T01:47:43.8766369Z   RUNNER_INFO: []
2024-12-18T01:47:43.8766628Z   DEPENDENCIES: {}
2024-12-18T01:47:43.8766879Z ##[endgroup]
2024-12-18T01:47:43.8793328Z + [[ ! -d test/test-reports ]]
2024-12-18T01:47:43.8793631Z + [[ false == \t\r\u\e ]]
2024-12-18T01:47:43.8796388Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/release/2.6/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1734486463, "schema_version": "v3", "name": "cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 2, 2, linux.g5.4xlarge.nvidia.gpu)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/release/2.6", "head_sha": "0cdf8b1d09254cfda66191d1bd01e3041c3c76f7", "workflow_id": 12383255690, "run_attempt": 1, "job_id": 34567153040}' --runners '[]' --dependencies '{}'
2024-12-18T01:47:44.0349983Z INFO:root:Upload test/test-reports/inference_timm_models.json to s3://ossci-benchmarks/v3/pytorch/pytorch/12383255690/34567153040/inference_timm_models.json
2024-12-18T01:47:44.0734426Z INFO:botocore.credentials:Found credentials from IAM Role: gh-ci-github-action-runners-runner-role
2024-12-18T01:47:44.2882594Z INFO:root:Upload test/test-reports/training_timm_models.json to s3://ossci-benchmarks/v3/pytorch/pytorch/12383255690/34567153040/training_timm_models.json
2024-12-18T01:47:44.4089088Z INFO:root:Upload test/test-reports/training_timm_model_graph_breaks.json to s3://ossci-benchmarks/v3/pytorch/pytorch/12383255690/34567153040/training_timm_model_graph_breaks.json
2024-12-18T01:47:44.6045161Z ##[group]Run cat test/**/*_toprint.log || true
2024-12-18T01:47:44.6045571Z cat test/**/*_toprint.log || true
2024-12-18T01:47:44.6053895Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:47:44.6054287Z env:
2024-12-18T01:47:44.6054527Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:47:44.6054879Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:47:44.6055433Z   DOCKER_CONTAINER_ID: 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T01:47:44.6055916Z ##[endgroup]
2024-12-18T01:47:44.6145318Z cat: 'test/**/*_toprint.log': No such file or directory
2024-12-18T01:47:44.6177628Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2024-12-18T01:47:44.6177990Z kill "$MONITOR_SCRIPT_PID"
2024-12-18T01:47:44.6185988Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:47:44.6186472Z env:
2024-12-18T01:47:44.6186702Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:47:44.6187041Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:47:44.6187586Z   DOCKER_CONTAINER_ID: 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T01:47:44.6188073Z   MONITOR_SCRIPT_PID: 53441
2024-12-18T01:47:44.6188361Z ##[endgroup]
2024-12-18T01:47:44.6337462Z Prepare all required actions
2024-12-18T01:47:44.6337868Z Getting action download info
2024-12-18T01:47:44.8461289Z Download action repository 'actions/upload-artifact@v4' (SHA:6f51ac03b9356f520e9adb1b1b7802705f340c2b)
2024-12-18T01:47:45.1439996Z ##[group]Run ./.github/actions/upload-test-artifacts
2024-12-18T01:47:45.1440372Z with:
2024-12-18T01:47:45.1440769Z   file-suffix: test-inductor_timm-2-2-linux.g5.4xlarge.nvidia.gpu_34567153040
2024-12-18T01:47:45.1441245Z   s3-bucket: gha-artifacts
2024-12-18T01:47:45.1441523Z env:
2024-12-18T01:47:45.1441757Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:47:45.1442117Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:47:45.1442660Z   DOCKER_CONTAINER_ID: 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T01:47:45.1443154Z ##[endgroup]
2024-12-18T01:47:45.1472777Z ##[group]Run # Remove any previous test jsons if they exist
2024-12-18T01:47:45.1473272Z # Remove any previous test jsons if they exist
2024-12-18T01:47:45.1473646Z rm -f test-jsons-*.zip
2024-12-18T01:47:45.1474079Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json'
2024-12-18T01:47:45.1483059Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:47:45.1483442Z env:
2024-12-18T01:47:45.1483669Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:47:45.1484010Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:47:45.1484544Z   DOCKER_CONTAINER_ID: 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T01:47:45.1485183Z   FILE_SUFFIX: test-inductor_timm-2-2-linux.g5.4xlarge.nvidia.gpu_34567153040
2024-12-18T01:47:45.1485643Z ##[endgroup]
2024-12-18T01:47:45.1650177Z   adding: test/test-reports/inference_timm_models.json (deflated 99%)
2024-12-18T01:47:45.1691748Z   adding: test/test-reports/training_timm_models.json (deflated 99%)
2024-12-18T01:47:45.1797161Z   adding: test/test-reports/training_timm_model_graph_breaks.json (deflated 99%)
2024-12-18T01:47:45.1828711Z ##[group]Run # Remove any previous test reports if they exist
2024-12-18T01:47:45.1829179Z # Remove any previous test reports if they exist
2024-12-18T01:47:45.1829563Z rm -f test-reports-*.zip
2024-12-18T01:47:45.1830032Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv'
2024-12-18T01:47:45.1838424Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:47:45.1838828Z env:
2024-12-18T01:47:45.1839075Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:47:45.1839412Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:47:45.1840138Z   DOCKER_CONTAINER_ID: 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T01:47:45.1840763Z   FILE_SUFFIX: test-inductor_timm-2-2-linux.g5.4xlarge.nvidia.gpu_34567153040
2024-12-18T01:47:45.1841222Z ##[endgroup]
2024-12-18T01:47:45.1891938Z   adding: test/test-reports/inference_timm_models.csv (deflated 69%)
2024-12-18T01:47:45.1892575Z   adding: test/test-reports/training_timm_models.csv (deflated 66%)
2024-12-18T01:47:45.1897190Z   adding: test/test-reports/training_timm_model_graph_breaks.csv (deflated 97%)
2024-12-18T01:47:45.1897939Z   adding: test/test-reports/training_timm_model_graph_break_deduped.csv (deflated 74%)
2024-12-18T01:47:45.1927254Z ##[group]Run # Remove any previous usage logs if they exist
2024-12-18T01:47:45.1927714Z # Remove any previous usage logs if they exist
2024-12-18T01:47:45.1928088Z rm -f logs-*.zip
2024-12-18T01:47:45.1928558Z # this workflow is also run in bazel build test, but we dont generate usage reports for it
2024-12-18T01:47:45.1929095Z # so check to see if the file exists first
2024-12-18T01:47:45.1929460Z if [ -f 'usage_log.txt' ]; then
2024-12-18T01:47:45.1929837Z     zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt'
2024-12-18T01:47:45.1930190Z fi
2024-12-18T01:47:45.1930767Z if find "test/test-reports" -name "*.log" 2>/dev/null | grep -q .; then
2024-12-18T01:47:45.1931326Z     zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log'
2024-12-18T01:47:45.1931722Z fi
2024-12-18T01:47:45.1939714Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:47:45.1940096Z env:
2024-12-18T01:47:45.1940323Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:47:45.1940671Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:47:45.1941218Z   DOCKER_CONTAINER_ID: 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T01:47:45.1941851Z   FILE_SUFFIX: test-inductor_timm-2-2-linux.g5.4xlarge.nvidia.gpu_34567153040
2024-12-18T01:47:45.1942322Z ##[endgroup]
2024-12-18T01:47:45.2217179Z   adding: usage_log.txt (deflated 98%)
2024-12-18T01:47:45.2268985Z ##[group]Run # Remove any previous debugging artifacts if they exist
2024-12-18T01:47:45.2269506Z # Remove any previous debugging artifacts if they exist
2024-12-18T01:47:45.2269925Z rm -f debug-*.zip
2024-12-18T01:47:45.2270222Z if [ -d 'test/debug' ]; then
2024-12-18T01:47:45.2270586Z   zip -r "debug-${FILE_SUFFIX}.zip" test/debug
2024-12-18T01:47:45.2270930Z fi
2024-12-18T01:47:45.2280170Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:47:45.2280556Z env:
2024-12-18T01:47:45.2280787Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:47:45.2281136Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:47:45.2281682Z   DOCKER_CONTAINER_ID: 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T01:47:45.2282326Z   FILE_SUFFIX: test-inductor_timm-2-2-linux.g5.4xlarge.nvidia.gpu_34567153040
2024-12-18T01:47:45.2282787Z ##[endgroup]
2024-12-18T01:47:45.2389732Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-12-18T01:47:45.2390080Z with:
2024-12-18T01:47:45.2390323Z   s3-bucket: gha-artifacts
2024-12-18T01:47:45.2390659Z   s3-prefix: pytorch/pytorch/12383255690/1/artifact

2024-12-18T01:47:45.2391029Z   retention-days: 14
2024-12-18T01:47:45.2391297Z   if-no-files-found: warn
2024-12-18T01:47:45.2391586Z   path: test-jsons-*.zip
2024-12-18T01:47:45.2391856Z   name: artifact
2024-12-18T01:47:45.2392105Z   region: us-east-1
2024-12-18T01:47:45.2392347Z env:
2024-12-18T01:47:45.2392566Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:47:45.2392910Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:47:45.2393455Z   DOCKER_CONTAINER_ID: 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T01:47:45.2393944Z ##[endgroup]
2024-12-18T01:47:45.5962833Z NOTE: s3-prefix specified, ignoring name parameter
2024-12-18T01:47:45.5964482Z With the provided path, there will be 1 file uploaded
2024-12-18T01:47:45.5964942Z Uploading to s3 prefix: pytorch/pytorch/12383255690/1/artifact
2024-12-18T01:47:45.6021785Z Starting upload of test-jsons-test-inductor_timm-2-2-linux.g5.4xlarge.nvidia.gpu_34567153040.zip
2024-12-18T01:47:45.8389156Z Finished upload of test-jsons-test-inductor_timm-2-2-linux.g5.4xlarge.nvidia.gpu_34567153040.zip
2024-12-18T01:47:45.8680852Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-12-18T01:47:45.8681207Z with:
2024-12-18T01:47:45.8681443Z   s3-bucket: gha-artifacts
2024-12-18T01:47:45.8681770Z   s3-prefix: pytorch/pytorch/12383255690/1/artifact

2024-12-18T01:47:45.8682131Z   retention-days: 14
2024-12-18T01:47:45.8682402Z   if-no-files-found: error
2024-12-18T01:47:45.8682699Z   path: test-reports-*.zip
2024-12-18T01:47:45.8682980Z   name: artifact
2024-12-18T01:47:45.8683223Z   region: us-east-1
2024-12-18T01:47:45.8683467Z env:
2024-12-18T01:47:45.8683696Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:47:45.8684060Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:47:45.8684614Z   DOCKER_CONTAINER_ID: 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T01:47:45.8685093Z ##[endgroup]
2024-12-18T01:47:46.1980388Z NOTE: s3-prefix specified, ignoring name parameter
2024-12-18T01:47:46.1981487Z With the provided path, there will be 1 file uploaded
2024-12-18T01:47:46.1981964Z Uploading to s3 prefix: pytorch/pytorch/12383255690/1/artifact
2024-12-18T01:47:46.2039006Z Starting upload of test-reports-test-inductor_timm-2-2-linux.g5.4xlarge.nvidia.gpu_34567153040.zip
2024-12-18T01:47:46.3152623Z Finished upload of test-reports-test-inductor_timm-2-2-linux.g5.4xlarge.nvidia.gpu_34567153040.zip
2024-12-18T01:47:46.3443490Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-12-18T01:47:46.3443846Z with:
2024-12-18T01:47:46.3444073Z   s3-bucket: gha-artifacts
2024-12-18T01:47:46.3444406Z   s3-prefix: pytorch/pytorch/12383255690/1/artifact

2024-12-18T01:47:46.3444760Z   retention-days: 14
2024-12-18T01:47:46.3445042Z   if-no-files-found: ignore
2024-12-18T01:47:46.3445321Z   path: logs-*.zip
2024-12-18T01:47:46.3445559Z   name: artifact
2024-12-18T01:47:46.3445852Z   region: us-east-1
2024-12-18T01:47:46.3446185Z env:
2024-12-18T01:47:46.3446499Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:47:46.3446972Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:47:46.3447577Z   DOCKER_CONTAINER_ID: 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T01:47:46.3448060Z ##[endgroup]
2024-12-18T01:47:46.6817721Z NOTE: s3-prefix specified, ignoring name parameter
2024-12-18T01:47:46.6818175Z With the provided path, there will be 1 file uploaded
2024-12-18T01:47:46.6818621Z Uploading to s3 prefix: pytorch/pytorch/12383255690/1/artifact
2024-12-18T01:47:46.6875871Z Starting upload of logs-test-inductor_timm-2-2-linux.g5.4xlarge.nvidia.gpu_34567153040.zip
2024-12-18T01:47:46.7975834Z Finished upload of logs-test-inductor_timm-2-2-linux.g5.4xlarge.nvidia.gpu_34567153040.zip
2024-12-18T01:47:46.8266799Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-12-18T01:47:46.8267148Z with:
2024-12-18T01:47:46.8267376Z   s3-bucket: gha-artifacts
2024-12-18T01:47:46.8267707Z   s3-prefix: pytorch/pytorch/12383255690/1/artifact

2024-12-18T01:47:46.8268065Z   retention-days: 14
2024-12-18T01:47:46.8268344Z   if-no-files-found: ignore
2024-12-18T01:47:46.8268648Z   path: debug-*.zip
2024-12-18T01:47:46.8268887Z   name: artifact
2024-12-18T01:47:46.8269132Z   region: us-east-1
2024-12-18T01:47:46.8269372Z env:
2024-12-18T01:47:46.8269600Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:47:46.8269942Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:47:46.8270475Z   DOCKER_CONTAINER_ID: 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T01:47:46.8270956Z ##[endgroup]
2024-12-18T01:47:47.1507382Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded.
2024-12-18T01:47:47.1801497Z ##[group]Run # shellcheck disable=SC2156
2024-12-18T01:47:47.1802078Z # shellcheck disable=SC2156
2024-12-18T01:47:47.1802659Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2024-12-18T01:47:47.1811764Z shell: /usr/bin/bash -e {0}
2024-12-18T01:47:47.1812048Z env:
2024-12-18T01:47:47.1812275Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:47:47.1812620Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:47:47.1813158Z   DOCKER_CONTAINER_ID: 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T01:47:47.1813641Z ##[endgroup]
2024-12-18T01:47:47.4501182Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@release/2.6
2024-12-18T01:47:47.4501650Z with:
2024-12-18T01:47:47.4501863Z env:
2024-12-18T01:47:47.4502087Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:47:47.4502430Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:47:47.4502969Z   DOCKER_CONTAINER_ID: 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T01:47:47.4503470Z ##[endgroup]
2024-12-18T01:47:47.4524428Z ##[group]Run set -eou pipefail
2024-12-18T01:47:47.4524757Z set -eou pipefail
2024-12-18T01:47:47.4525032Z 
2024-12-18T01:47:47.4525397Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2024-12-18T01:47:47.4525858Z for _ in $(seq 1440); do
2024-12-18T01:47:47.4526197Z     # Break if no ssh session exists anymore
2024-12-18T01:47:47.4526546Z     if [ "$(who)" = "" ]; then
2024-12-18T01:47:47.4526848Z       break
2024-12-18T01:47:47.4527115Z     fi
2024-12-18T01:47:47.4527354Z     echo "."
2024-12-18T01:47:47.4527605Z     sleep 5
2024-12-18T01:47:47.4527850Z done
2024-12-18T01:47:47.4536502Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:47:47.4536885Z env:
2024-12-18T01:47:47.4537115Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:47:47.4537458Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:47:47.4537995Z   DOCKER_CONTAINER_ID: 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T01:47:47.4538474Z ##[endgroup]
2024-12-18T01:47:47.4566088Z Holding runner for 2 hours until all ssh sessions have logged out
2024-12-18T01:47:47.4663138Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2024-12-18T01:47:47.4663721Z # ignore expansion of "docker ps -q" since it could be empty
2024-12-18T01:47:47.4664159Z # shellcheck disable=SC2046
2024-12-18T01:47:47.4664529Z docker stop $(docker ps -q) || true
2024-12-18T01:47:47.4664891Z # Prune all of the docker images
2024-12-18T01:47:47.4665240Z docker system prune -af
2024-12-18T01:47:47.4674015Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:47:47.4674400Z env:
2024-12-18T01:47:47.4674627Z   GIT_DEFAULT_BRANCH: main
2024-12-18T01:47:47.4674979Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T01:47:47.4675546Z   DOCKER_CONTAINER_ID: 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T01:47:47.4676033Z ##[endgroup]
2024-12-18T01:47:48.2112211Z 6581d0cd7381
2024-12-18T01:47:49.7007923Z Deleted Containers:
2024-12-18T01:47:49.7008350Z 6581d0cd738195abaf48187d8e12cd281595ed6c961106739eaf853802632684
2024-12-18T01:47:49.7008678Z 
2024-12-18T01:48:01.6976472Z Deleted Images:
2024-12-18T01:48:01.6977438Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T01:48:01.6979169Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks@sha256:ff80d4c93d18e8ab7a2501f0e62b7c40cf97aa83fc9702e88b9c2d02cdcbecdb
2024-12-18T01:48:01.6980341Z deleted: sha256:67e93a8badaf799ceb664d7c453077c6c6785d56dc10c720be9358edeb91759b
2024-12-18T01:48:01.6981426Z deleted: sha256:7958cbbed07d0e7e404de5150d4669da383984a17de74e455563b7ecc409b2e8
2024-12-18T01:48:01.6982063Z deleted: sha256:dc6a8b10e3a8452ea47655ddad3b52816f12bcbe35298b3aa96aae8a0da9bc12
2024-12-18T01:48:01.6982719Z deleted: sha256:c92bf6e7cfbdecd0dfe86d452eac5dbe5aa5b2a6aa7ef2eaeffb21af2870cda7
2024-12-18T01:48:01.6983370Z deleted: sha256:f9597cb89d67c0c899e266adc5e5f04817474bad200712b3c256ed33c346b3e8
2024-12-18T01:48:01.6984007Z deleted: sha256:eb062308fcca86ab5a341555bf03af093d21f1d44dbe3dce6f9b7781b650b2f1
2024-12-18T01:48:01.6984845Z deleted: sha256:96f28e0a6369023dd4e75125d2454531ff686a14467f3e06d3b5c6da89893472
2024-12-18T01:48:01.6985459Z deleted: sha256:3e6e64b255956d1bf66c97393ae4325f7fc67414cd710e284c17190cb262c92d
2024-12-18T01:48:01.6986192Z deleted: sha256:a3c0a58b4f23e46b45dd1b385363731800042b3f04ca66f8867b99f3c9311f11
2024-12-18T01:48:01.6986831Z deleted: sha256:435e519973ffb9c85330ae8db1d1fb1fa75d673e2f27640c2ca7277c6352092b
2024-12-18T01:48:01.6987446Z deleted: sha256:69800681a8b6382d520ac4b8b8828bd5540a297961ddeb004e415bc46b33f80d
2024-12-18T01:48:01.6988075Z deleted: sha256:1e837ad662894f46afc56a73aa5ccb04064e4a974c9e3cad86ce9e5546b1d63d
2024-12-18T01:48:01.6988698Z deleted: sha256:688eccf2945022bd75f9f2782e2397ea54f1032c2435aa8f7dffa23cbcf2ae99
2024-12-18T01:48:01.6989318Z deleted: sha256:4855651bf4c3c8acad23029fda7d17f43bf32a69e8d575bb43f4491f0d653c1f
2024-12-18T01:48:01.6989947Z deleted: sha256:ca2d39907b7e10e3c7ae753b8bd230fac77c4e4b15980befd71173da8a36da93
2024-12-18T01:48:01.6990673Z deleted: sha256:d4f04a8342f272455eb816a413dc2dfcc75b9ebb9ef7f6fab32436621412ba6b
2024-12-18T01:48:01.6991317Z deleted: sha256:63ae01c22dc840b763ddc08bf4e145c14a04e0ff8a833b357ed45cb0962a8116
2024-12-18T01:48:01.6992028Z deleted: sha256:cf91e4695bf05d3607b7fa68e3ba3d6cc8762e6fac8773483f8116812f3d52c0
2024-12-18T01:48:01.6992765Z deleted: sha256:71baa7b72633376bb45367512dcb676c81b1e09964085a3151381530df7cdc28
2024-12-18T01:48:01.6993361Z deleted: sha256:68855433b69809d035da2a30cb22864ed1cd2795434571a0c77532de19344497
2024-12-18T01:48:01.6993965Z deleted: sha256:929280209b1f4e29860f386a519df32161cb608338d5338f24735201939dc91f
2024-12-18T01:48:01.6994575Z deleted: sha256:25133b63282efce5897b0e8fbf64d8b2a2625e15792dad95bdada277b78c777d
2024-12-18T01:48:01.6995198Z deleted: sha256:11fddfe713ea9ddd5acd8285319a606609b5c428111f15a019c8286e7164b9b4
2024-12-18T01:48:01.6995816Z deleted: sha256:b1464f3888e3b0a1694aa960a726584d01ae6b62df53fe8743d197d2d5051036
2024-12-18T01:48:01.6996470Z deleted: sha256:9627c2925d7a3936b439a6f519083e650d160bc2522d2a9244d6a1dfda02bc83
2024-12-18T01:48:01.6997091Z deleted: sha256:33880827ded8236cc91531c2f928d6de5c7cfb95a241da2653b7e8a8d1ea4881
2024-12-18T01:48:01.6997700Z deleted: sha256:6851b39610a99eb7ec6939a15e35657e144eb3e628a3856afa0a0adff49b8c2f
2024-12-18T01:48:01.6998317Z deleted: sha256:1a76c7f0af1e05689638a3792e8b9d4418648384aed1f56d3ccde5b0a9681d29
2024-12-18T01:48:01.6998936Z deleted: sha256:072f501c97aae0be55d1457426bc5f8238b0bb86eac9e92889b01924352d79a5
2024-12-18T01:48:01.6999551Z deleted: sha256:8277868848759cc86b200280dee8701d7e8328a811bf75330ed24aeaf490e69e
2024-12-18T01:48:01.7000160Z deleted: sha256:1da994d575d08f659512ea580648bfddd62020acc1aef2f7d4c6008d36a6f83f
2024-12-18T01:48:01.7000792Z deleted: sha256:ddf6d49d71559688ae65aeecccbbc58dd555887d4b3306d02a87d049114dd2f9
2024-12-18T01:48:01.7001417Z deleted: sha256:648d99cd03665c7b88aae7e3291f27daa65f496f7ab2c37f0ee62c72a4459151
2024-12-18T01:48:01.7002036Z deleted: sha256:6e57cd88598cddad4070d39467d8e1d4c387388f89d2970a55391653f9243e1a
2024-12-18T01:48:01.7002668Z deleted: sha256:9b0c5db353ac5d4e0995c2a70ae3c28390676afe60dbd11e4ade07c792eac4ce
2024-12-18T01:48:01.7003295Z deleted: sha256:8fa5311fe41b6caf697456e9678220a152bbf6a7645f4b7ff89ea9317ae04444
2024-12-18T01:48:01.7003914Z deleted: sha256:dc438d6465050b81bff4a8c5925e70c2167712dbab174c38f79b6a5d7af407af
2024-12-18T01:48:01.7004527Z deleted: sha256:29b8f0a42a1f3b944f2c54348348d173d322522b578e3d485428d67996f6adde
2024-12-18T01:48:01.7005129Z deleted: sha256:26231024334c7a704f9acba86430547a013e959014bfd7a31e26c6ed78521479
2024-12-18T01:48:01.7005858Z deleted: sha256:4166e9dd7564ae2f46ea2c3dcb8a8ef6bef9e06f4faf1742f4c6f21861967193
2024-12-18T01:48:01.7006539Z deleted: sha256:80a3113f256f146f0a2528ca69aa9244a1f9ba1ecdf54c3ef8ba0fb21507cbc4
2024-12-18T01:48:01.7007169Z deleted: sha256:e90bbba5d7c08e01c5bedb0a855cc5cdd371688e23994155674704963591296f
2024-12-18T01:48:01.7007796Z deleted: sha256:d3b3df431f230849965cc370c2f1853a81d78dd6eb3f7ac3bc3b4ca64aa9818a
2024-12-18T01:48:01.7008411Z deleted: sha256:7a33f9ca143b08c9f5d8c53974b2d6fe840739b5d2cd30694ba1c86db9393908
2024-12-18T01:48:01.7009179Z deleted: sha256:bad1d7dae968956b4a81369dedfbcf539fd5b61dd3051e2eccb5457174fe788c
2024-12-18T01:48:01.7009822Z deleted: sha256:08a420245ca6eed4aa5662d0fec04e7892165cdb3fd2f5b99428adbb4d6199c0
2024-12-18T01:48:01.7010449Z deleted: sha256:c3fda6b9a4d08a345824508dc5625f52e4d23163843d0caf337c60feda68a830
2024-12-18T01:48:01.7011089Z deleted: sha256:c74a8a3ca5e31670eed41e4bbb4a89f25c7c05dacabdf93cbd03984ead0330e2
2024-12-18T01:48:01.7011721Z deleted: sha256:4171d4c54045bd3035546906a7dd3b63c9ca748e0a34eeba27fe918f52cdcf85
2024-12-18T01:48:01.7012332Z deleted: sha256:f6a1d112657a4aeaca96ac2792a10ea09463dc3d633cb8572314814d5abf1ddb
2024-12-18T01:48:01.7012959Z deleted: sha256:af5d54d84b9c302254498eed8b0bbce1cdd705351bf5cc210d3f11c0deb269b3
2024-12-18T01:48:01.7013611Z deleted: sha256:45ae227353c6db8230b86fc1c4b8c50def31bd64895b879d91ea6e600669045d
2024-12-18T01:48:01.7014246Z deleted: sha256:6032ea27ad90071e3b75290e03fd0763a74155f513f356d21e56663a878b0ed9
2024-12-18T01:48:01.7015286Z deleted: sha256:d5aef8ace96787c48070530c09b68e4bcb23bc92a9fb0978220204f64bfbd97f
2024-12-18T01:48:01.7015913Z deleted: sha256:e3116153021ece2a3d29eccdce1f7e0f092316e889c1d534257789f6b5743d51
2024-12-18T01:48:01.7016532Z deleted: sha256:fb0f11460e5fa2e4895230ae84c97fe0552e9509dfbae60abf94e1b8a6563fb9
2024-12-18T01:48:01.7017156Z deleted: sha256:e262415e0abb361b284df9e05790ae27cc3a1ea7b8d4618e7eb3eca17cf1d18e
2024-12-18T01:48:01.7017781Z deleted: sha256:7f810eab57a6121f1a2aae664f7a5940d12f789661445ce3500ecd3b3688b46b
2024-12-18T01:48:01.7018476Z deleted: sha256:6ed71f55ff78520154c43468df2f05a92eaa28ee2aa2e28b8ea860b9ffb258ef
2024-12-18T01:48:01.7019093Z deleted: sha256:92a10d8d42a56a8516d0e9e14fec53263302fe6e6438ced263a3845680456dd7
2024-12-18T01:48:01.7019715Z deleted: sha256:ec67f912ff53e9b7164f48250da3bb4c1ffc94c3eb2c1c56c534a94fe55e3845
2024-12-18T01:48:01.7020340Z deleted: sha256:c85033e4f60ace56eaad80a8151b1bdd1619865b10a9ccee5638878b0a027c23
2024-12-18T01:48:01.7020965Z deleted: sha256:0e8680489f0a37851beb2cb5e42f0df129247a68b755b08a6367ea42180b3e7a
2024-12-18T01:48:01.7021575Z deleted: sha256:3a767fb38f6b1124e5fe9a230fdd383ccb188c38fe1d6ec252260676efe255c8
2024-12-18T01:48:01.7022206Z deleted: sha256:2cce59d2a43773cb7c6b83d239dab3a53659767f96bc0d47a70ee0d51cfa096f
2024-12-18T01:48:01.7022831Z deleted: sha256:51e50e75789ab5b4e40a66bb2f284e2da947e37c0e757c74d49fdeea5da9cbec
2024-12-18T01:48:01.7023477Z deleted: sha256:29e55e4feccba0660950ed92941d309527fe5f41bea609662d0e703b9c76b3a1
2024-12-18T01:48:01.7024145Z deleted: sha256:d15bee8fc455fcb45d16b0abc67f18c99ce1cf2c9e998ef5a100bf8373f2f8ab
2024-12-18T01:48:01.7024766Z deleted: sha256:363936db6a8869d44b6f0bee97a786b168093848e9c08c8042830fc7fccd956b
2024-12-18T01:48:01.7025375Z deleted: sha256:0f54d1f89ceaf71ded488007635a618fac1528bf3603d9415a593fe4926f8b2a
2024-12-18T01:48:01.7025999Z deleted: sha256:2f5796e9879cab7dac02bdbb178aec5301ac00029b38be8613333ac639c4dc3a
2024-12-18T01:48:01.7026672Z deleted: sha256:e68525d57f9da8f6c74b456d92869c4e60e451b88cb403addbc4ea917d8ab3ac
2024-12-18T01:48:01.7027295Z deleted: sha256:3386e25983944ce3b2ebdde10d840832d581a7dfafd1df66568822a18979e268
2024-12-18T01:48:01.7027911Z deleted: sha256:d3a036e6bd36f014b3a1540d5b848655ecdf72a20dc81c0da5f80430700cfa9a
2024-12-18T01:48:01.7028536Z deleted: sha256:96d0556708ff4ec0d5d727d5947d6cb29a9d6c16e1a1ee76a92dc85aa3ea8f25
2024-12-18T01:48:01.7029161Z deleted: sha256:f3c2fb3abccb3a33dbf204d3f0d8857537348e6c1d3cc87a431afe866435d41b
2024-12-18T01:48:01.7029986Z deleted: sha256:45cd6c9a41f12991ee60a89e775abb604b0892ef3936526c18184bbbf4de57da
2024-12-18T01:48:01.7030609Z deleted: sha256:b3cacd0508bbfd2fd668f30c539033e7c6b606bf669258c8e8a3cbf80bf050c6
2024-12-18T01:48:01.7031236Z deleted: sha256:de72ce930361a96c151a8a4725f8360f33df6c812b6ed6d3dd5b6e6710a5d2c0
2024-12-18T01:48:01.7031864Z deleted: sha256:2e5d223d2bfa0f47e628b631a540d20c284dabb380ad9ee4caec65bf8d0ecde2
2024-12-18T01:48:01.7032490Z deleted: sha256:14a85e604a0401093323d356b3bb7f75a8473abd712b4bf1a2ef01fa432b344c
2024-12-18T01:48:01.7033234Z deleted: sha256:15a95a1e8f47d7c18dcbf9a6e2fc530cddcb161c0093c663181f3e73dc9f599e
2024-12-18T01:48:01.7033919Z deleted: sha256:106e8431b412f51ccd75ea46a2d5cb4343b23273cbcf50188377cb93aa9a6d82
2024-12-18T01:48:01.7034294Z 
2024-12-18T01:48:01.7034415Z Total reclaimed space: 47.85GB
2024-12-18T01:48:01.7101959Z Post job cleanup.
2024-12-18T01:48:01.7160912Z Post job cleanup.
2024-12-18T01:48:01.7977225Z [command]/usr/bin/git version
2024-12-18T01:48:01.8034011Z git version 2.40.1
2024-12-18T01:48:01.8074013Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/11bd37bf-2af0-47f3-8f59-e0c2a5adf0b4' before making global git config changes
2024-12-18T01:48:01.8075119Z Adding repository directory to the temporary git global config as a safe directory
2024-12-18T01:48:01.8078178Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-12-18T01:48:01.8110866Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2024-12-18T01:48:01.8140880Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2024-12-18T01:48:01.8506814Z Entering 'android/libs/fbjni'
2024-12-18T01:48:01.8577245Z Entering 'third_party/FP16'
2024-12-18T01:48:01.8646880Z Entering 'third_party/FXdiv'
2024-12-18T01:48:01.8715671Z Entering 'third_party/NNPACK'
2024-12-18T01:48:01.8785742Z Entering 'third_party/NVTX'
2024-12-18T01:48:01.8853613Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T01:48:01.8922858Z Entering 'third_party/XNNPACK'
2024-12-18T01:48:01.9004712Z Entering 'third_party/benchmark'
2024-12-18T01:48:01.9073762Z Entering 'third_party/composable_kernel'
2024-12-18T01:48:01.9148101Z Entering 'third_party/cpp-httplib'
2024-12-18T01:48:01.9217396Z Entering 'third_party/cpuinfo'
2024-12-18T01:48:01.9289874Z Entering 'third_party/cudnn_frontend'
2024-12-18T01:48:01.9360648Z Entering 'third_party/cutlass'
2024-12-18T01:48:01.9436419Z Entering 'third_party/eigen'
2024-12-18T01:48:01.9507106Z Entering 'third_party/fbgemm'
2024-12-18T01:48:01.9578002Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T01:48:01.9645477Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T01:48:01.9713024Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T01:48:01.9786196Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T01:48:01.9856641Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T01:48:01.9926354Z Entering 'third_party/flatbuffers'
2024-12-18T01:48:01.9999114Z Entering 'third_party/fmt'
2024-12-18T01:48:02.0068111Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T01:48:02.0137596Z Entering 'third_party/gloo'
2024-12-18T01:48:02.0206688Z Entering 'third_party/googletest'
2024-12-18T01:48:02.0280008Z Entering 'third_party/ideep'
2024-12-18T01:48:02.0348378Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T01:48:02.0423685Z Entering 'third_party/ittapi'
2024-12-18T01:48:02.0492644Z Entering 'third_party/kineto'
2024-12-18T01:48:02.0559666Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T01:48:02.0625228Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T01:48:02.0694662Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T01:48:02.0762626Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T01:48:02.0831542Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T01:48:02.0897406Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T01:48:02.0968692Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T01:48:02.1038055Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T01:48:02.1107011Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T01:48:02.1176907Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T01:48:02.1248116Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T01:48:02.1315513Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T01:48:02.1385345Z Entering 'third_party/mimalloc'
2024-12-18T01:48:02.1455719Z Entering 'third_party/nccl/nccl'
2024-12-18T01:48:02.1527699Z Entering 'third_party/nlohmann'
2024-12-18T01:48:02.1598439Z Entering 'third_party/onnx'
2024-12-18T01:48:02.1680973Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T01:48:02.1752719Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T01:48:02.1823207Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T01:48:02.1887871Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T01:48:02.1954593Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T01:48:02.2024492Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T01:48:02.2090735Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T01:48:02.2161770Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T01:48:02.2227836Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T01:48:02.2292150Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T01:48:02.2362656Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T01:48:02.2434176Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T01:48:02.2522336Z Entering 'third_party/pocketfft'
2024-12-18T01:48:02.2590985Z Entering 'third_party/protobuf'
2024-12-18T01:48:02.2662027Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T01:48:02.2730731Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T01:48:02.2801372Z Entering 'third_party/psimd'
2024-12-18T01:48:02.2869762Z Entering 'third_party/pthreadpool'
2024-12-18T01:48:02.2938851Z Entering 'third_party/pybind11'
2024-12-18T01:48:02.3007640Z Entering 'third_party/python-peachpy'
2024-12-18T01:48:02.3080731Z Entering 'third_party/sleef'
2024-12-18T01:48:02.3150334Z Entering 'third_party/tensorpipe'
2024-12-18T01:48:02.3219895Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T01:48:02.3290715Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T01:48:02.3358626Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T01:48:02.3426332Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T01:48:02.3490675Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T01:48:02.3585651Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2024-12-18T01:48:02.3606958Z http.https://github.com/.extraheader
2024-12-18T01:48:02.3616738Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2024-12-18T01:48:02.3647662Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2024-12-18T01:48:02.3998079Z Entering 'android/libs/fbjni'
2024-12-18T01:48:02.4041978Z http.https://github.com/.extraheader
2024-12-18T01:48:02.4085489Z Entering 'third_party/FP16'
2024-12-18T01:48:02.4134020Z http.https://github.com/.extraheader
2024-12-18T01:48:02.4177398Z Entering 'third_party/FXdiv'
2024-12-18T01:48:02.4226870Z http.https://github.com/.extraheader
2024-12-18T01:48:02.4270565Z Entering 'third_party/NNPACK'
2024-12-18T01:48:02.4314287Z http.https://github.com/.extraheader
2024-12-18T01:48:02.4358268Z Entering 'third_party/NVTX'
2024-12-18T01:48:02.4401241Z http.https://github.com/.extraheader
2024-12-18T01:48:02.4445649Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T01:48:02.4488363Z http.https://github.com/.extraheader
2024-12-18T01:48:02.4534244Z Entering 'third_party/XNNPACK'
2024-12-18T01:48:02.4577969Z http.https://github.com/.extraheader
2024-12-18T01:48:02.4637375Z Entering 'third_party/benchmark'
2024-12-18T01:48:02.4681648Z http.https://github.com/.extraheader
2024-12-18T01:48:02.4725509Z Entering 'third_party/composable_kernel'
2024-12-18T01:48:02.4770065Z http.https://github.com/.extraheader
2024-12-18T01:48:02.4820435Z Entering 'third_party/cpp-httplib'
2024-12-18T01:48:02.4864893Z http.https://github.com/.extraheader
2024-12-18T01:48:02.4908411Z Entering 'third_party/cpuinfo'
2024-12-18T01:48:02.4953991Z http.https://github.com/.extraheader
2024-12-18T01:48:02.4998821Z Entering 'third_party/cudnn_frontend'
2024-12-18T01:48:02.5042878Z http.https://github.com/.extraheader
2024-12-18T01:48:02.5085850Z Entering 'third_party/cutlass'
2024-12-18T01:48:02.5129975Z http.https://github.com/.extraheader
2024-12-18T01:48:02.5181766Z Entering 'third_party/eigen'
2024-12-18T01:48:02.5225910Z http.https://github.com/.extraheader
2024-12-18T01:48:02.5271327Z Entering 'third_party/fbgemm'
2024-12-18T01:48:02.5315022Z http.https://github.com/.extraheader
2024-12-18T01:48:02.5357961Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T01:48:02.5401635Z http.https://github.com/.extraheader
2024-12-18T01:48:02.5445872Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T01:48:02.5487835Z http.https://github.com/.extraheader
2024-12-18T01:48:02.5533560Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T01:48:02.5576834Z http.https://github.com/.extraheader
2024-12-18T01:48:02.5628282Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T01:48:02.5671631Z http.https://github.com/.extraheader
2024-12-18T01:48:02.5714603Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T01:48:02.5757670Z http.https://github.com/.extraheader
2024-12-18T01:48:02.5802574Z Entering 'third_party/flatbuffers'
2024-12-18T01:48:02.5848168Z http.https://github.com/.extraheader
2024-12-18T01:48:02.5893885Z Entering 'third_party/fmt'
2024-12-18T01:48:02.5942951Z http.https://github.com/.extraheader
2024-12-18T01:48:02.5987070Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T01:48:02.6031227Z http.https://github.com/.extraheader
2024-12-18T01:48:02.6073630Z Entering 'third_party/gloo'
2024-12-18T01:48:02.6118537Z http.https://github.com/.extraheader
2024-12-18T01:48:02.6161259Z Entering 'third_party/googletest'
2024-12-18T01:48:02.6206921Z http.https://github.com/.extraheader
2024-12-18T01:48:02.6252288Z Entering 'third_party/ideep'
2024-12-18T01:48:02.6300565Z http.https://github.com/.extraheader
2024-12-18T01:48:02.6341653Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T01:48:02.6388169Z http.https://github.com/.extraheader
2024-12-18T01:48:02.6439341Z Entering 'third_party/ittapi'
2024-12-18T01:48:02.6482872Z http.https://github.com/.extraheader
2024-12-18T01:48:02.6525619Z Entering 'third_party/kineto'
2024-12-18T01:48:02.6569580Z http.https://github.com/.extraheader
2024-12-18T01:48:02.6611293Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T01:48:02.6656341Z http.https://github.com/.extraheader
2024-12-18T01:48:02.6698924Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T01:48:02.6742729Z http.https://github.com/.extraheader
2024-12-18T01:48:02.6788214Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T01:48:02.6836170Z http.https://github.com/.extraheader
2024-12-18T01:48:02.6880751Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T01:48:02.6924847Z http.https://github.com/.extraheader
2024-12-18T01:48:02.6969503Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T01:48:02.7012170Z http.https://github.com/.extraheader
2024-12-18T01:48:02.7054996Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T01:48:02.7102635Z http.https://github.com/.extraheader
2024-12-18T01:48:02.7152462Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T01:48:02.7195493Z http.https://github.com/.extraheader
2024-12-18T01:48:02.7239434Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T01:48:02.7282183Z http.https://github.com/.extraheader
2024-12-18T01:48:02.7327545Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T01:48:02.7370822Z http.https://github.com/.extraheader
2024-12-18T01:48:02.7416131Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T01:48:02.7462862Z http.https://github.com/.extraheader
2024-12-18T01:48:02.7510024Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T01:48:02.7554240Z http.https://github.com/.extraheader
2024-12-18T01:48:02.7597393Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T01:48:02.7640684Z http.https://github.com/.extraheader
2024-12-18T01:48:02.7688622Z Entering 'third_party/mimalloc'
2024-12-18T01:48:02.7733362Z http.https://github.com/.extraheader
2024-12-18T01:48:02.7777020Z Entering 'third_party/nccl/nccl'
2024-12-18T01:48:02.7820992Z http.https://github.com/.extraheader
2024-12-18T01:48:02.7864372Z Entering 'third_party/nlohmann'
2024-12-18T01:48:02.7907491Z http.https://github.com/.extraheader
2024-12-18T01:48:02.7951846Z Entering 'third_party/onnx'
2024-12-18T01:48:02.7994915Z http.https://github.com/.extraheader
2024-12-18T01:48:02.8052049Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T01:48:02.8094714Z http.https://github.com/.extraheader
2024-12-18T01:48:02.8143291Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T01:48:02.8187087Z http.https://github.com/.extraheader
2024-12-18T01:48:02.8232283Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T01:48:02.8276110Z http.https://github.com/.extraheader
2024-12-18T01:48:02.8320045Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T01:48:02.8362282Z http.https://github.com/.extraheader
2024-12-18T01:48:02.8404938Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T01:48:02.8448577Z http.https://github.com/.extraheader
2024-12-18T01:48:02.8492078Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T01:48:02.8535876Z http.https://github.com/.extraheader
2024-12-18T01:48:02.8580277Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T01:48:02.8622709Z http.https://github.com/.extraheader
2024-12-18T01:48:02.8665883Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T01:48:02.8708224Z http.https://github.com/.extraheader
2024-12-18T01:48:02.8751299Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T01:48:02.8794129Z http.https://github.com/.extraheader
2024-12-18T01:48:02.8836333Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T01:48:02.8879073Z http.https://github.com/.extraheader
2024-12-18T01:48:02.8925720Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T01:48:02.8968713Z http.https://github.com/.extraheader
2024-12-18T01:48:02.9018559Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T01:48:02.9061225Z http.https://github.com/.extraheader
2024-12-18T01:48:02.9126110Z Entering 'third_party/pocketfft'
2024-12-18T01:48:02.9169848Z http.https://github.com/.extraheader
2024-12-18T01:48:02.9211673Z Entering 'third_party/protobuf'
2024-12-18T01:48:02.9256596Z http.https://github.com/.extraheader
2024-12-18T01:48:02.9301785Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T01:48:02.9346943Z http.https://github.com/.extraheader
2024-12-18T01:48:02.9390246Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T01:48:02.9434264Z http.https://github.com/.extraheader
2024-12-18T01:48:02.9480922Z Entering 'third_party/psimd'
2024-12-18T01:48:02.9526349Z http.https://github.com/.extraheader
2024-12-18T01:48:02.9571695Z Entering 'third_party/pthreadpool'
2024-12-18T01:48:02.9615052Z http.https://github.com/.extraheader
2024-12-18T01:48:02.9658011Z Entering 'third_party/pybind11'
2024-12-18T01:48:02.9701435Z http.https://github.com/.extraheader
2024-12-18T01:48:02.9745277Z Entering 'third_party/python-peachpy'
2024-12-18T01:48:02.9788573Z http.https://github.com/.extraheader
2024-12-18T01:48:02.9831118Z Entering 'third_party/sleef'
2024-12-18T01:48:02.9873817Z http.https://github.com/.extraheader
2024-12-18T01:48:02.9919197Z Entering 'third_party/tensorpipe'
2024-12-18T01:48:02.9962459Z http.https://github.com/.extraheader
2024-12-18T01:48:03.0005508Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T01:48:03.0048618Z http.https://github.com/.extraheader
2024-12-18T01:48:03.0091014Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T01:48:03.0134430Z http.https://github.com/.extraheader
2024-12-18T01:48:03.0176743Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T01:48:03.0220197Z http.https://github.com/.extraheader
2024-12-18T01:48:03.0264042Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T01:48:03.0306352Z http.https://github.com/.extraheader
2024-12-18T01:48:03.0347679Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T01:48:03.0391183Z http.https://github.com/.extraheader
2024-12-18T01:48:03.0549749Z A job completed hook has been configured by the self-hosted runner administrator
2024-12-18T01:48:03.0576832Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh'
2024-12-18T01:48:03.0584723Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T01:48:03.0585121Z ##[endgroup]
2024-12-18T01:48:09.9956115Z Cleaning up orphan processes