2024-12-18T00:18:00.0473876Z Current runner version: '2.321.0'
2024-12-18T00:18:00.0480805Z Runner name: 'i-02d991818cc142eee'
2024-12-18T00:18:00.0481828Z Runner group name: 'Default'
2024-12-18T00:18:00.0483092Z Machine name: 'ip-10-0-72-182'
2024-12-18T00:18:00.0488751Z ##[group]GITHUB_TOKEN Permissions
2024-12-18T00:18:00.0491463Z Actions: read
2024-12-18T00:18:00.0492267Z Attestations: read
2024-12-18T00:18:00.0492991Z Checks: read
2024-12-18T00:18:00.0493716Z Contents: read
2024-12-18T00:18:00.0494468Z Deployments: read
2024-12-18T00:18:00.0495230Z Discussions: read
2024-12-18T00:18:00.0495955Z Issues: read
2024-12-18T00:18:00.0496723Z Metadata: read
2024-12-18T00:18:00.0497479Z Packages: read
2024-12-18T00:18:00.0498189Z Pages: read
2024-12-18T00:18:00.0498989Z PullRequests: read
2024-12-18T00:18:00.0499745Z RepositoryProjects: read
2024-12-18T00:18:00.0500510Z SecurityEvents: read
2024-12-18T00:18:00.0501335Z Statuses: read
2024-12-18T00:18:00.0502040Z ##[endgroup]
2024-12-18T00:18:00.0505762Z Secret source: Actions
2024-12-18T00:18:00.0506925Z Prepare workflow directory
2024-12-18T00:18:00.1164306Z Prepare all required actions
2024-12-18T00:18:00.1206972Z Getting action download info
2024-12-18T00:18:00.3268688Z Download action repository 'pytorch/test-infra@release/2.6' (SHA:eb0adf5a84668865394af69e26428b32c8105c1c)
2024-12-18T00:18:01.9488046Z Download action repository 'pytorch/pytorch@release/2.6' (SHA:0cdf8b1d09254cfda66191d1bd01e3041c3c76f7)
2024-12-18T00:18:16.1093652Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0)
2024-12-18T00:18:16.3322248Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2024-12-18T00:18:16.6404550Z Getting action download info
2024-12-18T00:18:16.7549674Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c)
2024-12-18T00:18:16.9984353Z Getting action download info
2024-12-18T00:18:17.0985330Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e)
2024-12-18T00:18:17.3204457Z Getting action download info
2024-12-18T00:18:17.4480351Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482)
2024-12-18T00:18:17.6197125Z Getting action download info
2024-12-18T00:18:17.7507281Z Download action repository 'pytorch/test-infra@main' (SHA:a07505a74641a4ff5123d635defac481ef28ef1e)
2024-12-18T00:18:19.3232874Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/release/2.6 (0cdf8b1d09254cfda66191d1bd01e3041c3c76f7)
2024-12-18T00:18:19.3234785Z ##[group] Inputs
2024-12-18T00:18:19.3235156Z   build-environment: linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-12-18T00:18:19.3237110Z   test-matrix: {"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}
2024-12-18T00:18:19.3239609Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:18:19.3240535Z   sync-tag: 
2024-12-18T00:18:19.3241260Z   timeout-minutes: 240
2024-12-18T00:18:19.3241534Z   use-gha: 
2024-12-18T00:18:19.3241757Z   dashboard-tag: 
2024-12-18T00:18:19.3242008Z   s3-bucket: gha-artifacts
2024-12-18T00:18:19.3242285Z   aws-role-to-assume: 
2024-12-18T00:18:19.3242822Z   disable-monitor: false
2024-12-18T00:18:19.3243097Z ##[endgroup]
2024-12-18T00:18:19.3243862Z Complete job name: cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:18:19.3776299Z A job started hook has been configured by the self-hosted runner administrator
2024-12-18T00:18:19.3875811Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh'
2024-12-18T00:18:19.3886929Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:18:19.3887531Z ##[endgroup]
2024-12-18T00:18:21.4252010Z Runner Type: linux.g5.4xlarge.nvidia.gpu
2024-12-18T00:18:21.4252501Z Instance Type: g5.4xlarge
2024-12-18T00:18:21.4252765Z AMI Name: unknown
2024-12-18T00:18:21.4288893Z AMI ID: ami-0fff1b9a61dec8a5f
2024-12-18T00:18:27.2358489Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@release/2.6
2024-12-18T00:18:27.2358978Z with:
2024-12-18T00:18:27.2359693Z   github-secret: ***
2024-12-18T00:18:27.2360410Z   instructions: All testing is done inside the container, to start an interactive session run:
  docker exec -it $(docker container ps --format '{{.ID}}') bash

2024-12-18T00:18:27.2361178Z   activate-with-label: false
2024-12-18T00:18:27.2361474Z   label: with-ssh
2024-12-18T00:18:27.2361722Z   remove-existing-keys: true
2024-12-18T00:18:27.2362001Z   fail-silently: true
2024-12-18T00:18:27.2362236Z env:
2024-12-18T00:18:27.2362451Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:18:27.2362716Z ##[endgroup]
2024-12-18T00:18:27.3582083Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info.
2024-12-18T00:18:27.3583405Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys
2024-12-18T00:18:27.3770134Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@release/2.6
2024-12-18T00:18:27.3770592Z with:
2024-12-18T00:18:27.3770796Z   no-sudo: true
2024-12-18T00:18:27.3771047Z   submodules: recursive
2024-12-18T00:18:27.3771298Z   fetch-depth: 0
2024-12-18T00:18:27.3771515Z env:
2024-12-18T00:18:27.3771719Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:18:27.3771967Z ##[endgroup]
2024-12-18T00:18:27.3852407Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T00:18:27.3853366Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T00:18:27.3862623Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:18:27.3863004Z env:
2024-12-18T00:18:27.3863214Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:18:27.3863477Z ##[endgroup]
2024-12-18T00:18:27.3963212Z ##[group]Run retry () {
2024-12-18T00:18:27.3963515Z retry () {
2024-12-18T00:18:27.3963885Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2024-12-18T00:18:27.3964357Z }
2024-12-18T00:18:27.3964613Z echo "${GITHUB_WORKSPACE}"
2024-12-18T00:18:27.3964940Z if [ -z "${NO_SUDO}" ]; then
2024-12-18T00:18:27.3965299Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2024-12-18T00:18:27.3965634Z else
2024-12-18T00:18:27.3965919Z   retry rm -rf "${GITHUB_WORKSPACE}"
2024-12-18T00:18:27.3966252Z fi
2024-12-18T00:18:27.3966505Z mkdir "${GITHUB_WORKSPACE}"
2024-12-18T00:18:27.3975403Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:18:27.3975788Z env:
2024-12-18T00:18:27.3976004Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:18:27.3976279Z   NO_SUDO: true
2024-12-18T00:18:27.3976517Z ##[endgroup]
2024-12-18T00:18:27.4006743Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-12-18T00:18:33.1007797Z ##[group]Run malfet/checkout@silent-checkout
2024-12-18T00:18:33.1008141Z with:
2024-12-18T00:18:33.1008397Z   ref: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:18:33.1008741Z   fetch-depth: 0
2024-12-18T00:18:33.1008986Z   submodules: recursive
2024-12-18T00:18:33.1009243Z   quiet-checkout: true
2024-12-18T00:18:33.1009515Z   repository: pytorch/pytorch
2024-12-18T00:18:33.1009892Z   token: ***
2024-12-18T00:18:33.1010118Z   ssh-strict: true
2024-12-18T00:18:33.1010626Z   persist-credentials: true
2024-12-18T00:18:33.1010900Z   clean: true
2024-12-18T00:18:33.1011144Z   sparse-checkout-cone-mode: true
2024-12-18T00:18:33.1011441Z   lfs: false
2024-12-18T00:18:33.1011675Z   set-safe-directory: true
2024-12-18T00:18:33.1011939Z env:
2024-12-18T00:18:33.1012148Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:18:33.1012411Z ##[endgroup]
2024-12-18T00:18:33.1906034Z Syncing repository: pytorch/pytorch
2024-12-18T00:18:33.1907185Z ##[group]Getting Git version info
2024-12-18T00:18:33.1907664Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2024-12-18T00:18:33.1908315Z [command]/usr/bin/git version
2024-12-18T00:18:33.1908599Z git version 2.40.1
2024-12-18T00:18:33.1925093Z ##[endgroup]
2024-12-18T00:18:33.1938235Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/5956a7b1-29ee-4124-8556-b158cd22c1fb' before making global git config changes
2024-12-18T00:18:33.1939164Z Adding repository directory to the temporary git global config as a safe directory
2024-12-18T00:18:33.1942259Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-12-18T00:18:33.1976056Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2024-12-18T00:18:33.1979118Z ##[group]Initializing the repository
2024-12-18T00:18:33.1981670Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-12-18T00:18:33.2014270Z hint: Using 'master' as the name for the initial branch. This default branch name
2024-12-18T00:18:33.2015216Z hint: is subject to change. To configure the initial branch name to use in all
2024-12-18T00:18:33.2016058Z hint: of your new repositories, which will suppress this warning, call:
2024-12-18T00:18:33.2016725Z hint: 
2024-12-18T00:18:33.2017158Z hint: 	git config --global init.defaultBranch <name>
2024-12-18T00:18:33.2017665Z hint: 
2024-12-18T00:18:33.2018125Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2024-12-18T00:18:33.2018944Z hint: 'development'. The just-created branch can be renamed via this command:
2024-12-18T00:18:33.2019572Z hint: 
2024-12-18T00:18:33.2019905Z hint: 	git branch -m <name>
2024-12-18T00:18:33.2021446Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2024-12-18T00:18:33.2030470Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2024-12-18T00:18:33.2062616Z ##[endgroup]
2024-12-18T00:18:33.2063086Z ##[group]Disabling automatic garbage collection
2024-12-18T00:18:33.2065623Z [command]/usr/bin/git config --local gc.auto 0
2024-12-18T00:18:33.2093817Z ##[endgroup]
2024-12-18T00:18:33.2094222Z ##[group]Setting up auth
2024-12-18T00:18:33.2098881Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2024-12-18T00:18:33.2126773Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2024-12-18T00:18:33.2467030Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2024-12-18T00:18:33.2494923Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2024-12-18T00:18:33.2837819Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2024-12-18T00:18:33.2884534Z ##[endgroup]
2024-12-18T00:18:33.2890429Z ##[group]Fetching the repository
2024-12-18T00:18:33.2891233Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2024-12-18T00:18:35.8553907Z remote: Enumerating objects: 1056563        
2024-12-18T00:18:35.8554728Z remote: Enumerating objects: 1057277, done.        
2024-12-18T00:18:35.8555684Z remote: Counting objects:   0% (1/714)        
2024-12-18T00:18:35.8556821Z remote: Counting objects:   1% (8/714)        
2024-12-18T00:18:35.8557218Z remote: Counting objects:   2% (15/714)        
2024-12-18T00:18:35.8557587Z remote: Counting objects:   3% (22/714)        
2024-12-18T00:18:35.8557953Z remote: Counting objects:   4% (29/714)        
2024-12-18T00:18:35.8558316Z remote: Counting objects:   5% (36/714)        
2024-12-18T00:18:35.8558678Z remote: Counting objects:   6% (43/714)        
2024-12-18T00:18:35.8559053Z remote: Counting objects:   7% (50/714)        
2024-12-18T00:18:35.8559419Z remote: Counting objects:   8% (58/714)        
2024-12-18T00:18:35.8559780Z remote: Counting objects:   9% (65/714)        
2024-12-18T00:18:35.8560144Z remote: Counting objects:  10% (72/714)        
2024-12-18T00:18:35.8560507Z remote: Counting objects:  11% (79/714)        
2024-12-18T00:18:35.8560863Z remote: Counting objects:  12% (86/714)        
2024-12-18T00:18:35.8561224Z remote: Counting objects:  13% (93/714)        
2024-12-18T00:18:35.8561595Z remote: Counting objects:  14% (100/714)        
2024-12-18T00:18:35.8561968Z remote: Counting objects:  15% (108/714)        
2024-12-18T00:18:35.8562336Z remote: Counting objects:  16% (115/714)        
2024-12-18T00:18:35.8562693Z remote: Counting objects:  17% (122/714)        
2024-12-18T00:18:35.8563058Z remote: Counting objects:  18% (129/714)        
2024-12-18T00:18:35.8563418Z remote: Counting objects:  19% (136/714)        
2024-12-18T00:18:35.8563949Z remote: Counting objects:  20% (143/714)        
2024-12-18T00:18:35.8564314Z remote: Counting objects:  21% (150/714)        
2024-12-18T00:18:35.8564665Z remote: Counting objects:  22% (158/714)        
2024-12-18T00:18:35.8565026Z remote: Counting objects:  23% (165/714)        
2024-12-18T00:18:35.8565385Z remote: Counting objects:  24% (172/714)        
2024-12-18T00:18:35.8565742Z remote: Counting objects:  25% (179/714)        
2024-12-18T00:18:35.8566102Z remote: Counting objects:  26% (186/714)        
2024-12-18T00:18:35.8566558Z remote: Counting objects:  27% (193/714)        
2024-12-18T00:18:35.8566956Z remote: Counting objects:  28% (200/714)        
2024-12-18T00:18:35.8567309Z remote: Counting objects:  29% (208/714)        
2024-12-18T00:18:35.8567671Z remote: Counting objects:  30% (215/714)        
2024-12-18T00:18:35.8568031Z remote: Counting objects:  31% (222/714)        
2024-12-18T00:18:35.8568390Z remote: Counting objects:  32% (229/714)        
2024-12-18T00:18:35.8568756Z remote: Counting objects:  33% (236/714)        
2024-12-18T00:18:35.8569118Z remote: Counting objects:  34% (243/714)        
2024-12-18T00:18:35.8569471Z remote: Counting objects:  35% (250/714)        
2024-12-18T00:18:35.8569829Z remote: Counting objects:  36% (258/714)        
2024-12-18T00:18:35.8570187Z remote: Counting objects:  37% (265/714)        
2024-12-18T00:18:35.8570544Z remote: Counting objects:  38% (272/714)        
2024-12-18T00:18:35.8570902Z remote: Counting objects:  39% (279/714)        
2024-12-18T00:18:35.8571259Z remote: Counting objects:  40% (286/714)        
2024-12-18T00:18:35.8571616Z remote: Counting objects:  41% (293/714)        
2024-12-18T00:18:35.8571972Z remote: Counting objects:  42% (300/714)        
2024-12-18T00:18:35.8572332Z remote: Counting objects:  43% (308/714)        
2024-12-18T00:18:35.8572692Z remote: Counting objects:  44% (315/714)        
2024-12-18T00:18:35.8573045Z remote: Counting objects:  45% (322/714)        
2024-12-18T00:18:35.8573411Z remote: Counting objects:  46% (329/714)        
2024-12-18T00:18:35.8573773Z remote: Counting objects:  47% (336/714)        
2024-12-18T00:18:35.8574134Z remote: Counting objects:  48% (343/714)        
2024-12-18T00:18:35.8574499Z remote: Counting objects:  49% (350/714)        
2024-12-18T00:18:35.8574857Z remote: Counting objects:  50% (357/714)        
2024-12-18T00:18:35.8575223Z remote: Counting objects:  51% (365/714)        
2024-12-18T00:18:35.8575581Z remote: Counting objects:  52% (372/714)        
2024-12-18T00:18:35.8576047Z remote: Counting objects:  53% (379/714)        
2024-12-18T00:18:35.8576411Z remote: Counting objects:  54% (386/714)        
2024-12-18T00:18:35.8576773Z remote: Counting objects:  55% (393/714)        
2024-12-18T00:18:35.8577126Z remote: Counting objects:  56% (400/714)        
2024-12-18T00:18:35.8577487Z remote: Counting objects:  57% (407/714)        
2024-12-18T00:18:35.8577850Z remote: Counting objects:  58% (415/714)        
2024-12-18T00:18:35.8578268Z remote: Counting objects:  59% (422/714)        
2024-12-18T00:18:35.8578640Z remote: Counting objects:  60% (429/714)        
2024-12-18T00:18:35.8579001Z remote: Counting objects:  61% (436/714)        
2024-12-18T00:18:35.8579361Z remote: Counting objects:  62% (443/714)        
2024-12-18T00:18:35.8579744Z remote: Counting objects:  63% (450/714)        
2024-12-18T00:18:35.8580176Z remote: Counting objects:  64% (457/714)        
2024-12-18T00:18:35.8580535Z remote: Counting objects:  65% (465/714)        
2024-12-18T00:18:35.8580896Z remote: Counting objects:  66% (472/714)        
2024-12-18T00:18:35.8581261Z remote: Counting objects:  67% (479/714)        
2024-12-18T00:18:35.8581615Z remote: Counting objects:  68% (486/714)        
2024-12-18T00:18:35.8581974Z remote: Counting objects:  69% (493/714)        
2024-12-18T00:18:35.8582396Z remote: Counting objects:  70% (500/714)        
2024-12-18T00:18:35.8582756Z remote: Counting objects:  71% (507/714)        
2024-12-18T00:18:35.8583115Z remote: Counting objects:  72% (515/714)        
2024-12-18T00:18:35.8583570Z remote: Counting objects:  73% (522/714)        
2024-12-18T00:18:35.8583932Z remote: Counting objects:  74% (529/714)        
2024-12-18T00:18:35.8584292Z remote: Counting objects:  75% (536/714)        
2024-12-18T00:18:35.8584651Z remote: Counting objects:  76% (543/714)        
2024-12-18T00:18:35.8585010Z remote: Counting objects:  77% (550/714)        
2024-12-18T00:18:35.8585376Z remote: Counting objects:  78% (557/714)        
2024-12-18T00:18:35.8585736Z remote: Counting objects:  79% (565/714)        
2024-12-18T00:18:35.8586102Z remote: Counting objects:  80% (572/714)        
2024-12-18T00:18:35.8586462Z remote: Counting objects:  81% (579/714)        
2024-12-18T00:18:35.8586822Z remote: Counting objects:  82% (586/714)        
2024-12-18T00:18:35.8587180Z remote: Counting objects:  83% (593/714)        
2024-12-18T00:18:35.8587533Z remote: Counting objects:  84% (600/714)        
2024-12-18T00:18:35.8587894Z remote: Counting objects:  85% (607/714)        
2024-12-18T00:18:35.8588260Z remote: Counting objects:  86% (615/714)        
2024-12-18T00:18:35.8588618Z remote: Counting objects:  87% (622/714)        
2024-12-18T00:18:35.8588982Z remote: Counting objects:  88% (629/714)        
2024-12-18T00:18:35.8589333Z remote: Counting objects:  89% (636/714)        
2024-12-18T00:18:35.8589699Z remote: Counting objects:  90% (643/714)        
2024-12-18T00:18:35.8590056Z remote: Counting objects:  91% (650/714)        
2024-12-18T00:18:35.8590416Z remote: Counting objects:  92% (657/714)        
2024-12-18T00:18:35.8590781Z remote: Counting objects:  93% (665/714)        
2024-12-18T00:18:35.8591135Z remote: Counting objects:  94% (672/714)        
2024-12-18T00:18:35.8591499Z remote: Counting objects:  95% (679/714)        
2024-12-18T00:18:35.8591856Z remote: Counting objects:  96% (686/714)        
2024-12-18T00:18:35.8592216Z remote: Counting objects:  97% (693/714)        
2024-12-18T00:18:35.8592576Z remote: Counting objects:  98% (700/714)        
2024-12-18T00:18:35.8592935Z remote: Counting objects:  99% (707/714)        
2024-12-18T00:18:35.8593309Z remote: Counting objects: 100% (714/714)        
2024-12-18T00:18:35.8593693Z remote: Counting objects: 100% (714/714), done.        
2024-12-18T00:18:35.8645716Z remote: Compressing objects:   0% (1/311)        
2024-12-18T00:18:35.8720293Z remote: Compressing objects:   1% (4/311)        
2024-12-18T00:18:35.8759841Z remote: Compressing objects:   2% (7/311)        
2024-12-18T00:18:35.8831922Z remote: Compressing objects:   3% (10/311)        
2024-12-18T00:18:35.8965529Z remote: Compressing objects:   4% (13/311)        
2024-12-18T00:18:35.9360156Z remote: Compressing objects:   5% (16/311)        
2024-12-18T00:18:35.9778998Z remote: Compressing objects:   6% (19/311)        
2024-12-18T00:18:36.0501988Z remote: Compressing objects:   7% (22/311)        
2024-12-18T00:18:36.0916900Z remote: Compressing objects:   8% (25/311)        
2024-12-18T00:18:36.1317390Z remote: Compressing objects:   9% (28/311)        
2024-12-18T00:18:36.1458651Z remote: Compressing objects:  10% (32/311)        
2024-12-18T00:18:36.1568305Z remote: Compressing objects:  11% (35/311)        
2024-12-18T00:18:36.1568706Z remote: Compressing objects:  12% (38/311)        
2024-12-18T00:18:36.1569086Z remote: Compressing objects:  13% (41/311)        
2024-12-18T00:18:36.1569463Z remote: Compressing objects:  14% (44/311)        
2024-12-18T00:18:36.1569840Z remote: Compressing objects:  15% (47/311)        
2024-12-18T00:18:36.1570218Z remote: Compressing objects:  16% (50/311)        
2024-12-18T00:18:36.1570600Z remote: Compressing objects:  17% (53/311)        
2024-12-18T00:18:36.1570975Z remote: Compressing objects:  18% (56/311)        
2024-12-18T00:18:36.1571353Z remote: Compressing objects:  19% (60/311)        
2024-12-18T00:18:36.1571730Z remote: Compressing objects:  20% (63/311)        
2024-12-18T00:18:36.1572116Z remote: Compressing objects:  21% (66/311)        
2024-12-18T00:18:36.1577437Z remote: Compressing objects:  22% (69/311)        
2024-12-18T00:18:36.1585732Z remote: Compressing objects:  23% (72/311)        
2024-12-18T00:18:36.1594181Z remote: Compressing objects:  24% (75/311)        
2024-12-18T00:18:36.1599833Z remote: Compressing objects:  25% (78/311)        
2024-12-18T00:18:36.1600215Z remote: Compressing objects:  26% (81/311)        
2024-12-18T00:18:36.1603702Z remote: Compressing objects:  27% (84/311)        
2024-12-18T00:18:36.1606232Z remote: Compressing objects:  28% (88/311)        
2024-12-18T00:18:36.1651871Z remote: Compressing objects:  29% (91/311)        
2024-12-18T00:18:36.1652581Z remote: Compressing objects:  30% (94/311)        
2024-12-18T00:18:36.1653273Z remote: Compressing objects:  31% (97/311)        
2024-12-18T00:18:36.1653964Z remote: Compressing objects:  32% (100/311)        
2024-12-18T00:18:36.1654658Z remote: Compressing objects:  33% (103/311)        
2024-12-18T00:18:36.1655343Z remote: Compressing objects:  34% (106/311)        
2024-12-18T00:18:36.1656031Z remote: Compressing objects:  35% (109/311)        
2024-12-18T00:18:36.1656717Z remote: Compressing objects:  36% (112/311)        
2024-12-18T00:18:36.1657284Z remote: Compressing objects:  37% (116/311)        
2024-12-18T00:18:36.1659608Z remote: Compressing objects:  38% (119/311)        
2024-12-18T00:18:36.1665005Z remote: Compressing objects:  39% (122/311)        
2024-12-18T00:18:36.1665424Z remote: Compressing objects:  40% (125/311)        
2024-12-18T00:18:36.1668022Z remote: Compressing objects:  41% (128/311)        
2024-12-18T00:18:36.1671512Z remote: Compressing objects:  42% (131/311)        
2024-12-18T00:18:36.1672213Z remote: Compressing objects:  43% (134/311)        
2024-12-18T00:18:36.1673830Z remote: Compressing objects:  44% (137/311)        
2024-12-18T00:18:36.1679774Z remote: Compressing objects:  45% (140/311)        
2024-12-18T00:18:36.1682980Z remote: Compressing objects:  46% (144/311)        
2024-12-18T00:18:36.1683377Z remote: Compressing objects:  47% (147/311)        
2024-12-18T00:18:36.1687339Z remote: Compressing objects:  48% (150/311)        
2024-12-18T00:18:36.1689214Z remote: Compressing objects:  49% (153/311)        
2024-12-18T00:18:36.1691022Z remote: Compressing objects:  50% (156/311)        
2024-12-18T00:18:36.1692815Z remote: Compressing objects:  51% (159/311)        
2024-12-18T00:18:36.1695160Z remote: Compressing objects:  52% (162/311)        
2024-12-18T00:18:36.1697553Z remote: Compressing objects:  53% (165/311)        
2024-12-18T00:18:36.1699585Z remote: Compressing objects:  54% (168/311)        
2024-12-18T00:18:36.1700846Z remote: Compressing objects:  55% (172/311)        
2024-12-18T00:18:36.1702597Z remote: Compressing objects:  56% (175/311)        
2024-12-18T00:18:36.1704281Z remote: Compressing objects:  57% (178/311)        
2024-12-18T00:18:36.1705709Z remote: Compressing objects:  58% (181/311)        
2024-12-18T00:18:36.1706799Z remote: Compressing objects:  59% (184/311)        
2024-12-18T00:18:36.1708432Z remote: Compressing objects:  60% (187/311)        
2024-12-18T00:18:36.1710022Z remote: Compressing objects:  61% (190/311)        
2024-12-18T00:18:36.1710920Z remote: Compressing objects:  62% (193/311)        
2024-12-18T00:18:36.1714054Z remote: Compressing objects:  63% (196/311)        
2024-12-18T00:18:36.1714915Z remote: Compressing objects:  64% (200/311)        
2024-12-18T00:18:36.1722213Z remote: Compressing objects:  65% (203/311)        
2024-12-18T00:18:36.1722600Z remote: Compressing objects:  66% (206/311)        
2024-12-18T00:18:36.1726964Z remote: Compressing objects:  67% (209/311)        
2024-12-18T00:18:36.1727606Z remote: Compressing objects:  68% (212/311)        
2024-12-18T00:18:36.1730537Z remote: Compressing objects:  69% (215/311)        
2024-12-18T00:18:36.1731673Z remote: Compressing objects:  70% (218/311)        
2024-12-18T00:18:36.1734559Z remote: Compressing objects:  71% (221/311)        
2024-12-18T00:18:36.1738509Z remote: Compressing objects:  72% (224/311)        
2024-12-18T00:18:36.1740485Z remote: Compressing objects:  73% (228/311)        
2024-12-18T00:18:36.1743055Z remote: Compressing objects:  74% (231/311)        
2024-12-18T00:18:36.1743687Z remote: Compressing objects:  75% (234/311)        
2024-12-18T00:18:36.1744795Z remote: Compressing objects:  76% (237/311)        
2024-12-18T00:18:36.1745253Z remote: Compressing objects:  77% (240/311)        
2024-12-18T00:18:36.1745691Z remote: Compressing objects:  78% (243/311)        
2024-12-18T00:18:36.1746569Z remote: Compressing objects:  79% (246/311)        
2024-12-18T00:18:36.1746972Z remote: Compressing objects:  80% (249/311)        
2024-12-18T00:18:36.1749283Z remote: Compressing objects:  81% (252/311)        
2024-12-18T00:18:36.1751261Z remote: Compressing objects:  82% (256/311)        
2024-12-18T00:18:36.1753479Z remote: Compressing objects:  83% (259/311)        
2024-12-18T00:18:36.1755581Z remote: Compressing objects:  84% (262/311)        
2024-12-18T00:18:36.1757042Z remote: Compressing objects:  85% (265/311)        
2024-12-18T00:18:36.1758639Z remote: Compressing objects:  86% (268/311)        
2024-12-18T00:18:36.1760009Z remote: Compressing objects:  87% (271/311)        
2024-12-18T00:18:36.1761124Z remote: Compressing objects:  88% (274/311)        
2024-12-18T00:18:36.1762206Z remote: Compressing objects:  89% (277/311)        
2024-12-18T00:18:36.1763805Z remote: Compressing objects:  90% (280/311)        
2024-12-18T00:18:36.1764573Z remote: Compressing objects:  91% (284/311)        
2024-12-18T00:18:36.1765565Z remote: Compressing objects:  92% (287/311)        
2024-12-18T00:18:36.1766409Z remote: Compressing objects:  93% (290/311)        
2024-12-18T00:18:36.1767769Z remote: Compressing objects:  94% (293/311)        
2024-12-18T00:18:36.1768375Z remote: Compressing objects:  95% (296/311)        
2024-12-18T00:18:36.1768807Z remote: Compressing objects:  96% (299/311)        
2024-12-18T00:18:36.1769279Z remote: Compressing objects:  97% (302/311)        
2024-12-18T00:18:36.1770316Z remote: Compressing objects:  98% (305/311)        
2024-12-18T00:18:36.1770803Z remote: Compressing objects:  99% (308/311)        
2024-12-18T00:18:36.1771742Z remote: Compressing objects: 100% (311/311)        
2024-12-18T00:18:36.1772150Z remote: Compressing objects: 100% (311/311), done.        
2024-12-18T00:19:00.7469715Z remote: Total 1057277 (delta 573), reused 407 (delta 403), pack-reused 1056563 (from 3)        
2024-12-18T00:19:23.5036569Z [command]/usr/bin/git rev-parse --verify --quiet 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7^{object}
2024-12-18T00:19:23.5062217Z 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:19:23.5068036Z ##[endgroup]
2024-12-18T00:19:23.5068856Z ##[group]Determining the checkout info
2024-12-18T00:19:23.5069780Z ##[endgroup]
2024-12-18T00:19:23.5070453Z ##[group]Checking out the ref
2024-12-18T00:19:23.5071602Z [command]/usr/bin/git checkout --quiet --force 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:19:25.2947106Z ##[endgroup]
2024-12-18T00:19:25.2947574Z ##[group]Setting up auth for fetching submodules
2024-12-18T00:19:25.2953240Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2024-12-18T00:19:25.2998624Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2024-12-18T00:19:25.3026317Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2024-12-18T00:19:25.3055127Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2024-12-18T00:19:25.3084810Z ##[endgroup]
2024-12-18T00:19:25.3085218Z ##[group]Fetching submodules
2024-12-18T00:19:25.3087302Z [command]/usr/bin/git submodule sync --recursive
2024-12-18T00:19:25.3453305Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2024-12-18T00:19:25.3807204Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2024-12-18T00:19:25.3809568Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2024-12-18T00:19:25.3812510Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2024-12-18T00:19:25.3815765Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2024-12-18T00:19:25.3819071Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX'
2024-12-18T00:19:25.3823152Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2024-12-18T00:19:25.3826420Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2024-12-18T00:19:25.3830347Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2024-12-18T00:19:25.3834674Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel'
2024-12-18T00:19:25.3838231Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib'
2024-12-18T00:19:25.3841966Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2024-12-18T00:19:25.3846228Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2024-12-18T00:19:25.3850005Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2024-12-18T00:19:25.3854030Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2024-12-18T00:19:25.3858106Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2024-12-18T00:19:25.3863547Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2024-12-18T00:19:25.3871305Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2024-12-18T00:19:25.3875782Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2024-12-18T00:19:25.3880044Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2024-12-18T00:19:25.3884655Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2024-12-18T00:19:25.3889062Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2024-12-18T00:19:25.3893679Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2024-12-18T00:19:25.3898353Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2024-12-18T00:19:25.3903205Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc'
2024-12-18T00:19:25.3907975Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2024-12-18T00:19:25.3912928Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2024-12-18T00:19:25.3917876Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2024-12-18T00:19:25.3923100Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp'
2024-12-18T00:19:25.3928030Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2024-12-18T00:19:25.3934090Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2024-12-18T00:19:25.3939007Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2024-12-18T00:19:25.3944940Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2024-12-18T00:19:25.3950631Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2024-12-18T00:19:25.3957005Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2024-12-18T00:19:25.3963592Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2024-12-18T00:19:25.3969101Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2024-12-18T00:19:25.4004008Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2024-12-18T00:19:25.6940498Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2024-12-18T00:19:25.9047333Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2024-12-18T00:19:26.0916038Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2024-12-18T00:19:26.3933723Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'...
2024-12-18T00:19:26.7211166Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2024-12-18T00:19:28.6302730Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2024-12-18T00:19:40.4710475Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2024-12-18T00:19:40.9044523Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'...
2024-12-18T00:19:43.2255143Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'...
2024-12-18T00:19:43.7476997Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2024-12-18T00:19:44.3926528Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2024-12-18T00:19:45.6587223Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2024-12-18T00:19:47.8156246Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2024-12-18T00:19:53.5932968Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2024-12-18T00:19:55.3319310Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2024-12-18T00:19:56.6591625Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2024-12-18T00:19:58.2885213Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2024-12-18T00:19:58.7346959Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2024-12-18T00:19:59.1154951Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2024-12-18T00:20:00.1146865Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2024-12-18T00:20:00.4910845Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2024-12-18T00:20:00.7420818Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2024-12-18T00:20:02.3228239Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'...
2024-12-18T00:20:03.2015800Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2024-12-18T00:20:03.5833064Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2024-12-18T00:20:10.8548585Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2024-12-18T00:20:12.9278328Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'...
2024-12-18T00:20:17.8060992Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2024-12-18T00:20:18.0419184Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2024-12-18T00:20:27.3037047Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2024-12-18T00:20:27.5137297Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2024-12-18T00:20:27.7444263Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2024-12-18T00:20:28.8696584Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2024-12-18T00:20:29.2368974Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2024-12-18T00:20:29.8863666Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2024-12-18T00:20:30.3167334Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2024-12-18T00:20:30.3315110Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2024-12-18T00:20:30.3435226Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2024-12-18T00:20:30.3752000Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2024-12-18T00:20:30.4196138Z Submodule path 'third_party/NVTX': checked out 'e170594ac7cf1dac584da473d4ca9301087090c1'
2024-12-18T00:20:30.4638280Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2024-12-18T00:20:31.3553345Z Submodule path 'third_party/XNNPACK': checked out '4ea82e595b36106653175dcb04b2aa532660d0d8'
2024-12-18T00:20:31.3835730Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2024-12-18T00:20:31.6859089Z Submodule path 'third_party/composable_kernel': checked out '50ee4267e27b875d149e642f4cebd47be1dc3b57'
2024-12-18T00:20:31.7422234Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006'
2024-12-18T00:20:31.8571490Z Submodule path 'third_party/cpuinfo': checked out '1e83a2fdd3102f65c6f1fb602c1b320486218a99'
2024-12-18T00:20:31.8976259Z Submodule path 'third_party/cudnn_frontend': checked out '936021bfed8c91dc416af1588b2c4eca631a9e45'
2024-12-18T00:20:32.5256976Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49'
2024-12-18T00:20:32.8147879Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2024-12-18T00:20:32.9208044Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e'
2024-12-18T00:20:32.9232864Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2024-12-18T00:20:32.9236281Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T00:20:32.9239629Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass'
2024-12-18T00:20:32.9243228Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2024-12-18T00:20:32.9246848Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T00:20:32.9278924Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2024-12-18T00:20:33.7749466Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2024-12-18T00:20:34.4213006Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'...
2024-12-18T00:20:36.6010997Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2024-12-18T00:20:37.6075733Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2024-12-18T00:20:37.9728547Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2024-12-18T00:20:38.0826954Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2024-12-18T00:20:38.5907208Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6'
2024-12-18T00:20:38.6619291Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2024-12-18T00:20:38.6770701Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1'
2024-12-18T00:20:38.8419324Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b'
2024-12-18T00:20:38.8869745Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592'
2024-12-18T00:20:38.9334406Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2024-12-18T00:20:38.9670863Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509'
2024-12-18T00:20:39.0185487Z Submodule path 'third_party/googletest': checked out 'b514bdc898e2951020cbdca1304b75f5950d1f59'
2024-12-18T00:20:39.0351043Z Submodule path 'third_party/ideep': checked out 'c7ccd5bdbe5434ba156f4e856dcef0601637334b'
2024-12-18T00:20:39.0371804Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2024-12-18T00:20:39.0402612Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2024-12-18T00:20:53.2266185Z Submodule path 'third_party/ideep/mkl-dnn': checked out '66f0cb9eb66affd2da3bf5f8d897376f04aae6af'
2024-12-18T00:20:53.2487983Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2024-12-18T00:20:53.3479744Z Submodule path 'third_party/kineto': checked out '338140f58a28d599da3434ced4fd2d75dd1a213d'
2024-12-18T00:20:53.3503506Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T00:20:53.3506847Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T00:20:53.3510260Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T00:20:53.3541837Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'...
2024-12-18T00:20:54.2384653Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2024-12-18T00:20:55.4673112Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2024-12-18T00:20:56.5786633Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e'
2024-12-18T00:20:56.5807296Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T00:20:56.5811076Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T00:20:56.5814441Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T00:20:56.5818239Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T00:20:56.5822039Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T00:20:56.5825921Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T00:20:56.5829728Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T00:20:56.5835105Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T00:20:56.5867504Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'...
2024-12-18T00:20:57.5401877Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'...
2024-12-18T00:20:57.9277268Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'...
2024-12-18T00:20:59.1367158Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'...
2024-12-18T00:20:59.4297078Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'...
2024-12-18T00:20:59.9092337Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'...
2024-12-18T00:21:00.9605942Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'...
2024-12-18T00:21:08.1343766Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'...
2024-12-18T00:21:08.5752575Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9'
2024-12-18T00:21:08.5978045Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400'
2024-12-18T00:21:08.6405333Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2024-12-18T00:21:08.6576398Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067'
2024-12-18T00:21:08.6595496Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T00:21:08.6627001Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'...
2024-12-18T00:21:08.9753021Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4'
2024-12-18T00:21:08.9977598Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446'
2024-12-18T00:21:09.0451191Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850'
2024-12-18T00:21:09.1721459Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5'
2024-12-18T00:21:09.1918984Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150'
2024-12-18T00:21:09.2369266Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164'
2024-12-18T00:21:09.3038326Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2024-12-18T00:21:09.3506894Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78'
2024-12-18T00:21:09.3861156Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290'
2024-12-18T00:21:09.5197198Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2024-12-18T00:21:10.0314211Z Submodule path 'third_party/onnx': checked out 'b8baa8446686496da4cc8fda09f2b6fe65c2a02c'
2024-12-18T00:21:10.0354785Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2024-12-18T00:21:10.0386997Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2024-12-18T00:21:11.1975059Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b'
2024-12-18T00:21:11.2895384Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878'
2024-12-18T00:21:11.2918999Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T00:21:11.2922437Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T00:21:11.2926034Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T00:21:11.2930193Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T00:21:11.2934710Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T00:21:11.2938439Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T00:21:11.2942489Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T00:21:11.2946533Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T00:21:11.2982202Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'...
2024-12-18T00:21:11.7745002Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'...
2024-12-18T00:21:12.8266163Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'...
2024-12-18T00:21:13.1587368Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'...
2024-12-18T00:21:20.2038425Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'...
2024-12-18T00:21:20.4891833Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'...
2024-12-18T00:21:20.7225613Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'...
2024-12-18T00:21:21.0885480Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'...
2024-12-18T00:21:28.4989085Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2'
2024-12-18T00:21:28.5468071Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1'
2024-12-18T00:21:28.5660307Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa'
2024-12-18T00:21:28.6892229Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d'
2024-12-18T00:21:28.7053323Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce'
2024-12-18T00:21:28.7232173Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5'
2024-12-18T00:21:28.7431365Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d'
2024-12-18T00:21:28.7450722Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T00:21:28.7454227Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T00:21:28.7485531Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'...
2024-12-18T00:21:30.6741660Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'...
2024-12-18T00:21:31.9759945Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4'
2024-12-18T00:21:32.0300934Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2024-12-18T00:21:32.6669302Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50'
2024-12-18T00:21:32.6815730Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe'
2024-12-18T00:21:33.0061717Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2024-12-18T00:21:33.0089912Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2024-12-18T00:21:33.0093303Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2024-12-18T00:21:33.0126983Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2024-12-18T00:21:33.4739937Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2024-12-18T00:21:34.4912134Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2024-12-18T00:21:34.5736742Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2024-12-18T00:21:34.5854408Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2024-12-18T00:21:34.6013080Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8'
2024-12-18T00:21:34.6439225Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4'
2024-12-18T00:21:34.6773882Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2024-12-18T00:21:34.7266284Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b'
2024-12-18T00:21:34.7600377Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2024-12-18T00:21:34.7621285Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2024-12-18T00:21:34.7624579Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2024-12-18T00:21:34.7627911Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2024-12-18T00:21:34.7632176Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T00:21:34.7662924Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2024-12-18T00:21:35.8469339Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2024-12-18T00:21:36.1049630Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2024-12-18T00:21:38.3886407Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2024-12-18T00:21:39.5698915Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2024-12-18T00:21:39.5897583Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2024-12-18T00:21:39.6621112Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2024-12-18T00:21:39.6968290Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2024-12-18T00:21:39.6987216Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T00:21:39.7018613Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2024-12-18T00:21:39.9490489Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2024-12-18T00:21:39.9538127Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2024-12-18T00:21:39.9901262Z Entering 'android/libs/fbjni'
2024-12-18T00:21:39.9958125Z Entering 'third_party/FP16'
2024-12-18T00:21:40.0009486Z Entering 'third_party/FXdiv'
2024-12-18T00:21:40.0062104Z Entering 'third_party/NNPACK'
2024-12-18T00:21:40.0112839Z Entering 'third_party/NVTX'
2024-12-18T00:21:40.0165294Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T00:21:40.0216735Z Entering 'third_party/XNNPACK'
2024-12-18T00:21:40.0285833Z Entering 'third_party/benchmark'
2024-12-18T00:21:40.0338782Z Entering 'third_party/composable_kernel'
2024-12-18T00:21:40.0397098Z Entering 'third_party/cpp-httplib'
2024-12-18T00:21:40.0447315Z Entering 'third_party/cpuinfo'
2024-12-18T00:21:40.0499138Z Entering 'third_party/cudnn_frontend'
2024-12-18T00:21:40.0551390Z Entering 'third_party/cutlass'
2024-12-18T00:21:40.0610472Z Entering 'third_party/eigen'
2024-12-18T00:21:40.0665946Z Entering 'third_party/fbgemm'
2024-12-18T00:21:40.0715716Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T00:21:40.0773022Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T00:21:40.0824375Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T00:21:40.0883082Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T00:21:40.0936090Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T00:21:40.0989383Z Entering 'third_party/flatbuffers'
2024-12-18T00:21:40.1044589Z Entering 'third_party/fmt'
2024-12-18T00:21:40.1096195Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T00:21:40.1149529Z Entering 'third_party/gloo'
2024-12-18T00:21:40.1201422Z Entering 'third_party/googletest'
2024-12-18T00:21:40.1255038Z Entering 'third_party/ideep'
2024-12-18T00:21:40.1304397Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T00:21:40.1362798Z Entering 'third_party/ittapi'
2024-12-18T00:21:40.1414243Z Entering 'third_party/kineto'
2024-12-18T00:21:40.1465613Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T00:21:40.1515829Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T00:21:40.1569998Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T00:21:40.1621340Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T00:21:40.1672386Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T00:21:40.1722634Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T00:21:40.1778266Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T00:21:40.1830870Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T00:21:40.1882231Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T00:21:40.1935664Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T00:21:40.1988470Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T00:21:40.2046253Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T00:21:40.2099504Z Entering 'third_party/mimalloc'
2024-12-18T00:21:40.2152800Z Entering 'third_party/nccl/nccl'
2024-12-18T00:21:40.2203722Z Entering 'third_party/nlohmann'
2024-12-18T00:21:40.2255836Z Entering 'third_party/onnx'
2024-12-18T00:21:40.2320395Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T00:21:40.2386029Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T00:21:40.2438651Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T00:21:40.2489211Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T00:21:40.2545938Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T00:21:40.2601141Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T00:21:40.2659480Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T00:21:40.2713460Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T00:21:40.2764451Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T00:21:40.2813159Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T00:21:40.2867581Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T00:21:40.2919312Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T00:21:40.2997371Z Entering 'third_party/pocketfft'
2024-12-18T00:21:40.3048605Z Entering 'third_party/protobuf'
2024-12-18T00:21:40.3102702Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T00:21:40.3153832Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T00:21:40.3208329Z Entering 'third_party/psimd'
2024-12-18T00:21:40.3260485Z Entering 'third_party/pthreadpool'
2024-12-18T00:21:40.3311858Z Entering 'third_party/pybind11'
2024-12-18T00:21:40.3364171Z Entering 'third_party/python-peachpy'
2024-12-18T00:21:40.3414592Z Entering 'third_party/sleef'
2024-12-18T00:21:40.3466495Z Entering 'third_party/tensorpipe'
2024-12-18T00:21:40.3516538Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T00:21:40.3566809Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T00:21:40.3616668Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T00:21:40.3672178Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T00:21:40.3720456Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T00:21:40.3794800Z ##[endgroup]
2024-12-18T00:21:40.3796379Z ##[group]Persisting credentials for submodules
2024-12-18T00:21:40.3800332Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :"
2024-12-18T00:21:40.4154787Z Entering 'android/libs/fbjni'
2024-12-18T00:21:40.4223110Z Entering 'third_party/FP16'
2024-12-18T00:21:40.4291327Z Entering 'third_party/FXdiv'
2024-12-18T00:21:40.4359195Z Entering 'third_party/NNPACK'
2024-12-18T00:21:40.4433641Z Entering 'third_party/NVTX'
2024-12-18T00:21:40.4503257Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T00:21:40.4573701Z Entering 'third_party/XNNPACK'
2024-12-18T00:21:40.4655612Z Entering 'third_party/benchmark'
2024-12-18T00:21:40.4724682Z Entering 'third_party/composable_kernel'
2024-12-18T00:21:40.4800591Z Entering 'third_party/cpp-httplib'
2024-12-18T00:21:40.4871782Z Entering 'third_party/cpuinfo'
2024-12-18T00:21:40.4939691Z Entering 'third_party/cudnn_frontend'
2024-12-18T00:21:40.5011511Z Entering 'third_party/cutlass'
2024-12-18T00:21:40.5085650Z Entering 'third_party/eigen'
2024-12-18T00:21:40.5154832Z Entering 'third_party/fbgemm'
2024-12-18T00:21:40.5220095Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T00:21:40.5287204Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T00:21:40.5353080Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T00:21:40.5424849Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T00:21:40.5490846Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T00:21:40.5558081Z Entering 'third_party/flatbuffers'
2024-12-18T00:21:40.5633174Z Entering 'third_party/fmt'
2024-12-18T00:21:40.5700722Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T00:21:40.5768691Z Entering 'third_party/gloo'
2024-12-18T00:21:40.5840915Z Entering 'third_party/googletest'
2024-12-18T00:21:40.5907489Z Entering 'third_party/ideep'
2024-12-18T00:21:40.5973044Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T00:21:40.6047289Z Entering 'third_party/ittapi'
2024-12-18T00:21:40.6115060Z Entering 'third_party/kineto'
2024-12-18T00:21:40.6182279Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T00:21:40.6247581Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T00:21:40.6315890Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T00:21:40.6384779Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T00:21:40.6452107Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T00:21:40.6522331Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T00:21:40.6594006Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T00:21:40.6666232Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T00:21:40.6735210Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T00:21:40.6802981Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T00:21:40.6875821Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T00:21:40.6942685Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T00:21:40.7011913Z Entering 'third_party/mimalloc'
2024-12-18T00:21:40.7081980Z Entering 'third_party/nccl/nccl'
2024-12-18T00:21:40.7149807Z Entering 'third_party/nlohmann'
2024-12-18T00:21:40.7218073Z Entering 'third_party/onnx'
2024-12-18T00:21:40.7303364Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T00:21:40.7375219Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T00:21:40.7445503Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T00:21:40.7510512Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T00:21:40.7575953Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T00:21:40.7642609Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T00:21:40.7709826Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T00:21:40.7775436Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T00:21:40.7840926Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T00:21:40.7904355Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T00:21:40.7973675Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T00:21:40.8043528Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T00:21:40.8130575Z Entering 'third_party/pocketfft'
2024-12-18T00:21:40.8203362Z Entering 'third_party/protobuf'
2024-12-18T00:21:40.8273338Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T00:21:40.8339384Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T00:21:40.8414565Z Entering 'third_party/psimd'
2024-12-18T00:21:40.8483551Z Entering 'third_party/pthreadpool'
2024-12-18T00:21:40.8551076Z Entering 'third_party/pybind11'
2024-12-18T00:21:40.8622277Z Entering 'third_party/python-peachpy'
2024-12-18T00:21:40.8689718Z Entering 'third_party/sleef'
2024-12-18T00:21:40.8764653Z Entering 'third_party/tensorpipe'
2024-12-18T00:21:40.8831452Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T00:21:40.8898075Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T00:21:40.8965327Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T00:21:40.9032016Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T00:21:40.9096387Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T00:21:40.9189145Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url"
2024-12-18T00:21:40.9542462Z Entering 'android/libs/fbjni'
2024-12-18T00:21:40.9604126Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2024-12-18T00:21:40.9626088Z Entering 'third_party/FP16'
2024-12-18T00:21:40.9688647Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2024-12-18T00:21:40.9710698Z Entering 'third_party/FXdiv'
2024-12-18T00:21:40.9773135Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2024-12-18T00:21:40.9794803Z Entering 'third_party/NNPACK'
2024-12-18T00:21:40.9857544Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2024-12-18T00:21:40.9880695Z Entering 'third_party/NVTX'
2024-12-18T00:21:40.9942287Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config	remote.origin.url
2024-12-18T00:21:40.9964725Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T00:21:41.0026403Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2024-12-18T00:21:41.0049369Z Entering 'third_party/XNNPACK'
2024-12-18T00:21:41.0111167Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2024-12-18T00:21:41.0148118Z Entering 'third_party/benchmark'
2024-12-18T00:21:41.0211330Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2024-12-18T00:21:41.0233652Z Entering 'third_party/composable_kernel'
2024-12-18T00:21:41.0295956Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config	remote.origin.url
2024-12-18T00:21:41.0324135Z Entering 'third_party/cpp-httplib'
2024-12-18T00:21:41.0387288Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config	remote.origin.url
2024-12-18T00:21:41.0408491Z Entering 'third_party/cpuinfo'
2024-12-18T00:21:41.0469609Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2024-12-18T00:21:41.0491436Z Entering 'third_party/cudnn_frontend'
2024-12-18T00:21:41.0552804Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2024-12-18T00:21:41.0574493Z Entering 'third_party/cutlass'
2024-12-18T00:21:41.0636014Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2024-12-18T00:21:41.0664926Z Entering 'third_party/eigen'
2024-12-18T00:21:41.0725049Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2024-12-18T00:21:41.0749102Z Entering 'third_party/fbgemm'
2024-12-18T00:21:41.0812422Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2024-12-18T00:21:41.0836182Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T00:21:41.0897713Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2024-12-18T00:21:41.0919006Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T00:21:41.0984599Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2024-12-18T00:21:41.1006307Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T00:21:41.1067384Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config	remote.origin.url
2024-12-18T00:21:41.1095289Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T00:21:41.1157342Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:41.1183629Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T00:21:41.1251821Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2024-12-18T00:21:41.1275740Z Entering 'third_party/flatbuffers'
2024-12-18T00:21:41.1337904Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2024-12-18T00:21:41.1361963Z Entering 'third_party/fmt'
2024-12-18T00:21:41.1422167Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2024-12-18T00:21:41.1444313Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T00:21:41.1505554Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2024-12-18T00:21:41.1527135Z Entering 'third_party/gloo'
2024-12-18T00:21:41.1589014Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2024-12-18T00:21:41.1611355Z Entering 'third_party/googletest'
2024-12-18T00:21:41.1677696Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:41.1700424Z Entering 'third_party/ideep'
2024-12-18T00:21:41.1762892Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2024-12-18T00:21:41.1782380Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T00:21:41.1843320Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2024-12-18T00:21:41.1873635Z Entering 'third_party/ittapi'
2024-12-18T00:21:41.1934294Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2024-12-18T00:21:41.1955787Z Entering 'third_party/kineto'
2024-12-18T00:21:41.2019703Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2024-12-18T00:21:41.2041435Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T00:21:41.2104064Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config	remote.origin.url
2024-12-18T00:21:41.2125294Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T00:21:41.2188584Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config	remote.origin.url
2024-12-18T00:21:41.2210307Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T00:21:41.2272533Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config	remote.origin.url
2024-12-18T00:21:41.2293197Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T00:21:41.2357036Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config	remote.origin.url
2024-12-18T00:21:41.2377992Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T00:21:41.2440448Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config	remote.origin.url
2024-12-18T00:21:41.2459025Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T00:21:41.2527526Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config	remote.origin.url
2024-12-18T00:21:41.2555030Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T00:21:41.2616895Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config	remote.origin.url
2024-12-18T00:21:41.2638490Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T00:21:41.2700394Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:41.2720585Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T00:21:41.2782826Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config	remote.origin.url
2024-12-18T00:21:41.2803755Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T00:21:41.2866862Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config	remote.origin.url
2024-12-18T00:21:41.2891855Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T00:21:41.2958096Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2024-12-18T00:21:41.2978598Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T00:21:41.3049148Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:41.3076350Z Entering 'third_party/mimalloc'
2024-12-18T00:21:41.3139578Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config	remote.origin.url
2024-12-18T00:21:41.3161681Z Entering 'third_party/nccl/nccl'
2024-12-18T00:21:41.3223570Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2024-12-18T00:21:41.3246741Z Entering 'third_party/nlohmann'
2024-12-18T00:21:41.3308677Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2024-12-18T00:21:41.3331973Z Entering 'third_party/onnx'
2024-12-18T00:21:41.3394543Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2024-12-18T00:21:41.3430141Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T00:21:41.3491435Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2024-12-18T00:21:41.3517338Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T00:21:41.3582599Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config	remote.origin.url
2024-12-18T00:21:41.3603022Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T00:21:41.3664775Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config	remote.origin.url
2024-12-18T00:21:41.3685439Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T00:21:41.3746995Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:41.3767680Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T00:21:41.3829060Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config	remote.origin.url
2024-12-18T00:21:41.3850013Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T00:21:41.3911619Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config	remote.origin.url
2024-12-18T00:21:41.3934214Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T00:21:41.3998387Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config	remote.origin.url
2024-12-18T00:21:41.4018608Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T00:21:41.4080328Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config	remote.origin.url
2024-12-18T00:21:41.4104066Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T00:21:41.4166191Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config	remote.origin.url
2024-12-18T00:21:41.4185243Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T00:21:41.4250275Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2024-12-18T00:21:41.4272863Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T00:21:41.4335224Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2024-12-18T00:21:41.4359119Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T00:21:41.4426107Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config	remote.origin.url
2024-12-18T00:21:41.4468355Z Entering 'third_party/pocketfft'
2024-12-18T00:21:41.4530549Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2024-12-18T00:21:41.4552209Z Entering 'third_party/protobuf'
2024-12-18T00:21:41.4613764Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2024-12-18T00:21:41.4636465Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T00:21:41.4698494Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2024-12-18T00:21:41.4719231Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T00:21:41.4785666Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:41.4809864Z Entering 'third_party/psimd'
2024-12-18T00:21:41.4873445Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2024-12-18T00:21:41.4895409Z Entering 'third_party/pthreadpool'
2024-12-18T00:21:41.4957959Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2024-12-18T00:21:41.4979780Z Entering 'third_party/pybind11'
2024-12-18T00:21:41.5041999Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2024-12-18T00:21:41.5064612Z Entering 'third_party/python-peachpy'
2024-12-18T00:21:41.5127605Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2024-12-18T00:21:41.5149279Z Entering 'third_party/sleef'
2024-12-18T00:21:41.5211976Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2024-12-18T00:21:41.5233570Z Entering 'third_party/tensorpipe'
2024-12-18T00:21:41.5298345Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2024-12-18T00:21:41.5320415Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T00:21:41.5382556Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:41.5403737Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T00:21:41.5465183Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2024-12-18T00:21:41.5485811Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T00:21:41.5547242Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2024-12-18T00:21:41.5568258Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T00:21:41.5629531Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2024-12-18T00:21:41.5648840Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T00:21:41.5715874Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2024-12-18T00:21:41.6391543Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2024-12-18T00:21:41.6747914Z Entering 'android/libs/fbjni'
2024-12-18T00:21:41.6800665Z Entering 'third_party/FP16'
2024-12-18T00:21:41.6852877Z Entering 'third_party/FXdiv'
2024-12-18T00:21:41.6905150Z Entering 'third_party/NNPACK'
2024-12-18T00:21:41.6957958Z Entering 'third_party/NVTX'
2024-12-18T00:21:41.7011801Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T00:21:41.7064078Z Entering 'third_party/XNNPACK'
2024-12-18T00:21:41.7133363Z Entering 'third_party/benchmark'
2024-12-18T00:21:41.7185347Z Entering 'third_party/composable_kernel'
2024-12-18T00:21:41.7244509Z Entering 'third_party/cpp-httplib'
2024-12-18T00:21:41.7296677Z Entering 'third_party/cpuinfo'
2024-12-18T00:21:41.7349856Z Entering 'third_party/cudnn_frontend'
2024-12-18T00:21:41.7402740Z Entering 'third_party/cutlass'
2024-12-18T00:21:41.7466734Z Entering 'third_party/eigen'
2024-12-18T00:21:41.7519801Z Entering 'third_party/fbgemm'
2024-12-18T00:21:41.7572746Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T00:21:41.7627761Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T00:21:41.7678892Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T00:21:41.7736581Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T00:21:41.7787009Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T00:21:41.7840815Z Entering 'third_party/flatbuffers'
2024-12-18T00:21:41.7897561Z Entering 'third_party/fmt'
2024-12-18T00:21:41.7948846Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T00:21:41.8001366Z Entering 'third_party/gloo'
2024-12-18T00:21:41.8054802Z Entering 'third_party/googletest'
2024-12-18T00:21:41.8107236Z Entering 'third_party/ideep'
2024-12-18T00:21:41.8158169Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T00:21:41.8221488Z Entering 'third_party/ittapi'
2024-12-18T00:21:41.8274405Z Entering 'third_party/kineto'
2024-12-18T00:21:41.8325699Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T00:21:41.8376139Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T00:21:41.8429261Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T00:21:41.8481452Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T00:21:41.8531913Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T00:21:41.8579780Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T00:21:41.8638611Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T00:21:41.8689170Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T00:21:41.8740978Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T00:21:41.8792560Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T00:21:41.8846290Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T00:21:41.8896376Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T00:21:41.8955095Z Entering 'third_party/mimalloc'
2024-12-18T00:21:41.9007502Z Entering 'third_party/nccl/nccl'
2024-12-18T00:21:41.9063307Z Entering 'third_party/nlohmann'
2024-12-18T00:21:41.9116502Z Entering 'third_party/onnx'
2024-12-18T00:21:41.9182423Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T00:21:41.9237916Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T00:21:41.9290493Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T00:21:41.9341220Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T00:21:41.9394651Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T00:21:41.9444892Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T00:21:41.9495757Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T00:21:41.9547580Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T00:21:41.9596486Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T00:21:41.9652040Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T00:21:41.9703981Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T00:21:41.9756941Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T00:21:41.9831829Z Entering 'third_party/pocketfft'
2024-12-18T00:21:41.9883747Z Entering 'third_party/protobuf'
2024-12-18T00:21:41.9936011Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T00:21:41.9988231Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T00:21:42.0042335Z Entering 'third_party/psimd'
2024-12-18T00:21:42.0094397Z Entering 'third_party/pthreadpool'
2024-12-18T00:21:42.0145236Z Entering 'third_party/pybind11'
2024-12-18T00:21:42.0196399Z Entering 'third_party/python-peachpy'
2024-12-18T00:21:42.0248347Z Entering 'third_party/sleef'
2024-12-18T00:21:42.0299522Z Entering 'third_party/tensorpipe'
2024-12-18T00:21:42.0350883Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T00:21:42.0404500Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T00:21:42.0454728Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T00:21:42.0506030Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T00:21:42.0554553Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T00:21:42.0629924Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2024-12-18T00:21:42.0988232Z Entering 'android/libs/fbjni'
2024-12-18T00:21:42.1040622Z Entering 'third_party/FP16'
2024-12-18T00:21:42.1091168Z Entering 'third_party/FXdiv'
2024-12-18T00:21:42.1141789Z Entering 'third_party/NNPACK'
2024-12-18T00:21:42.1194210Z Entering 'third_party/NVTX'
2024-12-18T00:21:42.1247252Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T00:21:42.1299999Z Entering 'third_party/XNNPACK'
2024-12-18T00:21:42.1369731Z Entering 'third_party/benchmark'
2024-12-18T00:21:42.1423196Z Entering 'third_party/composable_kernel'
2024-12-18T00:21:42.1492917Z Entering 'third_party/cpp-httplib'
2024-12-18T00:21:42.1545756Z Entering 'third_party/cpuinfo'
2024-12-18T00:21:42.1595865Z Entering 'third_party/cudnn_frontend'
2024-12-18T00:21:42.1648867Z Entering 'third_party/cutlass'
2024-12-18T00:21:42.1708175Z Entering 'third_party/eigen'
2024-12-18T00:21:42.1761728Z Entering 'third_party/fbgemm'
2024-12-18T00:21:42.1821575Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T00:21:42.1874004Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T00:21:42.1925776Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T00:21:42.1983612Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T00:21:42.2034498Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T00:21:42.2088578Z Entering 'third_party/flatbuffers'
2024-12-18T00:21:42.2150177Z Entering 'third_party/fmt'
2024-12-18T00:21:42.2203438Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T00:21:42.2254509Z Entering 'third_party/gloo'
2024-12-18T00:21:42.2305174Z Entering 'third_party/googletest'
2024-12-18T00:21:42.2356911Z Entering 'third_party/ideep'
2024-12-18T00:21:42.2405295Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T00:21:42.2479534Z Entering 'third_party/ittapi'
2024-12-18T00:21:42.2517228Z Entering 'third_party/kineto'
2024-12-18T00:21:42.2567135Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T00:21:42.2615641Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T00:21:42.2668430Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T00:21:42.2724556Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T00:21:42.2775686Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T00:21:42.2824395Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T00:21:42.2879816Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T00:21:42.2930157Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T00:21:42.2981872Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T00:21:42.3039885Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T00:21:42.3094069Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T00:21:42.3146796Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T00:21:42.3201592Z Entering 'third_party/mimalloc'
2024-12-18T00:21:42.3254498Z Entering 'third_party/nccl/nccl'
2024-12-18T00:21:42.3305094Z Entering 'third_party/nlohmann'
2024-12-18T00:21:42.3358939Z Entering 'third_party/onnx'
2024-12-18T00:21:42.3423321Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T00:21:42.3478176Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T00:21:42.3532087Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T00:21:42.3583027Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T00:21:42.3633721Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T00:21:42.3683411Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T00:21:42.3735088Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T00:21:42.3785004Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T00:21:42.3835013Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T00:21:42.3884108Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T00:21:42.3936386Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T00:21:42.3989182Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T00:21:42.4063092Z Entering 'third_party/pocketfft'
2024-12-18T00:21:42.4116016Z Entering 'third_party/protobuf'
2024-12-18T00:21:42.4171508Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T00:21:42.4221992Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T00:21:42.4277945Z Entering 'third_party/psimd'
2024-12-18T00:21:42.4327813Z Entering 'third_party/pthreadpool'
2024-12-18T00:21:42.4382150Z Entering 'third_party/pybind11'
2024-12-18T00:21:42.4437525Z Entering 'third_party/python-peachpy'
2024-12-18T00:21:42.4488577Z Entering 'third_party/sleef'
2024-12-18T00:21:42.4546278Z Entering 'third_party/tensorpipe'
2024-12-18T00:21:42.4597500Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T00:21:42.4648128Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T00:21:42.4697810Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T00:21:42.4748797Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T00:21:42.4797400Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T00:21:42.4873106Z ##[endgroup]
2024-12-18T00:21:42.4909482Z [command]/usr/bin/git log -1 --format='%H'
2024-12-18T00:21:42.4934654Z '0cdf8b1d09254cfda66191d1bd01e3041c3c76f7'
2024-12-18T00:21:42.5106176Z Prepare all required actions
2024-12-18T00:21:42.5106664Z Getting action download info
2024-12-18T00:21:42.7153195Z ##[group]Run ./.github/actions/setup-linux
2024-12-18T00:21:42.7153521Z env:
2024-12-18T00:21:42.7153732Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:42.7153994Z ##[endgroup]
2024-12-18T00:21:42.7199925Z ##[group]Run set -euo pipefail
2024-12-18T00:21:42.7200274Z set -euo pipefail
2024-12-18T00:21:42.7200588Z function get_ec2_metadata() {
2024-12-18T00:21:42.7200990Z   # Pulled from instance metadata endpoint for EC2
2024-12-18T00:21:42.7201671Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2024-12-18T00:21:42.7202296Z   category=$1
2024-12-18T00:21:42.7202681Z   # If it is GCP runner (runner name contains gcp), do not run this
2024-12-18T00:21:42.7203149Z   runner_name_str=i-02d991818cc142eee
2024-12-18T00:21:42.7203531Z   if [[ -f /.inarc ]]; then
2024-12-18T00:21:42.7203877Z     echo "ARC Runner, no info on ec2 metadata"
2024-12-18T00:21:42.7204262Z   elif [[ $runner_name_str == *"gcp"* ]]; then
2024-12-18T00:21:42.7204722Z     echo "Runner is from Google Cloud Platform, No info on ec2 metadata"
2024-12-18T00:21:42.7205140Z   else
2024-12-18T00:21:42.7205987Z     curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2024-12-18T00:21:42.7206876Z   fi
2024-12-18T00:21:42.7207097Z }
2024-12-18T00:21:42.7207370Z echo "ami-id: $(get_ec2_metadata ami-id)"
2024-12-18T00:21:42.7207787Z echo "instance-id: $(get_ec2_metadata instance-id)"
2024-12-18T00:21:42.7208261Z echo "instance-type: $(get_ec2_metadata instance-type)"
2024-12-18T00:21:42.7208678Z echo "system info $(uname -a)"
2024-12-18T00:21:42.7219606Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:42.7219974Z env:
2024-12-18T00:21:42.7220195Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:42.7220458Z ##[endgroup]
2024-12-18T00:21:42.7377479Z ami-id: ami-0fff1b9a61dec8a5f
2024-12-18T00:21:42.7491686Z instance-id: i-02d991818cc142eee
2024-12-18T00:21:42.7598362Z instance-type: g5.4xlarge
2024-12-18T00:21:42.7611762Z system info Linux ip-10-0-72-182.ec2.internal 6.1.109-118.189.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 10 08:59:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
2024-12-18T00:21:42.7649479Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T00:21:42.7650383Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T00:21:42.7659620Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:42.7660018Z env:
2024-12-18T00:21:42.7660240Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:42.7660495Z ##[endgroup]
2024-12-18T00:21:42.7730857Z ##[group]Run if systemctl is-active --quiet docker; then
2024-12-18T00:21:42.7731595Z if systemctl is-active --quiet docker; then
2024-12-18T00:21:42.7731985Z     echo "Docker daemon is running...";
2024-12-18T00:21:42.7732366Z else
2024-12-18T00:21:42.7732850Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2024-12-18T00:21:42.7733456Z fi
2024-12-18T00:21:42.7742619Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:42.7743228Z env:
2024-12-18T00:21:42.7743452Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:42.7743710Z ##[endgroup]
2024-12-18T00:21:42.7827928Z Docker daemon is running...
2024-12-18T00:21:42.7881068Z ##[group]Run nick-fields/retry@v3.0.0
2024-12-18T00:21:42.7881375Z with:
2024-12-18T00:21:42.7881593Z   shell: bash
2024-12-18T00:21:42.7882026Z   timeout_minutes: 5
2024-12-18T00:21:42.7882273Z   max_attempts: 3
2024-12-18T00:21:42.7882509Z   retry_wait_seconds: 30
2024-12-18T00:21:42.7884735Z   command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \
    --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"

# For LF Runners we need to make sure we also login to Meta's ECR docker registry too.
META_AWS_ACCOUNT_ID=308535385114
if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then
    aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \
        --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
fi

2024-12-18T00:21:42.7886991Z   polling_interval_seconds: 1
2024-12-18T00:21:42.7887280Z   warning_on_retry: true
2024-12-18T00:21:42.7887545Z   continue_on_error: false
2024-12-18T00:21:42.7887803Z env:
2024-12-18T00:21:42.7888024Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:42.7888296Z   AWS_RETRY_MODE: standard
2024-12-18T00:21:42.7888575Z   AWS_MAX_ATTEMPTS: 5
2024-12-18T00:21:42.7888841Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T00:21:42.7889112Z ##[endgroup]
2024-12-18T00:21:44.0516456Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-12-18T00:21:44.0517377Z Configure a credential helper to remove this warning. See
2024-12-18T00:21:44.0518149Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-12-18T00:21:44.0518632Z 
2024-12-18T00:21:44.0518751Z Login Succeeded
2024-12-18T00:21:44.9476429Z Command completed after 1 attempt(s).
2024-12-18T00:21:44.9561519Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2024-12-18T00:21:44.9562033Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2024-12-18T00:21:44.9562486Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2024-12-18T00:21:44.9572718Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:44.9573082Z env:
2024-12-18T00:21:44.9573298Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:44.9573555Z ##[endgroup]
2024-12-18T00:21:44.9678096Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2024-12-18T00:21:44.9678643Z # ignore expansion of "docker ps -q" since it could be empty
2024-12-18T00:21:44.9679068Z # shellcheck disable=SC2046
2024-12-18T00:21:44.9679394Z docker stop $(docker ps -q) || true
2024-12-18T00:21:44.9679736Z # Prune all of the docker images
2024-12-18T00:21:44.9680058Z docker system prune -af
2024-12-18T00:21:44.9688267Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:44.9688632Z env:
2024-12-18T00:21:44.9688851Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:44.9689103Z ##[endgroup]
2024-12-18T00:21:45.0014710Z "docker stop" requires at least 1 argument.
2024-12-18T00:21:45.0015094Z See 'docker stop --help'.
2024-12-18T00:21:45.0015281Z 
2024-12-18T00:21:45.0015475Z Usage:  docker stop [OPTIONS] CONTAINER [CONTAINER...]
2024-12-18T00:21:45.0015748Z 
2024-12-18T00:21:45.0015859Z Stop one or more running containers
2024-12-18T00:21:50.1315887Z Deleted Images:
2024-12-18T00:21:50.1316234Z untagged: pytorch/manylinux2_28-builder:cuda12.4
2024-12-18T00:21:50.1316876Z untagged: pytorch/manylinux2_28-builder@sha256:d831111be47c42997ed761363c535c3e3aef55a1dea5d1b6ac6785705dd8e3d6
2024-12-18T00:21:50.1317633Z deleted: sha256:45fd72e393900307ed7fe71c258ce5e557974227df378abb1d751efb528052d5
2024-12-18T00:21:50.1318261Z deleted: sha256:d733f715c0b74aa1caafaaf43352c8629c839619dcac0e264145ae53f4094de3
2024-12-18T00:21:50.1319214Z deleted: sha256:8fd0b215bb29c9288bc18886c286982ef4a1f71a9f67939ac92a4a6bba4478d5
2024-12-18T00:21:50.1319945Z deleted: sha256:64a54a6c17d6324abdd31f5a8b6add3c88308c789f94d64bcbc659443174edb1
2024-12-18T00:21:50.1320633Z deleted: sha256:eacfbed66ccc75c8601a1ba05d151166da0b7149a21d0f9f94f0c75c7f4b1050
2024-12-18T00:21:50.1321795Z deleted: sha256:5b9ca13b81da6943dfe2a3d1f5c4cc46f072494bfebe9e96781a869057fd65f4
2024-12-18T00:21:50.1338499Z deleted: sha256:3b4d4765a4705cfc0114341fca901a60c6600f5b0fc902315e809cfdee5c8b7c
2024-12-18T00:21:50.1339154Z deleted: sha256:47eefb5e64ff3f98cb14346a5692bc5fe0a96ee1fd04807337c49a8c6bc7c2fe
2024-12-18T00:21:50.1339914Z deleted: sha256:5a2bbbf1b6b7ed03f3ff5ce8cc2db99fcc0b82de9db286b37a7c5f2fa56e6ef3
2024-12-18T00:21:50.1340555Z deleted: sha256:801cf5eaa056f62be3e96f73675cda64de4144839ca8211e0daaadd58aea0670
2024-12-18T00:21:50.1341181Z deleted: sha256:92a43f5f90e4760d770a30785e699f646c0cf79f36f66c6591b55cfc0d0148f8
2024-12-18T00:21:50.1341809Z deleted: sha256:5b0c5704cf1bbd33789de39c50c7fba1e883f371e0446d06c5e654751e861271
2024-12-18T00:21:50.1342524Z deleted: sha256:f07b6a8ce777e08f37c16c0835e62281886aca2d8e14f28688e30338bfe1c7b7
2024-12-18T00:21:50.1343138Z deleted: sha256:f8a1d37a33fed4d3d3fc2c483674628dffe46283362698c49542f028c469ac7a
2024-12-18T00:21:50.1343767Z deleted: sha256:3b99d8907a5cccca3ae71d0768b72cb4af61d1970c41973c3a2ac0a618046686
2024-12-18T00:21:50.1344395Z deleted: sha256:878f35e5ddaee806bdea6be1f2f2465e3a38310789872c647134173088ef2f59
2024-12-18T00:21:50.1345014Z deleted: sha256:f4461bfaf198fa65cf21819111cf17047a17f312d6a2974ced6e416eeea204a9
2024-12-18T00:21:50.1345623Z deleted: sha256:690c69b1c30bf2c777824a06b939f80059366fa0474eb086c3d1e61509500089
2024-12-18T00:21:50.1346238Z deleted: sha256:1b248efab83d590da489a650f9cb3a737d1ae170172c8f71069ea82a9723e8a9
2024-12-18T00:21:50.1346859Z deleted: sha256:28cf4ca1f4fda594c8da6e45a98f809384b851f6b926b2178a048e3a22f514f8
2024-12-18T00:21:50.1347479Z deleted: sha256:9df1df3e9f00df659c7499f8c7d25543a1710f489b1289e9c9a8644682bb3155
2024-12-18T00:21:50.1348092Z deleted: sha256:81d131bc6d7193a1d5ceac7604279c789b3e48f50e907f45dd0c596496e1edd0
2024-12-18T00:21:50.1348709Z deleted: sha256:388ea69c907d09cf2d58fa917ed4f9ef7e78ae3e4e40235cc5922360ec7001c5
2024-12-18T00:21:50.1349340Z deleted: sha256:36f95ad2b5b3ee5959502c79522b64eb848b52dfb1d16052023690db4754f3c7
2024-12-18T00:21:50.1349953Z deleted: sha256:812e8afb11f2b15bf2115b56d2bbe14f4e33bd60b1196530267dd0c0b632d0d8
2024-12-18T00:21:50.1350571Z deleted: sha256:98491a257f4a8fab8b81b23c4d43b553d8b3df091e27ab864f52c1c04445e9e2
2024-12-18T00:21:50.1351182Z deleted: sha256:79db0df886564106a67e8d0f5f65070061abf80771e126978dddddae3c2a826a
2024-12-18T00:21:50.1351780Z deleted: sha256:482879d7fa41016e849aed5a388341d79601879304b4aad4b35ea3cf4121b7ce
2024-12-18T00:21:50.1352398Z deleted: sha256:a5d3eecefb86d1288d1094060fe133cd2ad1b68a11d4b08161e6b1a009390edf
2024-12-18T00:21:50.1352771Z 
2024-12-18T00:21:50.1352897Z Total reclaimed space: 13.08GB
2024-12-18T00:21:50.1399476Z ##[group]Run set +e
2024-12-18T00:21:50.1399785Z set +e
2024-12-18T00:21:50.1400033Z set -x
2024-12-18T00:21:50.1400268Z 
2024-12-18T00:21:50.1400517Z PT_DOMAIN=download.pytorch.org
2024-12-18T00:21:50.1401118Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400,
2024-12-18T00:21:50.1401868Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last
2024-12-18T00:21:50.1402398Z # one is returned at random
2024-12-18T00:21:50.1402802Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1)
2024-12-18T00:21:50.1403187Z 
2024-12-18T00:21:50.1403424Z if [ -z "${RESOLVED_IP}" ]; then
2024-12-18T00:21:50.1403867Z   echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..."
2024-12-18T00:21:50.1404387Z   RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1)
2024-12-18T00:21:50.1404980Z 
2024-12-18T00:21:50.1405228Z   if [ -z "${RESOLVED_IP}" ]; then
2024-12-18T00:21:50.1405610Z     echo "Couldn't resolve ${PT_DOMAIN}, exiting..."
2024-12-18T00:21:50.1405980Z     exit 1
2024-12-18T00:21:50.1406234Z   fi
2024-12-18T00:21:50.1406474Z fi
2024-12-18T00:21:50.1406868Z 
2024-12-18T00:21:50.1407134Z if grep -r "${PT_DOMAIN}" /etc/hosts; then
2024-12-18T00:21:50.1407507Z   # Clean up any old records first
2024-12-18T00:21:50.1407889Z   sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts
2024-12-18T00:21:50.1408219Z fi
2024-12-18T00:21:50.1408443Z 
2024-12-18T00:21:50.1408752Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts
2024-12-18T00:21:50.1409151Z cat /etc/hosts
2024-12-18T00:21:50.1418396Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:50.1418786Z env:
2024-12-18T00:21:50.1419009Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:50.1419286Z ##[endgroup]
2024-12-18T00:21:50.1449153Z + PT_DOMAIN=download.pytorch.org
2024-12-18T00:21:50.1455978Z ++ dig -4 +short download.pytorch.org
2024-12-18T00:21:50.1457385Z ++ tail -n1
2024-12-18T00:21:50.1740634Z + RESOLVED_IP=18.160.18.34
2024-12-18T00:21:50.1740993Z + '[' -z 18.160.18.34 ']'
2024-12-18T00:21:50.1741376Z + grep -r download.pytorch.org /etc/hosts
2024-12-18T00:21:50.1754499Z 3.167.99.70 download.pytorch.org
2024-12-18T00:21:50.1756602Z + sudo sed -i /download.pytorch.org/d /etc/hosts
2024-12-18T00:21:50.3130556Z + echo '18.160.18.34 download.pytorch.org'
2024-12-18T00:21:50.3131275Z + sudo tee -a /etc/hosts
2024-12-18T00:21:50.3468787Z 18.160.18.34 download.pytorch.org
2024-12-18T00:21:50.3490729Z + cat /etc/hosts
2024-12-18T00:21:50.3505799Z 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
2024-12-18T00:21:50.3509973Z ::1         localhost6 localhost6.localdomain6
2024-12-18T00:21:50.3510342Z 18.160.18.34 download.pytorch.org
2024-12-18T00:21:50.3665323Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@release/2.6
2024-12-18T00:21:50.3665826Z with:
2024-12-18T00:21:50.3666665Z   docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:50.3667620Z   docker-build-dir: .ci/docker
2024-12-18T00:21:50.3667921Z   working-directory: .
2024-12-18T00:21:50.3668283Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:50.3668677Z   force-push: false
2024-12-18T00:21:50.3668912Z env:
2024-12-18T00:21:50.3669134Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:50.3669399Z ##[endgroup]
2024-12-18T00:21:50.3698338Z ##[group]Run set -ex
2024-12-18T00:21:50.3698639Z set -ex
2024-12-18T00:21:50.3698878Z 
2024-12-18T00:21:50.3699275Z # If the docker build directory or the build script doesn't exist, the action will
2024-12-18T00:21:50.3699969Z # gracefully return the docker image name as it is.  Pulling docker image in Linux
2024-12-18T00:21:50.3700548Z # job could then download the pre-built image as usual
2024-12-18T00:21:50.3701065Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then
2024-12-18T00:21:50.3701539Z   echo "skip=true" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:50.3702101Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:50.3702522Z 
2024-12-18T00:21:50.3702885Z   echo "There is no Docker build script in ${REPO_NAME} repo, skipping..."
2024-12-18T00:21:50.3703319Z   exit 0
2024-12-18T00:21:50.3703565Z else
2024-12-18T00:21:50.3703846Z   echo "skip=false" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:50.3704188Z fi
2024-12-18T00:21:50.3704409Z 
2024-12-18T00:21:50.3704751Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then
2024-12-18T00:21:50.3706360Z   # The docker image name already includes the ECR prefix and tag, so we can just
2024-12-18T00:21:50.3706883Z   # use it as it is, but first let's extract the tag
2024-12-18T00:21:50.3707361Z   DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}')
2024-12-18T00:21:50.3707867Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:50.3708353Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:50.3708765Z else
2024-12-18T00:21:50.3709082Z   DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}")
2024-12-18T00:21:50.3709535Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:50.3710171Z   echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:50.3710720Z fi
2024-12-18T00:21:50.3720248Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:50.3720634Z env:
2024-12-18T00:21:50.3720861Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:50.3721134Z   REPO_NAME: pytorch
2024-12-18T00:21:50.3721992Z   DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:50.3722902Z   DOCKER_BUILD_DIR: .ci/docker
2024-12-18T00:21:50.3723289Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:50.3723680Z ##[endgroup]
2024-12-18T00:21:50.3755370Z + [[ ! -d .ci/docker ]]
2024-12-18T00:21:50.3756033Z + [[ ! -f .ci/docker/build.sh ]]
2024-12-18T00:21:50.3756525Z + echo skip=false
2024-12-18T00:21:50.3758049Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]]
2024-12-18T00:21:50.3762870Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:50.3763860Z ++ awk -F '[:,]' '{print $2}'
2024-12-18T00:21:50.3789104Z + DOCKER_TAG=45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:50.3789606Z + echo docker-tag=45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:50.3790612Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:50.3833562Z ##[group]Run set +e
2024-12-18T00:21:50.3834001Z set +e
2024-12-18T00:21:50.3834475Z set -x
2024-12-18T00:21:50.3834816Z 
2024-12-18T00:21:50.3835069Z login() {
2024-12-18T00:21:50.3835728Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2024-12-18T00:21:50.3836369Z }
2024-12-18T00:21:50.3836713Z 
2024-12-18T00:21:50.3837060Z retry () {
2024-12-18T00:21:50.3837432Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2024-12-18T00:21:50.3837844Z }
2024-12-18T00:21:50.3838190Z 
2024-12-18T00:21:50.3838517Z retry login "${DOCKER_REGISTRY}"
2024-12-18T00:21:50.3838973Z 
2024-12-18T00:21:50.3839390Z START_TIME=$(date +%s)
2024-12-18T00:21:50.3839740Z # Wait up to 90 minutes
2024-12-18T00:21:50.3840267Z while [[ $(( $(date +%s) - 5400 )) -lt $START_TIME ]]; do
2024-12-18T00:21:50.3840845Z   # Check if image already exists, if it does then skip building it
2024-12-18T00:21:50.3841401Z   if docker manifest inspect "${DOCKER_IMAGE}"; then
2024-12-18T00:21:50.3841940Z     exit 0
2024-12-18T00:21:50.3842273Z   fi
2024-12-18T00:21:50.3842533Z 
2024-12-18T00:21:50.3843113Z   # NB: This flag is used by Docker build workflow to push the image to ECR, so we can
2024-12-18T00:21:50.3844027Z   # use this to differentiate between the Docker build and regular build jobs. For the
2024-12-18T00:21:50.3844813Z   # latter, it will wait for the Docker images to become available before continuing
2024-12-18T00:21:50.3845508Z   if [ "${DOCKER_PUSH:-false}" == "true" ]; then
2024-12-18T00:21:50.3846054Z     # It's a Docker build job, let's build the image
2024-12-18T00:21:50.3846516Z     break
2024-12-18T00:21:50.3846866Z   else
2024-12-18T00:21:50.3847315Z     # It's a regular build job, wait for the image to become available
2024-12-18T00:21:50.3847859Z     sleep 300
2024-12-18T00:21:50.3848203Z   fi
2024-12-18T00:21:50.3848523Z done
2024-12-18T00:21:50.3848864Z 
2024-12-18T00:21:50.3849306Z # NB: This part requires a full checkout. Otherwise, the merge base will
2024-12-18T00:21:50.3849968Z # be empty.  The default action would be to continue rebuild the image
2024-12-18T00:21:50.3850706Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then
2024-12-18T00:21:50.3851270Z   # if we're on the base branch then use the parent commit
2024-12-18T00:21:50.3851798Z   MERGE_BASE=$(git rev-parse HEAD~)
2024-12-18T00:21:50.3852236Z else
2024-12-18T00:21:50.3852670Z   # otherwise we're on a PR, so use the most recent base commit
2024-12-18T00:21:50.3853250Z   MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION")
2024-12-18T00:21:50.3853734Z fi
2024-12-18T00:21:50.3854043Z 
2024-12-18T00:21:50.3854421Z if [[ -z "${MERGE_BASE}" ]]; then
2024-12-18T00:21:50.3854859Z   echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:50.3855283Z 
2024-12-18T00:21:50.3856139Z   echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..."
2024-12-18T00:21:50.3856780Z   exit 0
2024-12-18T00:21:50.3857088Z fi
2024-12-18T00:21:50.3857454Z 
2024-12-18T00:21:50.3857853Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then
2024-12-18T00:21:50.3858597Z   echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit"
2024-12-18T00:21:50.3859331Z   exit 1
2024-12-18T00:21:50.3859651Z fi
2024-12-18T00:21:50.3860015Z 
2024-12-18T00:21:50.3860443Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}")
2024-12-18T00:21:50.3861170Z # If no image exists but the hash is the same as the previous hash then we should error out here
2024-12-18T00:21:50.3862057Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then
2024-12-18T00:21:50.3862840Z   echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch"
2024-12-18T00:21:50.3863667Z   echo "         Will re-build docker image to store in local cache, TTS may be longer"
2024-12-18T00:21:50.3864285Z fi
2024-12-18T00:21:50.3864601Z 
2024-12-18T00:21:50.3864945Z echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:50.3874083Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:50.3874580Z env:
2024-12-18T00:21:50.3874979Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:50.3875428Z   DOCKER_BUILD_DIR: .ci/docker
2024-12-18T00:21:50.3875889Z   BASE_REVISION: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:21:50.3876881Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:50.3877966Z   DOCKER_TAG: 45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:50.3878543Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:50.3879124Z   DOCKER_PUSH: 
2024-12-18T00:21:50.3879401Z ##[endgroup]
2024-12-18T00:21:50.3912248Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:50.3912697Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:50.3915492Z + aws ecr get-login-password --region us-east-1
2024-12-18T00:21:50.3917380Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:50.9447338Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-12-18T00:21:50.9448126Z Configure a credential helper to remove this warning. See
2024-12-18T00:21:50.9448816Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-12-18T00:21:50.9449289Z 
2024-12-18T00:21:50.9449726Z Login Succeeded
2024-12-18T00:21:50.9477164Z ++ date +%s
2024-12-18T00:21:50.9490109Z + START_TIME=1734481310
2024-12-18T00:21:50.9493495Z ++ date +%s
2024-12-18T00:21:50.9504263Z + [[ 1734475910 -lt 1734481310 ]]
2024-12-18T00:21:50.9505466Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:51.1782540Z {
2024-12-18T00:21:51.1782882Z 	"schemaVersion": 2,
2024-12-18T00:21:51.1783420Z 	"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
2024-12-18T00:21:51.1783863Z 	"config": {
2024-12-18T00:21:51.1784186Z 		"mediaType": "application/vnd.docker.container.image.v1+json",
2024-12-18T00:21:51.1784597Z 		"size": 52680,
2024-12-18T00:21:51.1785075Z 		"digest": "sha256:67e93a8badaf799ceb664d7c453077c6c6785d56dc10c720be9358edeb91759b"
2024-12-18T00:21:51.1785562Z 	},
2024-12-18T00:21:51.1785754Z 	"layers": [
2024-12-18T00:21:51.1785963Z 		{
2024-12-18T00:21:51.1786294Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1786709Z 			"size": 28584506,
2024-12-18T00:21:51.1787413Z 			"digest": "sha256:80888bc6716fcbb8874e75ac88898d3e38e6f1bc55678f0e97ca9d706b7f3733"
2024-12-18T00:21:51.1787902Z 		},
2024-12-18T00:21:51.1788094Z 		{
2024-12-18T00:21:51.1788427Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1788861Z 			"size": 7944698,
2024-12-18T00:21:51.1789272Z 			"digest": "sha256:fbcd35dc5bc3a7bda41926aadd083020f942b001ebac6f1d30480f0f065394c0"
2024-12-18T00:21:51.1789761Z 		},
2024-12-18T00:21:51.1789957Z 		{
2024-12-18T00:21:51.1790302Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1790717Z 			"size": 57593527,
2024-12-18T00:21:51.1791143Z 			"digest": "sha256:c7232af9ae05f7de83f8d6171bd0c35a4dd0a85ebafb15b950dbc08f89ea5fb5"
2024-12-18T00:21:51.1791621Z 		},
2024-12-18T00:21:51.1791813Z 		{
2024-12-18T00:21:51.1792137Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1792546Z 			"size": 185,
2024-12-18T00:21:51.1792953Z 			"digest": "sha256:db6cdef1932a0d9ca6ef9a539e08d491f66d1b1ed81926ae1525375bdd8100cc"
2024-12-18T00:21:51.1793430Z 		},
2024-12-18T00:21:51.1793619Z 		{
2024-12-18T00:21:51.1793935Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1794346Z 			"size": 6886,
2024-12-18T00:21:51.1794735Z 			"digest": "sha256:56dc8550293751a1604e97ac949cfae82ba20cb2a28e034737bafd7382559609"
2024-12-18T00:21:51.1795197Z 		},
2024-12-18T00:21:51.1795385Z 		{
2024-12-18T00:21:51.1795704Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1796115Z 			"size": 1374287670,
2024-12-18T00:21:51.1796526Z 			"digest": "sha256:30c0ea6140d07e2a8deb70d780f277c63cf61836ff33d66eef944728a4bef6bd"
2024-12-18T00:21:51.1796989Z 		},
2024-12-18T00:21:51.1797178Z 		{
2024-12-18T00:21:51.1797496Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1797905Z 			"size": 62641,
2024-12-18T00:21:51.1798310Z 			"digest": "sha256:71bdb1a72c2d6dc97bbdbca82383f0260c4ee87556701e8e606c08a6bb0f0da5"
2024-12-18T00:21:51.1799018Z 		},
2024-12-18T00:21:51.1799209Z 		{
2024-12-18T00:21:51.1799527Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1799936Z 			"size": 1685,
2024-12-18T00:21:51.1800329Z 			"digest": "sha256:4829486be7c30f19f4136fa56adbb3de206ed0bbf0705b59fb2147406778ce38"
2024-12-18T00:21:51.1800790Z 		},
2024-12-18T00:21:51.1800976Z 		{
2024-12-18T00:21:51.1801300Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1801703Z 			"size": 1523,
2024-12-18T00:21:51.1802095Z 			"digest": "sha256:1f4e68d7b5e4224ba1da78ef461ff7f01e8d59c09d39281277521384105a9441"
2024-12-18T00:21:51.1802560Z 		},
2024-12-18T00:21:51.1802746Z 		{
2024-12-18T00:21:51.1803067Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1803475Z 			"size": 2626741021,
2024-12-18T00:21:51.1803884Z 			"digest": "sha256:7c373e2d9b7e82a6878d4a31293dd857915a0fe47d07dce541cea03b043d57fc"
2024-12-18T00:21:51.1804351Z 		},
2024-12-18T00:21:51.1804546Z 		{
2024-12-18T00:21:51.1804864Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1805270Z 			"size": 86545,
2024-12-18T00:21:51.1805656Z 			"digest": "sha256:6223811417458a3c93b84ee3b65f8b08d9e2828b926f0aed863041610d7d95d4"
2024-12-18T00:21:51.1806110Z 		},
2024-12-18T00:21:51.1806296Z 		{
2024-12-18T00:21:51.1806616Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1807021Z 			"size": 1824,
2024-12-18T00:21:51.1807426Z 			"digest": "sha256:f003d065e31e5e8c0cea490c88a912dd331bddcbd2f5ff22ccc692a5343bcbeb"
2024-12-18T00:21:51.1807889Z 		},
2024-12-18T00:21:51.1808076Z 		{
2024-12-18T00:21:51.1808398Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1808806Z 			"size": 246420139,
2024-12-18T00:21:51.1809218Z 			"digest": "sha256:cd6ce71168153e485b99493ce07032bbd9fd5cefceb68ef33ce088244306a653"
2024-12-18T00:21:51.1809681Z 		},
2024-12-18T00:21:51.1809867Z 		{
2024-12-18T00:21:51.1810287Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1810959Z 			"size": 703,
2024-12-18T00:21:51.1811399Z 			"digest": "sha256:e843066a8f68042266cfe814f1729975619fb8f733f371be115e8b2fe81cbc9d"
2024-12-18T00:21:51.1811853Z 		},
2024-12-18T00:21:51.1812039Z 		{
2024-12-18T00:21:51.1812359Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1812769Z 			"size": 1256,
2024-12-18T00:21:51.1813173Z 			"digest": "sha256:bac50ca2930bc4b1afa8a2f1b4b0d1d499d63b7922b784af690c3b31ec0c2f70"
2024-12-18T00:21:51.1813647Z 		},
2024-12-18T00:21:51.1813840Z 		{
2024-12-18T00:21:51.1814160Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1814567Z 			"size": 484,
2024-12-18T00:21:51.1814966Z 			"digest": "sha256:57e52c6751f4d865ada4cae919859fe563fe5dcbb93ca82cb4746f98caccbd3a"
2024-12-18T00:21:51.1815430Z 		},
2024-12-18T00:21:51.1815615Z 		{
2024-12-18T00:21:51.1815941Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1816347Z 			"size": 91715265,
2024-12-18T00:21:51.1816761Z 			"digest": "sha256:a9c5c14d77c035c7501566b1247024dfd241bd6bc12fbe2bca28ec7d90b26069"
2024-12-18T00:21:51.1817224Z 		},
2024-12-18T00:21:51.1817411Z 		{
2024-12-18T00:21:51.1817731Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1818135Z 			"size": 3706,
2024-12-18T00:21:51.1818543Z 			"digest": "sha256:b86099b8619ad88e8fe6c8e693180d52e0f2fbeac5927f0b403c8a9ef76d7f6b"
2024-12-18T00:21:51.1819016Z 		},
2024-12-18T00:21:51.1819205Z 		{
2024-12-18T00:21:51.1819530Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1819929Z 			"size": 1860,
2024-12-18T00:21:51.1820339Z 			"digest": "sha256:3c8ea8a600fff6cce4c1b36acc5f3949ecd664e55b5eaf4a50fe5499351574a6"
2024-12-18T00:21:51.1820817Z 		},
2024-12-18T00:21:51.1821014Z 		{
2024-12-18T00:21:51.1821337Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1821851Z 			"size": 701,
2024-12-18T00:21:51.1822366Z 			"digest": "sha256:bebd539a1d598fe86afc0da77bf16ae4c7ebfb9f031f21ed7634150fbe23f996"
2024-12-18T00:21:51.1822856Z 		},
2024-12-18T00:21:51.1823044Z 		{
2024-12-18T00:21:51.1823370Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1823770Z 			"size": 478,
2024-12-18T00:21:51.1824162Z 			"digest": "sha256:292b18ec2dab9bd60d00192387c1269d39b0b4886b782585d2c879808218fc66"
2024-12-18T00:21:51.1824627Z 		},
2024-12-18T00:21:51.1824816Z 		{
2024-12-18T00:21:51.1825135Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1825535Z 			"size": 2941869535,
2024-12-18T00:21:51.1825942Z 			"digest": "sha256:76e3817b4b4cec0a5b005161867dc257681b792271ba9d23c9b79b3a6801dcd9"
2024-12-18T00:21:51.1826402Z 		},
2024-12-18T00:21:51.1826587Z 		{
2024-12-18T00:21:51.1826909Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1827314Z 			"size": 380,
2024-12-18T00:21:51.1827717Z 			"digest": "sha256:50b1f245e9f5be8670d8ba5790e3b96450eee3428dcac7060367db48ae00234e"
2024-12-18T00:21:51.1828180Z 		},
2024-12-18T00:21:51.1828371Z 		{
2024-12-18T00:21:51.1828690Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1829095Z 			"size": 12901,
2024-12-18T00:21:51.1829535Z 			"digest": "sha256:ad93cf5205f24ada86b5f7eab5b63ed97ae7858368f6fdbce373343d2568d66e"
2024-12-18T00:21:51.1830323Z 		},
2024-12-18T00:21:51.1830530Z 		{
2024-12-18T00:21:51.1830846Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1831263Z 			"size": 863,
2024-12-18T00:21:51.1831675Z 			"digest": "sha256:8d17c59eaf7d84d49e3660380debcc70734b2a178e6cab86f0d4fd3e7a8100d1"
2024-12-18T00:21:51.1832144Z 		},
2024-12-18T00:21:51.1832334Z 		{
2024-12-18T00:21:51.1832650Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1833058Z 			"size": 106,
2024-12-18T00:21:51.1833621Z 			"digest": "sha256:b36a8ccf5c8c8ef0bf3e168ec9f568a0ada36c49103c9cacfef6dd072ddd681a"
2024-12-18T00:21:51.1834118Z 		},
2024-12-18T00:21:51.1834306Z 		{
2024-12-18T00:21:51.1834621Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1835041Z 			"size": 504,
2024-12-18T00:21:51.1835439Z 			"digest": "sha256:aa0a3b87e9e34ab15903206018bfd9fb1eb54fd21f05a61ef254557703890732"
2024-12-18T00:21:51.1835905Z 		},
2024-12-18T00:21:51.1836095Z 		{
2024-12-18T00:21:51.1836411Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1836824Z 			"size": 121477314,
2024-12-18T00:21:51.1837236Z 			"digest": "sha256:72a632052c5be4900c60ce09e32f8a0fd3b637ab0b7685dc5b07bda4776ddd15"
2024-12-18T00:21:51.1837709Z 		},
2024-12-18T00:21:51.1837892Z 		{
2024-12-18T00:21:51.1838205Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1838613Z 			"size": 109,
2024-12-18T00:21:51.1839025Z 			"digest": "sha256:ceed6316bc249ffffee9563cb7505023b3e70fba306f187bd612fece63508263"
2024-12-18T00:21:51.1839504Z 		},
2024-12-18T00:21:51.1839693Z 		{
2024-12-18T00:21:51.1840004Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1840409Z 			"size": 489,
2024-12-18T00:21:51.1840812Z 			"digest": "sha256:d0ca0f491f809a9f1dedd391f5b25cc3f59d8e95e7f76299f18ec3c931e56ed1"
2024-12-18T00:21:51.1858247Z 		},
2024-12-18T00:21:51.1858470Z 		{
2024-12-18T00:21:51.1858806Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1859229Z 			"size": 296,
2024-12-18T00:21:51.1859628Z 			"digest": "sha256:cf02c706cd567c781eeb567c90960801dc1ef78872b73fbc047df898951ecbb8"
2024-12-18T00:21:51.1860100Z 		},
2024-12-18T00:21:51.1860291Z 		{
2024-12-18T00:21:51.1860615Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1861026Z 			"size": 103,
2024-12-18T00:21:51.1861425Z 			"digest": "sha256:16073d3f107dab837d7edb5e6044449660fec7e2ded263ca81d5dcfea5beb486"
2024-12-18T00:21:51.1862177Z 		},
2024-12-18T00:21:51.1862368Z 		{
2024-12-18T00:21:51.1862694Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1863109Z 			"size": 1473,
2024-12-18T00:21:51.1863511Z 			"digest": "sha256:d234faf2f1bfb6cb9d4ba14103226d20d540e568b47690b67c2a2b2c5a49645f"
2024-12-18T00:21:51.1863986Z 		},
2024-12-18T00:21:51.1864175Z 		{
2024-12-18T00:21:51.1864500Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1864912Z 			"size": 430258758,
2024-12-18T00:21:51.1865331Z 			"digest": "sha256:3f0ade86af1a5bc72c9088735bb1a7ab9819fa20fdd96437fc080b2c711dfd3d"
2024-12-18T00:21:51.1865805Z 		},
2024-12-18T00:21:51.1865995Z 		{
2024-12-18T00:21:51.1866315Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1866726Z 			"size": 164,
2024-12-18T00:21:51.1867122Z 			"digest": "sha256:034b6a03f5f751ed12f7505175ce3443eb3ed7b34d02d1bb25de0ee52d93a44b"
2024-12-18T00:21:51.1867589Z 		},
2024-12-18T00:21:51.1867782Z 		{
2024-12-18T00:21:51.1868111Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1868521Z 			"size": 564,
2024-12-18T00:21:51.1868917Z 			"digest": "sha256:21f8f066852dd484ccbf79470452fe8005cde08773dd4fedc07fed783a0b9bf8"
2024-12-18T00:21:51.1869386Z 		},
2024-12-18T00:21:51.1869571Z 		{
2024-12-18T00:21:51.1869890Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1870299Z 			"size": 35865656,
2024-12-18T00:21:51.1870702Z 			"digest": "sha256:9dc1f315a979ec860a882d7df33584e02c1d18f86293521839ce42e21082e639"
2024-12-18T00:21:51.1871152Z 		},
2024-12-18T00:21:51.1871340Z 		{
2024-12-18T00:21:51.1871665Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1872072Z 			"size": 104,
2024-12-18T00:21:51.1872458Z 			"digest": "sha256:cd6a6063178b945c150fe21a31df6e33203295dd814802e0658835e810fb2bf9"
2024-12-18T00:21:51.1872912Z 		},
2024-12-18T00:21:51.1873097Z 		{
2024-12-18T00:21:51.1873520Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1873930Z 			"size": 425,
2024-12-18T00:21:51.1874311Z 			"digest": "sha256:2116738f4ea9299766a55629d39b881c4d33fc4fc0ce915965be1a850fcd7aaa"
2024-12-18T00:21:51.1874765Z 		},
2024-12-18T00:21:51.1874948Z 		{
2024-12-18T00:21:51.1875271Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1875681Z 			"size": 20262160,
2024-12-18T00:21:51.1875949Z + exit 0
2024-12-18T00:21:51.1876330Z 			"digest": "sha256:d7fa134316d91d2ec39bceb121619d44cdfdf1c7ac30c72768a9fd48f9763bd1"
2024-12-18T00:21:51.1876796Z 		},
2024-12-18T00:21:51.1876973Z 		{
2024-12-18T00:21:51.1877291Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1877699Z 			"size": 643,
2024-12-18T00:21:51.1878102Z 			"digest": "sha256:af03dc3fd15ea8f5f1fa41d514400f6a1d963985cf046b09d789f7f0359b2aba"
2024-12-18T00:21:51.1878567Z 		},
2024-12-18T00:21:51.1878755Z 		{
2024-12-18T00:21:51.1879081Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1879490Z 			"size": 701,
2024-12-18T00:21:51.1879967Z 			"digest": "sha256:bebd539a1d598fe86afc0da77bf16ae4c7ebfb9f031f21ed7634150fbe23f996"
2024-12-18T00:21:51.1880447Z 		},
2024-12-18T00:21:51.1880639Z 		{
2024-12-18T00:21:51.1880958Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1881367Z 			"size": 142,
2024-12-18T00:21:51.1881773Z 			"digest": "sha256:fd9de3dc425828e7b8984d4ced05f4102762137b81a5dc67104fc6b0805c570d"
2024-12-18T00:21:51.1882234Z 		},
2024-12-18T00:21:51.1882423Z 		{
2024-12-18T00:21:51.1882743Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1883147Z 			"size": 135,
2024-12-18T00:21:51.1883545Z 			"digest": "sha256:b85093661f7768ac691f4b3ee5f553b9ab14cb2a6877c8edc6133d74caa88f69"
2024-12-18T00:21:51.1884011Z 		},
2024-12-18T00:21:51.1884200Z 		{
2024-12-18T00:21:51.1884527Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1885026Z 			"size": 4995482131,
2024-12-18T00:21:51.1885453Z 			"digest": "sha256:ab6ef6c2ac4d23af7a3249cee1d4761f63a89513796ba720da7cc425c8afa6d2"
2024-12-18T00:21:51.1885918Z 		},
2024-12-18T00:21:51.1886110Z 		{
2024-12-18T00:21:51.1886431Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1886833Z 			"size": 196,
2024-12-18T00:21:51.1887225Z 			"digest": "sha256:56c896077120aaa4cbcc2203662d13b4ea14cf998e9657e91b252535ceae5348"
2024-12-18T00:21:51.1887679Z 		},
2024-12-18T00:21:51.1887865Z 		{
2024-12-18T00:21:51.1888182Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1888584Z 			"size": 564,
2024-12-18T00:21:51.1888985Z 			"digest": "sha256:d88e9c5e7c70fe36bd1bd2e94d43d4192d94a1f28e323234a73a75a2d1da48f1"
2024-12-18T00:21:51.1889443Z 		},
2024-12-18T00:21:51.1889625Z 		{
2024-12-18T00:21:51.1889947Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1890363Z 			"size": 43164612,
2024-12-18T00:21:51.1890801Z 			"digest": "sha256:53617ee83302d7f97eb56f6756d78d18f8dc41bf252bf2d04cb643198f400f6f"
2024-12-18T00:21:51.1891257Z 		},
2024-12-18T00:21:51.1891441Z 		{
2024-12-18T00:21:51.1891758Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1892158Z 			"size": 106,
2024-12-18T00:21:51.1892564Z 			"digest": "sha256:3edbde10d4043bf2ebf32fddfa55c8d0923435465637df7dc8f4addcc03a0b84"
2024-12-18T00:21:51.1893035Z 		},
2024-12-18T00:21:51.1893217Z 		{
2024-12-18T00:21:51.1893531Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1893936Z 			"size": 1402,
2024-12-18T00:21:51.1894336Z 			"digest": "sha256:f4038f7524c3ed705e7560db6dd6efe76f548363f16bc7a93cfe360d5bc6dcaa"
2024-12-18T00:21:51.1894794Z 		},
2024-12-18T00:21:51.1894985Z 		{
2024-12-18T00:21:51.1895301Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1895705Z 			"size": 701,
2024-12-18T00:21:51.1896222Z 			"digest": "sha256:bebd539a1d598fe86afc0da77bf16ae4c7ebfb9f031f21ed7634150fbe23f996"
2024-12-18T00:21:51.1896693Z 		},
2024-12-18T00:21:51.1896879Z 		{
2024-12-18T00:21:51.1897197Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1897608Z 			"size": 137,
2024-12-18T00:21:51.1898002Z 			"digest": "sha256:4a51173f77576ab68fba2f862dd4d12b9ff1f250145c090520b05f88f5c81391"
2024-12-18T00:21:51.1898451Z 		},
2024-12-18T00:21:51.1898638Z 		{
2024-12-18T00:21:51.1898953Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1899359Z 			"size": 120,
2024-12-18T00:21:51.1899752Z 			"digest": "sha256:ed6a71d41274e5b1eea4025d24a4851d6880b902b08f7d4dd0618ea81154f575"
2024-12-18T00:21:51.1900208Z 		},
2024-12-18T00:21:51.1900395Z 		{
2024-12-18T00:21:51.1900713Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1901126Z 			"size": 4886226234,
2024-12-18T00:21:51.1901549Z 			"digest": "sha256:389dc939e2a414ef26b642b1e8657a710358afa6a6be3ebc4380b01ffacbbacc"
2024-12-18T00:21:51.1902065Z 		},
2024-12-18T00:21:51.1902265Z 		{
2024-12-18T00:21:51.1902622Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1903025Z 			"size": 174,
2024-12-18T00:21:51.1903424Z 			"digest": "sha256:0341eaeed33df462f8bd28003bb9b09739b0786abba58ccc17ba4cec535ee4ef"
2024-12-18T00:21:51.1903886Z 		},
2024-12-18T00:21:51.1904068Z 		{
2024-12-18T00:21:51.1904383Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1904780Z 			"size": 908,
2024-12-18T00:21:51.1905177Z 			"digest": "sha256:537cdb2d31685c0dbaf612d24633539a141cfcb69d90c5ce932923a6e4ab4474"
2024-12-18T00:21:51.1905625Z 		},
2024-12-18T00:21:51.1905813Z 		{
2024-12-18T00:21:51.1906131Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1906534Z 			"size": 701,
2024-12-18T00:21:51.1906952Z 			"digest": "sha256:bebd539a1d598fe86afc0da77bf16ae4c7ebfb9f031f21ed7634150fbe23f996"
2024-12-18T00:21:51.1907506Z 		},
2024-12-18T00:21:51.1907691Z 		{
2024-12-18T00:21:51.1908006Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1908409Z 			"size": 135,
2024-12-18T00:21:51.1908799Z 			"digest": "sha256:76d4a07024975dd727ff7b1c8873c513043bc9a5d310b469e51c3b110fdb6084"
2024-12-18T00:21:51.1909249Z 		},
2024-12-18T00:21:51.1909442Z 		{
2024-12-18T00:21:51.1909749Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1910148Z 			"size": 32,
2024-12-18T00:21:51.1910546Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-12-18T00:21:51.1911006Z 		},
2024-12-18T00:21:51.1911187Z 		{
2024-12-18T00:21:51.1911495Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1911898Z 			"size": 159,
2024-12-18T00:21:51.1912277Z 			"digest": "sha256:3287a539d61e3913b93941ecd49e55887b5122d63403e0fe9e7a3a63b6ed03c6"
2024-12-18T00:21:51.1912778Z 		},
2024-12-18T00:21:51.1912964Z 		{
2024-12-18T00:21:51.1913273Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1913673Z 			"size": 1932,
2024-12-18T00:21:51.1914060Z 			"digest": "sha256:9b763e24a7d42c731eb337815675c9478832146e73305559d0db1c88a0731492"
2024-12-18T00:21:51.1914503Z 		},
2024-12-18T00:21:51.1914682Z 		{
2024-12-18T00:21:51.1914994Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1915398Z 			"size": 196810750,
2024-12-18T00:21:51.1915801Z 			"digest": "sha256:f6ef3d4c814e330a230273579dc978970563774df75f73fba78181ddcf6ab98e"
2024-12-18T00:21:51.1916254Z 		},
2024-12-18T00:21:51.1916434Z 		{
2024-12-18T00:21:51.1916745Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1917147Z 			"size": 165,
2024-12-18T00:21:51.1917535Z 			"digest": "sha256:06914b00eb49b8005245cba4a631c34d0a2aaf8bf6291bc6477705206de11a06"
2024-12-18T00:21:51.1917994Z 		},
2024-12-18T00:21:51.1918266Z 		{
2024-12-18T00:21:51.1918587Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1918982Z 			"size": 7943,
2024-12-18T00:21:51.1919364Z 			"digest": "sha256:724553524224508e03de9422ca47b1a1c047e53cd305c2a1344147bb0d82d1d0"
2024-12-18T00:21:51.1919808Z 		},
2024-12-18T00:21:51.1919990Z 		{
2024-12-18T00:21:51.1920300Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1920711Z 			"size": 8070,
2024-12-18T00:21:51.1921116Z 			"digest": "sha256:47d9fb6dae26902b0fdb744916689adbd42add22a476d1c5ec89ae4e30b91e39"
2024-12-18T00:21:51.1921584Z 		},
2024-12-18T00:21:51.1921772Z 		{
2024-12-18T00:21:51.1922083Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1922486Z 			"size": 303,
2024-12-18T00:21:51.1922895Z 			"digest": "sha256:acce7b690dcb2fde77ba3ce65df70452fc7c98de73a42fe72652e477f86fc62d"
2024-12-18T00:21:51.1923366Z 		},
2024-12-18T00:21:51.1923553Z 		{
2024-12-18T00:21:51.1923871Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1924286Z 			"size": 7633763,
2024-12-18T00:21:51.1924700Z 			"digest": "sha256:58dc9f64def75bcb2abecdf39b798c3800e837d37da971e766a05731ea5537c2"
2024-12-18T00:21:51.1925174Z 		},
2024-12-18T00:21:51.1925360Z 		{
2024-12-18T00:21:51.1925678Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1926100Z 			"size": 108,
2024-12-18T00:21:51.1926509Z 			"digest": "sha256:94dfe5ef1b7e91ebd6a1a5acc1d7732e051334d5102d8a69c509c1d2f6e95598"
2024-12-18T00:21:51.1926985Z 		},
2024-12-18T00:21:51.1927166Z 		{
2024-12-18T00:21:51.1927490Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1927895Z 			"size": 54145665,
2024-12-18T00:21:51.1928296Z 			"digest": "sha256:40a2c9135b0683a7e2432748a9a6ff19926f6a2a49722d844c07e307320f7362"
2024-12-18T00:21:51.1928755Z 		},
2024-12-18T00:21:51.1928938Z 		{
2024-12-18T00:21:51.1929277Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1929782Z 			"size": 484,
2024-12-18T00:21:51.1930442Z 			"digest": "sha256:8839d6b0c8218eccb426367e5bfe066b172b8c6059bae8473e915e11c4e582a9"
2024-12-18T00:21:51.1930918Z 		},
2024-12-18T00:21:51.1931103Z 		{
2024-12-18T00:21:51.1931426Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1931839Z 			"size": 1374859271,
2024-12-18T00:21:51.1932255Z 			"digest": "sha256:1c588f0a9e97f3e192252a179d987d1ba1e5201ba4b6e74c429074e9e34df1b2"
2024-12-18T00:21:51.1932740Z 		},
2024-12-18T00:21:51.1932947Z 		{
2024-12-18T00:21:51.1933268Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1933673Z 			"size": 106,
2024-12-18T00:21:51.1934072Z 			"digest": "sha256:6727e7e279360b8b66f0137a15b357efede06e69bfd0800ebad6ed7c8ed71df4"
2024-12-18T00:21:51.1934539Z 		},
2024-12-18T00:21:51.1934718Z 		{
2024-12-18T00:21:51.1935041Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1935457Z 			"size": 568,
2024-12-18T00:21:51.1935861Z 			"digest": "sha256:3a55100d1957b103c43e314deb55697c459aa6fa225f91e7b1c69bc4c01dca4e"
2024-12-18T00:21:51.1936317Z 		},
2024-12-18T00:21:51.1936499Z 		{
2024-12-18T00:21:51.1936817Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1937223Z 			"size": 303706071,
2024-12-18T00:21:51.1937626Z 			"digest": "sha256:3d6f160d335161c5321e13ce2823ec68379a6a3401b07bd7378602a69c85b80c"
2024-12-18T00:21:51.1938085Z 		},
2024-12-18T00:21:51.1938262Z 		{
2024-12-18T00:21:51.1938578Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1938983Z 			"size": 111,
2024-12-18T00:21:51.1939379Z 			"digest": "sha256:b2a8bb947a87562144fe67a39c280d4dee50cbe9d8ff349140f5db6c377517e6"
2024-12-18T00:21:51.1939843Z 		},
2024-12-18T00:21:51.1940020Z 		{
2024-12-18T00:21:51.1940341Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1940755Z 			"size": 529,
2024-12-18T00:21:51.1941325Z 			"digest": "sha256:69a74b2360f4ede95654274ce7ef6cc1597a7234cc17e86d5cdf4c98286d97c3"
2024-12-18T00:21:51.1941795Z 		},
2024-12-18T00:21:51.1942032Z 		{
2024-12-18T00:21:51.1942363Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1942811Z 			"size": 48289052,
2024-12-18T00:21:51.1943219Z 			"digest": "sha256:f9e12a253d89c16863b9d4d90138ffa673447c9b7f27afcba54a2a679e58b5ab"
2024-12-18T00:21:51.1943678Z 		},
2024-12-18T00:21:51.1943859Z 		{
2024-12-18T00:21:51.1944178Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1944583Z 			"size": 106,
2024-12-18T00:21:51.1944984Z 			"digest": "sha256:8fdb06d51b96d538045b9566a381de3dc25e59d1adb9d0a4e567afcb3213f676"
2024-12-18T00:21:51.1945449Z 		},
2024-12-18T00:21:51.1945629Z 		{
2024-12-18T00:21:51.1945951Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1946354Z 			"size": 32,
2024-12-18T00:21:51.1946767Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-12-18T00:21:51.1947241Z 		},
2024-12-18T00:21:51.1947422Z 		{
2024-12-18T00:21:51.1947744Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1948154Z 			"size": 32,
2024-12-18T00:21:51.1948560Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-12-18T00:21:51.1949026Z 		},
2024-12-18T00:21:51.1949211Z 		{
2024-12-18T00:21:51.1949532Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1949937Z 			"size": 32,
2024-12-18T00:21:51.1950336Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-12-18T00:21:51.1950801Z 		},
2024-12-18T00:21:51.1950979Z 		{
2024-12-18T00:21:51.1951299Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:51.1951704Z 			"size": 32,
2024-12-18T00:21:51.1952112Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-12-18T00:21:51.1952698Z 		}
2024-12-18T00:21:51.1952887Z 	]
2024-12-18T00:21:51.1953081Z }
2024-12-18T00:21:51.1992406Z ##[group]Run set -eux
2024-12-18T00:21:51.1992705Z set -eux
2024-12-18T00:21:51.1993551Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin
2024-12-18T00:21:51.2003146Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:51.2003520Z env:
2024-12-18T00:21:51.2003738Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:51.2004000Z ##[endgroup]
2024-12-18T00:21:51.2036800Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token
2024-12-18T00:21:51.2037370Z + jq --raw-output .SecretString
2024-12-18T00:21:51.2038379Z + jq -r .docker_hub_readonly_token
2024-12-18T00:21:51.2039926Z + docker login --username pytorchbot --password-stdin
2024-12-18T00:21:51.8245052Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-12-18T00:21:51.8245683Z Configure a credential helper to remove this warning. See
2024-12-18T00:21:51.8246256Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-12-18T00:21:51.8246635Z 
2024-12-18T00:21:51.8246777Z Login Succeeded
2024-12-18T00:21:51.8342936Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/}
2024-12-18T00:21:51.8343296Z tag=${ECR_DOCKER_IMAGE##*/}
2024-12-18T00:21:51.8343686Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}"
2024-12-18T00:21:51.8352468Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:51.8352843Z env:
2024-12-18T00:21:51.8353057Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:51.8353899Z   ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:51.8354755Z ##[endgroup]
2024-12-18T00:21:51.8384718Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks-45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:51.8435336Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@release/2.6
2024-12-18T00:21:51.8435795Z with:
2024-12-18T00:21:51.8436594Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:51.8437574Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:51.8437968Z env:
2024-12-18T00:21:51.8438185Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:51.8438456Z ##[endgroup]
2024-12-18T00:21:51.8463263Z ##[group]Run set -x
2024-12-18T00:21:51.8463608Z set -x
2024-12-18T00:21:51.8463876Z set +e
2024-12-18T00:21:51.8464146Z 
2024-12-18T00:21:51.8464358Z login() {
2024-12-18T00:21:51.8464888Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2024-12-18T00:21:51.8465500Z }
2024-12-18T00:21:51.8465746Z 
2024-12-18T00:21:51.8465987Z retry () {
2024-12-18T00:21:51.8466262Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2024-12-18T00:21:51.8466569Z }
2024-12-18T00:21:51.8466785Z 
2024-12-18T00:21:51.8467019Z retry login "${DOCKER_REGISTRY}"
2024-12-18T00:21:51.8467321Z 
2024-12-18T00:21:51.8467529Z set -e
2024-12-18T00:21:51.8467857Z # ignore output since only exit code is used for conditional
2024-12-18T00:21:51.8468335Z # only pull docker image if it's not available locally
2024-12-18T00:21:51.8468873Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2024-12-18T00:21:51.8469366Z   retry docker pull "${DOCKER_IMAGE}"
2024-12-18T00:21:51.8469676Z fi
2024-12-18T00:21:51.8477700Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:51.8478236Z env:
2024-12-18T00:21:51.8478448Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:51.8479283Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:51.8480236Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:51.8480618Z ##[endgroup]
2024-12-18T00:21:51.8506319Z + set +e
2024-12-18T00:21:51.8506626Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:51.8507050Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:51.8510012Z + aws ecr get-login-password --region us-east-1
2024-12-18T00:21:51.8511826Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:52.4012331Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-12-18T00:21:52.4013248Z Configure a credential helper to remove this warning. See
2024-12-18T00:21:52.4013936Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-12-18T00:21:52.4014336Z 
2024-12-18T00:21:52.4014440Z Login Succeeded
2024-12-18T00:21:52.4036763Z + set -e
2024-12-18T00:21:52.4037909Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:52.4197727Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:52.4199225Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:52.6587146Z 45e1356b47a284893081276eff3000b7b534f3b1: Pulling from pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks
2024-12-18T00:21:52.6592521Z 80888bc6716f: Pulling fs layer
2024-12-18T00:21:52.6592978Z fbcd35dc5bc3: Pulling fs layer
2024-12-18T00:21:52.6593821Z c7232af9ae05: Pulling fs layer
2024-12-18T00:21:52.6594177Z db6cdef1932a: Pulling fs layer
2024-12-18T00:21:52.6594506Z 56dc85502937: Pulling fs layer
2024-12-18T00:21:52.6594882Z 30c0ea6140d0: Pulling fs layer
2024-12-18T00:21:52.6595272Z 71bdb1a72c2d: Pulling fs layer
2024-12-18T00:21:52.6595629Z 4829486be7c3: Pulling fs layer
2024-12-18T00:21:52.6595902Z 1f4e68d7b5e4: Pulling fs layer
2024-12-18T00:21:52.6596183Z 7c373e2d9b7e: Pulling fs layer
2024-12-18T00:21:52.6596457Z 622381141745: Pulling fs layer
2024-12-18T00:21:52.6596729Z db6cdef1932a: Waiting
2024-12-18T00:21:52.6596984Z f003d065e31e: Pulling fs layer
2024-12-18T00:21:52.6597255Z cd6ce7116815: Pulling fs layer
2024-12-18T00:21:52.6597530Z e843066a8f68: Pulling fs layer
2024-12-18T00:21:52.6597804Z bac50ca2930b: Pulling fs layer
2024-12-18T00:21:52.6598085Z 57e52c6751f4: Pulling fs layer
2024-12-18T00:21:52.6598377Z a9c5c14d77c0: Pulling fs layer
2024-12-18T00:21:52.6598653Z b86099b8619a: Pulling fs layer
2024-12-18T00:21:52.6598939Z 3c8ea8a600ff: Pulling fs layer
2024-12-18T00:21:52.6599219Z bebd539a1d59: Pulling fs layer
2024-12-18T00:21:52.6599529Z 56dc85502937: Waiting
2024-12-18T00:21:52.6599779Z 292b18ec2dab: Pulling fs layer
2024-12-18T00:21:52.6600058Z 76e3817b4b4c: Pulling fs layer
2024-12-18T00:21:52.6600317Z 30c0ea6140d0: Waiting
2024-12-18T00:21:52.6600563Z 50b1f245e9f5: Pulling fs layer
2024-12-18T00:21:52.6600827Z 71bdb1a72c2d: Waiting
2024-12-18T00:21:52.6601076Z ad93cf5205f2: Pulling fs layer
2024-12-18T00:21:52.6601336Z 4829486be7c3: Waiting
2024-12-18T00:21:52.6601582Z 8d17c59eaf7d: Pulling fs layer
2024-12-18T00:21:52.6601852Z 1f4e68d7b5e4: Waiting
2024-12-18T00:21:52.6602099Z b36a8ccf5c8c: Pulling fs layer
2024-12-18T00:21:52.6602367Z 7c373e2d9b7e: Waiting
2024-12-18T00:21:52.6602606Z aa0a3b87e9e3: Pulling fs layer
2024-12-18T00:21:52.6602871Z 622381141745: Waiting
2024-12-18T00:21:52.6603306Z 72a632052c5b: Pulling fs layer
2024-12-18T00:21:52.6603582Z ceed6316bc24: Pulling fs layer
2024-12-18T00:21:52.6603855Z f003d065e31e: Waiting
2024-12-18T00:21:52.6604102Z d0ca0f491f80: Pulling fs layer
2024-12-18T00:21:52.6604380Z cf02c706cd56: Pulling fs layer
2024-12-18T00:21:52.6604646Z cd6ce7116815: Waiting
2024-12-18T00:21:52.6604891Z 16073d3f107d: Pulling fs layer
2024-12-18T00:21:52.6605154Z e843066a8f68: Waiting
2024-12-18T00:21:52.6605381Z a9c5c14d77c0: Waiting
2024-12-18T00:21:52.6605629Z d234faf2f1bf: Pulling fs layer
2024-12-18T00:21:52.6605899Z bac50ca2930b: Waiting
2024-12-18T00:21:52.6622708Z 3f0ade86af1a: Pulling fs layer
2024-12-18T00:21:52.6623147Z b86099b8619a: Waiting
2024-12-18T00:21:52.6623487Z 57e52c6751f4: Waiting
2024-12-18T00:21:52.6623752Z 034b6a03f5f7: Pulling fs layer
2024-12-18T00:21:52.6624030Z 3c8ea8a600ff: Waiting
2024-12-18T00:21:52.6624284Z 21f8f066852d: Pulling fs layer
2024-12-18T00:21:52.6624563Z ad93cf5205f2: Waiting
2024-12-18T00:21:52.6624806Z 76e3817b4b4c: Waiting
2024-12-18T00:21:52.6625065Z 9dc1f315a979: Pulling fs layer
2024-12-18T00:21:52.6625339Z 8d17c59eaf7d: Waiting
2024-12-18T00:21:52.6625586Z bebd539a1d59: Waiting
2024-12-18T00:21:52.6625831Z 50b1f245e9f5: Waiting
2024-12-18T00:21:52.6626072Z b36a8ccf5c8c: Waiting
2024-12-18T00:21:52.6626317Z cd6a6063178b: Pulling fs layer
2024-12-18T00:21:52.6626590Z aa0a3b87e9e3: Waiting
2024-12-18T00:21:52.6626836Z ceed6316bc24: Waiting
2024-12-18T00:21:52.6627088Z 2116738f4ea9: Pulling fs layer
2024-12-18T00:21:52.6627356Z 72a632052c5b: Waiting
2024-12-18T00:21:52.6627587Z 292b18ec2dab: Waiting
2024-12-18T00:21:52.6627842Z d7fa134316d9: Pulling fs layer
2024-12-18T00:21:52.6628116Z d0ca0f491f80: Waiting
2024-12-18T00:21:52.6628356Z cf02c706cd56: Waiting
2024-12-18T00:21:52.6628609Z af03dc3fd15e: Pulling fs layer
2024-12-18T00:21:52.6628876Z d234faf2f1bf: Waiting
2024-12-18T00:21:52.6629116Z 16073d3f107d: Waiting
2024-12-18T00:21:52.6629357Z 3f0ade86af1a: Waiting
2024-12-18T00:21:52.6629595Z 2116738f4ea9: Waiting
2024-12-18T00:21:52.6629839Z 9dc1f315a979: Waiting
2024-12-18T00:21:52.6630351Z 034b6a03f5f7: Waiting
2024-12-18T00:21:52.6630817Z fd9de3dc4258: Pulling fs layer
2024-12-18T00:21:52.6631101Z cd6a6063178b: Waiting
2024-12-18T00:21:52.6631343Z 21f8f066852d: Waiting
2024-12-18T00:21:52.6631638Z b85093661f77: Pulling fs layer
2024-12-18T00:21:52.6631908Z af03dc3fd15e: Waiting
2024-12-18T00:21:52.6632139Z d7fa134316d9: Waiting
2024-12-18T00:21:52.6632390Z ab6ef6c2ac4d: Pulling fs layer
2024-12-18T00:21:52.6632670Z 56c896077120: Pulling fs layer
2024-12-18T00:21:52.6632936Z fd9de3dc4258: Waiting
2024-12-18T00:21:52.6633179Z d88e9c5e7c70: Pulling fs layer
2024-12-18T00:21:52.6633459Z 53617ee83302: Pulling fs layer
2024-12-18T00:21:52.6633738Z 3edbde10d404: Pulling fs layer
2024-12-18T00:21:52.6634016Z f4038f7524c3: Pulling fs layer
2024-12-18T00:21:52.6634288Z 4a51173f7757: Pulling fs layer
2024-12-18T00:21:52.6634552Z ed6a71d41274: Pulling fs layer
2024-12-18T00:21:52.6634828Z 389dc939e2a4: Pulling fs layer
2024-12-18T00:21:52.6635114Z 0341eaeed33d: Pulling fs layer
2024-12-18T00:21:52.6635391Z 537cdb2d3168: Pulling fs layer
2024-12-18T00:21:52.6635667Z ab6ef6c2ac4d: Waiting
2024-12-18T00:21:52.6635900Z f4038f7524c3: Waiting
2024-12-18T00:21:52.6636147Z 76d4a0702497: Pulling fs layer
2024-12-18T00:21:52.6636411Z 56c896077120: Waiting
2024-12-18T00:21:52.6636647Z 0341eaeed33d: Waiting
2024-12-18T00:21:52.6636888Z d88e9c5e7c70: Waiting
2024-12-18T00:21:52.6637117Z 4a51173f7757: Waiting
2024-12-18T00:21:52.6637351Z 389dc939e2a4: Waiting
2024-12-18T00:21:52.6637587Z 537cdb2d3168: Waiting
2024-12-18T00:21:52.6637839Z 4f4fb700ef54: Pulling fs layer
2024-12-18T00:21:52.6638112Z ed6a71d41274: Waiting
2024-12-18T00:21:52.6638342Z 53617ee83302: Waiting
2024-12-18T00:21:52.6638580Z 3edbde10d404: Waiting
2024-12-18T00:21:52.6638827Z 3287a539d61e: Pulling fs layer
2024-12-18T00:21:52.6639101Z 4f4fb700ef54: Waiting
2024-12-18T00:21:52.6639329Z 76d4a0702497: Waiting
2024-12-18T00:21:52.6639576Z 9b763e24a7d4: Pulling fs layer
2024-12-18T00:21:52.6640003Z f6ef3d4c814e: Pulling fs layer
2024-12-18T00:21:52.6640290Z 06914b00eb49: Pulling fs layer
2024-12-18T00:21:52.6640572Z 724553524224: Pulling fs layer
2024-12-18T00:21:52.6640843Z 47d9fb6dae26: Pulling fs layer
2024-12-18T00:21:52.6641126Z acce7b690dcb: Pulling fs layer
2024-12-18T00:21:52.6641415Z 58dc9f64def7: Pulling fs layer
2024-12-18T00:21:52.6641700Z 94dfe5ef1b7e: Pulling fs layer
2024-12-18T00:21:52.6641981Z 40a2c9135b06: Pulling fs layer
2024-12-18T00:21:52.6642256Z 8839d6b0c821: Pulling fs layer
2024-12-18T00:21:52.6642534Z 1c588f0a9e97: Pulling fs layer
2024-12-18T00:21:52.6642812Z 6727e7e27936: Pulling fs layer
2024-12-18T00:21:52.6643090Z 3a55100d1957: Pulling fs layer
2024-12-18T00:21:52.6643367Z 3d6f160d3351: Pulling fs layer
2024-12-18T00:21:52.6643642Z b2a8bb947a87: Pulling fs layer
2024-12-18T00:21:52.6643915Z 3287a539d61e: Waiting
2024-12-18T00:21:52.6644158Z acce7b690dcb: Waiting
2024-12-18T00:21:52.6644416Z 69a74b2360f4: Pulling fs layer
2024-12-18T00:21:52.6644684Z 58dc9f64def7: Waiting
2024-12-18T00:21:52.6644923Z 9b763e24a7d4: Waiting
2024-12-18T00:21:52.6645157Z 8839d6b0c821: Waiting
2024-12-18T00:21:52.6645405Z f9e12a253d89: Pulling fs layer
2024-12-18T00:21:52.6645677Z 94dfe5ef1b7e: Waiting
2024-12-18T00:21:52.6645923Z 1c588f0a9e97: Waiting
2024-12-18T00:21:52.6646153Z 6727e7e27936: Waiting
2024-12-18T00:21:52.6646391Z f6ef3d4c814e: Waiting
2024-12-18T00:21:52.6646628Z 724553524224: Waiting
2024-12-18T00:21:52.6646862Z 06914b00eb49: Waiting
2024-12-18T00:21:52.6647105Z 8fdb06d51b96: Pulling fs layer
2024-12-18T00:21:52.6647379Z 47d9fb6dae26: Waiting
2024-12-18T00:21:52.6647617Z 40a2c9135b06: Waiting
2024-12-18T00:21:52.6647852Z 3a55100d1957: Waiting
2024-12-18T00:21:52.6648093Z 3d6f160d3351: Waiting
2024-12-18T00:21:52.6648324Z 69a74b2360f4: Waiting
2024-12-18T00:21:52.6648563Z 8fdb06d51b96: Waiting
2024-12-18T00:21:52.7838096Z fbcd35dc5bc3: Verifying Checksum
2024-12-18T00:21:52.7839210Z fbcd35dc5bc3: Download complete
2024-12-18T00:21:52.8676024Z db6cdef1932a: Verifying Checksum
2024-12-18T00:21:52.8676477Z db6cdef1932a: Download complete
2024-12-18T00:21:52.9627107Z 56dc85502937: Download complete
2024-12-18T00:21:53.0166483Z 80888bc6716f: Verifying Checksum
2024-12-18T00:21:53.0166797Z 80888bc6716f: Download complete
2024-12-18T00:21:53.0954844Z 71bdb1a72c2d: Download complete
2024-12-18T00:21:53.1816816Z 4829486be7c3: Download complete
2024-12-18T00:21:53.2573084Z 1f4e68d7b5e4: Verifying Checksum
2024-12-18T00:21:53.2573539Z 1f4e68d7b5e4: Download complete
2024-12-18T00:21:53.2960085Z c7232af9ae05: Verifying Checksum
2024-12-18T00:21:53.2960532Z c7232af9ae05: Download complete
2024-12-18T00:21:53.4607734Z 622381141745: Download complete
2024-12-18T00:21:53.5502858Z f003d065e31e: Download complete
2024-12-18T00:21:54.1102243Z 80888bc6716f: Pull complete
2024-12-18T00:21:54.3938916Z fbcd35dc5bc3: Pull complete
2024-12-18T00:21:55.1412274Z c7232af9ae05: Pull complete
2024-12-18T00:21:55.1661070Z db6cdef1932a: Pull complete
2024-12-18T00:21:55.1967857Z 56dc85502937: Pull complete
2024-12-18T00:21:56.0826358Z cd6ce7116815: Verifying Checksum
2024-12-18T00:21:56.0826714Z cd6ce7116815: Download complete
2024-12-18T00:21:56.1710180Z e843066a8f68: Verifying Checksum
2024-12-18T00:21:56.1710573Z e843066a8f68: Download complete
2024-12-18T00:21:56.2920272Z bac50ca2930b: Download complete
2024-12-18T00:21:56.3782075Z 57e52c6751f4: Verifying Checksum
2024-12-18T00:21:56.3782522Z 57e52c6751f4: Download complete
2024-12-18T00:21:57.3590345Z a9c5c14d77c0: Verifying Checksum
2024-12-18T00:21:57.3590700Z a9c5c14d77c0: Download complete
2024-12-18T00:21:57.4456713Z b86099b8619a: Download complete
2024-12-18T00:21:57.5329116Z 3c8ea8a600ff: Verifying Checksum
2024-12-18T00:21:57.5329548Z 3c8ea8a600ff: Download complete
2024-12-18T00:21:57.6220622Z bebd539a1d59: Verifying Checksum
2024-12-18T00:21:57.6221067Z bebd539a1d59: Download complete
2024-12-18T00:21:57.7078360Z 292b18ec2dab: Download complete
2024-12-18T00:22:06.7737556Z 30c0ea6140d0: Verifying Checksum
2024-12-18T00:22:06.7737917Z 30c0ea6140d0: Download complete
2024-12-18T00:22:06.8685202Z 50b1f245e9f5: Verifying Checksum
2024-12-18T00:22:06.8685712Z 50b1f245e9f5: Download complete
2024-12-18T00:22:06.9687260Z ad93cf5205f2: Verifying Checksum
2024-12-18T00:22:06.9687718Z ad93cf5205f2: Download complete
2024-12-18T00:22:07.0539206Z 8d17c59eaf7d: Download complete
2024-12-18T00:22:07.1360034Z b36a8ccf5c8c: Verifying Checksum
2024-12-18T00:22:07.1360502Z b36a8ccf5c8c: Download complete
2024-12-18T00:22:07.3476462Z aa0a3b87e9e3: Verifying Checksum
2024-12-18T00:22:07.3476885Z aa0a3b87e9e3: Download complete
2024-12-18T00:22:08.6266491Z 72a632052c5b: Verifying Checksum
2024-12-18T00:22:08.6266933Z 72a632052c5b: Download complete
2024-12-18T00:22:08.7061703Z ceed6316bc24: Download complete
2024-12-18T00:22:08.8037588Z d0ca0f491f80: Download complete
2024-12-18T00:22:08.8782463Z cf02c706cd56: Verifying Checksum
2024-12-18T00:22:08.8782820Z cf02c706cd56: Download complete
2024-12-18T00:22:08.9688990Z 16073d3f107d: Verifying Checksum
2024-12-18T00:22:08.9689385Z 16073d3f107d: Download complete
2024-12-18T00:22:09.0385111Z d234faf2f1bf: Download complete
2024-12-18T00:22:13.4003069Z 3f0ade86af1a: Verifying Checksum
2024-12-18T00:22:13.4003555Z 3f0ade86af1a: Download complete
2024-12-18T00:22:13.4912199Z 034b6a03f5f7: Download complete
2024-12-18T00:22:13.5794050Z 21f8f066852d: Verifying Checksum
2024-12-18T00:22:13.5794417Z 21f8f066852d: Download complete
2024-12-18T00:22:14.0196347Z 9dc1f315a979: Verifying Checksum
2024-12-18T00:22:14.0196697Z 9dc1f315a979: Download complete
2024-12-18T00:22:14.1234117Z cd6a6063178b: Verifying Checksum
2024-12-18T00:22:14.1234510Z cd6a6063178b: Download complete
2024-12-18T00:22:14.2129656Z 2116738f4ea9: Download complete
2024-12-18T00:22:14.4837215Z d7fa134316d9: Verifying Checksum
2024-12-18T00:22:14.4837548Z d7fa134316d9: Download complete
2024-12-18T00:22:14.5768582Z af03dc3fd15e: Verifying Checksum
2024-12-18T00:22:14.5769006Z af03dc3fd15e: Download complete
2024-12-18T00:22:14.6595310Z fd9de3dc4258: Verifying Checksum
2024-12-18T00:22:14.6595682Z fd9de3dc4258: Download complete
2024-12-18T00:22:14.7416917Z b85093661f77: Verifying Checksum
2024-12-18T00:22:14.7417698Z b85093661f77: Download complete
2024-12-18T00:22:18.7396936Z 30c0ea6140d0: Pull complete
2024-12-18T00:22:18.9289801Z 71bdb1a72c2d: Pull complete
2024-12-18T00:22:19.0195049Z 4829486be7c3: Pull complete
2024-12-18T00:22:19.1922824Z 1f4e68d7b5e4: Pull complete
2024-12-18T00:22:19.5940070Z 7c373e2d9b7e: Verifying Checksum
2024-12-18T00:22:19.5940543Z 7c373e2d9b7e: Download complete
2024-12-18T00:22:19.6826211Z 56c896077120: Verifying Checksum
2024-12-18T00:22:19.6826555Z 56c896077120: Download complete
2024-12-18T00:22:19.7592833Z d88e9c5e7c70: Verifying Checksum
2024-12-18T00:22:19.7593330Z d88e9c5e7c70: Download complete
2024-12-18T00:22:20.2582316Z 53617ee83302: Verifying Checksum
2024-12-18T00:22:20.2582983Z 53617ee83302: Download complete
2024-12-18T00:22:20.3549947Z 3edbde10d404: Verifying Checksum
2024-12-18T00:22:20.3550395Z 3edbde10d404: Download complete
2024-12-18T00:22:20.4414567Z f4038f7524c3: Download complete
2024-12-18T00:22:20.5239649Z 4a51173f7757: Download complete
2024-12-18T00:22:20.6104028Z ed6a71d41274: Verifying Checksum
2024-12-18T00:22:20.6104621Z ed6a71d41274: Download complete
2024-12-18T00:22:27.1947138Z 76e3817b4b4c: Verifying Checksum
2024-12-18T00:22:27.1947523Z 76e3817b4b4c: Download complete
2024-12-18T00:22:27.2784993Z 0341eaeed33d: Download complete
2024-12-18T00:22:27.3715154Z 537cdb2d3168: Download complete
2024-12-18T00:22:27.4826449Z 76d4a0702497: Verifying Checksum
2024-12-18T00:22:27.4826818Z 76d4a0702497: Download complete
2024-12-18T00:22:27.4910654Z 4f4fb700ef54: Verifying Checksum
2024-12-18T00:22:27.4911169Z 4f4fb700ef54: Download complete
2024-12-18T00:22:27.5808813Z 3287a539d61e: Verifying Checksum
2024-12-18T00:22:27.5809296Z 3287a539d61e: Download complete
2024-12-18T00:22:27.6674504Z 9b763e24a7d4: Verifying Checksum
2024-12-18T00:22:27.6674974Z 9b763e24a7d4: Download complete
2024-12-18T00:22:29.6893656Z f6ef3d4c814e: Verifying Checksum
2024-12-18T00:22:29.6894326Z f6ef3d4c814e: Download complete
2024-12-18T00:22:29.7631381Z 06914b00eb49: Verifying Checksum
2024-12-18T00:22:29.7631788Z 06914b00eb49: Download complete
2024-12-18T00:22:29.8704199Z 724553524224: Download complete
2024-12-18T00:22:29.9461088Z 47d9fb6dae26: Verifying Checksum
2024-12-18T00:22:29.9461497Z 47d9fb6dae26: Download complete
2024-12-18T00:22:30.0219635Z acce7b690dcb: Download complete
2024-12-18T00:22:30.1600949Z 58dc9f64def7: Verifying Checksum
2024-12-18T00:22:30.1601410Z 58dc9f64def7: Download complete
2024-12-18T00:22:30.2384341Z 94dfe5ef1b7e: Download complete
2024-12-18T00:22:30.8284887Z 40a2c9135b06: Verifying Checksum
2024-12-18T00:22:30.8285230Z 40a2c9135b06: Download complete
2024-12-18T00:22:30.9040136Z 8839d6b0c821: Verifying Checksum
2024-12-18T00:22:30.9040479Z 8839d6b0c821: Download complete
2024-12-18T00:23:04.6458094Z 1c588f0a9e97: Verifying Checksum
2024-12-18T00:23:04.6458434Z 1c588f0a9e97: Download complete
2024-12-18T00:23:04.7445860Z 6727e7e27936: Verifying Checksum
2024-12-18T00:23:04.7446194Z 6727e7e27936: Download complete
2024-12-18T00:23:04.8499578Z 3a55100d1957: Verifying Checksum
2024-12-18T00:23:04.8499888Z 3a55100d1957: Download complete
2024-12-18T00:23:13.9843859Z 3d6f160d3351: Verifying Checksum
2024-12-18T00:23:13.9844324Z 3d6f160d3351: Download complete
2024-12-18T00:23:14.0867992Z b2a8bb947a87: Verifying Checksum
2024-12-18T00:23:14.0868327Z b2a8bb947a87: Download complete
2024-12-18T00:23:14.1759216Z 69a74b2360f4: Verifying Checksum
2024-12-18T00:23:14.1759541Z 69a74b2360f4: Download complete
2024-12-18T00:23:15.6950704Z f9e12a253d89: Verifying Checksum
2024-12-18T00:23:15.6951059Z f9e12a253d89: Download complete
2024-12-18T00:23:15.9145503Z 8fdb06d51b96: Download complete
2024-12-18T00:23:46.0484749Z ab6ef6c2ac4d: Verifying Checksum
2024-12-18T00:23:46.0487193Z ab6ef6c2ac4d: Download complete
2024-12-18T00:23:49.4925219Z 389dc939e2a4: Verifying Checksum
2024-12-18T00:23:49.4925545Z 389dc939e2a4: Download complete
2024-12-18T00:24:01.9032158Z 7c373e2d9b7e: Pull complete
2024-12-18T00:24:02.0965752Z 622381141745: Pull complete
2024-12-18T00:24:02.2898465Z f003d065e31e: Pull complete
2024-12-18T00:24:09.6505035Z cd6ce7116815: Pull complete
2024-12-18T00:24:09.7601715Z e843066a8f68: Pull complete
2024-12-18T00:24:09.8938473Z bac50ca2930b: Pull complete
2024-12-18T00:24:10.0875089Z 57e52c6751f4: Pull complete
2024-12-18T00:24:12.3538189Z a9c5c14d77c0: Pull complete
2024-12-18T00:24:12.5788789Z b86099b8619a: Pull complete
2024-12-18T00:24:12.8103971Z 3c8ea8a600ff: Pull complete
2024-12-18T00:24:13.0539211Z bebd539a1d59: Pull complete
2024-12-18T00:24:13.1715870Z 292b18ec2dab: Pull complete
2024-12-18T00:25:05.5605969Z 76e3817b4b4c: Pull complete
2024-12-18T00:25:05.6979834Z 50b1f245e9f5: Pull complete
2024-12-18T00:25:05.8639844Z ad93cf5205f2: Pull complete
2024-12-18T00:25:05.9845393Z 8d17c59eaf7d: Pull complete
2024-12-18T00:25:06.1070247Z b36a8ccf5c8c: Pull complete
2024-12-18T00:25:06.3184353Z aa0a3b87e9e3: Pull complete
2024-12-18T00:25:08.7555295Z 72a632052c5b: Pull complete
2024-12-18T00:25:08.9237525Z ceed6316bc24: Pull complete
2024-12-18T00:25:09.0597879Z d0ca0f491f80: Pull complete
2024-12-18T00:25:09.2585206Z cf02c706cd56: Pull complete
2024-12-18T00:25:09.3544172Z 16073d3f107d: Pull complete
2024-12-18T00:25:09.4965525Z d234faf2f1bf: Pull complete
2024-12-18T00:25:17.8735150Z 3f0ade86af1a: Pull complete
2024-12-18T00:25:18.0607915Z 034b6a03f5f7: Pull complete
2024-12-18T00:25:18.2508420Z 21f8f066852d: Pull complete
2024-12-18T00:25:19.0073187Z 9dc1f315a979: Pull complete
2024-12-18T00:25:19.1952350Z cd6a6063178b: Pull complete
2024-12-18T00:25:19.4045978Z 2116738f4ea9: Pull complete
2024-12-18T00:25:19.8104164Z d7fa134316d9: Pull complete
2024-12-18T00:25:19.9988720Z af03dc3fd15e: Pull complete
2024-12-18T00:25:20.3506414Z fd9de3dc4258: Pull complete
2024-12-18T00:25:20.5716645Z b85093661f77: Pull complete
2024-12-18T00:25:50.7675100Z ab6ef6c2ac4d: Pull complete
2024-12-18T00:25:50.9946324Z 56c896077120: Pull complete
2024-12-18T00:25:51.2342478Z d88e9c5e7c70: Pull complete
2024-12-18T00:25:53.8071386Z 53617ee83302: Pull complete
2024-12-18T00:25:54.0453609Z 3edbde10d404: Pull complete
2024-12-18T00:25:54.2689213Z f4038f7524c3: Pull complete
2024-12-18T00:25:54.7185475Z 4a51173f7757: Pull complete
2024-12-18T00:25:54.9518366Z ed6a71d41274: Pull complete
2024-12-18T00:27:25.7157272Z 389dc939e2a4: Pull complete
2024-12-18T00:27:25.9445973Z 0341eaeed33d: Pull complete
2024-12-18T00:27:26.1680582Z 537cdb2d3168: Pull complete
2024-12-18T00:27:26.5935275Z 76d4a0702497: Pull complete
2024-12-18T00:27:26.8268706Z 4f4fb700ef54: Pull complete
2024-12-18T00:27:27.0683435Z 3287a539d61e: Pull complete
2024-12-18T00:27:27.2974321Z 9b763e24a7d4: Pull complete
2024-12-18T00:27:34.1813743Z f6ef3d4c814e: Pull complete
2024-12-18T00:27:34.4065046Z 06914b00eb49: Pull complete
2024-12-18T00:27:34.6357470Z 724553524224: Pull complete
2024-12-18T00:27:34.8753000Z 47d9fb6dae26: Pull complete
2024-12-18T00:27:35.1045145Z acce7b690dcb: Pull complete
2024-12-18T00:27:36.4733878Z 58dc9f64def7: Pull complete
2024-12-18T00:27:36.7089587Z 94dfe5ef1b7e: Pull complete
2024-12-18T00:27:38.3981615Z 40a2c9135b06: Pull complete
2024-12-18T00:27:38.6314095Z 8839d6b0c821: Pull complete
2024-12-18T00:27:51.4899498Z 1c588f0a9e97: Pull complete
2024-12-18T00:27:51.7175266Z 6727e7e27936: Pull complete
2024-12-18T00:27:51.9624288Z 3a55100d1957: Pull complete
2024-12-18T00:27:54.7876069Z 3d6f160d3351: Pull complete
2024-12-18T00:27:55.0270368Z b2a8bb947a87: Pull complete
2024-12-18T00:27:55.2542898Z 69a74b2360f4: Pull complete
2024-12-18T00:27:55.8969807Z f9e12a253d89: Pull complete
2024-12-18T00:27:56.1321018Z 8fdb06d51b96: Pull complete
2024-12-18T00:27:57.0849116Z Digest: sha256:ff80d4c93d18e8ab7a2501f0e62b7c40cf97aa83fc9702e88b9c2d02cdcbecdb
2024-12-18T00:27:57.1330678Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:27:57.1615165Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:27:57.1672933Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T00:27:57.1673853Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T00:27:57.1684796Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:27:57.1685166Z env:
2024-12-18T00:27:57.1685386Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:27:57.1685646Z ##[endgroup]
2024-12-18T00:27:57.1959553Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@release/2.6
2024-12-18T00:27:57.1959992Z with:
2024-12-18T00:27:57.1960214Z   driver-version: 550.54.15
2024-12-18T00:27:57.1960467Z env:
2024-12-18T00:27:57.1960680Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:27:57.1960957Z ##[endgroup]
2024-12-18T00:27:57.2085476Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2024-12-18T00:27:57.2085904Z with:
2024-12-18T00:27:57.2086130Z   timeout_minutes: 10
2024-12-18T00:27:57.2086383Z   max_attempts: 3
2024-12-18T00:27:57.2110765Z   command: # Is it disgusting to have a full shell script here in this github action? Sure
# But is it the best way to make it so that this action relies on nothing else? Absolutely
set -eou pipefail

DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID)
DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run"

install_nvidia_docker2_amzn2() {
    (
        set -x
        # Needed for yum-config-manager
        sudo yum install -y yum-utils
        if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then
          YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo"
        else
          # Amazon Linux 2
          YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo"
        fi

        sudo yum-config-manager --add-repo "${YUM_REPO_URL}"
        sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2
        sudo systemctl restart docker
    )
}

install_nvidia_docker2_ubuntu20() {
    (
        set -x
        # Install nvidia-driver package if not installed
        status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)"
        if [ ! $? = 0 ] || [ ! "$status" = installed ]; then
          sudo apt-get install -y nvidia-docker2 nvidia-container-toolkit-1.16.2
          sudo systemctl restart docker
        fi
    )
}

pre_install_nvidia_driver_amzn2() {
    (
        # Purge any nvidia driver installed from RHEL repo
        sudo yum remove -y nvidia-driver-latest-dkms
    )
}

install_nvidia_driver_common() {
    (
        # Try to gather more information about the runner and its existing NVIDIA driver if any
        echo "Before installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        HAS_NVIDIA_DRIVER=0
        # Check if NVIDIA driver has already been installed
        if [ -x "$(command -v nvidia-smi)" ]; then
            set +e
            # The driver exists, check its version next. Also check only the first GPU if there are more than one of them
            # so that the same driver version is not print over multiple lines
            INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
            NVIDIA_SMI_STATUS=$?

            if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing"
            elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing"
            else
                HAS_NVIDIA_DRIVER=1
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation"
            fi
            set -e
        fi

        if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then
            # CAUTION: this may need to be updated in future
            if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then
                  sudo yum groupinstall -y "Development Tools"
                  # ensure our kernel install is the same as our underlying kernel,
                  # groupinstall "Development Tools" has a habit of mismatching kernel headers
                  sudo yum install -y "kernel-devel-uname-r == $(uname -r)"
                  sudo modprobe backlight
            fi
            sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN"

            set +e
            sudo /bin/bash /tmp/nvidia_driver -s --no-drm
            NVIDIA_INSTALLATION_STATUS=$?

            RESET_GPU=0
            if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then
                sudo cat /var/log/nvidia-installer.log
                # Fail to install NVIDIA driver, try to reset the GPU
                RESET_GPU=1
            elif [ -x "$(command -v nvidia-smi)" ]; then
                # Check again if nvidia-smi works even if the driver installation completes successfully
                INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
                NVIDIA_SMI_STATUS=$?

                if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                    RESET_GPU=1
                fi
            fi

            if [ "$RESET_GPU" -eq 1 ]; then
                NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1)
                # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this
                # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388
                for PCI_ID in $NVIDIA_DEVICES; do
                    DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable)

                    echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)"
                    # This requires sudo permission of course
                    echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset
                    sleep 1
                done
            fi

            sudo rm -fv /tmp/nvidia_driver
            set -e
        fi
    )
}

post_install_nvidia_driver_common() {
    (
        sudo modprobe nvidia || true
        echo "After installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        (
            set +e

            nvidia-smi
            # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in
            # the case where the driver has already crashed as it still can get the driver version
            # and some basic information like the bus ID.  However, the rest of the information
            # would be missing (ERR!), for example:
            #
            # +-----------------------------------------------------------------------------+
            # | NVIDIA-SMI 525.89.02    Driver Version: 525.89.02    CUDA Version: 12.0     |
            # |-------------------------------+----------------------+----------------------+
            # | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
            # | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
            # |                               |                      |               MIG M. |
            # |===============================+======================+======================|
            # |   0  ERR!                Off  | 00000000:00:1E.0 Off |                 ERR! |
            # |ERR!  ERR! ERR!    ERR! / ERR! |   4184MiB / 23028MiB |    ERR!      Default |
            # |                               |                      |                 ERR! |
            # +-------------------------------+----------------------+----------------------+
            #
            # +-----------------------------------------------------------------------------+
            # | Processes:                                                                  |
            # |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
            # |        ID   ID                                                   Usage      |
            # |=============================================================================|
            # +-----------------------------------------------------------------------------+
            #
            # This should be reported as a failure instead as it will guarantee to fail when
            # Docker tries to run with --gpus all
            #
            # So, the correct check here is to query one of the missing piece of info like
            # GPU name, so that the command can fail accordingly
            nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0
            NVIDIA_SMI_STATUS=$?

            # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285
            if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then
                echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}"
            else
                echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}"
                exit ${NVIDIA_SMI_STATUS}
            fi
            set -e
        )
    )
}

install_nvidia_driver_amzn2() {
    (
        set -x
        pre_install_nvidia_driver_amzn2
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

install_nvidia_driver_ubuntu20() {
    (
        set -x
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

echo "== Installing nvidia driver ${DRIVER_FN} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_driver_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_driver_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac

# Install container toolkit based on distribution
echo "== Installing nvidia container toolkit for ${DISTRIBUTION} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_docker2_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_docker2_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac
echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}"

# Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with
# more than one GPUs. This just needs to be run once. The command fails
# on subsequent runs and complains that the mode is already on, but that's
# ok
sudo nvidia-persistenced || true
# This should show persistence mode ON
nvidia-smi

2024-12-18T00:27:57.2136156Z   retry_wait_seconds: 10
2024-12-18T00:27:57.2136441Z   polling_interval_seconds: 1
2024-12-18T00:27:57.2136727Z   warning_on_retry: true
2024-12-18T00:27:57.2136999Z   continue_on_error: false
2024-12-18T00:27:57.2137262Z env:
2024-12-18T00:27:57.2137478Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:27:57.2137748Z   DRIVER_VERSION: 550.54.15
2024-12-18T00:27:57.2138016Z ##[endgroup]
2024-12-18T00:27:57.3026556Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run ==
2024-12-18T00:27:57.3027549Z + pre_install_nvidia_driver_amzn2
2024-12-18T00:27:57.3029251Z + sudo yum remove -y nvidia-driver-latest-dkms
2024-12-18T00:27:58.1100520Z No match for argument: nvidia-driver-latest-dkms
2024-12-18T00:27:58.1101262Z No packages marked for removal.
2024-12-18T00:27:58.1310862Z Dependencies resolved.
2024-12-18T00:27:58.1320735Z Nothing to do.
2024-12-18T00:27:58.1321164Z Complete!
2024-12-18T00:27:58.2097471Z + install_nvidia_driver_common
2024-12-18T00:27:58.2100537Z + echo 'Before installing NVIDIA driver'
2024-12-18T00:27:58.2101592Z Before installing NVIDIA driver
2024-12-18T00:27:58.2104044Z + lspci
2024-12-18T00:27:58.2226042Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma]
2024-12-18T00:27:58.2226768Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2024-12-18T00:27:58.2227435Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
2024-12-18T00:27:58.2228002Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111
2024-12-18T00:27:58.2228699Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller
2024-12-18T00:27:58.2229423Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2024-12-18T00:27:58.2229946Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1)
2024-12-18T00:27:58.2230696Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller
2024-12-18T00:27:58.2231366Z + lsmod
2024-12-18T00:27:58.2275865Z Module                  Size  Used by
2024-12-18T00:27:58.2276345Z ib_core               454656  0
2024-12-18T00:27:58.2276717Z xt_nat                 16384  0
2024-12-18T00:27:58.2277093Z nvidia_modeset       1351680  0
2024-12-18T00:27:58.2277419Z video                  65536  1 nvidia_modeset
2024-12-18T00:27:58.2277814Z wmi                    36864  1 video
2024-12-18T00:27:58.2278217Z nvidia_uvm           4706304  0
2024-12-18T00:27:58.2278647Z nvidia              54071296  7 nvidia_uvm,nvidia_modeset
2024-12-18T00:27:58.2279115Z drm                   602112  1 nvidia
2024-12-18T00:27:58.2279442Z drm_panel_orientation_quirks    32768  1 drm
2024-12-18T00:27:58.2279835Z backlight              24576  3 video,drm,nvidia_modeset
2024-12-18T00:27:58.2280206Z i2c_core              106496  2 nvidia,drm
2024-12-18T00:27:58.2280514Z veth                   36864  0
2024-12-18T00:27:58.2280791Z xt_conntrack           16384  1
2024-12-18T00:27:58.2281081Z nft_chain_nat          16384  3
2024-12-18T00:27:58.2281355Z xt_MASQUERADE          20480  1
2024-12-18T00:27:58.2281689Z nf_nat                 57344  3 xt_nat,nft_chain_nat,xt_MASQUERADE
2024-12-18T00:27:58.2282056Z nf_conntrack_netlink    57344  0
2024-12-18T00:27:58.2282710Z nf_conntrack          184320  5 xt_conntrack,nf_nat,xt_nat,nf_conntrack_netlink,xt_MASQUERADE
2024-12-18T00:27:58.2283215Z nf_defrag_ipv6         24576  1 nf_conntrack
2024-12-18T00:27:58.2283548Z nf_defrag_ipv4         16384  1 nf_conntrack
2024-12-18T00:27:58.2283862Z xfrm_user              57344  1
2024-12-18T00:27:58.2284152Z xfrm_algo              16384  1 xfrm_user
2024-12-18T00:27:58.2284462Z xt_addrtype            16384  2
2024-12-18T00:27:58.2284738Z nft_compat             20480  4
2024-12-18T00:27:58.2285056Z nf_tables             311296  57 nft_compat,nft_chain_nat
2024-12-18T00:27:58.2285495Z nfnetlink              20480  4 nft_compat,nf_conntrack_netlink,nf_tables
2024-12-18T00:27:58.2285893Z br_netfilter           36864  0
2024-12-18T00:27:58.2286192Z bridge                323584  1 br_netfilter
2024-12-18T00:27:58.2286504Z stp                    16384  1 bridge
2024-12-18T00:27:58.2286800Z llc                    16384  2 bridge,stp
2024-12-18T00:27:58.2287098Z overlay               167936  0
2024-12-18T00:27:58.2287370Z tls                   114688  0
2024-12-18T00:27:58.2287633Z nls_ascii              16384  1
2024-12-18T00:27:58.2287900Z nls_cp437              20480  1
2024-12-18T00:27:58.2288157Z vfat                   24576  1
2024-12-18T00:27:58.2288420Z sunrpc                692224  1
2024-12-18T00:27:58.2288695Z fat                    86016  1 vfat
2024-12-18T00:27:58.2288988Z ena                   167936  0
2024-12-18T00:27:58.2289266Z i8042                  45056  0
2024-12-18T00:27:58.2289567Z serio                  28672  3 i8042
2024-12-18T00:27:58.2289853Z button                 24576  0
2024-12-18T00:27:58.2290125Z ghash_clmulni_intel    16384  0
2024-12-18T00:27:58.2290406Z sch_fq_codel           20480  17
2024-12-18T00:27:58.2290683Z dm_mod                188416  0
2024-12-18T00:27:58.2290942Z fuse                  163840  1
2024-12-18T00:27:58.2291204Z loop                   36864  0
2024-12-18T00:27:58.2291472Z dax                    45056  1 dm_mod
2024-12-18T00:27:58.2291770Z configfs               57344  1
2024-12-18T00:27:58.2292045Z dmi_sysfs              20480  0
2024-12-18T00:27:58.2292313Z crc32_pclmul           16384  0
2024-12-18T00:27:58.2292586Z crc32c_intel           24576  0
2024-12-18T00:27:58.2292858Z efivarfs               24576  1
2024-12-18T00:27:58.2293143Z + modinfo nvidia
2024-12-18T00:27:58.2293559Z filename:       /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko
2024-12-18T00:27:58.2294044Z alias:          char-major-195-*
2024-12-18T00:27:58.2294334Z version:        550.54.15
2024-12-18T00:27:58.2294600Z supported:      external
2024-12-18T00:27:58.2294859Z license:        NVIDIA
2024-12-18T00:27:58.2295149Z firmware:       nvidia/550.54.15/gsp_tu10x.bin
2024-12-18T00:27:58.2295604Z firmware:       nvidia/550.54.15/gsp_ga10x.bin
2024-12-18T00:27:58.2295951Z srcversion:     833721318DA517F0C2FEC97
2024-12-18T00:27:58.2296291Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2024-12-18T00:27:58.2296653Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2024-12-18T00:27:58.2297020Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2024-12-18T00:27:58.2297354Z depends:        i2c-core,drm
2024-12-18T00:27:58.2297624Z retpoline:      Y
2024-12-18T00:27:58.2297862Z name:           nvidia
2024-12-18T00:27:58.2298248Z vermagic:       6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 
2024-12-18T00:27:58.2298753Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2024-12-18T00:27:58.2299262Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2024-12-18T00:27:58.2299712Z parm:           NVreg_ResmanDebugLevel:int
2024-12-18T00:27:58.2300039Z parm:           NVreg_RmLogonRC:int
2024-12-18T00:27:58.2300363Z parm:           NVreg_ModifyDeviceFiles:int
2024-12-18T00:27:58.2300698Z parm:           NVreg_DeviceFileUID:int
2024-12-18T00:27:58.2301021Z parm:           NVreg_DeviceFileGID:int
2024-12-18T00:27:58.2301339Z parm:           NVreg_DeviceFileMode:int
2024-12-18T00:27:58.2301724Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2024-12-18T00:27:58.2302342Z parm:           NVreg_UsePageAttributeTable:int
2024-12-18T00:27:58.2302696Z parm:           NVreg_EnablePCIeGen3:int
2024-12-18T00:27:58.2303013Z parm:           NVreg_EnableMSI:int
2024-12-18T00:27:58.2303313Z parm:           NVreg_TCEBypassMode:int
2024-12-18T00:27:58.2303649Z parm:           NVreg_EnableStreamMemOPs:int
2024-12-18T00:27:58.2304035Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2024-12-18T00:27:58.2304459Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2024-12-18T00:27:58.2304868Z parm:           NVreg_EnableS0ixPowerManagement:int
2024-12-18T00:27:58.2305298Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2024-12-18T00:27:58.2305739Z parm:           NVreg_DynamicPowerManagement:int
2024-12-18T00:27:58.2306188Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2024-12-18T00:27:58.2306629Z parm:           NVreg_EnableGpuFirmware:int
2024-12-18T00:27:58.2306986Z parm:           NVreg_EnableGpuFirmwareLogs:int
2024-12-18T00:27:58.2307385Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2024-12-18T00:27:58.2307785Z parm:           NVreg_EnableUserNUMAManagement:int
2024-12-18T00:27:58.2308147Z parm:           NVreg_MemoryPoolSize:int
2024-12-18T00:27:58.2308491Z parm:           NVreg_KMallocHeapMaxSize:int
2024-12-18T00:27:58.2308844Z parm:           NVreg_VMallocHeapMaxSize:int
2024-12-18T00:27:58.2309192Z parm:           NVreg_IgnoreMMIOCheck:int
2024-12-18T00:27:58.2309518Z parm:           NVreg_NvLinkDisable:int
2024-12-18T00:27:58.2309898Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2024-12-18T00:27:58.2310287Z parm:           NVreg_RegisterPCIDriver:int
2024-12-18T00:27:58.2310645Z parm:           NVreg_EnableResizableBar:int
2024-12-18T00:27:58.2311003Z parm:           NVreg_EnableDbgBreakpoint:int
2024-12-18T00:27:58.2311367Z parm:           NVreg_EnableNonblockingOpen:int
2024-12-18T00:27:58.2311732Z parm:           NVreg_RegistryDwords:charp
2024-12-18T00:27:58.2312105Z parm:           NVreg_RegistryDwordsPerDevice:charp
2024-12-18T00:27:58.2312457Z parm:           NVreg_RmMsg:charp
2024-12-18T00:27:58.2312765Z parm:           NVreg_GpuBlacklist:charp
2024-12-18T00:27:58.2313107Z parm:           NVreg_TemporaryFilePath:charp
2024-12-18T00:27:58.2313455Z parm:           NVreg_ExcludedGpus:charp
2024-12-18T00:27:58.2313792Z parm:           NVreg_DmaRemapPeerMmio:int
2024-12-18T00:27:58.2314144Z parm:           NVreg_RmNvlinkBandwidth:charp
2024-12-18T00:27:58.2314495Z parm:           NVreg_ImexChannelCount:int
2024-12-18T00:27:58.2314824Z parm:           rm_firmware_active:charp
2024-12-18T00:27:58.2315146Z + HAS_NVIDIA_DRIVER=0
2024-12-18T00:27:58.2315404Z ++ command -v nvidia-smi
2024-12-18T00:27:58.2315772Z + '[' -x /usr/bin/nvidia-smi ']'
2024-12-18T00:27:58.2316044Z + set +e
2024-12-18T00:27:58.2316369Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0
2024-12-18T00:27:58.2571726Z + INSTALLED_DRIVER_VERSION=550.54.15
2024-12-18T00:27:58.2572049Z + NVIDIA_SMI_STATUS=0
2024-12-18T00:27:58.2573362Z + '[' 0 -ne 0 ']'
2024-12-18T00:27:58.2573603Z + '[' 550.54.15 '!=' 550.54.15 ']'
2024-12-18T00:27:58.2573879Z + HAS_NVIDIA_DRIVER=1
2024-12-18T00:27:58.2574327Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation'
2024-12-18T00:27:58.2574841Z + set -e
2024-12-18T00:27:58.2575047Z + '[' 1 -eq 0 ']'
2024-12-18T00:27:58.2575454Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation
2024-12-18T00:27:58.2575967Z + post_install_nvidia_driver_common
2024-12-18T00:27:58.2579379Z + sudo modprobe nvidia
2024-12-18T00:27:58.3628610Z + echo 'After installing NVIDIA driver'
2024-12-18T00:27:58.3629100Z + lspci
2024-12-18T00:27:58.3629328Z After installing NVIDIA driver
2024-12-18T00:27:58.3747190Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma]
2024-12-18T00:27:58.3747730Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2024-12-18T00:27:58.3748532Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
2024-12-18T00:27:58.3749092Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111
2024-12-18T00:27:58.3749583Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller
2024-12-18T00:27:58.3750135Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2024-12-18T00:27:58.3750643Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1)
2024-12-18T00:27:58.3751142Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller
2024-12-18T00:27:58.3751566Z + lsmod
2024-12-18T00:27:58.3780416Z Module                  Size  Used by
2024-12-18T00:27:58.3780763Z ib_core               454656  0
2024-12-18T00:27:58.3781118Z xt_nat                 16384  0
2024-12-18T00:27:58.3781397Z nvidia_modeset       1351680  0
2024-12-18T00:27:58.3781696Z video                  65536  1 nvidia_modeset
2024-12-18T00:27:58.3782012Z wmi                    36864  1 video
2024-12-18T00:27:58.3782387Z nvidia_uvm           4706304  0
2024-12-18T00:27:58.3782694Z nvidia              54071296  7 nvidia_uvm,nvidia_modeset
2024-12-18T00:27:58.3783036Z drm                   602112  1 nvidia
2024-12-18T00:27:58.3783359Z drm_panel_orientation_quirks    32768  1 drm
2024-12-18T00:27:58.3783736Z backlight              24576  3 video,drm,nvidia_modeset
2024-12-18T00:27:58.3784094Z i2c_core              106496  2 nvidia,drm
2024-12-18T00:27:58.3784383Z veth                   36864  0
2024-12-18T00:27:58.3784650Z xt_conntrack           16384  1
2024-12-18T00:27:58.3784916Z nft_chain_nat          16384  3
2024-12-18T00:27:58.3785188Z xt_MASQUERADE          20480  1
2024-12-18T00:27:58.3785516Z nf_nat                 57344  3 xt_nat,nft_chain_nat,xt_MASQUERADE
2024-12-18T00:27:58.3785866Z nf_conntrack_netlink    57344  0
2024-12-18T00:27:58.3786303Z nf_conntrack          184320  5 xt_conntrack,nf_nat,xt_nat,nf_conntrack_netlink,xt_MASQUERADE
2024-12-18T00:27:58.3786800Z nf_defrag_ipv6         24576  1 nf_conntrack
2024-12-18T00:27:58.3787128Z nf_defrag_ipv4         16384  1 nf_conntrack
2024-12-18T00:27:58.3787433Z xfrm_user              57344  1
2024-12-18T00:27:58.3787709Z xfrm_algo              16384  1 xfrm_user
2024-12-18T00:27:58.3788008Z xt_addrtype            16384  2
2024-12-18T00:27:58.3788276Z nft_compat             20480  4
2024-12-18T00:27:58.3788592Z nf_tables             311296  57 nft_compat,nft_chain_nat
2024-12-18T00:27:58.3789021Z nfnetlink              20480  4 nft_compat,nf_conntrack_netlink,nf_tables
2024-12-18T00:27:58.3789402Z br_netfilter           36864  0
2024-12-18T00:27:58.3789691Z bridge                323584  1 br_netfilter
2024-12-18T00:27:58.3790185Z stp                    16384  1 bridge
2024-12-18T00:27:58.3790493Z llc                    16384  2 bridge,stp
2024-12-18T00:27:58.3790786Z overlay               167936  0
2024-12-18T00:27:58.3791039Z tls                   114688  0
2024-12-18T00:27:58.3791301Z nls_ascii              16384  1
2024-12-18T00:27:58.3791566Z nls_cp437              20480  1
2024-12-18T00:27:58.3791825Z vfat                   24576  1
2024-12-18T00:27:58.3792083Z sunrpc                692224  1
2024-12-18T00:27:58.3792338Z fat                    86016  1 vfat
2024-12-18T00:27:58.3792630Z ena                   167936  0
2024-12-18T00:27:58.3792883Z i8042                  45056  0
2024-12-18T00:27:58.3793153Z serio                  28672  3 i8042
2024-12-18T00:27:58.3793437Z button                 24576  0
2024-12-18T00:27:58.3793699Z ghash_clmulni_intel    16384  0
2024-12-18T00:27:58.3793973Z sch_fq_codel           20480  17
2024-12-18T00:27:58.3794242Z dm_mod                188416  0
2024-12-18T00:27:58.3794500Z fuse                  163840  1
2024-12-18T00:27:58.3794766Z loop                   36864  0
2024-12-18T00:27:58.3795023Z dax                    45056  1 dm_mod
2024-12-18T00:27:58.3795307Z configfs               57344  1
2024-12-18T00:27:58.3795579Z dmi_sysfs              20480  0
2024-12-18T00:27:58.3795940Z crc32_pclmul           16384  0
2024-12-18T00:27:58.3796207Z crc32c_intel           24576  0
2024-12-18T00:27:58.3796464Z efivarfs               24576  1
2024-12-18T00:27:58.3796727Z + modinfo nvidia
2024-12-18T00:27:58.3797451Z filename:       /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko
2024-12-18T00:27:58.3797934Z alias:          char-major-195-*
2024-12-18T00:27:58.3798232Z version:        550.54.15
2024-12-18T00:27:58.3798488Z supported:      external
2024-12-18T00:27:58.3798750Z license:        NVIDIA
2024-12-18T00:27:58.3799055Z firmware:       nvidia/550.54.15/gsp_tu10x.bin
2024-12-18T00:27:58.3799409Z firmware:       nvidia/550.54.15/gsp_ga10x.bin
2024-12-18T00:27:58.3799743Z srcversion:     833721318DA517F0C2FEC97
2024-12-18T00:27:58.3800072Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2024-12-18T00:27:58.3800416Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2024-12-18T00:27:58.3800763Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2024-12-18T00:27:58.3801091Z depends:        i2c-core,drm
2024-12-18T00:27:58.3801358Z retpoline:      Y
2024-12-18T00:27:58.3801585Z name:           nvidia
2024-12-18T00:27:58.3801946Z vermagic:       6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 
2024-12-18T00:27:58.3802434Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2024-12-18T00:27:58.3802896Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2024-12-18T00:27:58.3803338Z parm:           NVreg_ResmanDebugLevel:int
2024-12-18T00:27:58.3803655Z parm:           NVreg_RmLogonRC:int
2024-12-18T00:27:58.3803962Z parm:           NVreg_ModifyDeviceFiles:int
2024-12-18T00:27:58.3804287Z parm:           NVreg_DeviceFileUID:int
2024-12-18T00:27:58.3804602Z parm:           NVreg_DeviceFileGID:int
2024-12-18T00:27:58.3804919Z parm:           NVreg_DeviceFileMode:int
2024-12-18T00:27:58.3805298Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2024-12-18T00:27:58.3805693Z parm:           NVreg_UsePageAttributeTable:int
2024-12-18T00:27:58.3806044Z parm:           NVreg_EnablePCIeGen3:int
2024-12-18T00:27:58.3806360Z parm:           NVreg_EnableMSI:int
2024-12-18T00:27:58.3806664Z parm:           NVreg_TCEBypassMode:int
2024-12-18T00:27:58.3806996Z parm:           NVreg_EnableStreamMemOPs:int
2024-12-18T00:27:58.3807368Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2024-12-18T00:27:58.3807783Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2024-12-18T00:27:58.3808180Z parm:           NVreg_EnableS0ixPowerManagement:int
2024-12-18T00:27:58.3808658Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2024-12-18T00:27:58.3809087Z parm:           NVreg_DynamicPowerManagement:int
2024-12-18T00:27:58.3809627Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2024-12-18T00:27:58.3810051Z parm:           NVreg_EnableGpuFirmware:int
2024-12-18T00:27:58.3810406Z parm:           NVreg_EnableGpuFirmwareLogs:int
2024-12-18T00:27:58.3810791Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2024-12-18T00:27:58.3811188Z parm:           NVreg_EnableUserNUMAManagement:int
2024-12-18T00:27:58.3811543Z parm:           NVreg_MemoryPoolSize:int
2024-12-18T00:27:58.3811876Z parm:           NVreg_KMallocHeapMaxSize:int
2024-12-18T00:27:58.3812221Z parm:           NVreg_VMallocHeapMaxSize:int
2024-12-18T00:27:58.3812563Z parm:           NVreg_IgnoreMMIOCheck:int
2024-12-18T00:27:58.3812892Z parm:           NVreg_NvLinkDisable:int
2024-12-18T00:27:58.3813255Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2024-12-18T00:27:58.3813623Z parm:           NVreg_RegisterPCIDriver:int
2024-12-18T00:27:58.3813975Z parm:           NVreg_EnableResizableBar:int
2024-12-18T00:27:58.3814324Z parm:           NVreg_EnableDbgBreakpoint:int
2024-12-18T00:27:58.3814690Z parm:           NVreg_EnableNonblockingOpen:int
2024-12-18T00:27:58.3815044Z parm:           NVreg_RegistryDwords:charp
2024-12-18T00:27:58.3815392Z parm:           NVreg_RegistryDwordsPerDevice:charp
2024-12-18T00:27:58.3815732Z parm:           NVreg_RmMsg:charp
2024-12-18T00:27:58.3816112Z parm:           NVreg_GpuBlacklist:charp
2024-12-18T00:27:58.3816447Z parm:           NVreg_TemporaryFilePath:charp
2024-12-18T00:27:58.3816787Z parm:           NVreg_ExcludedGpus:charp
2024-12-18T00:27:58.3817109Z parm:           NVreg_DmaRemapPeerMmio:int
2024-12-18T00:27:58.3817452Z parm:           NVreg_RmNvlinkBandwidth:charp
2024-12-18T00:27:58.3817796Z parm:           NVreg_ImexChannelCount:int
2024-12-18T00:27:58.3818121Z parm:           rm_firmware_active:charp
2024-12-18T00:27:58.3818418Z + set +e
2024-12-18T00:27:58.3818615Z + nvidia-smi
2024-12-18T00:27:58.4010937Z Wed Dec 18 00:27:58 2024       
2024-12-18T00:27:58.4011720Z +-----------------------------------------------------------------------------------------+
2024-12-18T00:27:58.4012762Z | NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
2024-12-18T00:27:58.4013787Z |-----------------------------------------+------------------------+----------------------+
2024-12-18T00:27:58.4014826Z | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
2024-12-18T00:27:58.4015953Z | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
2024-12-18T00:27:58.4016853Z |                                         |                        |               MIG M. |
2024-12-18T00:27:58.4017549Z |=========================================+========================+======================|
2024-12-18T00:27:58.4293504Z |   0  NVIDIA A10G                    On  |   00000000:00:1E.0 Off |                    0 |
2024-12-18T00:27:58.4294458Z |  0%   22C    P8             16W /  300W |       0MiB /  23028MiB |      0%      Default |
2024-12-18T00:27:58.4295289Z |                                         |                        |                  N/A |
2024-12-18T00:27:58.4296116Z +-----------------------------------------+------------------------+----------------------+
2024-12-18T00:27:58.4297893Z                                                                                          
2024-12-18T00:27:58.4298621Z +-----------------------------------------------------------------------------------------+
2024-12-18T00:27:58.4299086Z | Processes:                                                                              |
2024-12-18T00:27:58.4299580Z |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
2024-12-18T00:27:58.4300044Z |        ID   ID                                                               Usage      |
2024-12-18T00:27:58.4300418Z |=========================================================================================|
2024-12-18T00:27:58.4305507Z |  No running processes found                                                             |
2024-12-18T00:27:58.4306008Z +-----------------------------------------------------------------------------------------+
2024-12-18T00:27:58.7271898Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0
2024-12-18T00:27:58.7467775Z NVIDIA A10G
2024-12-18T00:27:58.7535164Z + NVIDIA_SMI_STATUS=0
2024-12-18T00:27:58.7535467Z + '[' 0 -eq 0 ']'
2024-12-18T00:27:58.7535731Z + echo 'INFO: Ignoring allowed status 0'
2024-12-18T00:27:58.7536037Z + set -e
2024-12-18T00:27:58.7536270Z INFO: Ignoring allowed status 0
2024-12-18T00:27:58.7544933Z == Installing nvidia container toolkit for amzn2023 ==
2024-12-18T00:27:58.7548825Z + sudo yum install -y yum-utils
2024-12-18T00:27:59.4148255Z Last metadata expiration check: 2:41:05 ago on Tue Dec 17 21:46:54 2024.
2024-12-18T00:27:59.4373133Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed.
2024-12-18T00:27:59.4707791Z Dependencies resolved.
2024-12-18T00:27:59.4846996Z Nothing to do.
2024-12-18T00:27:59.4847280Z Complete!
2024-12-18T00:27:59.5434926Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]]
2024-12-18T00:27:59.5435550Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
2024-12-18T00:27:59.5436533Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
2024-12-18T00:27:59.7989900Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
2024-12-18T00:27:59.8773041Z + sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2
2024-12-18T00:28:00.3894132Z nvidia-container-toolkit                         11 kB/s | 833  B     00:00    
2024-12-18T00:28:00.4126068Z Package nvidia-docker2-2.14.0-1.noarch is already installed.
2024-12-18T00:28:00.4132319Z Package nvidia-container-toolkit-1.16.2-1.x86_64 is already installed.
2024-12-18T00:28:00.4464921Z Dependencies resolved.
2024-12-18T00:28:00.4605063Z Nothing to do.
2024-12-18T00:28:00.4605579Z Complete!
2024-12-18T00:28:00.6236330Z + sudo systemctl restart docker
2024-12-18T00:28:59.7498439Z nvidia-persistenced failed to initialize. Check syslog for more details.
2024-12-18T00:28:59.7732898Z Wed Dec 18 00:28:59 2024       
2024-12-18T00:28:59.7733749Z +-----------------------------------------------------------------------------------------+
2024-12-18T00:28:59.7734834Z | NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
2024-12-18T00:28:59.7735853Z |-----------------------------------------+------------------------+----------------------+
2024-12-18T00:28:59.7736648Z | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
2024-12-18T00:28:59.7737392Z | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
2024-12-18T00:28:59.7737882Z |                                         |                        |               MIG M. |
2024-12-18T00:28:59.7738248Z |=========================================+========================+======================|
2024-12-18T00:28:59.7910572Z |   0  NVIDIA A10G                    On  |   00000000:00:1E.0 Off |                    0 |
2024-12-18T00:28:59.7911207Z |  0%   22C    P8             16W /  300W |       0MiB /  23028MiB |      0%      Default |
2024-12-18T00:28:59.7911763Z |                                         |                        |                  N/A |
2024-12-18T00:28:59.7912221Z +-----------------------------------------+------------------------+----------------------+
2024-12-18T00:28:59.7915788Z                                                                                          
2024-12-18T00:28:59.7916374Z +-----------------------------------------------------------------------------------------+
2024-12-18T00:28:59.7916995Z | Processes:                                                                              |
2024-12-18T00:28:59.7917570Z |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
2024-12-18T00:28:59.7918404Z |        ID   ID                                                               Usage      |
2024-12-18T00:28:59.7918775Z |=========================================================================================|
2024-12-18T00:28:59.7921161Z |  No running processes found                                                             |
2024-12-18T00:28:59.7921830Z +-----------------------------------------------------------------------------------------+
2024-12-18T00:29:00.3454981Z Command completed after 1 attempt(s).
2024-12-18T00:29:00.3550575Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84
2024-12-18T00:29:00.3551167Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84
2024-12-18T00:29:00.3551665Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2024-12-18T00:29:00.3552131Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}"
2024-12-18T00:29:00.3564354Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:00.3564731Z env:
2024-12-18T00:29:00.3564937Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:00.3565266Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:00.3565623Z ##[endgroup]
2024-12-18T00:29:00.6086913Z Defaulting to user installation because normal site-packages is not writeable
2024-12-18T00:29:00.6259319Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.9/site-packages (5.9.1)
2024-12-18T00:29:00.6264867Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.9/site-packages (11.525.84)
2024-12-18T00:29:00.7643344Z Prepare all required actions
2024-12-18T00:29:00.7644509Z Getting action download info
2024-12-18T00:29:00.9197898Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6)
2024-12-18T00:29:01.1440659Z Download action repository 'actions/download-artifact@v4' (SHA:fa0a91b85d4f404e444e00e005971372dc801d16)
2024-12-18T00:29:01.4260611Z ##[group]Run ./.github/actions/download-build-artifacts
2024-12-18T00:29:01.4260973Z with:
2024-12-18T00:29:01.4261233Z   name: linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-12-18T00:29:01.4261574Z   s3-bucket: gha-artifacts
2024-12-18T00:29:01.4261873Z env:
2024-12-18T00:29:01.4262205Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:01.4262539Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:01.4262896Z ##[endgroup]
2024-12-18T00:29:01.4295730Z ##[group]Run seemethere/download-artifact-s3@v4
2024-12-18T00:29:01.4296064Z with:
2024-12-18T00:29:01.4296342Z   name: linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-12-18T00:29:01.4296727Z   s3-bucket: gha-artifacts
2024-12-18T00:29:01.4297034Z   region: us-east-1
2024-12-18T00:29:01.4297273Z env:
2024-12-18T00:29:01.4297487Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:01.4297823Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:01.4298181Z ##[endgroup]
2024-12-18T00:29:01.8892364Z (node:184620) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2024-12-18T00:29:01.8892949Z 
2024-12-18T00:29:01.8893173Z Please migrate your code to use AWS SDK for JavaScript (v3).
2024-12-18T00:29:01.8893794Z For more information, check the migration guide at https://a.co/7PzMCcy
2024-12-18T00:29:01.8894462Z (Use `node --trace-warnings ...` to show where the warning was created)
2024-12-18T00:29:02.0121920Z Found 1 objects with prefix pytorch/pytorch/12383255690/linux-focal-cuda12.4-py3.10-gcc9-sm86/
2024-12-18T00:29:02.0122807Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2024-12-18T00:29:09.8061091Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2024-12-18T00:29:09.8067603Z Artifact download has finished successfully
2024-12-18T00:29:09.8425217Z ##[group]Run unzip -o artifacts.zip
2024-12-18T00:29:09.8425557Z unzip -o artifacts.zip
2024-12-18T00:29:09.8435766Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:09.8436143Z env:
2024-12-18T00:29:09.8436366Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:09.8436713Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:09.8437077Z ##[endgroup]
2024-12-18T00:29:09.8493106Z Archive:  artifacts.zip
2024-12-18T00:29:09.8493751Z    creating: dist/
2024-12-18T00:29:12.0293782Z   inflating: dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl  
2024-12-18T00:29:12.0294278Z    creating: build/custom_test_artifacts/
2024-12-18T00:29:12.0294678Z    creating: build/custom_test_artifacts/custom-op-build/
2024-12-18T00:29:12.0295190Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2024-12-18T00:29:12.0295770Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/
2024-12-18T00:29:12.0303977Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml  
2024-12-18T00:29:12.0304635Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/
2024-12-18T00:29:12.0305285Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2024-12-18T00:29:12.0305982Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/
2024-12-18T00:29:12.0306916Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-12-18T00:29:12.0308363Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2024-12-18T00:29:12.0310810Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2024-12-18T00:29:12.0311518Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-12-18T00:29:12.0312214Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-12-18T00:29:12.0313765Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2024-12-18T00:29:12.0315879Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2024-12-18T00:29:12.0317313Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2024-12-18T00:29:12.0318266Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2024-12-18T00:29:12.0320524Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2024-12-18T00:29:12.0321617Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2024-12-18T00:29:12.0322343Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2024-12-18T00:29:12.0323045Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2024-12-18T00:29:12.0365915Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2024-12-18T00:29:12.0406901Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2024-12-18T00:29:12.0408743Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2024-12-18T00:29:12.0454055Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2024-12-18T00:29:12.0455044Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2024-12-18T00:29:12.0456044Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2024-12-18T00:29:12.0457069Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2024-12-18T00:29:12.0458064Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2024-12-18T00:29:12.0459046Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2024-12-18T00:29:12.0460033Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2024-12-18T00:29:12.0461013Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2024-12-18T00:29:12.0462391Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2024-12-18T00:29:12.0463307Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2024-12-18T00:29:12.0464318Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2024-12-18T00:29:12.0465234Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2024-12-18T00:29:12.0466351Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2024-12-18T00:29:12.0467423Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2024-12-18T00:29:12.0469612Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2024-12-18T00:29:12.0543468Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2024-12-18T00:29:12.0616755Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2024-12-18T00:29:12.0617588Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2024-12-18T00:29:12.0618287Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/
2024-12-18T00:29:12.0618883Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2024-12-18T00:29:12.0619504Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2024-12-18T00:29:12.0620249Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2024-12-18T00:29:12.0621067Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2024-12-18T00:29:12.0622046Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2024-12-18T00:29:12.0622835Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2024-12-18T00:29:12.0623558Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2024-12-18T00:29:12.0624302Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2024-12-18T00:29:12.0625052Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2024-12-18T00:29:12.0625798Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2024-12-18T00:29:12.0626547Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2024-12-18T00:29:12.0627316Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2024-12-18T00:29:12.0648954Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2024-12-18T00:29:12.0793208Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2024-12-18T00:29:12.0793904Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2024-12-18T00:29:12.0794659Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2024-12-18T00:29:12.0795509Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2024-12-18T00:29:12.0796318Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2024-12-18T00:29:12.0797081Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2024-12-18T00:29:12.0797867Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2024-12-18T00:29:12.0798659Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2024-12-18T00:29:12.0799442Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2024-12-18T00:29:12.0800258Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2024-12-18T00:29:12.0801056Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2024-12-18T00:29:12.0822894Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2024-12-18T00:29:12.0905465Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2024-12-18T00:29:12.0906463Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2024-12-18T00:29:12.0907386Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2024-12-18T00:29:12.0908259Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2024-12-18T00:29:12.0908902Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2024-12-18T00:29:12.0910057Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2024-12-18T00:29:12.0910788Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2024-12-18T00:29:12.0913944Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2024-12-18T00:29:12.0914675Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2024-12-18T00:29:12.0915559Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2024-12-18T00:29:12.1036590Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2024-12-18T00:29:12.1099539Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2024-12-18T00:29:12.1100050Z    creating: build/custom_test_artifacts/jit-hook-build/
2024-12-18T00:29:12.1100523Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2024-12-18T00:29:12.1101100Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/
2024-12-18T00:29:12.1108420Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml  
2024-12-18T00:29:12.1109060Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/
2024-12-18T00:29:12.1109690Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2024-12-18T00:29:12.1110365Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/
2024-12-18T00:29:12.1111010Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-12-18T00:29:12.1113109Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2024-12-18T00:29:12.1114389Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2024-12-18T00:29:12.1115069Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-12-18T00:29:12.1115738Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-12-18T00:29:12.1119079Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2024-12-18T00:29:12.1120513Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2024-12-18T00:29:12.1122797Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2024-12-18T00:29:12.1123999Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2024-12-18T00:29:12.1126360Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2024-12-18T00:29:12.1127334Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2024-12-18T00:29:12.1128049Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2024-12-18T00:29:12.1128731Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2024-12-18T00:29:12.1170564Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2024-12-18T00:29:12.1211551Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2024-12-18T00:29:12.1212564Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2024-12-18T00:29:12.1260138Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2024-12-18T00:29:12.1261127Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2024-12-18T00:29:12.1262330Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2024-12-18T00:29:12.1263355Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2024-12-18T00:29:12.1264357Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2024-12-18T00:29:12.1265363Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2024-12-18T00:29:12.1266366Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2024-12-18T00:29:12.1267358Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2024-12-18T00:29:12.1268344Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2024-12-18T00:29:12.1269278Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2024-12-18T00:29:12.1270177Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2024-12-18T00:29:12.1271033Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2024-12-18T00:29:12.1271908Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2024-12-18T00:29:12.1272755Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2024-12-18T00:29:12.1274068Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2024-12-18T00:29:12.1348275Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2024-12-18T00:29:12.1421861Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2024-12-18T00:29:12.1422726Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2024-12-18T00:29:12.1423416Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/
2024-12-18T00:29:12.1424002Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2024-12-18T00:29:12.1424624Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2024-12-18T00:29:12.1425273Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2024-12-18T00:29:12.1426015Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2024-12-18T00:29:12.1426858Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2024-12-18T00:29:12.1427666Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2024-12-18T00:29:12.1428411Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2024-12-18T00:29:12.1429191Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2024-12-18T00:29:12.1430148Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2024-12-18T00:29:12.1431573Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2024-12-18T00:29:12.1432372Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2024-12-18T00:29:12.1433494Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2024-12-18T00:29:12.1455088Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2024-12-18T00:29:12.1520222Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2024-12-18T00:29:12.1521114Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2024-12-18T00:29:12.1521871Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2024-12-18T00:29:12.1522554Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2024-12-18T00:29:12.1523184Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2024-12-18T00:29:12.1523800Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2024-12-18T00:29:12.1524436Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2024-12-18T00:29:12.1527079Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2024-12-18T00:29:12.1528155Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2024-12-18T00:29:12.1529180Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2024-12-18T00:29:12.1578558Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2024-12-18T00:29:12.1579091Z    creating: build/custom_test_artifacts/custom-backend-build/
2024-12-18T00:29:12.1579620Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2024-12-18T00:29:12.1580233Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/
2024-12-18T00:29:12.1587637Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml  
2024-12-18T00:29:12.1588338Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/
2024-12-18T00:29:12.1589048Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2024-12-18T00:29:12.1589789Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/
2024-12-18T00:29:12.1590521Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-12-18T00:29:12.1592090Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2024-12-18T00:29:12.1594041Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2024-12-18T00:29:12.1594784Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-12-18T00:29:12.1595535Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-12-18T00:29:12.1597669Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2024-12-18T00:29:12.1599260Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2024-12-18T00:29:12.1601345Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2024-12-18T00:29:12.1602201Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2024-12-18T00:29:12.1604083Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2024-12-18T00:29:12.1605481Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2024-12-18T00:29:12.1606300Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2024-12-18T00:29:12.1607050Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2024-12-18T00:29:12.1648982Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2024-12-18T00:29:12.1691144Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2024-12-18T00:29:12.1693576Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2024-12-18T00:29:12.1739354Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2024-12-18T00:29:12.1740548Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2024-12-18T00:29:12.1741625Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2024-12-18T00:29:12.1742780Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2024-12-18T00:29:12.1743857Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2024-12-18T00:29:12.1744908Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2024-12-18T00:29:12.1745959Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2024-12-18T00:29:12.1747008Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2024-12-18T00:29:12.1748041Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2024-12-18T00:29:12.1749011Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2024-12-18T00:29:12.1749969Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2024-12-18T00:29:12.1750906Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2024-12-18T00:29:12.1751843Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2024-12-18T00:29:12.1752758Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2024-12-18T00:29:12.1753681Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2024-12-18T00:29:12.1827318Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2024-12-18T00:29:12.1901496Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2024-12-18T00:29:12.1902568Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2024-12-18T00:29:12.1903361Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/
2024-12-18T00:29:12.1904014Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2024-12-18T00:29:12.1904707Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2024-12-18T00:29:12.1905407Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2024-12-18T00:29:12.1906210Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2024-12-18T00:29:12.1907106Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2024-12-18T00:29:12.1907983Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2024-12-18T00:29:12.1908984Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2024-12-18T00:29:12.1909818Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2024-12-18T00:29:12.1910811Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2024-12-18T00:29:12.1911666Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2024-12-18T00:29:12.1912504Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2024-12-18T00:29:12.1913340Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2024-12-18T00:29:12.1917874Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2024-12-18T00:29:12.2040059Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2024-12-18T00:29:12.2040902Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2024-12-18T00:29:12.2041741Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2024-12-18T00:29:12.2042679Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2024-12-18T00:29:12.2043573Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2024-12-18T00:29:12.2044416Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2024-12-18T00:29:12.2045296Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2024-12-18T00:29:12.2046178Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2024-12-18T00:29:12.2047055Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2024-12-18T00:29:12.2047944Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2024-12-18T00:29:12.2048814Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2024-12-18T00:29:12.2069454Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2024-12-18T00:29:12.2124671Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2024-12-18T00:29:12.2125629Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2024-12-18T00:29:12.2126450Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2024-12-18T00:29:12.2127470Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2024-12-18T00:29:12.2128872Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2024-12-18T00:29:12.2130868Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2024-12-18T00:29:12.2131553Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2024-12-18T00:29:12.2134390Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2024-12-18T00:29:12.2135282Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2024-12-18T00:29:12.2136166Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2024-12-18T00:29:12.2237190Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2024-12-18T00:29:12.2280301Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2024-12-18T00:29:12.2280948Z    creating: build/lib/
2024-12-18T00:29:12.2369707Z   inflating: build/lib/libprotobuf-lite.a  
2024-12-18T00:29:12.2823912Z   inflating: build/lib/libprotobuf.a  
2024-12-18T00:29:12.2834119Z   inflating: build/lib/libpthreadpool.a  
2024-12-18T00:29:12.2842470Z   inflating: build/lib/libcpuinfo.a  
2024-12-18T00:29:12.2850671Z   inflating: build/lib/libcpuinfo_internals.a  
2024-12-18T00:29:12.2851284Z   inflating: build/lib/libclog.a     
2024-12-18T00:29:12.2869767Z   inflating: build/lib/libnnpack.a   
2024-12-18T00:29:12.2872503Z   inflating: build/lib/libnnpack_reference_layers.a  
2024-12-18T00:29:12.3059830Z   inflating: build/lib/libmicrokernels-prod.a  
2024-12-18T00:29:12.3128268Z   inflating: build/lib/libgtest.a    
2024-12-18T00:29:12.3203710Z   inflating: build/lib/libbenchmark.a  
2024-12-18T00:29:12.3267323Z   inflating: build/lib/libasmjit.a   
2024-12-18T00:29:12.3274920Z   inflating: build/lib/libittnotify.a  
2024-12-18T00:29:12.3302552Z   inflating: build/lib/libtensorpipe_uv.a  
2024-12-18T00:29:12.3430400Z   inflating: build/lib/libgloo.a     
2024-12-18T00:29:12.3450855Z   inflating: build/lib/libfmt.a      
2024-12-18T00:29:12.3550361Z   inflating: build/lib/libc10.so     
2024-12-18T00:29:12.3552206Z   inflating: build/lib/libcaffe2_nvrtc.so  
2024-12-18T00:29:12.3553724Z   inflating: build/lib/libtorch_global_deps.so  
2024-12-18T00:29:12.4057196Z   inflating: build/lib/libprotoc.a   
2024-12-18T00:29:12.4076411Z   inflating: build/lib/libpytorch_qnnpack.a  
2024-12-18T00:29:12.4093721Z   inflating: build/lib/libgmock.a    
2024-12-18T00:29:12.4094363Z   inflating: build/lib/libgtest_main.a  
2024-12-18T00:29:12.4095339Z   inflating: build/lib/libbenchmark_main.a  
2024-12-18T00:29:12.4667543Z   inflating: build/lib/libtensorpipe.a  
2024-12-18T00:29:12.5055863Z   inflating: build/lib/libgloo_cuda.a  
2024-12-18T00:29:13.5335911Z   inflating: build/lib/libdnnl.a     
2024-12-18T00:29:13.5396031Z   inflating: build/lib/libc10_cuda.so  
2024-12-18T00:29:13.5396702Z   inflating: build/lib/libgmock_main.a  
2024-12-18T00:29:13.6651797Z   inflating: build/lib/libfbgemm.a   
2024-12-18T00:29:13.6906881Z   inflating: build/lib/libtensorpipe_cuda.a  
2024-12-18T00:29:13.7414342Z   inflating: build/lib/libkineto.a   
2024-12-18T00:29:13.7502857Z   inflating: build/lib/libXNNPACK.a  
2024-12-18T00:29:13.7545378Z   inflating: build/lib/libonnx_proto.a  
2024-12-18T00:29:13.8274471Z   inflating: build/lib/libonnx.a     
2024-12-18T00:29:13.9139904Z   inflating: build/lib/libmicrokernels-all.a  
2024-12-18T00:29:16.4330500Z   inflating: build/lib/libtorch_cpu.so  
2024-12-18T00:29:16.4336015Z   inflating: build/lib/libunbox_lib.a  
2024-12-18T00:29:16.4340097Z   inflating: build/lib/libshm.so     
2024-12-18T00:29:18.5239741Z   inflating: build/lib/libtorch_cuda.so  
2024-12-18T00:29:18.5240640Z   inflating: build/lib/libtorch.so   
2024-12-18T00:29:18.5242735Z   inflating: build/lib/libc10d_cuda_test.so  
2024-12-18T00:29:19.4021357Z   inflating: build/lib/libtorch_cuda_linalg.so  
2024-12-18T00:29:19.6101173Z   inflating: build/lib/libtorch_python.so  
2024-12-18T00:29:19.6171322Z   inflating: build/lib/libtorchbind_test.so  
2024-12-18T00:29:19.6191599Z   inflating: build/lib/libjitbackend_test.so  
2024-12-18T00:29:19.6216584Z   inflating: build/lib/libbackend_with_compiler.so  
2024-12-18T00:29:19.6241986Z   inflating: build/lib/libaoti_custom_ops.so  
2024-12-18T00:29:19.6276420Z   inflating: build/lib/libnnapi_backend.so  
2024-12-18T00:29:19.6276829Z    creating: build/bin/
2024-12-18T00:29:19.6330617Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2024-12-18T00:29:19.6385454Z   inflating: build/bin/c10_DeviceGuard_test  
2024-12-18T00:29:19.6440853Z   inflating: build/bin/c10_Device_test  
2024-12-18T00:29:19.6528098Z   inflating: build/bin/c10_DispatchKeySet_test  
2024-12-18T00:29:19.6560494Z   inflating: build/bin/c10_Scalar_test  
2024-12-18T00:29:19.6612435Z   inflating: build/bin/c10_StreamGuard_test  
2024-12-18T00:29:19.6668592Z   inflating: build/bin/c10_SymInt_test  
2024-12-18T00:29:19.6725964Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2024-12-18T00:29:19.6785495Z   inflating: build/bin/c10_InlineStreamGuard_test  
2024-12-18T00:29:19.6845223Z   inflating: build/bin/c10_SizesAndStrides_test  
2024-12-18T00:29:19.6919261Z   inflating: build/bin/c10_cow_test  
2024-12-18T00:29:19.6972701Z   inflating: build/bin/c10_ArrayRef_test  
2024-12-18T00:29:19.7029059Z   inflating: build/bin/c10_Bitset_test  
2024-12-18T00:29:19.7081189Z   inflating: build/bin/c10_ConstexprCrc_test  
2024-12-18T00:29:19.7134359Z   inflating: build/bin/c10_DeadlockDetection_test  
2024-12-18T00:29:19.7189058Z   inflating: build/bin/c10_Half_test  
2024-12-18T00:29:19.7248409Z   inflating: build/bin/c10_LeftRight_test  
2024-12-18T00:29:19.7306832Z   inflating: build/bin/c10_Metaprogramming_test  
2024-12-18T00:29:19.7363459Z   inflating: build/bin/c10_NetworkFlow_test  
2024-12-18T00:29:19.7416916Z   inflating: build/bin/c10_Synchronized_test  
2024-12-18T00:29:19.7476661Z   inflating: build/bin/c10_ThreadLocal_test  
2024-12-18T00:29:19.7531382Z   inflating: build/bin/c10_TypeIndex_test  
2024-12-18T00:29:19.7586540Z   inflating: build/bin/c10_TypeList_test  
2024-12-18T00:29:19.7638400Z   inflating: build/bin/c10_TypeTraits_test  
2024-12-18T00:29:19.7693758Z   inflating: build/bin/c10_accumulate_test  
2024-12-18T00:29:19.7753287Z   inflating: build/bin/c10_bfloat16_test  
2024-12-18T00:29:19.7807036Z   inflating: build/bin/c10_bit_cast_test  
2024-12-18T00:29:19.7867811Z   inflating: build/bin/c10_complex_math_test  
2024-12-18T00:29:19.7926032Z   inflating: build/bin/c10_complex_test  
2024-12-18T00:29:19.7979993Z   inflating: build/bin/c10_error_test  
2024-12-18T00:29:19.8036712Z   inflating: build/bin/c10_exception_test  
2024-12-18T00:29:19.8090599Z   inflating: build/bin/c10_flags_test  
2024-12-18T00:29:19.8144264Z   inflating: build/bin/c10_generic_math_test  
2024-12-18T00:29:19.8315658Z   inflating: build/bin/c10_intrusive_ptr_test  
2024-12-18T00:29:19.8370004Z   inflating: build/bin/c10_irange_test  
2024-12-18T00:29:19.8426278Z   inflating: build/bin/c10_lazy_test  
2024-12-18T00:29:19.8487266Z   inflating: build/bin/c10_logging_test  
2024-12-18T00:29:19.8566829Z   inflating: build/bin/c10_optional_test  
2024-12-18T00:29:19.8633136Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2024-12-18T00:29:19.8690116Z   inflating: build/bin/c10_registry_test  
2024-12-18T00:29:19.8850083Z   inflating: build/bin/c10_small_vector_test  
2024-12-18T00:29:19.8905332Z   inflating: build/bin/c10_ssize_test  
2024-12-18T00:29:19.8960766Z   inflating: build/bin/c10_string_util_test  
2024-12-18T00:29:19.9023939Z   inflating: build/bin/c10_string_view_test  
2024-12-18T00:29:19.9078155Z   inflating: build/bin/c10_tempfile_test  
2024-12-18T00:29:19.9138102Z   inflating: build/bin/c10_typeid_test  
2024-12-18T00:29:19.9186367Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2024-12-18T00:29:19.9633513Z   inflating: build/bin/protoc-3.13.0.0  
2024-12-18T00:29:20.0081093Z   inflating: build/bin/protoc        
2024-12-18T00:29:20.0137351Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test  
2024-12-18T00:29:20.0193778Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream  
2024-12-18T00:29:20.0249916Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device  
2024-12-18T00:29:20.0305788Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes  
2024-12-18T00:29:20.0361561Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads  
2024-12-18T00:29:20.0417706Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks  
2024-12-18T00:29:20.0473830Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block  
2024-12-18T00:29:20.0526023Z   inflating: build/bin/c10_cuda_CUDATest  
2024-12-18T00:29:20.0926083Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2024-12-18T00:29:20.1336871Z   inflating: build/bin/vec_test_all_types_AVX512  
2024-12-18T00:29:20.1763212Z   inflating: build/bin/vec_test_all_types_AVX2  
2024-12-18T00:29:20.1818636Z   inflating: build/bin/BackoffTest   
2024-12-18T00:29:20.1875348Z   inflating: build/bin/FileStoreTest  
2024-12-18T00:29:20.1934619Z   inflating: build/bin/TCPStoreTest  
2024-12-18T00:29:20.1990844Z   inflating: build/bin/HashStoreTest  
2024-12-18T00:29:20.2004495Z   inflating: build/bin/ProcessGroupMPITest  
2024-12-18T00:29:20.2062134Z   inflating: build/bin/test_edge_op_registration  
2024-12-18T00:29:20.2066100Z   inflating: build/bin/torch_shm_manager  
2024-12-18T00:29:20.2069124Z   inflating: build/bin/example_allreduce  
2024-12-18T00:29:20.2127526Z   inflating: build/bin/test_dist_autograd  
2024-12-18T00:29:20.2200646Z   inflating: build/bin/test_cpp_rpc  
2024-12-18T00:29:20.2203816Z   inflating: build/bin/parallel_benchmark  
2024-12-18T00:29:20.2273867Z   inflating: build/bin/test_mobile_nnc  
2024-12-18T00:29:20.2283116Z   inflating: build/bin/aot_model_compiler_test  
2024-12-18T00:29:20.2635613Z   inflating: build/bin/test_lazy     
2024-12-18T00:29:20.3822226Z   inflating: build/bin/test_api      
2024-12-18T00:29:20.3900348Z   inflating: build/bin/Dict_test     
2024-12-18T00:29:20.3956161Z   inflating: build/bin/Dimname_test  
2024-12-18T00:29:20.4024489Z   inflating: build/bin/MaybeOwned_test  
2024-12-18T00:29:20.4086417Z   inflating: build/bin/NamedTensor_test  
2024-12-18T00:29:20.4148580Z   inflating: build/bin/apply_utils_test  
2024-12-18T00:29:20.4211962Z   inflating: build/bin/atest         
2024-12-18T00:29:20.4278530Z   inflating: build/bin/basic         
2024-12-18T00:29:20.4337745Z   inflating: build/bin/broadcast_test  
2024-12-18T00:29:20.4391756Z   inflating: build/bin/cpu_allocator_test  
2024-12-18T00:29:20.4454090Z   inflating: build/bin/cpu_generator_test  
2024-12-18T00:29:20.4510892Z   inflating: build/bin/cpu_profiling_allocator_test  
2024-12-18T00:29:20.4607544Z   inflating: build/bin/cpu_rng_test  
2024-12-18T00:29:20.4661501Z   inflating: build/bin/dispatch_key_set_test  
2024-12-18T00:29:20.4715809Z   inflating: build/bin/dlconvertor_test  
2024-12-18T00:29:20.4778270Z   inflating: build/bin/extension_backend_test  
2024-12-18T00:29:20.4836581Z   inflating: build/bin/half_test     
2024-12-18T00:29:20.4936517Z   inflating: build/bin/ivalue_test   
2024-12-18T00:29:20.4989778Z   inflating: build/bin/lazy_tensor_test  
2024-12-18T00:29:20.5047841Z   inflating: build/bin/math_kernel_test  
2024-12-18T00:29:20.5105064Z   inflating: build/bin/memory_format_test  
2024-12-18T00:29:20.5161519Z   inflating: build/bin/memory_overlapping_test  
2024-12-18T00:29:20.5217675Z   inflating: build/bin/mobile_memory_cleanup  
2024-12-18T00:29:20.5277578Z   inflating: build/bin/native_test   
2024-12-18T00:29:20.5331792Z   inflating: build/bin/operator_name_test  
2024-12-18T00:29:20.5386527Z   inflating: build/bin/operators_test  
2024-12-18T00:29:20.5442274Z   inflating: build/bin/packedtensoraccessor_test  
2024-12-18T00:29:20.5512800Z   inflating: build/bin/pow_test      
2024-12-18T00:29:20.5573754Z   inflating: build/bin/quantized_test  
2024-12-18T00:29:20.5627172Z   inflating: build/bin/reduce_ops_test  
2024-12-18T00:29:20.5682229Z   inflating: build/bin/reportMemoryUsage_test  
2024-12-18T00:29:20.5743399Z   inflating: build/bin/scalar_tensor_test  
2024-12-18T00:29:20.5805736Z   inflating: build/bin/scalar_test   
2024-12-18T00:29:20.5861423Z   inflating: build/bin/StorageUtils_test  
2024-12-18T00:29:20.5917036Z   inflating: build/bin/stride_properties_test  
2024-12-18T00:29:20.6000752Z   inflating: build/bin/tensor_iterator_test  
2024-12-18T00:29:20.6058362Z   inflating: build/bin/test_parallel  
2024-12-18T00:29:20.6061466Z   inflating: build/bin/thread_init_test  
2024-12-18T00:29:20.6120496Z   inflating: build/bin/type_ptr_test  
2024-12-18T00:29:20.6183878Z   inflating: build/bin/type_test     
2024-12-18T00:29:20.6239882Z   inflating: build/bin/undefined_tensor_test  
2024-12-18T00:29:20.6241682Z   inflating: build/bin/verify_api_visibility  
2024-12-18T00:29:20.6315535Z   inflating: build/bin/legacy_vmap_test  
2024-12-18T00:29:20.6370937Z   inflating: build/bin/weakref_test  
2024-12-18T00:29:20.6425737Z   inflating: build/bin/wrapdim_test  
2024-12-18T00:29:20.6480758Z   inflating: build/bin/xla_tensor_test  
2024-12-18T00:29:20.6544172Z   inflating: build/bin/IListRef_test  
2024-12-18T00:29:20.6656115Z   inflating: build/bin/List_test     
2024-12-18T00:29:20.6725630Z   inflating: build/bin/KernelFunction_test  
2024-12-18T00:29:20.6851650Z   inflating: build/bin/kernel_function_legacy_test  
2024-12-18T00:29:20.6952351Z   inflating: build/bin/kernel_function_test  
2024-12-18T00:29:20.7084868Z   inflating: build/bin/kernel_lambda_legacy_test  
2024-12-18T00:29:20.7192355Z   inflating: build/bin/kernel_lambda_test  
2024-12-18T00:29:20.7257805Z   inflating: build/bin/kernel_stackbased_test  
2024-12-18T00:29:20.7358302Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2024-12-18T00:29:20.7412596Z   inflating: build/bin/CppSignature_test  
2024-12-18T00:29:20.7472060Z   inflating: build/bin/backend_fallback_test  
2024-12-18T00:29:20.7524396Z   inflating: build/bin/op_allowlist_test  
2024-12-18T00:29:20.7831184Z   inflating: build/bin/op_registration_test  
2024-12-18T00:29:20.7898746Z   inflating: build/bin/inline_container_test  
2024-12-18T00:29:20.7955651Z   inflating: build/bin/cuda_apply_test  
2024-12-18T00:29:20.8011569Z   inflating: build/bin/cuda_allocator_test  
2024-12-18T00:29:20.8070019Z   inflating: build/bin/cuda_caching_host_allocator_test  
2024-12-18T00:29:20.8132971Z   inflating: build/bin/cuda_atomic_ops_test  
2024-12-18T00:29:20.8207211Z   inflating: build/bin/cuda_complex_math_test  
2024-12-18T00:29:20.8270243Z   inflating: build/bin/cuda_complex_test  
2024-12-18T00:29:20.8322876Z   inflating: build/bin/cuda_device_test  
2024-12-18T00:29:20.8385286Z   inflating: build/bin/cuda_cub_test  
2024-12-18T00:29:20.8440211Z   inflating: build/bin/cuda_dlconvertor_test  
2024-12-18T00:29:20.8508827Z   inflating: build/bin/cuda_distributions_test  
2024-12-18T00:29:20.8569430Z   inflating: build/bin/cuda_generator_test  
2024-12-18T00:29:20.8622892Z   inflating: build/bin/cuda_half_test  
2024-12-18T00:29:20.8678319Z   inflating: build/bin/cuda_integer_divider_test  
2024-12-18T00:29:20.8731172Z   inflating: build/bin/cuda_optional_test  
2024-12-18T00:29:20.8787107Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2024-12-18T00:29:20.8843178Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2024-12-18T00:29:20.8896808Z   inflating: build/bin/cuda_allocatorTraceTracker_test  
2024-12-18T00:29:20.8960853Z   inflating: build/bin/cuda_stream_test  
2024-12-18T00:29:20.9014500Z   inflating: build/bin/cuda_cudnn_test  
2024-12-18T00:29:20.9069726Z   inflating: build/bin/cuda_vectorized_test  
2024-12-18T00:29:20.9084204Z   inflating: build/bin/tutorial_tensorexpr  
2024-12-18T00:29:20.9153820Z   inflating: build/bin/ProcessGroupGlooTest  
2024-12-18T00:29:20.9214635Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2024-12-18T00:29:20.9283145Z   inflating: build/bin/ProcessGroupNCCLTest  
2024-12-18T00:29:20.9348896Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2024-12-18T00:29:21.0188131Z   inflating: build/bin/test_tensorexpr  
2024-12-18T00:29:21.0772474Z   inflating: build/bin/test_jit      
2024-12-18T00:29:21.0772843Z    creating: .additional_ci_files/
2024-12-18T00:29:21.0863241Z   inflating: .additional_ci_files/test-times.json  
2024-12-18T00:29:21.1221423Z   inflating: .additional_ci_files/test-class-times.json  
2024-12-18T00:29:21.1257762Z ##[group]Run rm artifacts.zip
2024-12-18T00:29:21.1258082Z rm artifacts.zip
2024-12-18T00:29:21.1266960Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:21.1267334Z env:
2024-12-18T00:29:21.1267556Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:21.1267887Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:21.1268247Z ##[endgroup]
2024-12-18T00:29:21.2428159Z ##[group]Run df -H
2024-12-18T00:29:21.2428426Z df -H
2024-12-18T00:29:21.2437650Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:21.2438033Z env:
2024-12-18T00:29:21.2438259Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:21.2438599Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:21.2438959Z ##[endgroup]
2024-12-18T00:29:21.2486477Z Filesystem        Size  Used Avail Use% Mounted on
2024-12-18T00:29:21.2486871Z devtmpfs          4.2M     0  4.2M   0% /dev
2024-12-18T00:29:21.2487207Z tmpfs              34G     0   34G   0% /dev/shm
2024-12-18T00:29:21.2487535Z tmpfs              14G  553k   14G   1% /run
2024-12-18T00:29:21.2487856Z /dev/nvme0n1p1    161G   60G  102G  37% /
2024-12-18T00:29:21.2488178Z tmpfs              34G   29k   34G   1% /tmp
2024-12-18T00:29:21.2488506Z /dev/nvme0n1p128   11M  1.4M  9.2M  13% /boot/efi
2024-12-18T00:29:21.2488860Z tmpfs             6.7G     0  6.7G   0% /run/user/0
2024-12-18T00:29:21.2522311Z Prepare all required actions
2024-12-18T00:29:21.2522687Z Getting action download info
2024-12-18T00:29:21.4049789Z ##[group]Run ./.github/actions/download-td-artifacts
2024-12-18T00:29:21.4050136Z with:
2024-12-18T00:29:21.4050339Z env:
2024-12-18T00:29:21.4050551Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:21.4050881Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:21.4051241Z ##[endgroup]
2024-12-18T00:29:21.4083632Z ##[group]Run seemethere/download-artifact-s3@v4
2024-12-18T00:29:21.4083973Z with:
2024-12-18T00:29:21.4084185Z   name: td_results
2024-12-18T00:29:21.4084431Z   s3-bucket: gha-artifacts
2024-12-18T00:29:21.4084694Z   region: us-east-1
2024-12-18T00:29:21.4084921Z env:
2024-12-18T00:29:21.4085132Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:21.4085464Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:21.4085820Z ##[endgroup]
2024-12-18T00:29:21.8652666Z (node:184641) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2024-12-18T00:29:21.8653159Z 
2024-12-18T00:29:21.8653371Z Please migrate your code to use AWS SDK for JavaScript (v3).
2024-12-18T00:29:21.8653899Z For more information, check the migration guide at https://a.co/7PzMCcy
2024-12-18T00:29:21.8654451Z (Use `node --trace-warnings ...` to show where the warning was created)
2024-12-18T00:29:21.9751399Z Found 0 objects with prefix pytorch/pytorch/12383255690/td_results/
2024-12-18T00:29:21.9757453Z Artifact download has finished successfully
2024-12-18T00:29:22.0102892Z ##[group]Run mkdir -p .additional_ci_files
2024-12-18T00:29:22.0103264Z mkdir -p .additional_ci_files
2024-12-18T00:29:22.0103696Z mv td_results.json .additional_ci_files/td_results.json || true
2024-12-18T00:29:22.0112539Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:22.0112912Z env:
2024-12-18T00:29:22.0113134Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:22.0113471Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:22.0113822Z ##[endgroup]
2024-12-18T00:29:22.0183341Z mv: cannot stat 'td_results.json': No such file or directory
2024-12-18T00:29:22.0217558Z ##[group]Run .github/scripts/parse_ref.py
2024-12-18T00:29:22.0217945Z .github/scripts/parse_ref.py
2024-12-18T00:29:22.0226034Z shell: /usr/bin/bash -e {0}
2024-12-18T00:29:22.0226294Z env:
2024-12-18T00:29:22.0226505Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:22.0226841Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:22.0227194Z ##[endgroup]
2024-12-18T00:29:22.0502685Z Prepare all required actions
2024-12-18T00:29:22.0551662Z ##[group]Run ./.github/actions/get-workflow-job-id
2024-12-18T00:29:22.0552009Z with:
2024-12-18T00:29:22.0552407Z   github-token: ***
2024-12-18T00:29:22.0552651Z env:
2024-12-18T00:29:22.0552869Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:22.0553204Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:22.0553562Z ##[endgroup]
2024-12-18T00:29:22.0571978Z ##[group]Run set -eux
2024-12-18T00:29:22.0572252Z set -eux
2024-12-18T00:29:22.0572883Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2024-12-18T00:29:22.0582137Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:22.0582517Z env:
2024-12-18T00:29:22.0582733Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:22.0583071Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:22.0583567Z   GITHUB_TOKEN: ***
2024-12-18T00:29:22.0583803Z ##[endgroup]
2024-12-18T00:29:22.0610481Z + python3 .github/scripts/get_workflow_job_id.py 12383255690 i-02d991818cc142eee
2024-12-18T00:29:23.0894176Z setting job-id=34567153284
2024-12-18T00:29:23.0894735Z setting job-name=cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:23.1070617Z Prepare all required actions
2024-12-18T00:29:23.1070996Z Getting action download info
2024-12-18T00:29:23.2372352Z ##[group]Run ./.github/actions/filter-test-configs
2024-12-18T00:29:23.2372702Z with:
2024-12-18T00:29:23.2373112Z   github-token: ***
2024-12-18T00:29:23.2374878Z   test-matrix: {"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}
2024-12-18T00:29:23.2376895Z   job-name: cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:23.2377418Z env:
2024-12-18T00:29:23.2377848Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:23.2378211Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:23.2378566Z ##[endgroup]
2024-12-18T00:29:23.2415466Z ##[group]Run nick-fields/retry@v3.0.0
2024-12-18T00:29:23.2415811Z with:
2024-12-18T00:29:23.2416016Z   shell: bash
2024-12-18T00:29:23.2416235Z   timeout_minutes: 10
2024-12-18T00:29:23.2416479Z   max_attempts: 5
2024-12-18T00:29:23.2416713Z   retry_wait_seconds: 30
2024-12-18T00:29:23.2417482Z   command: set -eux
# PyYAML 6.0 doesn't work with MacOS x86 anymore
# This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2
python3 -m pip install requests==2.27.1 pyyaml==6.0.1

2024-12-18T00:29:23.2418312Z   polling_interval_seconds: 1
2024-12-18T00:29:23.2418585Z   warning_on_retry: true
2024-12-18T00:29:23.2418851Z   continue_on_error: false
2024-12-18T00:29:23.2419105Z env:
2024-12-18T00:29:23.2419316Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:23.2419645Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:23.2420312Z   GITHUB_TOKEN: ***
2024-12-18T00:29:23.2420545Z ##[endgroup]
2024-12-18T00:29:23.3403004Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1
2024-12-18T00:29:23.5776773Z Defaulting to user installation because normal site-packages is not writeable
2024-12-18T00:29:23.5954352Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.9/site-packages (2.27.1)
2024-12-18T00:29:23.5958134Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.9/site-packages (6.0.1)
2024-12-18T00:29:23.6076316Z Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2.0.12)
2024-12-18T00:29:23.6079809Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2024.12.14)
2024-12-18T00:29:23.6088268Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10)
2024-12-18T00:29:23.6093620Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10)
2024-12-18T00:29:24.3181915Z Command completed after 1 attempt(s).
2024-12-18T00:29:24.3252540Z ##[group]Run set -x
2024-12-18T00:29:24.3252800Z set -x
2024-12-18T00:29:24.3253030Z 
2024-12-18T00:29:24.3253431Z # Use relative path here as this could be checked out anywhere, not necessarily
2024-12-18T00:29:24.3253913Z # in runner workspace
2024-12-18T00:29:24.3254289Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py"
2024-12-18T00:29:24.3263336Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:24.3263709Z env:
2024-12-18T00:29:24.3263933Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:24.3264269Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:24.3264615Z ##[endgroup]
2024-12-18T00:29:24.3295019Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py
2024-12-18T00:29:24.3541359Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}"
2024-12-18T00:29:24.3541769Z echo "Workflow: ${GITHUB_WORKFLOW}"
2024-12-18T00:29:24.3542230Z echo "Job name: ${JOB_NAME}"
2024-12-18T00:29:24.3542522Z 
2024-12-18T00:29:24.3542904Z # Use relative path here as this could be checked out anywhere, not necessarily
2024-12-18T00:29:24.3543386Z # in runner workspace
2024-12-18T00:29:24.3543807Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \
2024-12-18T00:29:24.3544281Z   --workflow "${GITHUB_WORKFLOW}" \
2024-12-18T00:29:24.3544613Z   --job-name "${JOB_NAME}" \
2024-12-18T00:29:24.3546494Z   --test-matrix "{"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \
2024-12-18T00:29:24.3548386Z   --selected-test-configs "" \
2024-12-18T00:29:24.3548719Z   --pr-number "${PR_NUMBER}" \
2024-12-18T00:29:24.3549035Z   --tag "${TAG}" \
2024-12-18T00:29:24.3549335Z   --event-name "${EVENT_NAME}" \
2024-12-18T00:29:24.3549667Z   --schedule "${SCHEDULE}" \
2024-12-18T00:29:24.3549986Z   --branch "${HEAD_BRANCH}"
2024-12-18T00:29:24.3558797Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:24.3559170Z env:
2024-12-18T00:29:24.3559388Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:24.3559733Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:24.3560286Z   GITHUB_TOKEN: ***
2024-12-18T00:29:24.3560754Z   JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:24.3561282Z   PR_NUMBER: 
2024-12-18T00:29:24.3561506Z   TAG: 
2024-12-18T00:29:24.3561721Z   EVENT_NAME: push
2024-12-18T00:29:24.3561968Z   SCHEDULE: 
2024-12-18T00:29:24.3562198Z   HEAD_BRANCH: 
2024-12-18T00:29:24.3562428Z ##[endgroup]
2024-12-18T00:29:24.3591061Z Workflow: inductor
2024-12-18T00:29:24.3591980Z Job name: cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:24.6234910Z ##[group]Run echo "Filtered matrix:"
2024-12-18T00:29:24.6235248Z echo "Filtered matrix:"
2024-12-18T00:29:24.6237077Z echo "{"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}"
2024-12-18T00:29:24.6239024Z 
2024-12-18T00:29:24.6239228Z echo
2024-12-18T00:29:24.6239516Z echo "Is the current job unstable? False"
2024-12-18T00:29:24.6239853Z 
2024-12-18T00:29:24.6240068Z echo
2024-12-18T00:29:24.6240342Z echo "Is keep-going label set? False"
2024-12-18T00:29:24.6240654Z 
2024-12-18T00:29:24.6240875Z echo
2024-12-18T00:29:24.6241117Z echo "Renabled issues? "
2024-12-18T00:29:24.6249624Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:24.6250002Z env:
2024-12-18T00:29:24.6250214Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:24.6250718Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:24.6251099Z ##[endgroup]
2024-12-18T00:29:24.6283561Z Filtered matrix:
2024-12-18T00:29:24.6285255Z {include: [{config: inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}]}
2024-12-18T00:29:24.6286880Z 
2024-12-18T00:29:24.6286993Z Is the current job unstable? False
2024-12-18T00:29:24.6287208Z 
2024-12-18T00:29:24.6287316Z Is keep-going label set? False
2024-12-18T00:29:24.6287504Z 
2024-12-18T00:29:24.6287600Z Renabled issues? 
2024-12-18T00:29:24.6329522Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2024-12-18T00:29:24.6330312Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2024-12-18T00:29:24.6338753Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:24.6339123Z env:
2024-12-18T00:29:24.6339343Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:24.6339687Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:24.6340046Z   JOB_TIMEOUT: 240
2024-12-18T00:29:24.6340276Z ##[endgroup]
2024-12-18T00:29:24.6416128Z ##[group]Run set -x
2024-12-18T00:29:24.6416432Z set -x
2024-12-18T00:29:24.6416661Z 
2024-12-18T00:29:24.6416913Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2024-12-18T00:29:24.6417304Z   TEST_COMMAND=.ci/pytorch/multigpu-test.sh
2024-12-18T00:29:24.6417700Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2024-12-18T00:29:24.6418068Z   TEST_COMMAND=.ci/onnx/test.sh
2024-12-18T00:29:24.6418373Z else
2024-12-18T00:29:24.6418629Z   TEST_COMMAND=.ci/pytorch/test.sh
2024-12-18T00:29:24.6418945Z fi
2024-12-18T00:29:24.6419154Z 
2024-12-18T00:29:24.6419504Z # detached container should get cleaned up by teardown_ec2_linux
2024-12-18T00:29:24.6420038Z # TODO: Stop building test binaries as part of the build phase
2024-12-18T00:29:24.6420527Z # Used for GPU_FLAG since that doesn't play nice
2024-12-18T00:29:24.6420937Z # shellcheck disable=SC2086,SC2090
2024-12-18T00:29:24.6421288Z container_name=$(docker run \
2024-12-18T00:29:24.6421611Z   ${GPU_FLAG:-} \
2024-12-18T00:29:24.6421921Z   ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \
2024-12-18T00:29:24.6422372Z   -e BUILD_ENVIRONMENT \
2024-12-18T00:29:24.6422678Z   -e PR_NUMBER \
2024-12-18T00:29:24.6422966Z   -e GITHUB_ACTIONS \
2024-12-18T00:29:24.6423283Z   -e GITHUB_REPOSITORY \
2024-12-18T00:29:24.6423592Z   -e GITHUB_WORKFLOW \
2024-12-18T00:29:24.6423895Z   -e GITHUB_JOB \
2024-12-18T00:29:24.6424177Z   -e GITHUB_RUN_ID \
2024-12-18T00:29:24.6424476Z   -e GITHUB_RUN_NUMBER \
2024-12-18T00:29:24.6424787Z   -e GITHUB_RUN_ATTEMPT \
2024-12-18T00:29:24.6425079Z   -e JOB_ID \
2024-12-18T00:29:24.6425518Z   -e JOB_NAME \
2024-12-18T00:29:24.6425801Z   -e BASE_SHA \
2024-12-18T00:29:24.6426069Z   -e BRANCH \
2024-12-18T00:29:24.6426343Z   -e SHA1 \
2024-12-18T00:29:24.6426611Z   -e AWS_DEFAULT_REGION \
2024-12-18T00:29:24.6426927Z   -e IN_WHEEL_TEST \
2024-12-18T00:29:24.6427220Z   -e SHARD_NUMBER \
2024-12-18T00:29:24.6427508Z   -e TEST_CONFIG \
2024-12-18T00:29:24.6427796Z   -e NUM_TEST_SHARDS \
2024-12-18T00:29:24.6428097Z   -e REENABLED_ISSUES \
2024-12-18T00:29:24.6428423Z   -e CONTINUE_THROUGH_ERROR \
2024-12-18T00:29:24.6428767Z   -e VERBOSE_TEST_LOGS \
2024-12-18T00:29:24.6429092Z   -e TEST_SHOWLOCALS \
2024-12-18T00:29:24.6429410Z   -e NO_TEST_TIMEOUT \
2024-12-18T00:29:24.6429710Z   -e NO_TD \
2024-12-18T00:29:24.6430213Z   -e TD_DISTRIBUTED \
2024-12-18T00:29:24.6430589Z   -e PR_LABELS \
2024-12-18T00:29:24.6430900Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2024-12-18T00:29:24.6431240Z   -e SCCACHE_BUCKET \
2024-12-18T00:29:24.6431533Z   -e SCCACHE_REGION \
2024-12-18T00:29:24.6431821Z   -e SCCACHE_S3_KEY_PREFIX \
2024-12-18T00:29:24.6432127Z   -e XLA_CUDA \
2024-12-18T00:29:24.6432425Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2024-12-18T00:29:24.6432807Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2024-12-18T00:29:24.6433197Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2024-12-18T00:29:24.6433577Z   -e SKIP_SCCACHE_INITIALIZATION=1 \
2024-12-18T00:29:24.6433938Z   -e HUGGING_FACE_HUB_TOKEN \
2024-12-18T00:29:24.6434286Z   -e SCRIBE_GRAPHQL_ACCESS_TOKEN \
2024-12-18T00:29:24.6434619Z   -e DASHBOARD_TAG \
2024-12-18T00:29:24.6434909Z   -e IS_A100_RUNNER \
2024-12-18T00:29:24.6435201Z   -e ARTIFACTS_FILE_SUFFIX \
2024-12-18T00:29:24.6435563Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2024-12-18T00:29:24.6435968Z   --security-opt seccomp=unconfined \
2024-12-18T00:29:24.6436501Z   --cap-add=SYS_PTRACE \
2024-12-18T00:29:24.6436793Z   --ipc=host \
2024-12-18T00:29:24.6437071Z   --shm-size="${SHM_SIZE}" \
2024-12-18T00:29:24.6437356Z   --tty \
2024-12-18T00:29:24.6437597Z   --detach \
2024-12-18T00:29:24.6437867Z   --name="${container_name}" \
2024-12-18T00:29:24.6438175Z   --user jenkins \
2024-12-18T00:29:24.6438528Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2024-12-18T00:29:24.6438920Z   -w /var/lib/jenkins/workspace \
2024-12-18T00:29:24.6439244Z   "${DOCKER_IMAGE}"
2024-12-18T00:29:24.6439511Z )
2024-12-18T00:29:24.6439806Z # Propagate download.pytorch.org IP to container
2024-12-18T00:29:24.6440456Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts"
2024-12-18T00:29:24.6441269Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}"
2024-12-18T00:29:24.6442031Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"
2024-12-18T00:29:24.6450372Z shell: /usr/bin/bash -e {0}
2024-12-18T00:29:24.6450648Z env:
2024-12-18T00:29:24.6450873Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:24.6451209Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:24.6451664Z   BUILD_ENVIRONMENT: linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-12-18T00:29:24.6452037Z   PR_NUMBER: 
2024-12-18T00:29:24.6452294Z   GITHUB_REPOSITORY: pytorch/pytorch
2024-12-18T00:29:24.6452617Z   GITHUB_WORKFLOW: inductor
2024-12-18T00:29:24.6452890Z   GITHUB_JOB: test
2024-12-18T00:29:24.6453142Z   GITHUB_RUN_ID: 12383255690
2024-12-18T00:29:24.6453412Z   GITHUB_RUN_NUMBER: 109429
2024-12-18T00:29:24.6453680Z   GITHUB_RUN_ATTEMPT: 1
2024-12-18T00:29:24.6453931Z   JOB_ID: 34567153284
2024-12-18T00:29:24.6454535Z   JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:24.6455060Z   BRANCH: release/2.6
2024-12-18T00:29:24.6455338Z   SHA1: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:24.6455710Z   BASE_SHA: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:24.6456061Z   TEST_CONFIG: inductor_torchbench
2024-12-18T00:29:24.6456345Z   SHARD_NUMBER: 1
2024-12-18T00:29:24.6456577Z   NUM_TEST_SHARDS: 2
2024-12-18T00:29:24.6456813Z   REENABLED_ISSUES: 
2024-12-18T00:29:24.6457072Z   CONTINUE_THROUGH_ERROR: False
2024-12-18T00:29:24.6457353Z   VERBOSE_TEST_LOGS: False
2024-12-18T00:29:24.6457623Z   TEST_SHOWLOCALS: False
2024-12-18T00:29:24.6457884Z   NO_TEST_TIMEOUT: False
2024-12-18T00:29:24.6458125Z   NO_TD: False
2024-12-18T00:29:24.6458353Z   TD_DISTRIBUTED: False
2024-12-18T00:29:24.6458659Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2024-12-18T00:29:24.6459018Z   SCCACHE_REGION: us-east-1
2024-12-18T00:29:24.6459299Z   SCCACHE_S3_KEY_PREFIX: inductor
2024-12-18T00:29:24.6459581Z   SHM_SIZE: 2g
2024-12-18T00:29:24.6460391Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:29:24.6461255Z   XLA_CUDA: 
2024-12-18T00:29:24.6461611Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2024-12-18T00:29:24.6462141Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2024-12-18T00:29:24.6462463Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2024-12-18T00:29:24.6462757Z   DASHBOARD_TAG: 
2024-12-18T00:29:24.6463162Z   HUGGING_FACE_HUB_TOKEN: ***
2024-12-18T00:29:24.6463592Z   SCRIBE_GRAPHQL_ACCESS_TOKEN: ***
2024-12-18T00:29:24.6463913Z   IS_A100_RUNNER: 0
2024-12-18T00:29:24.6464364Z   ARTIFACTS_FILE_SUFFIX: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_34567153284
2024-12-18T00:29:24.6464876Z ##[endgroup]
2024-12-18T00:29:24.6491317Z + [[ inductor_torchbench == \m\u\l\t\i\g\p\u ]]
2024-12-18T00:29:24.6491838Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *onnx* ]]
2024-12-18T00:29:24.6492470Z + TEST_COMMAND=.ci/pytorch/test.sh
2024-12-18T00:29:24.6500351Z +++ nproc --ignore=2
2024-12-18T00:29:24.6522803Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_REGION -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e IS_A100_RUNNER -e ARTIFACTS_FILE_SUFFIX --env-file=/tmp/github_env_12383255690 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:29:36.4907928Z + container_name=5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T00:29:36.4909875Z + grep download.pytorch.org /etc/hosts
2024-12-18T00:29:36.4911645Z + docker exec -i 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f sudo bash -c '/bin/cat >> /etc/hosts'
2024-12-18T00:29:36.5999167Z + echo DOCKER_CONTAINER_ID=5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T00:29:36.6001585Z ++ echo dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl
2024-12-18T00:29:36.6004632Z + docker exec -t 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f sh -c 'python3 -m pip install dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh'
2024-12-18T00:29:36.9956508Z Processing ./dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl (from torch==2.6.0a0+git0cdf8b1)
2024-12-18T00:29:37.3271689Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.16.1)
2024-12-18T00:29:37.3274520Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (4.12.2)
2024-12-18T00:29:37.3278720Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (2.8.8)
2024-12-18T00:29:37.3281762Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.1.4)
2024-12-18T00:29:37.3286194Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (2024.10.0)
2024-12-18T00:29:37.3292493Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.13.1)
2024-12-18T00:29:37.3310023Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.3.0)
2024-12-18T00:29:37.3324083Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.3.0)
2024-12-18T00:29:37.3343360Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.22.4)
2024-12-18T00:29:37.3738015Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.0.2)
2024-12-18T00:29:37.9780475Z Installing collected packages: torch
2024-12-18T00:29:48.6873266Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
2024-12-18T00:29:48.6875071Z timm 1.0.9.dev0 requires torchvision, which is not installed.
2024-12-18T00:29:48.6875605Z Successfully installed torch-2.6.0a0+git0cdf8b1
2024-12-18T00:29:48.7671486Z + export TERM=vt100
2024-12-18T00:29:48.7671783Z + TERM=vt100
2024-12-18T00:29:48.7675339Z ++ dirname .ci/pytorch/test.sh
2024-12-18T00:29:48.7686844Z + source .ci/pytorch/common.sh
2024-12-18T00:29:48.7690399Z +++ dirname .ci/pytorch/common.sh
2024-12-18T00:29:48.7701243Z ++ source .ci/pytorch/common_utils.sh
2024-12-18T00:29:48.7702261Z +++ declare -f -t trap_add
2024-12-18T00:29:48.7708320Z ++ set -ex
2024-12-18T00:29:48.7708626Z ++ [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *rocm* ]]
2024-12-18T00:29:48.7708986Z ++ BUILD_TEST_LIBTORCH=0
2024-12-18T00:29:48.7709588Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *rocm* ]]
2024-12-18T00:29:48.7710072Z + [[ -d /var/lib/jenkins/workspace ]]
2024-12-18T00:29:48.7714308Z ++ stat -c %u /var/lib/jenkins/workspace
2024-12-18T00:29:48.7734489Z + WORKSPACE_ORIGINAL_OWNER_ID=1000
2024-12-18T00:29:48.7734827Z + trap_add cleanup_workspace EXIT
2024-12-18T00:29:48.7735134Z + trap_add_cmd=cleanup_workspace
2024-12-18T00:29:48.7735415Z + shift
2024-12-18T00:29:48.7735637Z + for trap_add_name in "$@"
2024-12-18T00:29:48.7742713Z +++ trap -p EXIT
2024-12-18T00:29:48.7745926Z ++ eval 'extract_trap_cmd '
2024-12-18T00:29:48.7746702Z +++ extract_trap_cmd
2024-12-18T00:29:48.7747024Z +++ printf '%s\n' ''
2024-12-18T00:29:48.7747296Z ++ printf '%s\n' cleanup_workspace
2024-12-18T00:29:48.7750820Z + trap -- '
2024-12-18T00:29:48.7751152Z cleanup_workspace' EXIT
2024-12-18T00:29:48.7751573Z + sudo chown -R jenkins /var/lib/jenkins/workspace
2024-12-18T00:29:49.4783262Z + git config --global --add safe.directory /var/lib/jenkins/workspace
2024-12-18T00:29:49.4807911Z + echo 'Environment variables:'
2024-12-18T00:29:49.4808703Z Environment variables:
2024-12-18T00:29:49.4809249Z + env
2024-12-18T00:29:49.4820259Z INSTALLED_DB=yes
2024-12-18T00:29:49.4820604Z NV_LIBCUBLAS_VERSION=12.4.5.8-1
2024-12-18T00:29:49.4820949Z NVIDIA_VISIBLE_DEVICES=all
2024-12-18T00:29:49.4821305Z NV_NVML_DEV_VERSION=12.4.127-1
2024-12-18T00:29:49.4821840Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-12-18T00:29:49.4822525Z CONTINUE_THROUGH_ERROR=False
2024-12-18T00:29:49.4822906Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.21.5-1+cuda12.4
2024-12-18T00:29:49.4823290Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.21.5-1
2024-12-18T00:29:49.4823752Z BUILD_ENVIRONMENT=linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-12-18T00:29:49.4824143Z HOSTNAME=5a407e2ba2ce
2024-12-18T00:29:49.4824767Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_34558096-485c-47b7-924a-2f4f48507737
2024-12-18T00:29:49.4825392Z GITHUB_ACTION=__self
2024-12-18T00:29:49.4825660Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2024-12-18T00:29:49.4830686Z NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536
2024-12-18T00:29:49.4835136Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-4=12.4.5.8-1
2024-12-18T00:29:49.4835505Z NV_NVTX_VERSION=12.4.127-1
2024-12-18T00:29:49.4835778Z GITHUB_RUN_NUMBER=109429
2024-12-18T00:29:49.4836054Z TEST_CONFIG=inductor_torchbench
2024-12-18T00:29:49.4836347Z GITHUB_REPOSITORY_OWNER_ID=21003710
2024-12-18T00:29:49.4836675Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2024-12-18T00:29:49.4836992Z IS_A100_RUNNER=0
2024-12-18T00:29:49.4837244Z NV_CUDA_CUDART_DEV_VERSION=12.4.127-1
2024-12-18T00:29:49.4837561Z NV_LIBCUSPARSE_VERSION=12.3.1.170-1
2024-12-18T00:29:49.4838066Z SCRIBE_GRAPHQL_ACCESS_TOKEN=***
2024-12-18T00:29:49.4838356Z NV_LIBNPP_VERSION=12.2.5.30-1
2024-12-18T00:29:49.4838648Z GITHUB_TRIGGERING_ACTOR=malfet
2024-12-18T00:29:49.4838985Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2024-12-18T00:29:49.4839343Z GITHUB_REF_TYPE=branch
2024-12-18T00:29:49.4839606Z TORCH_CUDA_ARCH_LIST=Maxwell
2024-12-18T00:29:49.4839879Z NCCL_VERSION=2.21.5-1
2024-12-18T00:29:49.4840178Z BASE_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:49.4840516Z XLA_CUDA=
2024-12-18T00:29:49.4840846Z HUGGING_FACE_HUB_TOKEN=***
2024-12-18T00:29:49.4843027Z ***
2024-12-18T00:29:49.4843262Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2024-12-18T00:29:49.4843563Z GITHUB_REPOSITORY_ID=65600975
2024-12-18T00:29:49.4843842Z GITHUB_ACTIONS=true
2024-12-18T00:29:49.4844098Z NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:49.4844631Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4=12.4.127-1
2024-12-18T00:29:49.4844998Z NV_LIBNPP_PACKAGE=libnpp-12-4=12.2.5.30-1
2024-12-18T00:29:49.4845362Z SHA1=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:49.4845715Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2024-12-18T00:29:49.4846073Z GITHUB_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:49.4846628Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/release/2.6
2024-12-18T00:29:49.4847137Z UCC_HOME=/usr
2024-12-18T00:29:49.4847370Z NV_LIBCUBLAS_DEV_VERSION=12.4.5.8-1
2024-12-18T00:29:49.4847670Z VERBOSE_TEST_LOGS=False
2024-12-18T00:29:49.4847930Z NVIDIA_PRODUCT_NAME=CUDA
2024-12-18T00:29:49.4848238Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-4
2024-12-18T00:29:49.4848590Z GITHUB_REF=refs/heads/release/2.6
2024-12-18T00:29:49.4848886Z NV_CUDA_CUDART_VERSION=12.4.127-1
2024-12-18T00:29:49.4849167Z SHARD_NUMBER=1
2024-12-18T00:29:49.4849403Z GITHUB_REF_PROTECTED=true
2024-12-18T00:29:49.4849672Z HOME=/var/lib/jenkins
2024-12-18T00:29:49.4849943Z GITHUB_API_URL=https://api.github.com
2024-12-18T00:29:49.4850272Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2024-12-18T00:29:49.4850624Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb
2024-12-18T00:29:49.4850980Z SCCACHE_S3_KEY_PREFIX=inductor
2024-12-18T00:29:49.4851260Z CUDA_VERSION=12.4.1
2024-12-18T00:29:49.4851531Z NV_LIBCUBLAS_PACKAGE=libcublas-12-4=12.4.5.8-1
2024-12-18T00:29:49.4851860Z NUM_TEST_SHARDS=2
2024-12-18T00:29:49.4852092Z UCX_HOME=/usr
2024-12-18T00:29:49.4852434Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-4=12.4.1-1
2024-12-18T00:29:49.4853178Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_34558096-485c-47b7-924a-2f4f48507737
2024-12-18T00:29:49.4854035Z JOB_NAME=cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:49.4854851Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_34558096-485c-47b7-924a-2f4f48507737
2024-12-18T00:29:49.4855650Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2024-12-18T00:29:49.4856251Z GITHUB_EVENT_NAME=push
2024-12-18T00:29:49.4856510Z DASHBOARD_TAG=
2024-12-18T00:29:49.4856765Z GITHUB_RUN_ID=12383255690
2024-12-18T00:29:49.4857073Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-4=12.2.5.30-1
2024-12-18T00:29:49.4857440Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-4
2024-12-18T00:29:49.4858118Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_34558096-485c-47b7-924a-2f4f48507737
2024-12-18T00:29:49.4858777Z GITHUB_ACTOR=malfet
2024-12-18T00:29:49.4859032Z NV_LIBNPP_DEV_VERSION=12.2.5.30-1
2024-12-18T00:29:49.4859317Z PR_NUMBER=
2024-12-18T00:29:49.4859531Z GITHUB_RUN_ATTEMPT=1
2024-12-18T00:29:49.4860015Z ANACONDA_PYTHON_VERSION=3.10
2024-12-18T00:29:49.4860358Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2024-12-18T00:29:49.4860741Z TERM=vt100
2024-12-18T00:29:49.4861001Z NV_LIBCUSPARSE_DEV_VERSION=12.3.1.170-1
2024-12-18T00:29:49.4861313Z INSTALLED_VISION=yes
2024-12-18T00:29:49.4861561Z BRANCH=release/2.6
2024-12-18T00:29:49.4861815Z SCCACHE_REGION=us-east-1
2024-12-18T00:29:49.4862194Z OPENSSL_ROOT_DIR=/opt/openssl
2024-12-18T00:29:49.4862581Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2024-12-18T00:29:49.4862902Z CUDA_PATH=/usr/local/cuda
2024-12-18T00:29:49.4863425Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2024-12-18T00:29:49.4864006Z GITHUB_SERVER_URL=https://github.com
2024-12-18T00:29:49.4864371Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b
2024-12-18T00:29:49.4864724Z REENABLED_ISSUES=
2024-12-18T00:29:49.4864947Z SHLVL=1
2024-12-18T00:29:49.4865152Z MAX_JOBS=14
2024-12-18T00:29:49.4865374Z NV_CUDA_LIB_VERSION=12.4.1-1
2024-12-18T00:29:49.4865643Z NVARCH=x86_64
2024-12-18T00:29:49.4865870Z GITHUB_ACTOR_ID=2453524
2024-12-18T00:29:49.4866202Z GITHUB_WORKFLOW_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:49.4866699Z GITHUB_REF_NAME=release/2.6
2024-12-18T00:29:49.4867004Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-4
2024-12-18T00:29:49.4867455Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2024-12-18T00:29:49.4867884Z GITHUB_JOB=test
2024-12-18T00:29:49.4868140Z NV_LIBNCCL_PACKAGE=libnccl2=2.21.5-1+cuda12.4
2024-12-18T00:29:49.4868555Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2024-12-18T00:29:49.4868937Z NO_TEST_TIMEOUT=False
2024-12-18T00:29:49.4869188Z TD_DISTRIBUTED=False
2024-12-18T00:29:49.4869458Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.1-1
2024-12-18T00:29:49.4869778Z GITHUB_REPOSITORY=pytorch/pytorch
2024-12-18T00:29:49.4870080Z NV_NVPROF_VERSION=12.4.127-1
2024-12-18T00:29:49.4870357Z GITHUB_RETENTION_DAYS=90
2024-12-18T00:29:49.4870620Z OPENSSL_DIR=/opt/openssl
2024-12-18T00:29:49.4870882Z GITHUB_ACTION_REPOSITORY=
2024-12-18T00:29:49.4871658Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-12-18T00:29:49.4872453Z GITHUB_BASE_REF=
2024-12-18T00:29:49.4872918Z ARTIFACTS_FILE_SUFFIX=test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_34567153284
2024-12-18T00:29:49.4873460Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2024-12-18T00:29:49.4873738Z CI=true
2024-12-18T00:29:49.4873954Z NV_LIBNCCL_PACKAGE_VERSION=2.21.5-1
2024-12-18T00:29:49.4874280Z GITHUB_REPOSITORY_OWNER=pytorch
2024-12-18T00:29:49.4874559Z JOB_ID=34567153284
2024-12-18T00:29:49.4874801Z INSTALLED_PROTOBUF=yes
2024-12-18T00:29:49.4875054Z GITHUB_HEAD_REF=
2024-12-18T00:29:49.4875280Z GITHUB_ACTION_REF=
2024-12-18T00:29:49.4875572Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2024-12-18T00:29:49.4875930Z TEST_SHOWLOCALS=False
2024-12-18T00:29:49.4876187Z GITHUB_WORKFLOW=inductor
2024-12-18T00:29:49.4876456Z DEBIAN_FRONTEND=noninteractive
2024-12-18T00:29:49.4877076Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_34558096-485c-47b7-924a-2f4f48507737
2024-12-18T00:29:49.4877705Z NO_TD=False
2024-12-18T00:29:49.4877939Z SKIP_SCCACHE_INITIALIZATION=1
2024-12-18T00:29:49.4878219Z _=/usr/bin/env
2024-12-18T00:29:49.4878636Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2024-12-18T00:29:49.5055443Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch
2024-12-18T00:29:49.5056794Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2024-12-18T00:29:49.5058211Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib
2024-12-18T00:29:49.5059411Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test
2024-12-18T00:29:49.5060279Z + BUILD_DIR=build
2024-12-18T00:29:49.5060627Z + BUILD_RENAMED_DIR=build_renamed
2024-12-18T00:29:49.5060914Z + BUILD_BIN_DIR=build/bin
2024-12-18T00:29:49.5061181Z + SHARD_NUMBER=1
2024-12-18T00:29:49.5061408Z + NUM_TEST_SHARDS=2
2024-12-18T00:29:49.5061642Z + export VALGRIND=ON
2024-12-18T00:29:49.5061875Z + VALGRIND=ON
2024-12-18T00:29:49.5062275Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *clang9* ]]
2024-12-18T00:29:49.5062789Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *xpu* ]]
2024-12-18T00:29:49.5063135Z + [[ 0 == \1 ]]
2024-12-18T00:29:49.5063402Z + [[ False == \1 ]]
2024-12-18T00:29:49.5063746Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *bazel* ]]
2024-12-18T00:29:49.5064122Z ++ realpath build/custom_test_artifacts
2024-12-18T00:29:49.5076680Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2024-12-18T00:29:49.5077259Z + [[ -n '' ]]
2024-12-18T00:29:49.5077506Z + echo 'Environment variables'
2024-12-18T00:29:49.5077794Z Environment variables
2024-12-18T00:29:49.5078034Z + env
2024-12-18T00:29:49.5085858Z INSTALLED_DB=yes
2024-12-18T00:29:49.5086117Z NV_LIBCUBLAS_VERSION=12.4.5.8-1
2024-12-18T00:29:49.5086440Z NVIDIA_VISIBLE_DEVICES=all
2024-12-18T00:29:49.5086775Z NV_NVML_DEV_VERSION=12.4.127-1
2024-12-18T00:29:49.5087198Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-12-18T00:29:49.5087874Z CONTINUE_THROUGH_ERROR=False
2024-12-18T00:29:49.5088288Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.21.5-1+cuda12.4
2024-12-18T00:29:49.5088671Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.21.5-1
2024-12-18T00:29:49.5089120Z BUILD_ENVIRONMENT=linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-12-18T00:29:49.5089575Z HOSTNAME=5a407e2ba2ce
2024-12-18T00:29:49.5090273Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_34558096-485c-47b7-924a-2f4f48507737
2024-12-18T00:29:49.5090895Z GITHUB_ACTION=__self
2024-12-18T00:29:49.5091160Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2024-12-18T00:29:49.5095501Z NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536
2024-12-18T00:29:49.5100254Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-4=12.4.5.8-1
2024-12-18T00:29:49.5100623Z NV_NVTX_VERSION=12.4.127-1
2024-12-18T00:29:49.5100894Z GITHUB_RUN_NUMBER=109429
2024-12-18T00:29:49.5101171Z TEST_CONFIG=inductor_torchbench
2024-12-18T00:29:49.5101477Z GITHUB_REPOSITORY_OWNER_ID=21003710
2024-12-18T00:29:49.5101804Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2024-12-18T00:29:49.5102426Z IS_A100_RUNNER=0
2024-12-18T00:29:49.5102688Z NV_CUDA_CUDART_DEV_VERSION=12.4.127-1
2024-12-18T00:29:49.5103008Z NV_LIBCUSPARSE_VERSION=12.3.1.170-1
2024-12-18T00:29:49.5103473Z SCRIBE_GRAPHQL_ACCESS_TOKEN=***
2024-12-18T00:29:49.5103763Z NV_LIBNPP_VERSION=12.2.5.30-1
2024-12-18T00:29:49.5104062Z GITHUB_TRIGGERING_ACTOR=malfet
2024-12-18T00:29:49.5104397Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2024-12-18T00:29:49.5104750Z GITHUB_REF_TYPE=branch
2024-12-18T00:29:49.5105013Z TORCH_CUDA_ARCH_LIST=Maxwell
2024-12-18T00:29:49.5105279Z NCCL_VERSION=2.21.5-1
2024-12-18T00:29:49.5105609Z BASE_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:49.5105939Z XLA_CUDA=
2024-12-18T00:29:49.5106283Z HUGGING_FACE_HUB_TOKEN=***
2024-12-18T00:29:49.5106847Z ***
2024-12-18T00:29:49.5107068Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2024-12-18T00:29:49.5107383Z GITHUB_REPOSITORY_ID=65600975
2024-12-18T00:29:49.5107653Z GITHUB_ACTIONS=true
2024-12-18T00:29:49.5107911Z NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:49.5108246Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4=12.4.127-1
2024-12-18T00:29:49.5108613Z NV_LIBNPP_PACKAGE=libnpp-12-4=12.2.5.30-1
2024-12-18T00:29:49.5108967Z SHA1=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:49.5109312Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2024-12-18T00:29:49.5109666Z GITHUB_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:49.5110212Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/release/2.6
2024-12-18T00:29:49.5110707Z UCC_HOME=/usr
2024-12-18T00:29:49.5110941Z NV_LIBCUBLAS_DEV_VERSION=12.4.5.8-1
2024-12-18T00:29:49.5111229Z VERBOSE_TEST_LOGS=False
2024-12-18T00:29:49.5111485Z NVIDIA_PRODUCT_NAME=CUDA
2024-12-18T00:29:49.5111789Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-4
2024-12-18T00:29:49.5112137Z GITHUB_REF=refs/heads/release/2.6
2024-12-18T00:29:49.5112539Z NV_CUDA_CUDART_VERSION=12.4.127-1
2024-12-18T00:29:49.5112819Z SHARD_NUMBER=1
2024-12-18T00:29:49.5113059Z GITHUB_REF_PROTECTED=true
2024-12-18T00:29:49.5113323Z HOME=/var/lib/jenkins
2024-12-18T00:29:49.5113599Z GITHUB_API_URL=https://api.github.com
2024-12-18T00:29:49.5113932Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2024-12-18T00:29:49.5114278Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb
2024-12-18T00:29:49.5114630Z SCCACHE_S3_KEY_PREFIX=inductor
2024-12-18T00:29:49.5114912Z CUDA_VERSION=12.4.1
2024-12-18T00:29:49.5115184Z NV_LIBCUBLAS_PACKAGE=libcublas-12-4=12.4.5.8-1
2024-12-18T00:29:49.5115514Z NUM_TEST_SHARDS=2
2024-12-18T00:29:49.5115735Z UCX_HOME=/usr
2024-12-18T00:29:49.5116077Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-4=12.4.1-1
2024-12-18T00:29:49.5116803Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_34558096-485c-47b7-924a-2f4f48507737
2024-12-18T00:29:49.5117645Z JOB_NAME=cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:49.5118468Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_34558096-485c-47b7-924a-2f4f48507737
2024-12-18T00:29:49.5119251Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2024-12-18T00:29:49.5119731Z GITHUB_EVENT_NAME=push
2024-12-18T00:29:49.5119983Z DASHBOARD_TAG=
2024-12-18T00:29:49.5120218Z GITHUB_RUN_ID=12383255690
2024-12-18T00:29:49.5120524Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-4=12.2.5.30-1
2024-12-18T00:29:49.5120887Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-4
2024-12-18T00:29:49.5121548Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_34558096-485c-47b7-924a-2f4f48507737
2024-12-18T00:29:49.5122193Z GITHUB_ACTOR=malfet
2024-12-18T00:29:49.5122442Z NV_LIBNPP_DEV_VERSION=12.2.5.30-1
2024-12-18T00:29:49.5122726Z PR_NUMBER=
2024-12-18T00:29:49.5122942Z GITHUB_RUN_ATTEMPT=1
2024-12-18T00:29:49.5123179Z VALGRIND=ON
2024-12-18T00:29:49.5123420Z ANACONDA_PYTHON_VERSION=3.10
2024-12-18T00:29:49.5123753Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2024-12-18T00:29:49.5124205Z TERM=vt100
2024-12-18T00:29:49.5124444Z NV_LIBCUSPARSE_DEV_VERSION=12.3.1.170-1
2024-12-18T00:29:49.5124750Z INSTALLED_VISION=yes
2024-12-18T00:29:49.5124996Z BRANCH=release/2.6
2024-12-18T00:29:49.5125243Z SCCACHE_REGION=us-east-1
2024-12-18T00:29:49.5125520Z OPENSSL_ROOT_DIR=/opt/openssl
2024-12-18T00:29:49.5125810Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2024-12-18T00:29:49.5126134Z CUDA_PATH=/usr/local/cuda
2024-12-18T00:29:49.5126644Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2024-12-18T00:29:49.5127206Z GITHUB_SERVER_URL=https://github.com
2024-12-18T00:29:49.5127559Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b
2024-12-18T00:29:49.5127900Z REENABLED_ISSUES=
2024-12-18T00:29:49.5128120Z SHLVL=1
2024-12-18T00:29:49.5128329Z MAX_JOBS=14
2024-12-18T00:29:49.5128550Z NV_CUDA_LIB_VERSION=12.4.1-1
2024-12-18T00:29:49.5128819Z NVARCH=x86_64
2024-12-18T00:29:49.5129048Z GITHUB_ACTOR_ID=2453524
2024-12-18T00:29:49.5129387Z GITHUB_WORKFLOW_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:49.5129773Z GITHUB_REF_NAME=release/2.6
2024-12-18T00:29:49.5130415Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-4
2024-12-18T00:29:49.5130905Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2024-12-18T00:29:49.5131332Z GITHUB_JOB=test
2024-12-18T00:29:49.5131600Z NV_LIBNCCL_PACKAGE=libnccl2=2.21.5-1+cuda12.4
2024-12-18T00:29:49.5132006Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2024-12-18T00:29:49.5132393Z NO_TEST_TIMEOUT=False
2024-12-18T00:29:49.5132650Z TD_DISTRIBUTED=False
2024-12-18T00:29:49.5132915Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.1-1
2024-12-18T00:29:49.5133244Z GITHUB_REPOSITORY=pytorch/pytorch
2024-12-18T00:29:49.5133556Z NV_NVPROF_VERSION=12.4.127-1
2024-12-18T00:29:49.5133836Z GITHUB_RETENTION_DAYS=90
2024-12-18T00:29:49.5134105Z OPENSSL_DIR=/opt/openssl
2024-12-18T00:29:49.5134534Z GITHUB_ACTION_REPOSITORY=
2024-12-18T00:29:49.5135305Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-12-18T00:29:49.5136087Z GITHUB_BASE_REF=
2024-12-18T00:29:49.5136548Z ARTIFACTS_FILE_SUFFIX=test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_34567153284
2024-12-18T00:29:49.5137082Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2024-12-18T00:29:49.5137356Z CI=true
2024-12-18T00:29:49.5137572Z NV_LIBNCCL_PACKAGE_VERSION=2.21.5-1
2024-12-18T00:29:49.5137883Z GITHUB_REPOSITORY_OWNER=pytorch
2024-12-18T00:29:49.5138158Z JOB_ID=34567153284
2024-12-18T00:29:49.5138394Z INSTALLED_PROTOBUF=yes
2024-12-18T00:29:49.5138646Z GITHUB_HEAD_REF=
2024-12-18T00:29:49.5138874Z GITHUB_ACTION_REF=
2024-12-18T00:29:49.5139161Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2024-12-18T00:29:49.5139513Z TEST_SHOWLOCALS=False
2024-12-18T00:29:49.5139771Z GITHUB_WORKFLOW=inductor
2024-12-18T00:29:49.5140051Z DEBIAN_FRONTEND=noninteractive
2024-12-18T00:29:49.5140661Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_34558096-485c-47b7-924a-2f4f48507737
2024-12-18T00:29:49.5141270Z NO_TD=False
2024-12-18T00:29:49.5141502Z SKIP_SCCACHE_INITIALIZATION=1
2024-12-18T00:29:49.5141780Z _=/usr/bin/env
2024-12-18T00:29:49.5142065Z + echo 'Testing pytorch'
2024-12-18T00:29:49.5142326Z Testing pytorch
2024-12-18T00:29:49.5142563Z + export LANG=C.UTF-8
2024-12-18T00:29:49.5142804Z + LANG=C.UTF-8
2024-12-18T00:29:49.5143025Z + PR_NUMBER=
2024-12-18T00:29:49.5143268Z + [[ inductor_torchbench == \d\e\f\a\u\l\t ]]
2024-12-18T00:29:49.5143628Z + [[ inductor_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]]
2024-12-18T00:29:49.5143981Z + [[ inductor_torchbench == \s\l\o\w ]]
2024-12-18T00:29:49.5144386Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *slow-gradcheck* ]]
2024-12-18T00:29:49.5144846Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *cuda* ]]
2024-12-18T00:29:49.5145235Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2024-12-18T00:29:49.5145576Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2024-12-18T00:29:49.5146038Z + [[ inductor_torchbench == *crossref* ]]
2024-12-18T00:29:49.5146414Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *rocm* ]]
2024-12-18T00:29:49.5146825Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *xpu* ]]
2024-12-18T00:29:49.5147238Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]]
2024-12-18T00:29:49.5147625Z + pip_install --user ninja==1.10.2
2024-12-18T00:29:49.5148015Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T00:29:49.5148503Z + python3 -m pip install --progress-bar off --user ninja==1.10.2
2024-12-18T00:29:50.8511428Z Collecting ninja==1.10.2
2024-12-18T00:29:50.8684485Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB)
2024-12-18T00:29:50.9296640Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2024-12-18T00:29:51.5375426Z Installing collected packages: ninja
2024-12-18T00:29:51.5455335Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2024-12-18T00:29:51.5456264Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2024-12-18T00:29:51.5940575Z Successfully installed ninja-1.10.2
2024-12-18T00:29:51.6750457Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-12-18T00:29:51.6752190Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-12-18T00:29:51.6753163Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *aarch64* ]]
2024-12-18T00:29:51.6753538Z + install_tlparse
2024-12-18T00:29:51.6754032Z + pip_install --user tlparse==0.3.25
2024-12-18T00:29:51.6754433Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T00:29:51.6754929Z + python3 -m pip install --progress-bar off --user tlparse==0.3.25
2024-12-18T00:29:52.0572206Z Collecting tlparse==0.3.25
2024-12-18T00:29:52.0843771Z   Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB)
2024-12-18T00:29:52.0977920Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB)
2024-12-18T00:29:52.6814372Z Installing collected packages: tlparse
2024-12-18T00:29:52.7154120Z Successfully installed tlparse-0.3.25
2024-12-18T00:29:52.7945559Z ++ python -m site --user-base
2024-12-18T00:29:52.8189295Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-12-18T00:29:52.8191651Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *asan* ]]
2024-12-18T00:29:52.8192136Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *-debug* ]]
2024-12-18T00:29:52.8192705Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]]
2024-12-18T00:29:52.8193509Z + echo 'We are not in debug mode: linux-focal-cuda12.4-py3.10-gcc9-sm86. Expect the assertion to pass'
2024-12-18T00:29:52.8194248Z We are not in debug mode: linux-focal-cuda12.4-py3.10-gcc9-sm86. Expect the assertion to pass
2024-12-18T00:29:52.8194760Z + cd test
2024-12-18T00:29:52.8195110Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)'
2024-12-18T00:29:54.4960440Z + [[ inductor_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2024-12-18T00:29:54.4960892Z + [[ inductor_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2024-12-18T00:29:54.4965349Z + DYNAMO_BENCHMARK_FLAGS=()
2024-12-18T00:29:54.4965794Z + [[ inductor_torchbench == *pr_time_benchmarks* ]]
2024-12-18T00:29:54.4966166Z + [[ inductor_torchbench == *dynamo_eager* ]]
2024-12-18T00:29:54.4966543Z + [[ inductor_torchbench == *aot_eager* ]]
2024-12-18T00:29:54.4966891Z + [[ inductor_torchbench == *aot_inductor* ]]
2024-12-18T00:29:54.4967486Z + [[ inductor_torchbench == *inductor* ]]
2024-12-18T00:29:54.4967825Z + [[ inductor_torchbench != *perf* ]]
2024-12-18T00:29:54.4968161Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor)
2024-12-18T00:29:54.4968498Z + [[ inductor_torchbench == *dynamic* ]]
2024-12-18T00:29:54.4968820Z + [[ inductor_torchbench == *cpu* ]]
2024-12-18T00:29:54.4969139Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda)
2024-12-18T00:29:54.4997821Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *libtorch* ]]
2024-12-18T00:29:54.4998276Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *-bazel-* ]]
2024-12-18T00:29:54.5001511Z + cd test
2024-12-18T00:29:54.5002350Z + python -c 'import torch; print(torch.__config__.show())'
2024-12-18T00:29:55.9873706Z PyTorch built with:
2024-12-18T00:29:55.9873996Z   - GCC 9.4
2024-12-18T00:29:55.9874218Z   - C++ Version: 201703
2024-12-18T00:29:55.9874781Z   - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2024-12-18T00:29:55.9875563Z   - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af)
2024-12-18T00:29:55.9876006Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2024-12-18T00:29:55.9876350Z   - LAPACK is enabled (usually provided by MKL)
2024-12-18T00:29:55.9876682Z   - NNPACK is enabled
2024-12-18T00:29:55.9876947Z   - CPU capability usage: AVX2
2024-12-18T00:29:55.9877229Z   - CUDA Runtime 12.4
2024-12-18T00:29:55.9877575Z   - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86
2024-12-18T00:29:55.9877972Z   - CuDNN 90.1
2024-12-18T00:29:55.9878206Z   - Magma 2.6.1
2024-12-18T00:29:55.9883094Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7, CUDA_VERSION=12.4, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.6.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 
2024-12-18T00:29:55.9889494Z 
2024-12-18T00:29:56.3184919Z + cd test
2024-12-18T00:29:56.3185327Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2024-12-18T00:29:57.6877943Z ATen/Parallel:
2024-12-18T00:29:57.6878281Z 	at::get_num_threads() : 8
2024-12-18T00:29:57.6878580Z 	at::get_num_interop_threads() : 16
2024-12-18T00:29:57.6878900Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2024-12-18T00:29:57.6879195Z 	omp_get_max_threads() : 8
2024-12-18T00:29:57.6879735Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2024-12-18T00:29:57.6880322Z 	mkl_get_max_threads() : 8
2024-12-18T00:29:57.6880717Z Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af)
2024-12-18T00:29:57.6881163Z std::thread::hardware_concurrency() : 16
2024-12-18T00:29:57.6881481Z Environment variables:
2024-12-18T00:29:57.6881743Z 	OMP_NUM_THREADS : [not set]
2024-12-18T00:29:57.6882009Z 	MKL_NUM_THREADS : [not set]
2024-12-18T00:29:57.6882288Z ATen parallel backend: OpenMP
2024-12-18T00:29:57.6882492Z 
2024-12-18T00:29:57.9718168Z + [[ inductor_torchbench == *numpy_2* ]]
2024-12-18T00:29:57.9719091Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *aarch64* ]]
2024-12-18T00:29:57.9719526Z + [[ inductor_torchbench == *backward* ]]
2024-12-18T00:29:57.9719870Z + [[ inductor_torchbench == *xla* ]]
2024-12-18T00:29:57.9720206Z + [[ inductor_torchbench == *executorch* ]]
2024-12-18T00:29:57.9720562Z + [[ inductor_torchbench == \j\i\t\_\l\e\g\a\c\y ]]
2024-12-18T00:29:57.9720980Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *libtorch* ]]
2024-12-18T00:29:57.9721380Z + [[ inductor_torchbench == distributed ]]
2024-12-18T00:29:57.9721743Z + [[ inductor_torchbench == *inductor_distributed* ]]
2024-12-18T00:29:57.9722126Z + [[ inductor_torchbench == *inductor-halide* ]]
2024-12-18T00:29:57.9722652Z + [[ inductor_torchbench == *inductor-triton-cpu* ]]
2024-12-18T00:29:57.9723113Z + [[ inductor_torchbench == *inductor-micro-benchmark* ]]
2024-12-18T00:29:57.9723539Z + [[ inductor_torchbench == *huggingface* ]]
2024-12-18T00:29:57.9723907Z + [[ inductor_torchbench == *timm* ]]
2024-12-18T00:29:57.9724256Z + [[ inductor_torchbench == *torchbench* ]]
2024-12-18T00:29:57.9724711Z + [[ inductor_torchbench == *cpu* ]]
2024-12-18T00:29:57.9725119Z + install_torchaudio cuda
2024-12-18T00:29:57.9725411Z + local commit
2024-12-18T00:29:57.9725657Z ++ get_pinned_commit audio
2024-12-18T00:29:57.9725953Z ++ cat .github/ci_commit_pins/audio.txt
2024-12-18T00:29:57.9743698Z + commit=332760d4b300f00a0d862e3cfe1495db3b1a14f9
2024-12-18T00:29:57.9744181Z + [[ cuda == \c\u\d\a ]]
2024-12-18T00:29:57.9744537Z + TORCH_CUDA_ARCH_LIST='8.0;8.6'
2024-12-18T00:29:57.9745215Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@332760d4b300f00a0d862e3cfe1495db3b1a14f9
2024-12-18T00:29:57.9745918Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T00:29:57.9746746Z + python3 -m pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@332760d4b300f00a0d862e3cfe1495db3b1a14f9
2024-12-18T00:29:58.3104419Z Collecting git+https://github.com/pytorch/audio.git@332760d4b300f00a0d862e3cfe1495db3b1a14f9
2024-12-18T00:29:58.3109430Z   Cloning https://github.com/pytorch/audio.git (to revision 332760d4b300f00a0d862e3cfe1495db3b1a14f9) to /tmp/pip-req-build-gbq1xtwj
2024-12-18T00:29:58.3141012Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-gbq1xtwj
2024-12-18T00:29:59.1612263Z   Running command git rev-parse -q --verify 'sha^332760d4b300f00a0d862e3cfe1495db3b1a14f9'
2024-12-18T00:29:59.1639933Z   Running command git fetch -q https://github.com/pytorch/audio.git 332760d4b300f00a0d862e3cfe1495db3b1a14f9
2024-12-18T00:29:59.5856639Z   Running command git checkout -q 332760d4b300f00a0d862e3cfe1495db3b1a14f9
2024-12-18T00:29:59.7463546Z   Resolved https://github.com/pytorch/audio.git to commit 332760d4b300f00a0d862e3cfe1495db3b1a14f9
2024-12-18T00:29:59.7464393Z   Running command git submodule update --init --recursive -q
2024-12-18T00:30:02.2926581Z   Preparing metadata (setup.py) ... [?25l- done
2024-12-18T00:30:02.2956498Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.5.0a0+332760d) (2.6.0a0+git0cdf8b1)
2024-12-18T00:30:02.2982440Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.5.0a0+332760d) (3.16.1)
2024-12-18T00:30:02.2986095Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.5.0a0+332760d) (4.12.2)
2024-12-18T00:30:02.2989261Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.5.0a0+332760d) (2.8.8)
2024-12-18T00:30:02.2992386Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.5.0a0+332760d) (3.1.4)
2024-12-18T00:30:02.2995591Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.5.0a0+332760d) (2024.10.0)
2024-12-18T00:30:02.3000905Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.5.0a0+332760d) (1.13.1)
2024-12-18T00:30:02.3015107Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchaudio==2.5.0a0+332760d) (1.3.0)
2024-12-18T00:30:02.3491846Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.5.0a0+332760d) (3.0.2)
2024-12-18T00:30:02.3553486Z Building wheels for collected packages: torchaudio
2024-12-18T00:32:14.7042849Z   Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done
2024-12-18T00:32:14.7080525Z [?25h  Created wheel for torchaudio: filename=torchaudio-2.5.0a0+332760d-cp310-cp310-linux_x86_64.whl size=2326947 sha256=671feadc35bf90a776b1eda2e40f257827044068ba46b3dd1323cfb5f67cc773
2024-12-18T00:32:14.7083713Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/78/2e/5c/66e884f380e2436e82a6c71c80865fe12e30aca3d9350545dd
2024-12-18T00:32:14.7113040Z Successfully built torchaudio
2024-12-18T00:32:15.2259677Z Installing collected packages: torchaudio
2024-12-18T00:32:15.4441415Z Successfully installed torchaudio-2.5.0a0+332760d
2024-12-18T00:32:15.6272687Z + install_torchvision
2024-12-18T00:32:15.6273015Z + local orig_preload
2024-12-18T00:32:15.6273290Z + local commit
2024-12-18T00:32:15.6277612Z ++ get_pinned_commit vision
2024-12-18T00:32:15.6277920Z ++ cat .github/ci_commit_pins/vision.txt
2024-12-18T00:32:15.6305414Z + commit=d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:32:15.6305752Z + orig_preload=
2024-12-18T00:32:15.6305990Z + '[' -n '' ']'
2024-12-18T00:32:15.6306539Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:32:15.6307437Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T00:32:15.6308226Z + python3 -m pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:32:15.9675982Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:32:15.9679374Z   Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-dq96vjsz
2024-12-18T00:32:15.9718629Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-dq96vjsz
2024-12-18T00:32:17.4630979Z   Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f'
2024-12-18T00:32:17.4659181Z   Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:32:18.8529399Z   Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:32:19.1992506Z   Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:32:21.9764061Z   Preparing metadata (setup.py) ... [?25l- \ done
2024-12-18T00:32:21.9797413Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4)
2024-12-18T00:32:21.9800669Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.6.0a0+git0cdf8b1)
2024-12-18T00:32:21.9805195Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (11.0.0)
2024-12-18T00:32:21.9872837Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.16.1)
2024-12-18T00:32:21.9876578Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2)
2024-12-18T00:32:21.9880274Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8)
2024-12-18T00:32:21.9883374Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4)
2024-12-18T00:32:21.9886638Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.10.0)
2024-12-18T00:32:21.9891601Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.1)
2024-12-18T00:32:21.9905885Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0)
2024-12-18T00:32:22.0384667Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (3.0.2)
2024-12-18T00:32:22.0448131Z Building wheels for collected packages: torchvision
2024-12-18T00:33:35.2116012Z   Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done
2024-12-18T00:33:35.2148085Z [?25h  Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2028112 sha256=c2048513a2bccb956a4323f8ee8c746b8e456bf9f2a94148e41336945bd94b47
2024-12-18T00:33:35.2151049Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96
2024-12-18T00:33:35.2183679Z Successfully built torchvision
2024-12-18T00:33:35.7268592Z Installing collected packages: torchvision
2024-12-18T00:33:36.1467421Z Successfully installed torchvision-0.19.0a0+d23a6e1
2024-12-18T00:33:36.2892503Z + '[' -n '' ']'
2024-12-18T00:33:36.2892762Z + TORCH_CUDA_ARCH_LIST='8.0;8.6'
2024-12-18T00:33:36.2893126Z + pip_install git+https://github.com/pytorch/ao.git
2024-12-18T00:33:36.2893577Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T00:33:36.2894137Z + python3 -m pip install --progress-bar off git+https://github.com/pytorch/ao.git
2024-12-18T00:33:36.6264055Z Collecting git+https://github.com/pytorch/ao.git
2024-12-18T00:33:36.6267977Z   Cloning https://github.com/pytorch/ao.git to /tmp/pip-req-build-uvgy9eq9
2024-12-18T00:33:36.6297398Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/ao.git /tmp/pip-req-build-uvgy9eq9
2024-12-18T00:33:37.3726044Z   Resolved https://github.com/pytorch/ao.git to commit a5a53a2bd00f18401baaa00db4467f955704f71f
2024-12-18T00:33:39.9457405Z   Preparing metadata (setup.py) ... [?25l- done
2024-12-18T00:33:39.9498048Z [?25hBuilding wheels for collected packages: torchao
2024-12-18T00:35:03.9378688Z   Building wheel for torchao (setup.py) ... [?25l- \ | / - \ | / - \ done
2024-12-18T00:35:03.9417640Z [?25h  Created wheel for torchao: filename=torchao-0.8.0+gita5a53a2b-cp310-cp310-linux_x86_64.whl size=3081972 sha256=d291f0491a7ff07bb0a5960b9559aec07ad155d426d2eba4c07b621c5601577a
2024-12-18T00:35:03.9421662Z   Stored in directory: /tmp/pip-ephem-wheel-cache-pursz_x1/wheels/c0/df/d8/b1e2f5f1ea3bf141268a83a667137647b234f04c636e1e7a5e
2024-12-18T00:35:03.9456061Z Successfully built torchao
2024-12-18T00:35:04.5145239Z Installing collected packages: torchao
2024-12-18T00:35:05.1221346Z Successfully installed torchao-0.8.0+gita5a53a2b
2024-12-18T00:35:05.2631492Z + id=0
2024-12-18T00:35:05.2631763Z + pip_install opencv-python==4.8.0.74
2024-12-18T00:35:05.2632180Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T00:35:05.2632682Z + python3 -m pip install --progress-bar off opencv-python==4.8.0.74
2024-12-18T00:35:05.7477515Z Collecting opencv-python==4.8.0.74
2024-12-18T00:35:05.7694154Z   Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB)
2024-12-18T00:35:05.7789076Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.22.4)
2024-12-18T00:35:05.7859774Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB)
2024-12-18T00:35:06.9299505Z Installing collected packages: opencv-python
2024-12-18T00:35:07.8221429Z Successfully installed opencv-python-4.8.0.74
2024-12-18T00:35:07.9199905Z + [[ inductor_torchbench == *inductor_torchbench_smoketest_perf* ]]
2024-12-18T00:35:07.9200947Z + [[ inductor_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]]
2024-12-18T00:35:07.9201941Z + [[ inductor_torchbench == *torchbench_gcp_smoketest* ]]
2024-12-18T00:35:07.9202572Z + checkout_install_torchbench
2024-12-18T00:35:07.9202869Z + local commit
2024-12-18T00:35:07.9203170Z ++ get_pinned_commit torchbench
2024-12-18T00:35:07.9203533Z ++ cat .github/ci_commit_pins/torchbench.txt
2024-12-18T00:35:07.9225973Z + commit=766a5e3a189384659fd35a68c3b17b88c761aaac
2024-12-18T00:35:07.9226429Z + git clone https://github.com/pytorch/benchmark torchbench
2024-12-18T00:35:07.9254626Z Cloning into 'torchbench'...
2024-12-18T00:35:08.1439273Z remote: Enumerating objects: 34540, done.
2024-12-18T00:35:08.1440160Z remote: Counting objects:   0% (1/1664)
2024-12-18T00:35:08.1441128Z remote: Counting objects:   1% (17/1664)
2024-12-18T00:35:08.1441528Z remote: Counting objects:   2% (34/1664)
2024-12-18T00:35:08.1442034Z remote: Counting objects:   3% (50/1664)
2024-12-18T00:35:08.1442430Z remote: Counting objects:   4% (67/1664)
2024-12-18T00:35:08.1442904Z remote: Counting objects:   5% (84/1664)
2024-12-18T00:35:08.1443291Z remote: Counting objects:   6% (100/1664)
2024-12-18T00:35:08.1443686Z remote: Counting objects:   7% (117/1664)
2024-12-18T00:35:08.1444164Z remote: Counting objects:   8% (134/1664)
2024-12-18T00:35:08.1444567Z remote: Counting objects:   9% (150/1664)
2024-12-18T00:35:08.1445047Z remote: Counting objects:  10% (167/1664)
2024-12-18T00:35:08.1445444Z remote: Counting objects:  11% (184/1664)
2024-12-18T00:35:08.1445841Z remote: Counting objects:  12% (200/1664)
2024-12-18T00:35:08.1446236Z remote: Counting objects:  13% (217/1664)
2024-12-18T00:35:08.1446722Z remote: Counting objects:  14% (233/1664)
2024-12-18T00:35:08.1447131Z remote: Counting objects:  15% (250/1664)
2024-12-18T00:35:08.1447632Z remote: Counting objects:  16% (267/1664)
2024-12-18T00:35:08.1448029Z remote: Counting objects:  17% (283/1664)
2024-12-18T00:35:08.1448422Z remote: Counting objects:  18% (300/1664)
2024-12-18T00:35:08.1448818Z remote: Counting objects:  19% (317/1664)
2024-12-18T00:35:08.1449287Z remote: Counting objects:  20% (333/1664)
2024-12-18T00:35:08.1449686Z remote: Counting objects:  21% (350/1664)
2024-12-18T00:35:08.1450151Z remote: Counting objects:  22% (367/1664)
2024-12-18T00:35:08.1450623Z remote: Counting objects:  23% (383/1664)
2024-12-18T00:35:08.1451027Z remote: Counting objects:  24% (400/1664)
2024-12-18T00:35:08.1451512Z remote: Counting objects:  25% (416/1664)
2024-12-18T00:35:08.1451999Z remote: Counting objects:  26% (433/1664)
2024-12-18T00:35:08.1452506Z remote: Counting objects:  27% (450/1664)
2024-12-18T00:35:08.1452967Z remote: Counting objects:  28% (466/1664)
2024-12-18T00:35:08.1453447Z remote: Counting objects:  29% (483/1664)
2024-12-18T00:35:08.1453863Z remote: Counting objects:  30% (500/1664)
2024-12-18T00:35:08.1454515Z remote: Counting objects:  31% (516/1664)
2024-12-18T00:35:08.1455034Z remote: Counting objects:  32% (533/1664)
2024-12-18T00:35:08.1455432Z remote: Counting objects:  33% (550/1664)
2024-12-18T00:35:08.1455908Z remote: Counting objects:  34% (566/1664)
2024-12-18T00:35:08.1456310Z remote: Counting objects:  35% (583/1664)
2024-12-18T00:35:08.1456777Z remote: Counting objects:  36% (600/1664)
2024-12-18T00:35:08.1457177Z remote: Counting objects:  37% (616/1664)
2024-12-18T00:35:08.1457653Z remote: Counting objects:  38% (633/1664)
2024-12-18T00:35:08.1458054Z remote: Counting objects:  39% (649/1664)
2024-12-18T00:35:08.1458529Z remote: Counting objects:  40% (666/1664)
2024-12-18T00:35:08.1458920Z remote: Counting objects:  41% (683/1664)
2024-12-18T00:35:08.1459399Z remote: Counting objects:  42% (699/1664)
2024-12-18T00:35:08.1459806Z remote: Counting objects:  43% (716/1664)
2024-12-18T00:35:08.1460284Z remote: Counting objects:  44% (733/1664)
2024-12-18T00:35:08.1460687Z remote: Counting objects:  45% (749/1664)
2024-12-18T00:35:08.1461159Z remote: Counting objects:  46% (766/1664)
2024-12-18T00:35:08.1461559Z remote: Counting objects:  47% (783/1664)
2024-12-18T00:35:08.1462107Z remote: Counting objects:  48% (799/1664)
2024-12-18T00:35:08.1462533Z remote: Counting objects:  49% (816/1664)
2024-12-18T00:35:08.1462985Z remote: Counting objects:  50% (832/1664)
2024-12-18T00:35:08.1463403Z remote: Counting objects:  51% (849/1664)
2024-12-18T00:35:08.1463841Z remote: Counting objects:  52% (866/1664)
2024-12-18T00:35:08.1464259Z remote: Counting objects:  53% (882/1664)
2024-12-18T00:35:08.1464707Z remote: Counting objects:  54% (899/1664)
2024-12-18T00:35:08.1465130Z remote: Counting objects:  55% (916/1664)
2024-12-18T00:35:08.1465577Z remote: Counting objects:  56% (932/1664)
2024-12-18T00:35:08.1466175Z remote: Counting objects:  57% (949/1664)
2024-12-18T00:35:08.1466576Z remote: Counting objects:  58% (966/1664)
2024-12-18T00:35:08.1467049Z remote: Counting objects:  59% (982/1664)
2024-12-18T00:35:08.1467451Z remote: Counting objects:  60% (999/1664)
2024-12-18T00:35:08.1467921Z remote: Counting objects:  61% (1016/1664)
2024-12-18T00:35:08.1468323Z remote: Counting objects:  62% (1032/1664)
2024-12-18T00:35:08.1468807Z remote: Counting objects:  63% (1049/1664)
2024-12-18T00:35:08.1469215Z remote: Counting objects:  64% (1065/1664)
2024-12-18T00:35:08.1469691Z remote: Counting objects:  65% (1082/1664)
2024-12-18T00:35:08.1470096Z remote: Counting objects:  66% (1099/1664)
2024-12-18T00:35:08.1470565Z remote: Counting objects:  67% (1115/1664)
2024-12-18T00:35:08.1470964Z remote: Counting objects:  68% (1132/1664)
2024-12-18T00:35:08.1471458Z remote: Counting objects:  69% (1149/1664)
2024-12-18T00:35:08.1471869Z remote: Counting objects:  70% (1165/1664)
2024-12-18T00:35:08.1472345Z remote: Counting objects:  71% (1182/1664)
2024-12-18T00:35:08.1472754Z remote: Counting objects:  72% (1199/1664)
2024-12-18T00:35:08.1473224Z remote: Counting objects:  73% (1215/1664)
2024-12-18T00:35:08.1473626Z remote: Counting objects:  74% (1232/1664)
2024-12-18T00:35:08.1474103Z remote: Counting objects:  75% (1248/1664)
2024-12-18T00:35:08.1474508Z remote: Counting objects:  76% (1265/1664)
2024-12-18T00:35:08.1474986Z remote: Counting objects:  77% (1282/1664)
2024-12-18T00:35:08.1475392Z remote: Counting objects:  78% (1298/1664)
2024-12-18T00:35:08.1475862Z remote: Counting objects:  79% (1315/1664)
2024-12-18T00:35:08.1476279Z remote: Counting objects:  80% (1332/1664)
2024-12-18T00:35:08.1476742Z remote: Counting objects:  81% (1348/1664)
2024-12-18T00:35:08.1477168Z remote: Counting objects:  82% (1365/1664)
2024-12-18T00:35:08.1477619Z remote: Counting objects:  83% (1382/1664)
2024-12-18T00:35:08.1478066Z remote: Counting objects:  84% (1398/1664)
2024-12-18T00:35:08.1478510Z remote: Counting objects:  85% (1415/1664)
2024-12-18T00:35:08.1479094Z remote: Counting objects:  86% (1432/1664)
2024-12-18T00:35:08.1479517Z remote: Counting objects:  87% (1448/1664)
2024-12-18T00:35:08.1479994Z remote: Counting objects:  88% (1465/1664)
2024-12-18T00:35:08.1480400Z remote: Counting objects:  89% (1481/1664)
2024-12-18T00:35:08.1480881Z remote: Counting objects:  90% (1498/1664)
2024-12-18T00:35:08.1481290Z remote: Counting objects:  91% (1515/1664)
2024-12-18T00:35:08.1481769Z remote: Counting objects:  92% (1531/1664)
2024-12-18T00:35:08.1482172Z remote: Counting objects:  93% (1548/1664)
2024-12-18T00:35:08.1482703Z remote: Counting objects:  94% (1565/1664)
2024-12-18T00:35:08.1483103Z remote: Counting objects:  95% (1581/1664)
2024-12-18T00:35:08.1483574Z remote: Counting objects:  96% (1598/1664)
2024-12-18T00:35:08.1483979Z remote: Counting objects:  97% (1615/1664)
2024-12-18T00:35:08.1484389Z remote: Counting objects:  98% (1631/1664)
2024-12-18T00:35:08.1484782Z remote: Counting objects:  99% (1648/1664)
2024-12-18T00:35:08.1485273Z remote: Counting objects: 100% (1664/1664)
2024-12-18T00:35:08.1485713Z remote: Counting objects: 100% (1664/1664), done.
2024-12-18T00:35:08.1486142Z remote: Compressing objects:   0% (1/448)
2024-12-18T00:35:08.1513968Z remote: Compressing objects:   1% (5/448)
2024-12-18T00:35:08.1558606Z remote: Compressing objects:   2% (9/448)
2024-12-18T00:35:08.1633125Z remote: Compressing objects:   3% (14/448)
2024-12-18T00:35:08.1653607Z remote: Compressing objects:   4% (18/448)
2024-12-18T00:35:08.1726290Z remote: Compressing objects:   5% (23/448)
2024-12-18T00:35:08.1765099Z remote: Compressing objects:   6% (27/448)
2024-12-18T00:35:08.1781450Z remote: Compressing objects:   7% (32/448)
2024-12-18T00:35:08.1834278Z remote: Compressing objects:   8% (36/448)
2024-12-18T00:35:08.1849687Z remote: Compressing objects:   9% (41/448)
2024-12-18T00:35:08.1880369Z remote: Compressing objects:  10% (45/448)
2024-12-18T00:35:08.1895874Z remote: Compressing objects:  11% (50/448)
2024-12-18T00:35:08.1918783Z remote: Compressing objects:  12% (54/448)
2024-12-18T00:35:08.1938306Z remote: Compressing objects:  13% (59/448)
2024-12-18T00:35:08.1960935Z remote: Compressing objects:  14% (63/448)
2024-12-18T00:35:08.1978981Z remote: Compressing objects:  15% (68/448)
2024-12-18T00:35:08.1994968Z remote: Compressing objects:  16% (72/448)
2024-12-18T00:35:08.2007569Z remote: Compressing objects:  17% (77/448)
2024-12-18T00:35:08.2019762Z remote: Compressing objects:  18% (81/448)
2024-12-18T00:35:08.2034089Z remote: Compressing objects:  19% (86/448)
2024-12-18T00:35:08.2046629Z remote: Compressing objects:  20% (90/448)
2024-12-18T00:35:08.2057372Z remote: Compressing objects:  21% (95/448)
2024-12-18T00:35:08.2068051Z remote: Compressing objects:  22% (99/448)
2024-12-18T00:35:08.2075186Z remote: Compressing objects:  23% (104/448)
2024-12-18T00:35:08.2080871Z remote: Compressing objects:  24% (108/448)
2024-12-18T00:35:08.2084452Z remote: Compressing objects:  25% (112/448)
2024-12-18T00:35:08.2089942Z remote: Compressing objects:  26% (117/448)
2024-12-18T00:35:08.2093221Z remote: Compressing objects:  27% (121/448)
2024-12-18T00:35:08.2095717Z remote: Compressing objects:  28% (126/448)
2024-12-18T00:35:08.2098331Z remote: Compressing objects:  29% (130/448)
2024-12-18T00:35:08.2098823Z remote: Compressing objects:  30% (135/448)
2024-12-18T00:35:08.2099240Z remote: Compressing objects:  31% (139/448)
2024-12-18T00:35:08.2099658Z remote: Compressing objects:  32% (144/448)
2024-12-18T00:35:08.2100071Z remote: Compressing objects:  33% (148/448)
2024-12-18T00:35:08.2100485Z remote: Compressing objects:  34% (153/448)
2024-12-18T00:35:08.2112792Z remote: Compressing objects:  35% (157/448)
2024-12-18T00:35:08.2127720Z remote: Compressing objects:  36% (162/448)
2024-12-18T00:35:08.2144137Z remote: Compressing objects:  37% (166/448)
2024-12-18T00:35:08.2144838Z remote: Compressing objects:  38% (171/448)
2024-12-18T00:35:08.2145260Z remote: Compressing objects:  39% (175/448)
2024-12-18T00:35:08.2145662Z remote: Compressing objects:  40% (180/448)
2024-12-18T00:35:08.2146066Z remote: Compressing objects:  41% (184/448)
2024-12-18T00:35:08.2146466Z remote: Compressing objects:  42% (189/448)
2024-12-18T00:35:08.2146871Z remote: Compressing objects:  43% (193/448)
2024-12-18T00:35:08.2147275Z remote: Compressing objects:  44% (198/448)
2024-12-18T00:35:08.2147678Z remote: Compressing objects:  45% (202/448)
2024-12-18T00:35:08.2148079Z remote: Compressing objects:  46% (207/448)
2024-12-18T00:35:08.2148481Z remote: Compressing objects:  47% (211/448)
2024-12-18T00:35:08.2148884Z remote: Compressing objects:  48% (216/448)
2024-12-18T00:35:08.2158570Z remote: Compressing objects:  49% (220/448)
2024-12-18T00:35:08.2159024Z remote: Compressing objects:  50% (224/448)
2024-12-18T00:35:08.2159470Z remote: Compressing objects:  51% (229/448)
2024-12-18T00:35:08.2159935Z remote: Compressing objects:  52% (233/448)
2024-12-18T00:35:08.2160380Z remote: Compressing objects:  53% (238/448)
2024-12-18T00:35:08.2160946Z remote: Compressing objects:  54% (242/448)
2024-12-18T00:35:08.2161467Z remote: Compressing objects:  55% (247/448)
2024-12-18T00:35:08.2161948Z remote: Compressing objects:  56% (251/448)
2024-12-18T00:35:08.2162371Z remote: Compressing objects:  57% (256/448)
2024-12-18T00:35:08.2162839Z remote: Compressing objects:  58% (260/448)
2024-12-18T00:35:08.2163361Z remote: Compressing objects:  59% (265/448)
2024-12-18T00:35:08.2163769Z remote: Compressing objects:  60% (269/448)
2024-12-18T00:35:08.2164182Z remote: Compressing objects:  61% (274/448)
2024-12-18T00:35:08.2164673Z remote: Compressing objects:  62% (278/448)
2024-12-18T00:35:08.2165277Z remote: Compressing objects:  63% (283/448)
2024-12-18T00:35:08.2165793Z remote: Compressing objects:  64% (287/448)
2024-12-18T00:35:08.2166212Z remote: Compressing objects:  65% (292/448)
2024-12-18T00:35:08.2166624Z remote: Compressing objects:  66% (296/448)
2024-12-18T00:35:08.2167074Z remote: Compressing objects:  67% (301/448)
2024-12-18T00:35:08.2167523Z remote: Compressing objects:  68% (305/448)
2024-12-18T00:35:08.2167984Z remote: Compressing objects:  69% (310/448)
2024-12-18T00:35:08.2168510Z remote: Compressing objects:  70% (314/448)
2024-12-18T00:35:08.2168928Z remote: Compressing objects:  71% (319/448)
2024-12-18T00:35:08.2169352Z remote: Compressing objects:  72% (323/448)
2024-12-18T00:35:08.2169843Z remote: Compressing objects:  73% (328/448)
2024-12-18T00:35:08.2170244Z remote: Compressing objects:  74% (332/448)
2024-12-18T00:35:08.2170814Z remote: Compressing objects:  75% (336/448)
2024-12-18T00:35:08.2171261Z remote: Compressing objects:  76% (341/448)
2024-12-18T00:35:08.2171690Z remote: Compressing objects:  77% (345/448)
2024-12-18T00:35:08.2173930Z remote: Compressing objects:  78% (350/448)
2024-12-18T00:35:08.2174375Z remote: Compressing objects:  79% (354/448)
2024-12-18T00:35:08.2174861Z remote: Compressing objects:  80% (359/448)
2024-12-18T00:35:08.2175275Z remote: Compressing objects:  81% (363/448)
2024-12-18T00:35:08.2175829Z remote: Compressing objects:  82% (368/448)
2024-12-18T00:35:08.2176269Z remote: Compressing objects:  83% (372/448)
2024-12-18T00:35:08.2176683Z remote: Compressing objects:  84% (377/448)
2024-12-18T00:35:08.2178659Z remote: Compressing objects:  85% (381/448)
2024-12-18T00:35:08.2181156Z remote: Compressing objects:  86% (386/448)
2024-12-18T00:35:08.2183854Z remote: Compressing objects:  87% (390/448)
2024-12-18T00:35:08.2184295Z remote: Compressing objects:  88% (395/448)
2024-12-18T00:35:08.2186439Z remote: Compressing objects:  89% (399/448)
2024-12-18T00:35:08.2186911Z remote: Compressing objects:  90% (404/448)
2024-12-18T00:35:08.2187311Z remote: Compressing objects:  91% (408/448)
2024-12-18T00:35:08.2187927Z remote: Compressing objects:  92% (413/448)
2024-12-18T00:35:08.2188860Z remote: Compressing objects:  93% (417/448)
2024-12-18T00:35:08.2189281Z remote: Compressing objects:  94% (422/448)
2024-12-18T00:35:08.2189697Z remote: Compressing objects:  95% (426/448)
2024-12-18T00:35:08.2191741Z remote: Compressing objects:  96% (431/448)
2024-12-18T00:35:08.2192169Z remote: Compressing objects:  97% (435/448)
2024-12-18T00:35:08.2192632Z remote: Compressing objects:  98% (440/448)
2024-12-18T00:35:08.2193074Z remote: Compressing objects:  99% (444/448)
2024-12-18T00:35:08.2193830Z remote: Compressing objects: 100% (448/448)
2024-12-18T00:35:08.2194273Z remote: Compressing objects: 100% (448/448), done.
2024-12-18T00:35:08.2285374Z Receiving objects:   0% (1/34540)
2024-12-18T00:35:08.2352783Z Receiving objects:   1% (346/34540)
2024-12-18T00:35:08.2411161Z Receiving objects:   2% (691/34540)
2024-12-18T00:35:08.2450709Z Receiving objects:   3% (1037/34540)
2024-12-18T00:35:08.2486025Z Receiving objects:   4% (1382/34540)
2024-12-18T00:35:08.2523746Z Receiving objects:   5% (1727/34540)
2024-12-18T00:35:08.2570753Z Receiving objects:   6% (2073/34540)
2024-12-18T00:35:08.2604001Z Receiving objects:   7% (2418/34540)
2024-12-18T00:35:08.2640663Z Receiving objects:   8% (2764/34540)
2024-12-18T00:35:08.2720380Z Receiving objects:   9% (3109/34540)
2024-12-18T00:35:08.2776483Z Receiving objects:  10% (3454/34540)
2024-12-18T00:35:08.2856789Z Receiving objects:  11% (3800/34540)
2024-12-18T00:35:08.3064318Z Receiving objects:  12% (4145/34540)
2024-12-18T00:35:08.3192112Z Receiving objects:  13% (4491/34540)
2024-12-18T00:35:08.3295032Z Receiving objects:  14% (4836/34540)
2024-12-18T00:35:08.3361005Z Receiving objects:  15% (5181/34540)
2024-12-18T00:35:08.3401247Z Receiving objects:  16% (5527/34540)
2024-12-18T00:35:08.3472688Z Receiving objects:  17% (5872/34540)
2024-12-18T00:35:08.3507311Z Receiving objects:  18% (6218/34540)
2024-12-18T00:35:08.3536126Z Receiving objects:  19% (6563/34540)
2024-12-18T00:35:08.5708951Z Receiving objects:  20% (6908/34540)
2024-12-18T00:35:09.2222054Z Receiving objects:  21% (7254/34540)
2024-12-18T00:35:09.9055337Z Receiving objects:  21% (7375/34540), 71.29 MiB | 71.29 MiB/s
2024-12-18T00:35:09.9908995Z Receiving objects:  22% (7599/34540), 109.68 MiB | 73.11 MiB/s
2024-12-18T00:35:10.0698784Z Receiving objects:  23% (7945/34540), 109.68 MiB | 73.11 MiB/s
2024-12-18T00:35:10.1496398Z Receiving objects:  24% (8290/34540), 109.68 MiB | 73.11 MiB/s
2024-12-18T00:35:10.2221761Z Receiving objects:  25% (8635/34540), 109.68 MiB | 73.11 MiB/s
2024-12-18T00:35:10.2285069Z Receiving objects:  25% (8951/34540), 147.23 MiB | 73.61 MiB/s
2024-12-18T00:35:10.3091307Z Receiving objects:  26% (8981/34540), 147.23 MiB | 73.61 MiB/s
2024-12-18T00:35:10.3886237Z Receiving objects:  27% (9326/34540), 147.23 MiB | 73.61 MiB/s
2024-12-18T00:35:10.4683399Z Receiving objects:  28% (9672/34540), 147.23 MiB | 73.61 MiB/s
2024-12-18T00:35:10.6995113Z Receiving objects:  29% (10017/34540), 147.23 MiB | 73.61 MiB/s
2024-12-18T00:35:10.7601016Z Receiving objects:  30% (10362/34540), 147.23 MiB | 73.61 MiB/s
2024-12-18T00:35:10.9643825Z Receiving objects:  31% (10708/34540), 183.17 MiB | 73.26 MiB/s
2024-12-18T00:35:11.2222563Z Receiving objects:  32% (11053/34540), 183.17 MiB | 73.26 MiB/s
2024-12-18T00:35:11.2516622Z Receiving objects:  32% (11289/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.2731830Z Receiving objects:  33% (11399/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.2755735Z Receiving objects:  34% (11744/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.2775754Z Receiving objects:  35% (12089/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.2792912Z Receiving objects:  36% (12435/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.2806044Z Receiving objects:  37% (12780/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.2818996Z Receiving objects:  38% (13126/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.2839833Z Receiving objects:  39% (13471/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.2852120Z Receiving objects:  40% (13816/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.2857094Z Receiving objects:  41% (14162/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.2871205Z Receiving objects:  42% (14507/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.2893868Z Receiving objects:  43% (14853/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.2908641Z Receiving objects:  44% (15198/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.3107631Z Receiving objects:  45% (15543/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.3113689Z Receiving objects:  46% (15889/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.3122187Z Receiving objects:  47% (16234/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.3128086Z Receiving objects:  48% (16580/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.3146089Z Receiving objects:  49% (16925/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.3157939Z Receiving objects:  50% (17270/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.3163923Z Receiving objects:  51% (17616/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.3179523Z Receiving objects:  52% (17961/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.3191142Z Receiving objects:  53% (18307/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.3201912Z Receiving objects:  54% (18652/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.3339973Z Receiving objects:  55% (18997/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.3346794Z Receiving objects:  56% (19343/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.3354065Z Receiving objects:  57% (19688/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.3364322Z Receiving objects:  58% (20034/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.3373313Z Receiving objects:  59% (20379/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.3628271Z Receiving objects:  60% (20724/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4025944Z Receiving objects:  61% (21070/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4037727Z Receiving objects:  62% (21415/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4051444Z Receiving objects:  63% (21761/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4064512Z Receiving objects:  64% (22106/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4078041Z Receiving objects:  65% (22451/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4087708Z Receiving objects:  66% (22797/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4107490Z Receiving objects:  67% (23142/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4115116Z Receiving objects:  68% (23488/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4123847Z Receiving objects:  69% (23833/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4339004Z Receiving objects:  70% (24178/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4373078Z Receiving objects:  71% (24524/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4383625Z Receiving objects:  72% (24869/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4396364Z Receiving objects:  73% (25215/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4437425Z Receiving objects:  74% (25560/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4587938Z Receiving objects:  75% (25905/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4594781Z Receiving objects:  76% (26251/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4601764Z Receiving objects:  77% (26596/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4610072Z Receiving objects:  78% (26942/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4617988Z Receiving objects:  79% (27287/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4624702Z Receiving objects:  80% (27632/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4738865Z Receiving objects:  81% (27978/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4755688Z Receiving objects:  82% (28323/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4769555Z Receiving objects:  83% (28669/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4832492Z Receiving objects:  84% (29014/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4883175Z Receiving objects:  85% (29359/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4937447Z Receiving objects:  86% (29705/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.4974592Z Receiving objects:  87% (30050/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.5002392Z Receiving objects:  88% (30396/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.5067564Z Receiving objects:  89% (30741/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.5168663Z Receiving objects:  90% (31086/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.5257711Z Receiving objects:  91% (31432/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.5355376Z Receiving objects:  92% (31777/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.5370357Z Receiving objects:  93% (32123/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.5494510Z Receiving objects:  94% (32468/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.5509133Z Receiving objects:  95% (32813/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.5514911Z Receiving objects:  96% (33159/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.5519934Z Receiving objects:  97% (33504/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.5552948Z Receiving objects:  98% (33850/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.5583629Z Receiving objects:  99% (34195/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.5584547Z remote: Total 34540 (delta 1463), reused 1255 (delta 1216), pack-reused 32876 (from 5)
2024-12-18T00:35:11.5601632Z Receiving objects: 100% (34540/34540), 217.86 MiB | 72.62 MiB/s
2024-12-18T00:35:11.5602209Z Receiving objects: 100% (34540/34540), 234.54 MiB | 70.27 MiB/s, done.
2024-12-18T00:35:11.5636236Z Resolving deltas:   0% (0/18575)
2024-12-18T00:35:11.5648449Z Resolving deltas:   1% (186/18575)
2024-12-18T00:35:11.5672704Z Resolving deltas:   2% (392/18575)
2024-12-18T00:35:11.5693292Z Resolving deltas:   3% (602/18575)
2024-12-18T00:35:11.5704124Z Resolving deltas:   4% (761/18575)
2024-12-18T00:35:11.5714162Z Resolving deltas:   5% (932/18575)
2024-12-18T00:35:11.5726028Z Resolving deltas:   6% (1115/18575)
2024-12-18T00:35:11.5737607Z Resolving deltas:   7% (1302/18575)
2024-12-18T00:35:11.5748996Z Resolving deltas:   8% (1496/18575)
2024-12-18T00:35:11.5762953Z Resolving deltas:   9% (1672/18575)
2024-12-18T00:35:11.5775139Z Resolving deltas:  10% (1872/18575)
2024-12-18T00:35:11.5789515Z Resolving deltas:  11% (2056/18575)
2024-12-18T00:35:11.5801407Z Resolving deltas:  12% (2229/18575)
2024-12-18T00:35:11.5822055Z Resolving deltas:  13% (2423/18575)
2024-12-18T00:35:11.5845065Z Resolving deltas:  14% (2620/18575)
2024-12-18T00:35:11.5861249Z Resolving deltas:  15% (2859/18575)
2024-12-18T00:35:11.5908317Z Resolving deltas:  16% (2995/18575)
2024-12-18T00:35:11.5927516Z Resolving deltas:  17% (3190/18575)
2024-12-18T00:35:11.5930961Z Resolving deltas:  18% (3486/18575)
2024-12-18T00:35:11.5947958Z Resolving deltas:  19% (3531/18575)
2024-12-18T00:35:11.5963237Z Resolving deltas:  20% (3720/18575)
2024-12-18T00:35:11.5976363Z Resolving deltas:  21% (3901/18575)
2024-12-18T00:35:11.6012579Z Resolving deltas:  22% (4102/18575)
2024-12-18T00:35:11.6036299Z Resolving deltas:  23% (4285/18575)
2024-12-18T00:35:11.6057575Z Resolving deltas:  24% (4460/18575)
2024-12-18T00:35:11.6073267Z Resolving deltas:  25% (4681/18575)
2024-12-18T00:35:11.6092434Z Resolving deltas:  26% (4831/18575)
2024-12-18T00:35:11.6109517Z Resolving deltas:  27% (5016/18575)
2024-12-18T00:35:11.6119850Z Resolving deltas:  28% (5221/18575)
2024-12-18T00:35:11.6135459Z Resolving deltas:  29% (5388/18575)
2024-12-18T00:35:11.6157900Z Resolving deltas:  30% (5590/18575)
2024-12-18T00:35:11.6173625Z Resolving deltas:  31% (5767/18575)
2024-12-18T00:35:11.6194580Z Resolving deltas:  32% (5949/18575)
2024-12-18T00:35:11.6212920Z Resolving deltas:  33% (6160/18575)
2024-12-18T00:35:11.6229607Z Resolving deltas:  34% (6316/18575)
2024-12-18T00:35:11.6243873Z Resolving deltas:  35% (6505/18575)
2024-12-18T00:35:11.6268722Z Resolving deltas:  36% (6700/18575)
2024-12-18T00:35:11.6276443Z Resolving deltas:  37% (6919/18575)
2024-12-18T00:35:11.6290012Z Resolving deltas:  38% (7079/18575)
2024-12-18T00:35:11.6303704Z Resolving deltas:  39% (7295/18575)
2024-12-18T00:35:11.6329687Z Resolving deltas:  40% (7435/18575)
2024-12-18T00:35:11.6338399Z Resolving deltas:  41% (7617/18575)
2024-12-18T00:35:11.6360307Z Resolving deltas:  42% (7802/18575)
2024-12-18T00:35:11.6370954Z Resolving deltas:  43% (7997/18575)
2024-12-18T00:35:11.6380276Z Resolving deltas:  44% (8173/18575)
2024-12-18T00:35:11.6393357Z Resolving deltas:  45% (8360/18575)
2024-12-18T00:35:11.6413842Z Resolving deltas:  46% (8546/18575)
2024-12-18T00:35:11.6418824Z Resolving deltas:  47% (8829/18575)
2024-12-18T00:35:11.6429385Z Resolving deltas:  48% (8916/18575)
2024-12-18T00:35:11.6444551Z Resolving deltas:  49% (9114/18575)
2024-12-18T00:35:11.6455010Z Resolving deltas:  50% (9288/18575)
2024-12-18T00:35:11.6464835Z Resolving deltas:  51% (9544/18575)
2024-12-18T00:35:11.6467280Z Resolving deltas:  52% (9782/18575)
2024-12-18T00:35:11.6484417Z Resolving deltas:  53% (9850/18575)
2024-12-18T00:35:11.6492042Z Resolving deltas:  55% (10258/18575)
2024-12-18T00:35:11.6500397Z Resolving deltas:  56% (10428/18575)
2024-12-18T00:35:11.6507717Z Resolving deltas:  57% (10618/18575)
2024-12-18T00:35:11.6526784Z Resolving deltas:  58% (10780/18575)
2024-12-18T00:35:11.6533399Z Resolving deltas:  59% (11060/18575)
2024-12-18T00:35:11.6555385Z Resolving deltas:  60% (11171/18575)
2024-12-18T00:35:11.6562785Z Resolving deltas:  61% (11358/18575)
2024-12-18T00:35:11.6575102Z Resolving deltas:  62% (11528/18575)
2024-12-18T00:35:11.6592266Z Resolving deltas:  63% (11722/18575)
2024-12-18T00:35:11.6605557Z Resolving deltas:  64% (11892/18575)
2024-12-18T00:35:11.6642782Z Resolving deltas:  65% (12084/18575)
2024-12-18T00:35:11.6643085Z Resolving deltas:  67% (12629/18575)
2024-12-18T00:35:11.6655992Z Resolving deltas:  68% (12644/18575)
2024-12-18T00:35:11.6665575Z Resolving deltas:  69% (12817/18575)
2024-12-18T00:35:11.6681574Z Resolving deltas:  70% (13008/18575)
2024-12-18T00:35:11.6692703Z Resolving deltas:  71% (13193/18575)
2024-12-18T00:35:11.6706831Z Resolving deltas:  72% (13375/18575)
2024-12-18T00:35:11.6717791Z Resolving deltas:  73% (13579/18575)
2024-12-18T00:35:11.6731899Z Resolving deltas:  74% (13747/18575)
2024-12-18T00:35:11.6741520Z Resolving deltas:  75% (13934/18575)
2024-12-18T00:35:11.6758254Z Resolving deltas:  76% (14120/18575)
2024-12-18T00:35:11.6768383Z Resolving deltas:  77% (14306/18575)
2024-12-18T00:35:11.6780000Z Resolving deltas:  78% (14489/18575)
2024-12-18T00:35:11.6794155Z Resolving deltas:  79% (14682/18575)
2024-12-18T00:35:11.6802754Z Resolving deltas:  80% (14909/18575)
2024-12-18T00:35:11.6814304Z Resolving deltas:  81% (15062/18575)
2024-12-18T00:35:11.6823390Z Resolving deltas:  82% (15233/18575)
2024-12-18T00:35:11.6830976Z Resolving deltas:  83% (15418/18575)
2024-12-18T00:35:11.6848521Z Resolving deltas:  84% (15604/18575)
2024-12-18T00:35:11.6872913Z Resolving deltas:  85% (15791/18575)
2024-12-18T00:35:11.6883173Z Resolving deltas:  86% (16032/18575)
2024-12-18T00:35:11.6910987Z Resolving deltas:  87% (16162/18575)
2024-12-18T00:35:11.6921247Z Resolving deltas:  88% (16354/18575)
2024-12-18T00:35:11.6938333Z Resolving deltas:  89% (16538/18575)
2024-12-18T00:35:11.6964066Z Resolving deltas:  90% (16719/18575)
2024-12-18T00:35:11.7014110Z Resolving deltas:  91% (16909/18575)
2024-12-18T00:35:11.7042731Z Resolving deltas:  92% (17090/18575)
2024-12-18T00:35:11.7047786Z Resolving deltas:  93% (17416/18575)
2024-12-18T00:35:11.7054903Z Resolving deltas:  94% (17520/18575)
2024-12-18T00:35:11.7065616Z Resolving deltas:  95% (17649/18575)
2024-12-18T00:35:11.7073463Z Resolving deltas:  96% (17842/18575)
2024-12-18T00:35:11.7091569Z Resolving deltas:  97% (18020/18575)
2024-12-18T00:35:11.7112220Z Resolving deltas:  98% (18292/18575)
2024-12-18T00:35:11.7360722Z Resolving deltas:  99% (18390/18575)
2024-12-18T00:35:11.7361037Z Resolving deltas: 100% (18575/18575)
2024-12-18T00:35:11.7361352Z Resolving deltas: 100% (18575/18575), done.
2024-12-18T00:35:12.7507013Z + pushd torchbench
2024-12-18T00:35:12.7507325Z ~/workspace/torchbench ~/workspace
2024-12-18T00:35:12.7507703Z + git checkout 766a5e3a189384659fd35a68c3b17b88c761aaac
2024-12-18T00:35:12.8583520Z Note: switching to '766a5e3a189384659fd35a68c3b17b88c761aaac'.
2024-12-18T00:35:12.8583843Z 
2024-12-18T00:35:12.8584062Z You are in 'detached HEAD' state. You can look around, make experimental
2024-12-18T00:35:12.8584606Z changes and commit them, and you can discard any commits you make in this
2024-12-18T00:35:12.8585142Z state without impacting any branches by switching back to a branch.
2024-12-18T00:35:12.8585452Z 
2024-12-18T00:35:12.8585664Z If you want to create a new branch to retain commits you create, you may
2024-12-18T00:35:12.8586196Z do so (now or later) by using -c with the switch command. Example:
2024-12-18T00:35:12.8586486Z 
2024-12-18T00:35:12.8586611Z   git switch -c <new-branch-name>
2024-12-18T00:35:12.8586806Z 
2024-12-18T00:35:12.8586912Z Or undo this operation with:
2024-12-18T00:35:12.8587098Z 
2024-12-18T00:35:12.8587193Z   git switch -
2024-12-18T00:35:12.8587333Z 
2024-12-18T00:35:12.8587563Z Turn off this advice by setting config variable advice.detachedHead to false
2024-12-18T00:35:12.8587898Z 
2024-12-18T00:35:12.8588115Z HEAD is now at 766a5e3a Account for older numpy versions in #2514 (#2524)
2024-12-18T00:35:12.8597865Z + '[' '' ']'
2024-12-18T00:35:12.8598130Z + python install.py --continue_on_fail
2024-12-18T00:35:15.6066349Z checking packages torch, torchvision, torchaudio are installed...OK
2024-12-18T00:35:34.6667358Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK
2024-12-18T00:35:37.9432618Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK
2024-12-18T00:35:39.9780588Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK
2024-12-18T00:35:41.5550365Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK
2024-12-18T00:35:41.5788866Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK
2024-12-18T00:35:49.3431245Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK
2024-12-18T00:35:52.1178273Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK
2024-12-18T00:35:54.8810741Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK
2024-12-18T00:35:57.6484306Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK
2024-12-18T00:35:57.6487192Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found
2024-12-18T00:35:58.7126651Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK
2024-12-18T00:36:01.6529460Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK
2024-12-18T00:36:01.6766992Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK
2024-12-18T00:36:57.8326725Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK
2024-12-18T00:37:07.3696226Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK
2024-12-18T00:37:15.0107095Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK
2024-12-18T00:37:22.1438599Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK
2024-12-18T00:37:31.2613094Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK
2024-12-18T00:37:39.2925098Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK
2024-12-18T00:37:46.4182768Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK
2024-12-18T00:37:53.6927936Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK
2024-12-18T00:38:01.3697280Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK
2024-12-18T00:38:08.9360113Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK
2024-12-18T00:38:16.2125591Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK
2024-12-18T00:38:23.6409789Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK
2024-12-18T00:38:24.8640624Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK
2024-12-18T00:39:02.5139209Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK
2024-12-18T00:39:07.0931898Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK
2024-12-18T00:39:12.3505604Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK
2024-12-18T00:39:23.8888962Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK
2024-12-18T00:39:25.0850951Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK
2024-12-18T00:39:26.6561745Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK
2024-12-18T00:39:33.6473629Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK
2024-12-18T00:39:40.7266494Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK
2024-12-18T00:39:46.2389891Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK
2024-12-18T00:39:53.9361609Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK
2024-12-18T00:39:59.8464055Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK
2024-12-18T00:40:04.9435847Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK
2024-12-18T00:40:11.2046935Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK
2024-12-18T00:40:27.2043414Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK
2024-12-18T00:40:33.4311612Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK
2024-12-18T00:40:37.4204308Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK
2024-12-18T00:40:42.2474221Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK
2024-12-18T00:40:49.0650437Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK
2024-12-18T00:40:49.0653176Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found
2024-12-18T00:41:02.0202205Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK
2024-12-18T00:41:06.0146189Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK
2024-12-18T00:41:06.0148967Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found
2024-12-18T00:41:13.6477820Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK
2024-12-18T00:41:14.8822362Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK
2024-12-18T00:41:16.1315544Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK
2024-12-18T00:41:59.8689685Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK
2024-12-18T00:43:27.7466663Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK
2024-12-18T00:43:27.7759054Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK
2024-12-18T00:43:29.3473833Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK
2024-12-18T00:43:29.3722125Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK
2024-12-18T00:43:29.3972409Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK
2024-12-18T00:43:29.4220888Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK
2024-12-18T00:43:29.4462957Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK
2024-12-18T00:43:29.4705525Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK
2024-12-18T00:43:50.4636664Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK
2024-12-18T00:43:50.4639372Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found
2024-12-18T00:43:51.7227055Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK
2024-12-18T00:43:54.2402753Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK
2024-12-18T00:43:54.2647222Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK
2024-12-18T00:43:54.2896779Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK
2024-12-18T00:43:54.3149745Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK
2024-12-18T00:43:54.3397290Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK
2024-12-18T00:43:54.3642813Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK
2024-12-18T00:44:01.2019008Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK
2024-12-18T00:44:02.7267885Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK
2024-12-18T00:44:06.5705155Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK
2024-12-18T00:44:06.5951477Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK
2024-12-18T00:44:06.6196795Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK
2024-12-18T00:44:06.6442480Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK
2024-12-18T00:44:06.6682100Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK
2024-12-18T00:44:06.6929741Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK
2024-12-18T00:44:23.7773397Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK
2024-12-18T00:44:40.2842942Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK
2024-12-18T00:44:40.3092074Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK
2024-12-18T00:44:40.3095179Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found
2024-12-18T00:44:40.3097907Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found
2024-12-18T00:44:43.0114506Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK
2024-12-18T00:44:44.8845336Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK
2024-12-18T00:44:44.9092779Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK
2024-12-18T00:45:10.5610411Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK
2024-12-18T00:45:17.2351581Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK
2024-12-18T00:45:20.5061767Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK
2024-12-18T00:45:30.6993977Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK
2024-12-18T00:45:30.7245096Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK
2024-12-18T00:45:30.7495423Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK
2024-12-18T00:45:30.7737377Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK
2024-12-18T00:45:30.7978245Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK
2024-12-18T00:45:30.8218729Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK
2024-12-18T00:45:30.8464141Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK
2024-12-18T00:45:30.8705478Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK
2024-12-18T00:45:38.5664329Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK
2024-12-18T00:45:51.0390693Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK
2024-12-18T00:45:51.0646752Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK
2024-12-18T00:45:53.1100071Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK
2024-12-18T00:45:55.1412005Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK
2024-12-18T00:45:55.6713225Z + echo 'Print all dependencies after TorchBench is installed'
2024-12-18T00:45:55.6715396Z Print all dependencies after TorchBench is installed
2024-12-18T00:45:55.6715832Z + python -mpip freeze
2024-12-18T00:45:56.1649720Z absl-py==2.1.0
2024-12-18T00:45:56.1650167Z accelerate==1.2.1
2024-12-18T00:45:56.1650550Z aiohappyeyeballs==2.4.4
2024-12-18T00:45:56.1650891Z aiohttp==3.11.10
2024-12-18T00:45:56.1651209Z aiosignal==1.3.1
2024-12-18T00:45:56.1651532Z annotated-types==0.7.0
2024-12-18T00:45:56.1651931Z antlr4-python3-runtime==4.9.3
2024-12-18T00:45:56.1652276Z anyascii==0.3.2
2024-12-18T00:45:56.1652501Z astroid==3.3.6
2024-12-18T00:45:56.1652780Z astunparse==1.6.3
2024-12-18T00:45:56.1653098Z async-timeout==5.0.1
2024-12-18T00:45:56.1653429Z attrs==23.1.0
2024-12-18T00:45:56.1653707Z audioread==3.0.1
2024-12-18T00:45:56.1654033Z babel==2.16.0
2024-12-18T00:45:56.1654359Z beautifulsoup4==4.12.3
2024-12-18T00:45:56.1655381Z -e git+https://github.com/pytorch/benchmark@766a5e3a189384659fd35a68c3b17b88c761aaac#egg=bert_pytorch&subdirectory=torchbenchmark/models/BERT_pytorch
2024-12-18T00:45:56.1656370Z bibtexparser==2.0.0b8
2024-12-18T00:45:56.1656715Z black==24.10.0
2024-12-18T00:45:56.1657015Z blinker==1.9.0
2024-12-18T00:45:56.1657311Z blis==1.1.0
2024-12-18T00:45:56.1657592Z blobfile==3.0.0
2024-12-18T00:45:56.1657901Z bokeh==3.6.2
2024-12-18T00:45:56.1658183Z boto3==1.35.42
2024-12-18T00:45:56.1658483Z botocore==1.35.78
2024-12-18T00:45:56.1658779Z bs4==0.0.2
2024-12-18T00:45:56.1659483Z cachetools==5.5.0
2024-12-18T00:45:56.1659848Z cardboardlint==1.3.1
2024-12-18T00:45:56.1660187Z catalogue==2.0.10
2024-12-18T00:45:56.1660501Z certifi==2024.8.30
2024-12-18T00:45:56.1660827Z cffi==1.17.1
2024-12-18T00:45:56.1661159Z charset-normalizer==3.4.0
2024-12-18T00:45:56.1661488Z click==8.1.7
2024-12-18T00:45:56.1661796Z clldutils==3.24.0
2024-12-18T00:45:56.1662235Z cloudpathlib==0.20.0
2024-12-18T00:45:56.1662566Z cloudpickle==3.1.0
2024-12-18T00:45:56.1662883Z colorama==0.4.6
2024-12-18T00:45:56.1663196Z colorlog==6.9.0
2024-12-18T00:45:56.1663514Z confection==0.1.5
2024-12-18T00:45:56.1663824Z contourpy==1.2.1
2024-12-18T00:45:56.1664146Z coremltools==5.0b5
2024-12-18T00:45:56.1664445Z cryptography==44.0.0
2024-12-18T00:45:56.1664781Z csvw==3.5.1
2024-12-18T00:45:56.1665049Z cycler==0.12.1
2024-12-18T00:45:56.1665363Z cymem==2.0.10
2024-12-18T00:45:56.1665657Z Cython==3.0.11
2024-12-18T00:45:56.1665935Z DALL-E==0.1
2024-12-18T00:45:56.1666229Z datasets==3.2.0
2024-12-18T00:45:56.1666563Z decorator==5.1.1
2024-12-18T00:45:56.1666867Z defusedxml==0.7.1
2024-12-18T00:45:56.1667170Z Deprecated==1.2.15
2024-12-18T00:45:56.1667861Z detectron2 @ git+https://github.com/facebookresearch/detectron2.git@1a4df4d954089534daebbd7a0a45f08ece87056e
2024-12-18T00:45:56.1668535Z diffusers==0.30.3
2024-12-18T00:45:56.1668764Z dill==0.3.8
2024-12-18T00:45:56.1668978Z diskcache==5.6.3
2024-12-18T00:45:56.1669209Z distro==1.9.0
2024-12-18T00:45:56.1669421Z dlinfo==1.2.1
2024-12-18T00:45:56.1669649Z docker-pycreds==0.4.0
2024-12-18T00:45:56.1669900Z dominate==2.9.1
2024-12-18T00:45:56.1670420Z effdet @ git+https://github.com/rwightman/efficientdet-pytorch.git@d43c9e34cd62d22b4205831bb735f6dd83b8e881
2024-12-18T00:45:56.1671049Z einops==0.8.0
2024-12-18T00:45:56.1671289Z exceptiongroup==1.2.2
2024-12-18T00:45:56.1671546Z execnet==2.1.1
2024-12-18T00:45:56.1672113Z expecttest==0.2.1
2024-12-18T00:45:56.1672357Z FastNLP==0.6.0
2024-12-18T00:45:56.1672600Z fbscribelogger==0.1.7
2024-12-18T00:45:56.1672867Z ffmpeg-python==0.2.0
2024-12-18T00:45:56.1673117Z filelock==3.16.1
2024-12-18T00:45:56.1673347Z Flask==3.1.0
2024-12-18T00:45:56.1673578Z flatbuffers==2.0
2024-12-18T00:45:56.1673811Z fonttools==4.55.3
2024-12-18T00:45:56.1674044Z frozenlist==1.5.0
2024-12-18T00:45:56.1674272Z fsspec==2024.9.0
2024-12-18T00:45:56.1674500Z ftfy==6.3.1
2024-12-18T00:45:56.1674955Z functorch @ git+https://github.com/pytorch/functorch.git@b71aa0b4387b86c278132209b99538be48ef4c74
2024-12-18T00:45:56.1675487Z future==1.0.0
2024-12-18T00:45:56.1675729Z fvcore==0.1.5.post20221221
2024-12-18T00:45:56.1675986Z gdown==5.2.0
2024-12-18T00:45:56.1676208Z ghstack==0.8.0
2024-12-18T00:45:56.1676436Z gitdb==4.0.11
2024-12-18T00:45:56.1676666Z GitPython==3.1.43
2024-12-18T00:45:56.1676903Z google-auth==2.36.0
2024-12-18T00:45:56.1677148Z google-auth-oauthlib==1.0.0
2024-12-18T00:45:56.1677431Z grpcio==1.68.1
2024-12-18T00:45:56.1677655Z gym==0.26.2
2024-12-18T00:45:56.1677869Z gym-notices==0.0.8
2024-12-18T00:45:56.1678093Z h5py==3.12.1
2024-12-18T00:45:56.1678316Z higher==0.2.1
2024-12-18T00:45:56.1678550Z huggingface-hub==0.26.5
2024-12-18T00:45:56.1678805Z hydra-core==1.3.2
2024-12-18T00:45:56.1679038Z hypothesis==5.35.1
2024-12-18T00:45:56.1679265Z idna==3.10
2024-12-18T00:45:56.1679482Z imageio==2.36.1
2024-12-18T00:45:56.1679718Z importlib_metadata==8.5.0
2024-12-18T00:45:56.1679977Z inflect==7.4.0
2024-12-18T00:45:56.1680197Z iniconfig==2.0.0
2024-12-18T00:45:56.1680418Z iopath==0.1.9
2024-12-18T00:45:56.1680637Z isodate==0.7.2
2024-12-18T00:45:56.1680853Z isort==5.13.2
2024-12-18T00:45:56.1681076Z itsdangerous==2.2.0
2024-12-18T00:45:56.1681304Z Jinja2==3.1.4
2024-12-18T00:45:56.1681522Z jmespath==1.0.1
2024-12-18T00:45:56.1681745Z joblib==1.4.2
2024-12-18T00:45:56.1681966Z jsonpatch==1.33
2024-12-18T00:45:56.1682196Z jsonpointer==3.0.0
2024-12-18T00:45:56.1682424Z jsonschema==4.23.0
2024-12-18T00:45:56.1682695Z jsonschema-specifications==2024.10.1
2024-12-18T00:45:56.1683007Z junitparser==2.1.1
2024-12-18T00:45:56.1683241Z kaldi-io==0.9.8
2024-12-18T00:45:56.1683568Z kiwisolver==1.4.7
2024-12-18T00:45:56.1683791Z kornia==0.7.4
2024-12-18T00:45:56.1684008Z kornia_rs==0.1.7
2024-12-18T00:45:56.1684234Z lameenc==1.7.0
2024-12-18T00:45:56.1684459Z langcodes==3.5.0
2024-12-18T00:45:56.1684680Z langdetect==1.0.9
2024-12-18T00:45:56.1684916Z language-tags==1.2.0
2024-12-18T00:45:56.1685173Z language_data==1.3.0
2024-12-18T00:45:56.1685410Z lark==0.12.0
2024-12-18T00:45:56.1685625Z lazy_loader==0.4
2024-12-18T00:45:56.1685840Z libcst==1.5.1
2024-12-18T00:45:56.1686054Z librosa==0.9.2
2024-12-18T00:45:56.1686279Z lintrunner==0.12.5
2024-12-18T00:45:56.1686509Z llvmlite==0.38.1
2024-12-18T00:45:56.1686733Z lxml==5.3.0
2024-12-18T00:45:56.1686939Z marisa-trie==1.2.1
2024-12-18T00:45:56.1687170Z Markdown==3.7
2024-12-18T00:45:56.1687396Z markdown-it-py==3.0.0
2024-12-18T00:45:56.1687652Z MarkupSafe==3.0.2
2024-12-18T00:45:56.1687885Z matplotlib==3.8.4
2024-12-18T00:45:56.1688110Z mccabe==0.7.0
2024-12-18T00:45:56.1688320Z mdurl==0.1.2
2024-12-18T00:45:56.1688541Z ml_dtypes==0.5.0
2024-12-18T00:45:56.1688776Z MonkeyType==23.3.0
2024-12-18T00:45:56.1689011Z more-itertools==10.5.0
2024-12-18T00:45:56.1689257Z mpmath==1.3.0
2024-12-18T00:45:56.1689472Z msgpack==1.1.0
2024-12-18T00:45:56.1689692Z multidict==6.1.0
2024-12-18T00:45:56.1689922Z multiprocess==0.70.16
2024-12-18T00:45:56.1690159Z murmurhash==1.0.11
2024-12-18T00:45:56.1690386Z musdb==0.4.2
2024-12-18T00:45:56.1690600Z museval==0.4.1
2024-12-18T00:45:56.1690815Z mypy==1.13.0
2024-12-18T00:45:56.1691029Z mypy-extensions==1.0.0
2024-12-18T00:45:56.1691276Z networkx==2.8.8
2024-12-18T00:45:56.1691502Z ninja==1.10.2
2024-12-18T00:45:56.1691714Z nose==1.3.7
2024-12-18T00:45:56.1691919Z numba==0.55.2
2024-12-18T00:45:56.1692128Z numpy==1.21.2
2024-12-18T00:45:56.1692361Z nvidia-cublas-cu12==12.4.5.8
2024-12-18T00:45:56.1692652Z nvidia-cuda-cupti-cu12==12.4.127
2024-12-18T00:45:56.1693080Z nvidia-cuda-nvrtc-cu12==12.4.127
2024-12-18T00:45:56.1693381Z nvidia-cuda-runtime-cu12==12.4.127
2024-12-18T00:45:56.1693684Z nvidia-cudnn-cu12==9.1.0.70
2024-12-18T00:45:56.1693958Z nvidia-cufft-cu12==11.2.1.3
2024-12-18T00:45:56.1694256Z nvidia-curand-cu12==10.3.5.147
2024-12-18T00:45:56.1694544Z nvidia-cusolver-cu12==11.6.1.9
2024-12-18T00:45:56.1694861Z nvidia-cusparse-cu12==12.3.1.170
2024-12-18T00:45:56.1695249Z nvidia-ml-py==12.560.30
2024-12-18T00:45:56.1695516Z nvidia-nccl-cu12==2.21.5
2024-12-18T00:45:56.1695793Z nvidia-nvjitlink-cu12==12.4.127
2024-12-18T00:45:56.1696078Z nvidia-nvtx-cu12==12.4.127
2024-12-18T00:45:56.1696331Z oauthlib==3.2.2
2024-12-18T00:45:56.1696558Z omegaconf==2.3.0
2024-12-18T00:45:56.1696783Z onnx==1.17.0
2024-12-18T00:45:56.1697016Z onnxscript==0.1.0.dev20240817
2024-12-18T00:45:56.1697293Z opacus==1.5.2
2024-12-18T00:45:56.1697517Z opencv-python==4.8.0.74
2024-12-18T00:45:56.1697771Z opt-einsum==3.3.0
2024-12-18T00:45:56.1698001Z optree==0.13.0
2024-12-18T00:45:56.1698238Z packaging==24.2
2024-12-18T00:45:56.1698481Z pandas==2.0.3
2024-12-18T00:45:56.1698707Z parameterized==0.8.1
2024-12-18T00:45:56.1698956Z patch==1.16
2024-12-18T00:45:56.1699172Z pathspec==0.12.1
2024-12-18T00:45:56.1699412Z phonemizer==3.3.0
2024-12-18T00:45:56.1699640Z pillow==11.0.0
2024-12-18T00:45:56.1699866Z platformdirs==4.3.6
2024-12-18T00:45:56.1700100Z pluggy==1.5.0
2024-12-18T00:45:56.1700318Z ply==3.11
2024-12-18T00:45:56.1700525Z pooch==1.8.2
2024-12-18T00:45:56.1700772Z portalocker==3.0.0
2024-12-18T00:45:56.1701023Z preshed==3.0.9
2024-12-18T00:45:56.1701252Z prettytable==3.12.0
2024-12-18T00:45:56.1701495Z propcache==0.2.1
2024-12-18T00:45:56.1701720Z protobuf==3.20.2
2024-12-18T00:45:56.1702085Z psutil==6.1.0
2024-12-18T00:45:56.1702299Z PuLP==2.9.0
2024-12-18T00:45:56.1702512Z pwlf==2.2.1
2024-12-18T00:45:56.1702730Z py-cpuinfo==9.0.0
2024-12-18T00:45:56.1702955Z pyaml==24.12.1
2024-12-18T00:45:56.1703176Z pyarrow==18.1.0
2024-12-18T00:45:56.1703414Z pyasn1==0.6.1
2024-12-18T00:45:56.1703640Z pyasn1_modules==0.4.1
2024-12-18T00:45:56.1703892Z pyclipper==1.3.0.post6
2024-12-18T00:45:56.1704142Z pycocotools==2.0.8
2024-12-18T00:45:56.1704505Z pycparser==2.22
2024-12-18T00:45:56.1704745Z pycryptodomex==3.21.0
2024-12-18T00:45:56.1704993Z pydantic==2.10.3
2024-12-18T00:45:56.1705221Z pydantic_core==2.27.1
2024-12-18T00:45:56.1705463Z pyDOE==0.3.8
2024-12-18T00:45:56.1705700Z pydot==3.0.3
2024-12-18T00:45:56.1705998Z pygame==2.6.1
2024-12-18T00:45:56.1706269Z PyGithub==2.3.0
2024-12-18T00:45:56.1706491Z Pygments==2.15.0
2024-12-18T00:45:56.1706723Z PyJWT==2.10.1
2024-12-18T00:45:56.1706949Z pylatexenc==2.10
2024-12-18T00:45:56.1707179Z pylint==3.3.2
2024-12-18T00:45:56.1707401Z PyNaCl==1.5.0
2024-12-18T00:45:56.1707615Z pynvml==12.0.0
2024-12-18T00:45:56.1707843Z pyparsing==3.2.0
2024-12-18T00:45:56.1708080Z pypdfium2==4.30.0
2024-12-18T00:45:56.1708315Z pysbd==0.3.4
2024-12-18T00:45:56.1708528Z PySocks==1.7.1
2024-12-18T00:45:56.1708756Z pytest==8.3.4
2024-12-18T00:45:56.1709005Z pytest-benchmark==5.1.0
2024-12-18T00:45:56.1709265Z pytest-cpp==2.3.0
2024-12-18T00:45:56.1709514Z pytest-flakefinder==1.1.0
2024-12-18T00:45:56.1709802Z pytest-rerunfailures==14.0
2024-12-18T00:45:56.1710089Z pytest-subtests==0.13.1
2024-12-18T00:45:56.1710344Z pytest-xdist==3.3.1
2024-12-18T00:45:56.1710597Z python-dateutil==2.9.0.post0
2024-12-18T00:45:56.1711136Z python-doctr @ git+https://github.com/xuzhao9/doctr.git@7f8a3e61bd6480c4e7b7cde47af0aa2153309c56
2024-12-18T00:45:56.1712118Z pytorch-labs-segment-anything-fast @ git+https://github.com/pytorch-labs/segment-anything-fast.git@e6aadeb86f3ae1f58c3f98e2a91e251716e0f2aa
2024-12-18T00:45:56.1712864Z pytz==2024.2
2024-12-18T00:45:56.1713092Z PyWavelets==1.4.1
2024-12-18T00:45:56.1713335Z PyYAML==6.0.2
2024-12-18T00:45:56.1713560Z RapidFuzz==3.11.0
2024-12-18T00:45:56.1713782Z rdflib==7.1.1
2024-12-18T00:45:56.1714001Z redis==5.2.1
2024-12-18T00:45:56.1714221Z referencing==0.35.1
2024-12-18T00:45:56.1714453Z regex==2024.11.6
2024-12-18T00:45:56.1714803Z requests==2.32.3
2024-12-18T00:45:56.1715046Z requests-oauthlib==2.0.0
2024-12-18T00:45:56.1715309Z resampy==0.4.3
2024-12-18T00:45:56.1715535Z rfc3986==1.5.0
2024-12-18T00:45:56.1715748Z rich==13.9.4
2024-12-18T00:45:56.1715966Z rpds-py==0.22.3
2024-12-18T00:45:56.1716186Z rsa==4.9
2024-12-18T00:45:56.1716393Z s3transfer==0.10.4
2024-12-18T00:45:56.1716629Z safetensors==0.4.5
2024-12-18T00:45:56.1716869Z scikit-image==0.21.0
2024-12-18T00:45:56.1717108Z scikit-learn==1.6.0
2024-12-18T00:45:56.1717339Z scipy==1.10.1
2024-12-18T00:45:56.1717557Z segments==2.2.1
2024-12-18T00:45:56.1717792Z sentencepiece==0.2.0
2024-12-18T00:45:56.1718033Z sentry-sdk==2.19.2
2024-12-18T00:45:56.1718284Z setproctitle==1.3.4
2024-12-18T00:45:56.1718522Z shapely==2.0.6
2024-12-18T00:45:56.1718757Z shellingham==1.5.4
2024-12-18T00:45:56.1719000Z simplejson==3.19.3
2024-12-18T00:45:56.1719225Z six==1.17.0
2024-12-18T00:45:56.1719442Z smart-open==7.1.0
2024-12-18T00:45:56.1719676Z smmap==5.0.1
2024-12-18T00:45:56.1719917Z sortedcontainers==2.4.0
2024-12-18T00:45:56.1720174Z soundfile==0.12.1
2024-12-18T00:45:56.1720403Z soupsieve==2.6
2024-12-18T00:45:56.1720638Z soxr==0.5.0.post1
2024-12-18T00:45:56.1720872Z spacy==3.8.3
2024-12-18T00:45:56.1721099Z spacy-legacy==3.0.12
2024-12-18T00:45:56.1721343Z spacy-loggers==1.0.5
2024-12-18T00:45:56.1721584Z srsly==2.5.0
2024-12-18T00:45:56.1721802Z stempeg==0.2.3
2024-12-18T00:45:56.1722026Z submitit==1.5.2
2024-12-18T00:45:56.1722249Z sympy==1.13.1
2024-12-18T00:45:56.1722463Z tabulate==0.9.0
2024-12-18T00:45:56.1722702Z tb-nightly==2.13.0a20230426
2024-12-18T00:45:56.1722973Z tensorboard==2.13.0
2024-12-18T00:45:56.1723238Z tensorboard-data-server==0.7.2
2024-12-18T00:45:56.1723526Z tensorboardX==2.6.2.2
2024-12-18T00:45:56.1723768Z termcolor==2.5.0
2024-12-18T00:45:56.1723996Z thinc==8.3.3
2024-12-18T00:45:56.1724222Z threadpoolctl==3.5.0
2024-12-18T00:45:56.1724468Z thriftpy2==0.5.2
2024-12-18T00:45:56.1724692Z tifffile==2024.9.20
2024-12-18T00:45:56.1725230Z timm @ git+https://github.com/huggingface/pytorch-image-models.git@730b907b4d45a4713cbc425cbf224c46089fd514
2024-12-18T00:45:56.1725827Z tlparse==0.3.25
2024-12-18T00:45:56.1726178Z tokenizers==0.15.2
2024-12-18T00:45:56.1726413Z tomli==2.2.1
2024-12-18T00:45:56.1726627Z tomlkit==0.13.2
2024-12-18T00:45:56.1727431Z torch @ file:///var/lib/jenkins/workspace/dist/torch-2.6.0a0%2Bgit0cdf8b1-cp310-cp310-linux_x86_64.whl#sha256=dcadd2cd6f8025bfa9997113da51bf28d112b9db4fb82c725ed7dd046c33c456
2024-12-18T00:45:56.1728593Z torch_geometric @ git+https://github.com/pyg-team/pytorch_geometric.git@cabcd4097442ba60aa1efa11e1619dd9bb8fb527
2024-12-18T00:45:56.1729396Z torchao @ git+https://github.com/pytorch/ao.git@a5a53a2bd00f18401baaa00db4467f955704f71f
2024-12-18T00:45:56.1730556Z torchaudio @ git+https://github.com/pytorch/audio.git@332760d4b300f00a0d862e3cfe1495db3b1a14f9
2024-12-18T00:45:56.1731459Z torchmultimodal @ git+https://github.com/facebookresearch/multimodal.git@6569fcc03450c2360b50d772bf9b18ec3487fcf4
2024-12-18T00:45:56.1732323Z torchvision @ git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:45:56.1732846Z tornado==6.4.2
2024-12-18T00:45:56.1733076Z tqdm==4.67.1
2024-12-18T00:45:56.1733308Z transformers==4.38.1
2024-12-18T00:45:56.1733562Z treetable==0.2.5
2024-12-18T00:45:56.1734127Z -e git+https://github.com/triton-lang/triton@35c6c7c6284582b3f41c71c150e11b517acf074a#egg=triton&subdirectory=python
2024-12-18T00:45:56.1734751Z typeguard==4.4.1
2024-12-18T00:45:56.1734983Z typer==0.15.1
2024-12-18T00:45:56.1735221Z typing_extensions==4.12.2
2024-12-18T00:45:56.1735497Z tzdata==2024.2
2024-12-18T00:45:56.1735725Z Unidecode==1.3.8
2024-12-18T00:45:56.1735992Z unittest-xml-reporting==3.2.0
2024-12-18T00:45:56.1736294Z uritemplate==4.1.1
2024-12-18T00:45:56.1736539Z urllib3==2.2.3
2024-12-18T00:45:56.1736766Z visdom==0.2.4
2024-12-18T00:45:56.1736986Z wandb==0.19.1
2024-12-18T00:45:56.1737208Z wasabi==1.1.3
2024-12-18T00:45:56.1737435Z wcwidth==0.2.13
2024-12-18T00:45:56.1737968Z weasel==0.4.1
2024-12-18T00:45:56.1738204Z websocket-client==1.8.0
2024-12-18T00:45:56.1738457Z Werkzeug==3.1.3
2024-12-18T00:45:56.1738684Z wrapt==1.17.0
2024-12-18T00:45:56.1738913Z xdoctest==1.1.0
2024-12-18T00:45:56.1739140Z xxhash==3.5.0
2024-12-18T00:45:56.1739363Z xyzservices==2024.9.0
2024-12-18T00:45:56.1739605Z yacs==0.1.8
2024-12-18T00:45:56.1739814Z yarl==1.18.3
2024-12-18T00:45:56.1740033Z z3-solver==4.12.2.0
2024-12-18T00:45:56.1740265Z zipp==3.21.0
2024-12-18T00:45:56.1961336Z + popd
2024-12-18T00:45:56.1961594Z ~/workspace
2024-12-18T00:45:56.1961829Z + [[ inductor_torchbench != *cpu* ]]
2024-12-18T00:45:56.1962152Z + install_torchrec_and_fbgemm
2024-12-18T00:45:56.1962435Z + local torchrec_commit
2024-12-18T00:45:56.1966061Z ++ get_pinned_commit torchrec
2024-12-18T00:45:56.1966380Z ++ cat .github/ci_commit_pins/torchrec.txt
2024-12-18T00:45:56.2025135Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-12-18T00:45:56.2026037Z + local fbgemm_commit
2024-12-18T00:45:56.2030366Z ++ get_pinned_commit fbgemm
2024-12-18T00:45:56.2030675Z ++ cat .github/ci_commit_pins/fbgemm.txt
2024-12-18T00:45:56.2102902Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd
2024-12-18T00:45:56.2103431Z + pip_uninstall torchrec-nightly
2024-12-18T00:45:56.2103864Z + pip3 uninstall -y torchrec-nightly
2024-12-18T00:45:56.5965474Z WARNING: Skipping torchrec-nightly as it is not installed.
2024-12-18T00:45:56.6317384Z + pip_uninstall fbgemm-gpu-nightly
2024-12-18T00:45:56.6317785Z + pip3 uninstall -y fbgemm-gpu-nightly
2024-12-18T00:45:57.0049741Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed.
2024-12-18T00:45:57.0415356Z + pip_install setuptools-git-versioning scikit-build pyre-extensions
2024-12-18T00:45:57.0415915Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T00:45:57.0416536Z + python3 -m pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions
2024-12-18T00:45:57.5390195Z Collecting setuptools-git-versioning
2024-12-18T00:45:57.5587035Z   Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl.metadata (5.8 kB)
2024-12-18T00:45:57.6865690Z Collecting scikit-build
2024-12-18T00:45:57.6926681Z   Downloading scikit_build-0.18.1-py3-none-any.whl.metadata (18 kB)
2024-12-18T00:45:57.8045513Z Collecting pyre-extensions
2024-12-18T00:45:57.8106789Z   Downloading pyre_extensions-0.0.32-py3-none-any.whl.metadata (4.0 kB)
2024-12-18T00:45:57.8637112Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.2)
2024-12-18T00:45:57.8640323Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (75.1.0)
2024-12-18T00:45:57.9214447Z Collecting toml>=0.10.2 (from setuptools-git-versioning)
2024-12-18T00:45:57.9247216Z   Downloading toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB)
2024-12-18T00:45:57.9765756Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.9.0)
2024-12-18T00:45:57.9771686Z Requirement already satisfied: tomli in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (2.2.1)
2024-12-18T00:45:57.9775502Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.44.0)
2024-12-18T00:45:58.0323399Z Collecting typing-inspect (from pyre-extensions)
2024-12-18T00:45:58.0358095Z   Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB)
2024-12-18T00:45:58.0875831Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.12.2)
2024-12-18T00:45:58.0948356Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0)
2024-12-18T00:45:58.1008442Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl (10 kB)
2024-12-18T00:45:58.1575992Z Downloading scikit_build-0.18.1-py3-none-any.whl (85 kB)
2024-12-18T00:45:58.2138173Z Downloading pyre_extensions-0.0.32-py3-none-any.whl (12 kB)
2024-12-18T00:45:58.2649653Z Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)
2024-12-18T00:45:58.3213566Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB)
2024-12-18T00:45:59.2238902Z Installing collected packages: typing-inspect, toml, scikit-build, setuptools-git-versioning, pyre-extensions
2024-12-18T00:45:59.5988935Z Successfully installed pyre-extensions-0.0.32 scikit-build-0.18.1 setuptools-git-versioning-2.0.0 toml-0.10.2 typing-inspect-0.9.0
2024-12-18T00:45:59.7124744Z + [[ 0 == \1 ]]
2024-12-18T00:45:59.7125104Z + CUDA_PATH=/usr/local/cuda-12.1
2024-12-18T00:45:59.7126198Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu'
2024-12-18T00:45:59.7127395Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T00:45:59.7128426Z + python3 -m pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu'
2024-12-18T00:46:00.0798592Z Collecting fbgemm-gpu
2024-12-18T00:46:00.0801124Z   Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-o3lxa2j4/fbgemm-gpu_5ada129ad4804747b4d59ed9364b442c
2024-12-18T00:46:00.0829734Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-o3lxa2j4/fbgemm-gpu_5ada129ad4804747b4d59ed9364b442c
2024-12-18T00:46:01.3070487Z   Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd'
2024-12-18T00:46:01.3098571Z   Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd
2024-12-18T00:46:01.9438919Z   Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd
2024-12-18T00:46:02.4726516Z   Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd
2024-12-18T00:46:02.4727862Z   Running command git submodule update --init --recursive -q
2024-12-18T00:46:10.7174544Z   Preparing metadata (setup.py) ... [?25l- done
2024-12-18T00:46:10.7195577Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.21.2)
2024-12-18T00:46:10.7204489Z Building wheels for collected packages: fbgemm-gpu
2024-12-18T01:30:05.8171314Z   Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done
2024-12-18T01:30:06.0012713Z [?25h  Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=253178278 sha256=2f7a04254272f20d1c20a551d5f3906acdc71cb217b561c4bfc4f1cee7d83906
2024-12-18T01:30:06.0013929Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39
2024-12-18T01:30:06.0034744Z Successfully built fbgemm-gpu
2024-12-18T01:30:06.8487102Z Installing collected packages: fbgemm-gpu
2024-12-18T01:30:10.2294434Z Successfully installed fbgemm-gpu-0.4.1rc0.post421
2024-12-18T01:30:11.1423887Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-12-18T01:30:11.1424641Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T01:30:11.1425462Z + python3 -m pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-12-18T01:30:11.5105389Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-12-18T01:30:11.5109939Z   Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-5htwd2oc
2024-12-18T01:30:11.5140328Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-5htwd2oc
2024-12-18T01:30:12.6225966Z   Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2'
2024-12-18T01:30:12.6255532Z   Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-12-18T01:30:13.1009450Z   Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-12-18T01:30:13.6835201Z   Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-12-18T01:30:13.9582567Z   Preparing metadata (setup.py) ... [?25l- done
2024-12-18T01:30:13.9617922Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9)
2024-12-18T01:30:13.9620624Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.0.3)
2024-12-18T01:30:13.9623939Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0)
2024-12-18T01:30:14.0204793Z Collecting torchmetrics (from torchrec==0.3.2)
2024-12-18T01:30:14.0402280Z   Downloading torchmetrics-1.6.0-py3-none-any.whl.metadata (20 kB)
2024-12-18T01:30:14.0836202Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.67.1)
2024-12-18T01:30:14.0838870Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421)
2024-12-18T01:30:14.0849469Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.21.2)
2024-12-18T01:30:14.0858615Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (3.0.0)
2024-12-18T01:30:14.1129990Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0)
2024-12-18T01:30:14.1132763Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.2)
2024-12-18T01:30:14.1135047Z Requirement already satisfied: tzdata>=2022.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.2)
2024-12-18T01:30:14.1158079Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.2)
2024-12-18T01:30:14.1161826Z Requirement already satisfied: torch>=2.0.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.6.0a0+git0cdf8b1)
2024-12-18T01:30:14.1318143Z Collecting lightning-utilities>=0.8.0 (from torchmetrics->torchrec==0.3.2)
2024-12-18T01:30:14.1377801Z   Downloading lightning_utilities-0.11.9-py3-none-any.whl.metadata (5.2 kB)
2024-12-18T01:30:14.1463214Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (75.1.0)
2024-12-18T01:30:14.1467001Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.12.2)
2024-12-18T01:30:14.1486539Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.17.0)
2024-12-18T01:30:14.1518269Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=2.0.0->torchmetrics->torchrec==0.3.2) (3.16.1)
2024-12-18T01:30:14.1521686Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=2.0.0->torchmetrics->torchrec==0.3.2) (2.8.8)
2024-12-18T01:30:14.1524557Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=2.0.0->torchmetrics->torchrec==0.3.2) (3.1.4)
2024-12-18T01:30:14.1527334Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=2.0.0->torchmetrics->torchrec==0.3.2) (2024.9.0)
2024-12-18T01:30:14.1532912Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=2.0.0->torchmetrics->torchrec==0.3.2) (1.13.1)
2024-12-18T01:30:14.1548448Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch>=2.0.0->torchmetrics->torchrec==0.3.2) (1.3.0)
2024-12-18T01:30:14.1962158Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=2.0.0->torchmetrics->torchrec==0.3.2) (3.0.2)
2024-12-18T01:30:14.2114707Z Downloading torchmetrics-1.6.0-py3-none-any.whl (926 kB)
2024-12-18T01:30:14.2342481Z Downloading lightning_utilities-0.11.9-py3-none-any.whl (28 kB)
2024-12-18T01:30:14.2454373Z Building wheels for collected packages: torchrec
2024-12-18T01:30:14.8219682Z   Building wheel for torchrec (setup.py) ... [?25l- \ | done
2024-12-18T01:30:14.8228739Z [?25h  Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374487 sha256=3ccbb91474c7a73461630998f739013f26963c2592ac3976da92d22d93b4225d
2024-12-18T01:30:14.8229853Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8
2024-12-18T01:30:14.8253664Z Successfully built torchrec
2024-12-18T01:30:15.6172913Z Installing collected packages: lightning-utilities, torchmetrics, torchrec
2024-12-18T01:30:16.2585224Z Successfully installed lightning-utilities-0.11.9 torchmetrics-1.6.0 torchrec-0.3.2
2024-12-18T01:30:16.4050588Z + [[ 0 == \1 ]]
2024-12-18T01:30:16.4053983Z ++ pwd
2024-12-18T01:30:16.4057026Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench
2024-12-18T01:30:16.4057492Z + test_dynamo_benchmark torchbench 0
2024-12-18T01:30:16.4062893Z ++ pwd
2024-12-18T01:30:16.4066811Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2024-12-18T01:30:16.4067241Z + local suite=torchbench
2024-12-18T01:30:16.4067501Z + shift
2024-12-18T01:30:16.4067708Z + local shard_id=0
2024-12-18T01:30:16.4067946Z + shift
2024-12-18T01:30:16.4068192Z + [[ inductor_torchbench == *perf_compare* ]]
2024-12-18T01:30:16.4068542Z + [[ inductor_torchbench == *perf* ]]
2024-12-18T01:30:16.4068861Z + [[ inductor_torchbench == *cpu* ]]
2024-12-18T01:30:16.4069446Z + [[ inductor_torchbench == *aot_inductor* ]]
2024-12-18T01:30:16.4069917Z + test_single_dynamo_benchmark inference torchbench 0 --inference --bfloat16
2024-12-18T01:30:16.4072432Z ++ pwd
2024-12-18T01:30:16.4075351Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2024-12-18T01:30:16.4075838Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports
2024-12-18T01:30:16.4126576Z + local name=inference
2024-12-18T01:30:16.4126955Z + shift
2024-12-18T01:30:16.4127172Z + local suite=torchbench
2024-12-18T01:30:16.4127430Z + shift
2024-12-18T01:30:16.4127637Z + local shard_id=0
2024-12-18T01:30:16.4127866Z + shift
2024-12-18T01:30:16.4128077Z + partition_flags=()
2024-12-18T01:30:16.4128324Z + local partition_flags
2024-12-18T01:30:16.4128628Z + [[ -n 2 ]]
2024-12-18T01:30:16.4128846Z + [[ -n 0 ]]
2024-12-18T01:30:16.4129248Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id")
2024-12-18T01:30:16.4129773Z + [[ inductor_torchbench == *perf_compare* ]]
2024-12-18T01:30:16.4130390Z + [[ inductor_torchbench == *perf* ]]
2024-12-18T01:30:16.4130877Z + [[ inductor_torchbench == *_avx2* ]]
2024-12-18T01:30:16.4131206Z + [[ inductor_torchbench == *_avx512* ]]
2024-12-18T01:30:16.4132280Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --inductor --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv
2024-12-18T01:30:23.2896915Z 
2024-12-18T01:30:24.5859154Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:30:24.5859507Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:30:24.5859828Z cuda eval  torchrec_dlrm                      
2024-12-18T01:30:24.5900009Z Traceback (most recent call last):
2024-12-18T01:30:24.5901290Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2732, in validate_model
2024-12-18T01:30:24.5902605Z     self.model_iter_fn(model, example_inputs)
2024-12-18T01:30:24.5903764Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 444, in forward_pass
2024-12-18T01:30:24.5904843Z     return mod(*inputs)
2024-12-18T01:30:24.5906207Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:30:24.5906892Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:30:24.5907509Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:30:24.5908136Z     return forward_call(*args, **kwargs)
2024-12-18T01:30:24.5908732Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 893, in forward
2024-12-18T01:30:24.5909443Z     logits = self.model(batch.dense_features, batch.sparse_features)
2024-12-18T01:30:24.5910197Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:30:24.5910850Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:30:24.5911457Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:30:24.5912075Z     return forward_call(*args, **kwargs)
2024-12-18T01:30:24.5912650Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 572, in forward
2024-12-18T01:30:24.5913243Z     concatenated_dense = self.inter_arch(
2024-12-18T01:30:24.5913893Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:30:24.5914542Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:30:24.5915141Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:30:24.5915742Z     return forward_call(*args, **kwargs)
2024-12-18T01:30:24.5916315Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 284, in forward
2024-12-18T01:30:24.5916978Z     return self.crossnet(combined_values.reshape([B, -1]))
2024-12-18T01:30:24.5917850Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:30:24.5918504Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:30:24.5919107Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:30:24.5919714Z     return forward_call(*args, **kwargs)
2024-12-18T01:30:24.5920313Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/crossnet.py", line 175, in forward
2024-12-18T01:30:24.5920998Z     x_l_v = torch.nn.functional.linear(x_l, self.V_kernels[layer])
2024-12-18T01:30:24.5921613Z RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::BFloat16
2024-12-18T01:30:24.5922007Z 
2024-12-18T01:30:24.5922225Z The above exception was the direct cause of the following exception:
2024-12-18T01:30:24.5922541Z 
2024-12-18T01:30:24.5922659Z Traceback (most recent call last):
2024-12-18T01:30:24.5923133Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run
2024-12-18T01:30:24.5923634Z     ) = runner.load_model(
2024-12-18T01:30:24.5924122Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 372, in load_model
2024-12-18T01:30:24.5924692Z     self.validate_model(model, example_inputs)
2024-12-18T01:30:24.5925242Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2734, in validate_model
2024-12-18T01:30:24.5925802Z     raise RuntimeError("Eager run failed") from e
2024-12-18T01:30:24.5926150Z RuntimeError: Eager run failed
2024-12-18T01:30:24.5926343Z 
2024-12-18T01:30:24.5926435Z eager_fail_to_run
2024-12-18T01:30:28.7196785Z 
2024-12-18T01:30:31.0784282Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:30:31.0784852Z loading model: 0it [00:02, ?it/s]
2024-12-18T01:30:31.0842370Z cuda eval  BERT_pytorch                       
2024-12-18T01:30:52.0210104Z pass
2024-12-18T01:30:52.0245428Z TIMING: _recursive_pre_grad_passes:0.03552 _recursive_joint_graph_passes:0.71474 _recursive_post_grad_passes:0.91283 async_compile.wait:1.55381 code_gen:3.93529 inductor_compile:7.18913 backend_compile:14.39002 entire_frame_compile:18.74685 total_wall_time:18.74685
2024-12-18T01:30:52.0246963Z STATS: call_* op count: 1076 | FakeTensor.__torch_dispatch__:2319 | FakeTensorMode.__torch_dispatch__:28253 | ProxyTorchDispatchMode.__torch_dispatch__:12905
2024-12-18T01:30:52.0247831Z Dynamo produced 1 graphs covering 1076 ops with 0 graph breaks (0 unique)
2024-12-18T01:30:56.8852420Z 
2024-12-18T01:30:59.8213884Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:30:59.8214538Z loading model: 0it [00:02, ?it/s]
2024-12-18T01:30:59.8264185Z cuda eval  Background_Matting                 
2024-12-18T01:30:59.8496985Z pass_due_to_skip
2024-12-18T01:30:59.9031482Z TIMING: total_wall_time:0
2024-12-18T01:30:59.9032421Z STATS: call_* op count: 0
2024-12-18T01:30:59.9033253Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-12-18T01:31:04.0031895Z 
2024-12-18T01:31:07.2100236Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:31:07.2100617Z loading model: 0it [00:03, ?it/s]
2024-12-18T01:31:07.2124568Z cuda eval  LearningToPaint                    
2024-12-18T01:31:15.6950173Z pass
2024-12-18T01:31:15.6961220Z TIMING: _recursive_pre_grad_passes:0.00914 _recursive_joint_graph_passes:0.37175 _recursive_post_grad_passes:0.10745 async_compile.wait:0.66842 code_gen:1.58195 inductor_compile:2.56238 backend_compile:5.09684 entire_frame_compile:6.75195 total_wall_time:6.75195
2024-12-18T01:31:15.6962694Z STATS: call_* op count: 142 | FakeTensorMode.__torch_dispatch__:6321 | ProxyTorchDispatchMode.__torch_dispatch__:2930 | FakeTensor.__torch_dispatch__:1094
2024-12-18T01:31:15.6963519Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique)
2024-12-18T01:31:20.4924129Z 
2024-12-18T01:31:22.0261455Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/vgg16-397923af.pth
2024-12-18T01:31:22.0417239Z 
2024-12-18T01:31:22.0417518Z 
2024-12-18T01:31:22.1422201Z   0% 0.00/528M [00:00<?, ?B/s]
2024-12-18T01:31:22.1422444Z 
2024-12-18T01:31:22.2422830Z   8% 42.5M/528M [00:00<00:01, 445MB/s]
2024-12-18T01:31:22.2423080Z 
2024-12-18T01:31:22.3424739Z  16% 86.5M/528M [00:00<00:01, 454MB/s]
2024-12-18T01:31:22.3424988Z 
2024-12-18T01:31:22.4425420Z  25% 130M/528M [00:00<00:00, 457MB/s] 
2024-12-18T01:31:22.4425678Z 
2024-12-18T01:31:22.5445090Z  33% 174M/528M [00:00<00:00, 459MB/s]
2024-12-18T01:31:22.5445341Z 
2024-12-18T01:31:22.6445633Z  41% 218M/528M [00:00<00:00, 455MB/s]
2024-12-18T01:31:22.6445885Z 
2024-12-18T01:31:22.7446528Z  50% 262M/528M [00:00<00:00, 457MB/s]
2024-12-18T01:31:22.7446772Z 
2024-12-18T01:31:22.8448716Z  58% 306M/528M [00:00<00:00, 459MB/s]
2024-12-18T01:31:22.8448965Z 
2024-12-18T01:31:22.9450814Z  66% 351M/528M [00:00<00:00, 460MB/s]
2024-12-18T01:31:22.9451075Z 
2024-12-18T01:31:23.0452300Z  75% 395M/528M [00:00<00:00, 461MB/s]
2024-12-18T01:31:23.0452552Z 
2024-12-18T01:31:23.1454703Z  83% 439M/528M [00:01<00:00, 461MB/s]
2024-12-18T01:31:23.1454955Z 
2024-12-18T01:31:23.2456291Z  92% 483M/528M [00:01<00:00, 462MB/s]
2024-12-18T01:31:23.2456537Z 
2024-12-18T01:31:23.2478409Z 100% 527M/528M [00:01<00:00, 462MB/s]
2024-12-18T01:31:23.2478740Z 100% 528M/528M [00:01<00:00, 459MB/s]
2024-12-18T01:31:23.8602920Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2024-12-18T01:31:24.1856697Z 
2024-12-18T01:31:24.1857294Z loading model: 0it [00:03, ?it/s]
2024-12-18T01:31:24.1858240Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2024-12-18T01:31:24.1859238Z cuda eval  Super_SloMo                        
2024-12-18T01:31:35.4050471Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2024-12-18T01:32:15.4758046Z pass
2024-12-18T01:32:15.4814549Z TIMING: _recursive_pre_grad_passes:0.02405 _recursive_joint_graph_passes:0.31319 _recursive_post_grad_passes:0.38563 async_compile.wait:5.56111 code_gen:16.33041 inductor_compile:21.88933 backend_compile:27.67606 entire_frame_compile:30.99152 total_wall_time:30.99152
2024-12-18T01:32:15.4816060Z STATS: call_* op count: 752 | FakeTensorMode.__torch_dispatch__:30963 | ProxyTorchDispatchMode.__torch_dispatch__:14575 | FakeTensor.__torch_dispatch__:3778
2024-12-18T01:32:15.4816905Z Dynamo produced 1 graphs covering 752 ops with 0 graph breaks (0 unique)
2024-12-18T01:32:20.7988773Z 
2024-12-18T01:32:21.4445197Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/alexnet-owt-7be5be79.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/alexnet-owt-7be5be79.pth
2024-12-18T01:32:21.4603687Z 
2024-12-18T01:32:21.4603926Z 
2024-12-18T01:32:21.5607531Z   0% 0.00/233M [00:00<?, ?B/s]
2024-12-18T01:32:21.5607776Z 
2024-12-18T01:32:21.6610884Z  16% 38.4M/233M [00:00<00:00, 401MB/s]
2024-12-18T01:32:21.6611413Z 
2024-12-18T01:32:21.7613973Z  34% 79.6M/233M [00:00<00:00, 419MB/s]
2024-12-18T01:32:21.7614457Z 
2024-12-18T01:32:21.8616402Z  52% 121M/233M [00:00<00:00, 426MB/s] 
2024-12-18T01:32:21.8616867Z 
2024-12-18T01:32:21.9617213Z  70% 164M/233M [00:00<00:00, 433MB/s]
2024-12-18T01:32:21.9617670Z 
2024-12-18T01:32:22.0254306Z  88% 206M/233M [00:00<00:00, 438MB/s]
2024-12-18T01:32:22.0254637Z 100% 233M/233M [00:00<00:00, 433MB/s]
2024-12-18T01:32:22.8204506Z 
2024-12-18T01:32:22.8205178Z loading model: 0it [00:02, ?it/s]
2024-12-18T01:32:22.8208619Z cuda eval  alexnet                            
2024-12-18T01:32:28.9809322Z pass
2024-12-18T01:32:28.9812408Z TIMING: _recursive_pre_grad_passes:0.00466 _recursive_joint_graph_passes:0.79728 _recursive_post_grad_passes:0.03221 async_compile.wait:0.44916 code_gen:1.23264 inductor_compile:1.48551 backend_compile:3.19494 entire_frame_compile:3.60678 total_wall_time:3.60678
2024-12-18T01:32:28.9813987Z STATS: call_* op count: 44 | FakeTensorMode.__torch_dispatch__:1462 | ProxyTorchDispatchMode.__torch_dispatch__:415 | FakeTensor.__torch_dispatch__:89
2024-12-18T01:32:28.9815189Z Dynamo produced 1 graphs covering 44 ops with 0 graph breaks (0 unique)
2024-12-18T01:32:33.4985141Z 
2024-12-18T01:32:35.7813678Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:32:35.7814017Z loading model: 0it [00:02, ?it/s]
2024-12-18T01:32:35.7818138Z cuda eval  basic_gnn_edgecnn                  
2024-12-18T01:32:38.3718770Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_reduce.two Found from   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2771, in run_n_iterations
2024-12-18T01:32:38.3719841Z     return self.model_iter_fn(mod, inputs, collect_outputs=True)
2024-12-18T01:32:38.3720473Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 444, in forward_pass
2024-12-18T01:32:38.3721021Z     return mod(*inputs)
2024-12-18T01:32:38.3721608Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/models/basic_gnn.py", line 226, in forward
2024-12-18T01:32:38.3722256Z     x = self.convs[i](x, edge_index)
2024-12-18T01:32:38.3722645Z   File "/tmp/jenkins_pyg/tmp04957g98.py", line 233, in forward
2024-12-18T01:32:38.3723066Z     return self.propagate(edge_index, x=x, size=None)
2024-12-18T01:32:38.3723501Z   File "/tmp/jenkins_pyg/tmp04957g98.py", line 192, in propagate
2024-12-18T01:32:38.3724062Z     out = self.aggregate(out, index=kwargs.index, dim_size=kwargs.dim_size, ptr=kwargs.ptr)
2024-12-18T01:32:38.3724879Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/message_passing.py", line 604, in aggregate
2024-12-18T01:32:38.3725624Z     return self.aggr_module(inputs, index, ptr=ptr, dim_size=dim_size,
2024-12-18T01:32:38.3726309Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 109, in __call__
2024-12-18T01:32:38.3726974Z     return super().__call__(x, index, ptr, dim_size, dim, **kwargs)
2024-12-18T01:32:38.3727643Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/basic.py", line 48, in forward
2024-12-18T01:32:38.3728584Z     return self.reduce(x, index, ptr, dim_size, dim, reduce='max')
2024-12-18T01:32:38.3729262Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 155, in reduce
2024-12-18T01:32:38.3729891Z     return scatter(x, index, dim, dim_size, reduce)
2024-12-18T01:32:38.3730940Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 98, in scatter
2024-12-18T01:32:38.3731627Z     return src.new_zeros(size).scatter_reduce_(
2024-12-18T01:32:38.3731889Z 
2024-12-18T01:32:38.3731893Z 
2024-12-18T01:32:38.5625141Z pass
2024-12-18T01:32:38.5654348Z TIMING: _recursive_pre_grad_passes:0.00596 _recursive_joint_graph_passes:0.13744 _recursive_post_grad_passes:0.0394 async_compile.wait:0.33516 code_gen:0.52777 inductor_compile:0.73531 backend_compile:1.80735 entire_frame_compile:2.40559 total_wall_time:2.40559
2024-12-18T01:32:38.5657212Z STATS: call_* op count: 110 | FakeTensorMode.__torch_dispatch__:2150 | ProxyTorchDispatchMode.__torch_dispatch__:829 | FakeTensor.__torch_dispatch__:95
2024-12-18T01:32:38.5658713Z Dynamo produced 1 graphs covering 110 ops with 0 graph breaks (0 unique)
2024-12-18T01:32:43.0179033Z 
2024-12-18T01:32:45.2004822Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:32:45.2005168Z loading model: 0it [00:02, ?it/s]
2024-12-18T01:32:45.2006360Z cuda eval  basic_gnn_gcn                      
2024-12-18T01:32:48.0292594Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_add.default Found from   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/gcn_conv.py", line 100, in torch_dynamo_resume_in_gcn_norm_at_91
2024-12-18T01:32:48.0293901Z     deg = scatter(edge_weight, idx, dim=0, dim_size=num_nodes, reduce='sum')
2024-12-18T01:32:48.0294619Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 74, in scatter
2024-12-18T01:32:48.0295594Z     return src.new_zeros(size).scatter_add_(dim, index, src)
2024-12-18T01:32:48.0295876Z 
2024-12-18T01:32:48.0295881Z 
2024-12-18T01:32:48.5943009Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_add.default Found from   File "/tmp/jenkins_pyg/tmp_oby_6ru.py", line 243, in torch_dynamo_resume_in_forward_at_221
2024-12-18T01:32:48.5944032Z     out = self.propagate(edge_index, x=x, edge_weight=edge_weight,
2024-12-18T01:32:48.5944516Z   File "/tmp/jenkins_pyg/tmp_oby_6ru.py", line 188, in propagate
2024-12-18T01:32:48.5945076Z     out = self.aggregate(out, dim_size=kwargs.dim_size, ptr=kwargs.ptr, index=kwargs.index)
2024-12-18T01:32:48.5945910Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/message_passing.py", line 604, in aggregate
2024-12-18T01:32:48.5946715Z     return self.aggr_module(inputs, index, ptr=ptr, dim_size=dim_size,
2024-12-18T01:32:48.5947437Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 109, in __call__
2024-12-18T01:32:48.5948123Z     return super().__call__(x, index, ptr, dim_size, dim, **kwargs)
2024-12-18T01:32:48.5948807Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/basic.py", line 21, in forward
2024-12-18T01:32:48.5949494Z     return self.reduce(x, index, ptr, dim_size, dim, reduce='sum')
2024-12-18T01:32:48.5950166Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 155, in reduce
2024-12-18T01:32:48.5950790Z     return scatter(x, index, dim, dim_size, reduce)
2024-12-18T01:32:48.5951412Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 74, in scatter
2024-12-18T01:32:48.5952077Z     return src.new_zeros(size).scatter_add_(dim, index, src)
2024-12-18T01:32:48.5952355Z 
2024-12-18T01:32:48.5952364Z 
2024-12-18T01:32:49.1760728Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_add.default Found from   File "/tmp/jenkins_pyg/tmp_oby_6ru.py", line 243, in torch_dynamo_resume_in_forward_at_221
2024-12-18T01:32:49.1761718Z     out = self.propagate(edge_index, x=x, edge_weight=edge_weight,
2024-12-18T01:32:49.1762200Z   File "/tmp/jenkins_pyg/tmp_oby_6ru.py", line 188, in propagate
2024-12-18T01:32:49.1762764Z     out = self.aggregate(out, dim_size=kwargs.dim_size, ptr=kwargs.ptr, index=kwargs.index)
2024-12-18T01:32:49.1763587Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/message_passing.py", line 604, in aggregate
2024-12-18T01:32:49.1764342Z     return self.aggr_module(inputs, index, ptr=ptr, dim_size=dim_size,
2024-12-18T01:32:49.1765027Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 109, in __call__
2024-12-18T01:32:49.1765713Z     return super().__call__(x, index, ptr, dim_size, dim, **kwargs)
2024-12-18T01:32:49.1766389Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/basic.py", line 21, in forward
2024-12-18T01:32:49.1767074Z     return self.reduce(x, index, ptr, dim_size, dim, reduce='sum')
2024-12-18T01:32:49.1767743Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 155, in reduce
2024-12-18T01:32:49.1768371Z     return scatter(x, index, dim, dim_size, reduce)
2024-12-18T01:32:49.1768992Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 74, in scatter
2024-12-18T01:32:49.1769655Z     return src.new_zeros(size).scatter_add_(dim, index, src)
2024-12-18T01:32:49.1769928Z 
2024-12-18T01:32:49.1769933Z 
2024-12-18T01:32:49.4593006Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_add.default Found from   File "/tmp/jenkins_pyg/tmp_oby_6ru.py", line 243, in torch_dynamo_resume_in_forward_at_221
2024-12-18T01:32:49.4594935Z     out = self.propagate(edge_index, x=x, edge_weight=edge_weight,
2024-12-18T01:32:49.4595707Z   File "/tmp/jenkins_pyg/tmp_oby_6ru.py", line 188, in propagate
2024-12-18T01:32:49.4596455Z     out = self.aggregate(out, dim_size=kwargs.dim_size, ptr=kwargs.ptr, index=kwargs.index)
2024-12-18T01:32:49.4597761Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/message_passing.py", line 604, in aggregate
2024-12-18T01:32:49.4599013Z     return self.aggr_module(inputs, index, ptr=ptr, dim_size=dim_size,
2024-12-18T01:32:49.4600168Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 109, in __call__
2024-12-18T01:32:49.4601190Z     return super().__call__(x, index, ptr, dim_size, dim, **kwargs)
2024-12-18T01:32:49.4602304Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/basic.py", line 21, in forward
2024-12-18T01:32:49.4603461Z     return self.reduce(x, index, ptr, dim_size, dim, reduce='sum')
2024-12-18T01:32:49.4604623Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 155, in reduce
2024-12-18T01:32:49.4605707Z     return scatter(x, index, dim, dim_size, reduce)
2024-12-18T01:32:49.4606767Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 74, in scatter
2024-12-18T01:32:49.4607892Z     return src.new_zeros(size).scatter_add_(dim, index, src)
2024-12-18T01:32:49.4608313Z 
2024-12-18T01:32:49.4608321Z 
2024-12-18T01:32:49.6031861Z pass
2024-12-18T01:32:49.6146639Z TIMING: entire_frame_compile:3.20251 _recursive_pre_grad_passes:0.00714 _recursive_joint_graph_passes:0.13167 _recursive_post_grad_passes:0.01938 async_compile.wait:1.20328 code_gen:1.22148 inductor_compile:1.37579 backend_compile:2.51724 total_wall_time:3.20251
2024-12-18T01:32:49.6148157Z STATS: call_* op count: 57 | FakeTensorMode.__torch_dispatch__:1066 | FakeTensor.__torch_dispatch__:70 | ProxyTorchDispatchMode.__torch_dispatch__:218
2024-12-18T01:32:49.6149280Z Dynamo produced 6 graphs covering 57 ops with 6 graph breaks (1 unique)
2024-12-18T01:32:54.0777334Z 
2024-12-18T01:32:56.1645659Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:32:56.1646087Z loading model: 0it [00:02, ?it/s]
2024-12-18T01:32:56.1649205Z cuda eval  basic_gnn_gin                      
2024-12-18T01:32:58.7141036Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_add.default Found from   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2771, in run_n_iterations
2024-12-18T01:32:58.7142176Z     return self.model_iter_fn(mod, inputs, collect_outputs=True)
2024-12-18T01:32:58.7142803Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 444, in forward_pass
2024-12-18T01:32:58.7143354Z     return mod(*inputs)
2024-12-18T01:32:58.7143944Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/models/basic_gnn.py", line 226, in forward
2024-12-18T01:32:58.7144625Z     x = self.convs[i](x, edge_index)
2024-12-18T01:32:58.7145012Z   File "/tmp/jenkins_pyg/tmpns4fots8.py", line 225, in forward
2024-12-18T01:32:58.7145455Z     out = self.propagate(edge_index, x=x, size=size)
2024-12-18T01:32:58.7145880Z   File "/tmp/jenkins_pyg/tmpns4fots8.py", line 182, in propagate
2024-12-18T01:32:58.7146440Z     out = self.aggregate(out, ptr=kwargs.ptr, dim_size=kwargs.dim_size, index=kwargs.index)
2024-12-18T01:32:58.7147259Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/message_passing.py", line 604, in aggregate
2024-12-18T01:32:58.7148031Z     return self.aggr_module(inputs, index, ptr=ptr, dim_size=dim_size,
2024-12-18T01:32:58.7148727Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 109, in __call__
2024-12-18T01:32:58.7149424Z     return super().__call__(x, index, ptr, dim_size, dim, **kwargs)
2024-12-18T01:32:58.7150114Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/basic.py", line 21, in forward
2024-12-18T01:32:58.7151199Z     return self.reduce(x, index, ptr, dim_size, dim, reduce='sum')
2024-12-18T01:32:58.7151872Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 155, in reduce
2024-12-18T01:32:58.7152486Z     return scatter(x, index, dim, dim_size, reduce)
2024-12-18T01:32:58.7153105Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 74, in scatter
2024-12-18T01:32:58.7153757Z     return src.new_zeros(size).scatter_add_(dim, index, src)
2024-12-18T01:32:58.7154027Z 
2024-12-18T01:32:58.7154031Z 
2024-12-18T01:32:58.9408416Z pass
2024-12-18T01:32:58.9504378Z TIMING: _recursive_pre_grad_passes:0.00562 _recursive_joint_graph_passes:0.13711 _recursive_post_grad_passes:0.0518 async_compile.wait:0.20179 code_gen:0.35678 inductor_compile:0.56285 backend_compile:1.62141 entire_frame_compile:2.20105 total_wall_time:2.20105
2024-12-18T01:32:58.9505976Z STATS: call_* op count: 104 | FakeTensorMode.__torch_dispatch__:2016 | ProxyTorchDispatchMode.__torch_dispatch__:815 | FakeTensor.__torch_dispatch__:100
2024-12-18T01:32:58.9506946Z Dynamo produced 1 graphs covering 104 ops with 0 graph breaks (0 unique)
2024-12-18T01:33:03.3732142Z 
2024-12-18T01:33:05.4727590Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:33:05.4727949Z loading model: 0it [00:02, ?it/s]
2024-12-18T01:33:05.4731202Z cuda eval  basic_gnn_sage                     
2024-12-18T01:33:07.9480550Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_add.default Found from   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2771, in run_n_iterations
2024-12-18T01:33:07.9482150Z     return self.model_iter_fn(mod, inputs, collect_outputs=True)
2024-12-18T01:33:07.9483081Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 444, in forward_pass
2024-12-18T01:33:07.9483876Z     return mod(*inputs)
2024-12-18T01:33:07.9484776Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/models/basic_gnn.py", line 226, in forward
2024-12-18T01:33:07.9486088Z     x = self.convs[i](x, edge_index)
2024-12-18T01:33:07.9486683Z   File "/tmp/jenkins_pyg/tmpjr_iyhas.py", line 228, in forward
2024-12-18T01:33:07.9487326Z     out = self.propagate(edge_index, x=x, size=size)
2024-12-18T01:33:07.9487959Z   File "/tmp/jenkins_pyg/tmpjr_iyhas.py", line 182, in propagate
2024-12-18T01:33:07.9488797Z     out = self.aggregate(out, dim_size=kwargs.dim_size, ptr=kwargs.ptr, index=kwargs.index)
2024-12-18T01:33:07.9490023Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/message_passing.py", line 604, in aggregate
2024-12-18T01:33:07.9491159Z     return self.aggr_module(inputs, index, ptr=ptr, dim_size=dim_size,
2024-12-18T01:33:07.9492198Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 109, in __call__
2024-12-18T01:33:07.9493220Z     return super().__call__(x, index, ptr, dim_size, dim, **kwargs)
2024-12-18T01:33:07.9494222Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/basic.py", line 34, in forward
2024-12-18T01:33:07.9495246Z     return self.reduce(x, index, ptr, dim_size, dim, reduce='mean')
2024-12-18T01:33:07.9496249Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 155, in reduce
2024-12-18T01:33:07.9497185Z     return scatter(x, index, dim, dim_size, reduce)
2024-12-18T01:33:07.9498113Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 82, in scatter
2024-12-18T01:33:07.9499083Z     out = src.new_zeros(size).scatter_add_(dim, index, src)
2024-12-18T01:33:07.9499474Z 
2024-12-18T01:33:07.9499480Z 
2024-12-18T01:33:08.1437592Z pass
2024-12-18T01:33:08.1524155Z TIMING: _recursive_pre_grad_passes:0.0059 _recursive_joint_graph_passes:0.14051 _recursive_post_grad_passes:0.04472 async_compile.wait:0.17722 code_gen:0.34557 inductor_compile:0.55235 backend_compile:1.62448 entire_frame_compile:2.10369 total_wall_time:2.10369
2024-12-18T01:33:08.1525985Z STATS: call_* op count: 116 | FakeTensorMode.__torch_dispatch__:2082 | ProxyTorchDispatchMode.__torch_dispatch__:840 | FakeTensor.__torch_dispatch__:111
2024-12-18T01:33:08.1526844Z Dynamo produced 1 graphs covering 116 ops with 0 graph breaks (0 unique)
2024-12-18T01:33:12.5893878Z 
2024-12-18T01:33:16.4037501Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:33:16.4037981Z loading model: 0it [00:03, ?it/s]
2024-12-18T01:33:16.4126307Z cuda eval  cm3leon_generate                   
2024-12-18T01:35:51.0460584Z CUDAGraph supports dynamic shapes by recording a new graph for each distinct input size. Recording too many CUDAGraphs may lead to extra overhead. We have observed 51 distinct sizes. Please consider the following options for better performance: a) padding inputs to a few fixed number of shapes; or b) set torch._inductor.config.triton.cudagraph_skip_dynamic_graphs=True. Set torch._inductor.config.triton.cudagraph_dynamic_shape_warn_limit=None to silence this warning.
2024-12-18T01:35:51.0815337Z CUDAGraph supports dynamic shapes by recording a new graph for each distinct input size. Recording too many CUDAGraphs may lead to extra overhead. We have observed 51 distinct sizes. Please consider the following options for better performance: a) padding inputs to a few fixed number of shapes; or b) set torch._inductor.config.triton.cudagraph_skip_dynamic_graphs=True. Set torch._inductor.config.triton.cudagraph_dynamic_shape_warn_limit=None to silence this warning.
2024-12-18T01:37:16.3927754Z pass
2024-12-18T01:37:16.4352794Z TIMING: entire_frame_compile:208.86298 _recursive_pre_grad_passes:0.12732 _recursive_joint_graph_passes:24.30211 _recursive_post_grad_passes:7.12743 async_compile.precompile:0.16452 async_compile.wait:3.98754 code_gen:40.10587 inductor_compile:77.52475 backend_compile:166.25596 total_wall_time:208.86298
2024-12-18T01:37:16.4355116Z STATS: call_* op count: 4737 | FakeTensorMode.__torch_dispatch__:156217 | FakeTensor.__torch_dispatch__:23015 | ProxyTorchDispatchMode.__torch_dispatch__:62506 | attempt fast:572 | fast is_contiguous:560 | slow no contiguity match:12
2024-12-18T01:37:16.4356251Z Dynamo produced 10 graphs covering 4737 ops with 4 graph breaks (1 unique)
2024-12-18T01:37:49.6079790Z 
2024-12-18T01:37:50.2253473Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:37:50.2256027Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:37:50.2258701Z cuda eval  dcgan                              
2024-12-18T01:37:53.9628764Z pass
2024-12-18T01:37:53.9629853Z TIMING: _recursive_pre_grad_passes:0.00412 _recursive_joint_graph_passes:0.12458 _recursive_post_grad_passes:0.17882 async_compile.wait:0.2403 code_gen:0.58315 inductor_compile:0.99202 backend_compile:2.00173 entire_frame_compile:2.41203 total_wall_time:2.41203
2024-12-18T01:37:53.9631643Z STATS: call_* op count: 26 | FakeTensorMode.__torch_dispatch__:1336 | FakeTensor.__torch_dispatch__:189 | ProxyTorchDispatchMode.__torch_dispatch__:520
2024-12-18T01:37:53.9632496Z Dynamo produced 1 graphs covering 26 ops with 0 graph breaks (0 unique)
2024-12-18T01:37:58.3297041Z 
2024-12-18T01:38:01.3703542Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:38:01.3703890Z loading model: 0it [00:03, ?it/s]
2024-12-18T01:38:01.3722252Z cuda eval  demucs                             
2024-12-18T01:38:42.0178047Z W1218 01:38:42.016000 39417 site-packages/torch/_functorch/_aot_autograd/autograd_cache.py:856] [2/0_1] Bypassing autograd cache due to: Cannot cache a graph with functional tensor
2024-12-18T01:38:46.4455580Z pass
2024-12-18T01:38:46.4524898Z TIMING: entire_frame_compile:5.27929 _recursive_pre_grad_passes:0.00608 _recursive_joint_graph_passes:0.12871 _recursive_post_grad_passes:0.02517 async_compile.precompile:0.10808 async_compile.wait:0.50232 code_gen:1.58408 inductor_compile:2.26015 backend_compile:3.4639 total_wall_time:5.27929
2024-12-18T01:38:46.4528670Z STATS: call_* op count: 62 | FakeTensorMode.__torch_dispatch__:2298 | ProxyTorchDispatchMode.__torch_dispatch__:595 | FakeTensor.__torch_dispatch__:223
2024-12-18T01:38:46.4531463Z Dynamo produced 3 graphs covering 62 ops with 3 graph breaks (1 unique)
2024-12-18T01:38:50.9823352Z 
2024-12-18T01:38:51.3675309Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/densenet121-a639ec97.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/densenet121-a639ec97.pth
2024-12-18T01:38:51.3836055Z 
2024-12-18T01:38:51.3836304Z 
2024-12-18T01:38:51.4638929Z   0% 0.00/30.8M [00:00<?, ?B/s]
2024-12-18T01:38:51.4639276Z 100% 30.8M/30.8M [00:00<00:00, 404MB/s]
2024-12-18T01:38:52.5288225Z 
2024-12-18T01:38:52.5288896Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:38:52.5424230Z cuda eval  densenet121                        
2024-12-18T01:39:35.0373235Z pass
2024-12-18T01:39:35.0385787Z TIMING: _recursive_pre_grad_passes:0.04307 _recursive_joint_graph_passes:0.51347 _recursive_post_grad_passes:0.53577 async_compile.wait:1.9268 code_gen:11.60682 inductor_compile:17.39606 backend_compile:28.03192 entire_frame_compile:38.31724 total_wall_time:38.31724
2024-12-18T01:39:35.0387808Z STATS: call_* op count: 862 | FakeTensorMode.__torch_dispatch__:35242 | ProxyTorchDispatchMode.__torch_dispatch__:16359 | FakeTensor.__torch_dispatch__:6359
2024-12-18T01:39:35.0388847Z Dynamo produced 1 graphs covering 862 ops with 0 graph breaks (0 unique)
2024-12-18T01:39:40.8375498Z 
2024-12-18T01:39:44.8451875Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:39:44.8452237Z loading model: 0it [00:04, ?it/s]
2024-12-18T01:39:44.8452569Z cuda eval  detectron2_fasterrcnn_r_101_c4     
2024-12-18T01:39:44.8473020Z Traceback (most recent call last):
2024-12-18T01:39:44.8473810Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2732, in validate_model
2024-12-18T01:39:44.8474521Z     self.model_iter_fn(model, example_inputs)
2024-12-18T01:39:44.8475080Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 444, in forward_pass
2024-12-18T01:39:44.8475636Z     return mod(*inputs)
2024-12-18T01:39:44.8477232Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:39:44.8477899Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:39:44.8478495Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:39:44.8479112Z     return forward_call(*args, **kwargs)
2024-12-18T01:39:44.8479739Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2024-12-18T01:39:44.8480393Z     return self.inference(batched_inputs)
2024-12-18T01:39:44.8481041Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2024-12-18T01:39:44.8481776Z     results, _ = self.roi_heads(images, features, proposals, None)
2024-12-18T01:39:44.8482493Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:39:44.8483161Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:39:44.8483763Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:39:44.8484351Z     return forward_call(*args, **kwargs)
2024-12-18T01:39:44.8485009Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2024-12-18T01:39:44.8485691Z     box_features = self._shared_roi_transform(
2024-12-18T01:39:44.8486416Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2024-12-18T01:39:44.8487121Z     x = self.pooler(features, boxes)
2024-12-18T01:39:44.8487759Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:39:44.8488397Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:39:44.8489161Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:39:44.8489750Z     return forward_call(*args, **kwargs)
2024-12-18T01:39:44.8490345Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2024-12-18T01:39:44.8491005Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2024-12-18T01:39:44.8491670Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:39:44.8492293Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:39:44.8492882Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:39:44.8493478Z     return forward_call(*args, **kwargs)
2024-12-18T01:39:44.8494054Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2024-12-18T01:39:44.8494629Z     return roi_align(
2024-12-18T01:39:44.8495182Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-12-18T01:39:44.8495974Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-12-18T01:39:44.8496780Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-12-18T01:39:44.8497590Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-12-18T01:39:44.8498427Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-12-18T01:39:44.8499082Z     v1 = masked_index(y_low, x_low)
2024-12-18T01:39:44.8499682Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-12-18T01:39:44.8500300Z     return input[
2024-12-18T01:39:44.8502528Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 26886.45 GiB. GPU 0 has a total capacity of 21.98 GiB of which 18.44 GiB is free. Process 227771 has 3.53 GiB memory in use. Of the allocated memory 2.99 GiB is allocated by PyTorch, and 242.98 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-12-18T01:39:44.8504594Z 
2024-12-18T01:39:44.8504814Z The above exception was the direct cause of the following exception:
2024-12-18T01:39:44.8505134Z 
2024-12-18T01:39:44.8505252Z Traceback (most recent call last):
2024-12-18T01:39:44.8505745Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run
2024-12-18T01:39:44.8506233Z     ) = runner.load_model(
2024-12-18T01:39:44.8506736Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 372, in load_model
2024-12-18T01:39:44.8507309Z     self.validate_model(model, example_inputs)
2024-12-18T01:39:44.8507882Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2734, in validate_model
2024-12-18T01:39:44.8508465Z     raise RuntimeError("Eager run failed") from e
2024-12-18T01:39:44.8508814Z RuntimeError: Eager run failed
2024-12-18T01:39:44.8509008Z 
2024-12-18T01:39:44.8509101Z eager_fail_to_run
2024-12-18T01:39:49.0461536Z 
2024-12-18T01:39:53.9082811Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:39:53.9083195Z loading model: 0it [00:04, ?it/s]
2024-12-18T01:39:53.9083529Z cuda eval  detectron2_fasterrcnn_r_101_dc5    
2024-12-18T01:39:53.9100283Z Traceback (most recent call last):
2024-12-18T01:39:53.9100898Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2732, in validate_model
2024-12-18T01:39:53.9101472Z     self.model_iter_fn(model, example_inputs)
2024-12-18T01:39:53.9102439Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 444, in forward_pass
2024-12-18T01:39:53.9102981Z     return mod(*inputs)
2024-12-18T01:39:53.9103580Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:39:53.9104251Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:39:53.9104849Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:39:53.9105444Z     return forward_call(*args, **kwargs)
2024-12-18T01:39:53.9106075Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2024-12-18T01:39:53.9106728Z     return self.inference(batched_inputs)
2024-12-18T01:39:53.9107373Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2024-12-18T01:39:53.9108101Z     results, _ = self.roi_heads(images, features, proposals, None)
2024-12-18T01:39:53.9108817Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:39:53.9109451Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:39:53.9110040Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:39:53.9110629Z     return forward_call(*args, **kwargs)
2024-12-18T01:39:53.9111266Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2024-12-18T01:39:53.9111985Z     pred_instances = self._forward_box(features, proposals)
2024-12-18T01:39:53.9112702Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2024-12-18T01:39:53.9113500Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2024-12-18T01:39:53.9114278Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:39:53.9115091Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:39:53.9115688Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:39:53.9116274Z     return forward_call(*args, **kwargs)
2024-12-18T01:39:53.9116869Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2024-12-18T01:39:53.9117515Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2024-12-18T01:39:53.9118182Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:39:53.9118822Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:39:53.9119412Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:39:53.9120002Z     return forward_call(*args, **kwargs)
2024-12-18T01:39:53.9120584Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2024-12-18T01:39:53.9121156Z     return roi_align(
2024-12-18T01:39:53.9121694Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-12-18T01:39:53.9122510Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-12-18T01:39:53.9123311Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-12-18T01:39:53.9124079Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-12-18T01:39:53.9124924Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-12-18T01:39:53.9125587Z     v1 = masked_index(y_low, x_low)
2024-12-18T01:39:53.9126286Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-12-18T01:39:53.9126874Z     return input[
2024-12-18T01:39:53.9128842Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 13324.51 GiB. GPU 0 has a total capacity of 21.98 GiB of which 17.63 GiB is free. Process 227894 has 4.34 GiB memory in use. Of the allocated memory 3.82 GiB is allocated by PyTorch, and 195.43 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-12-18T01:39:53.9131071Z 
2024-12-18T01:39:53.9131282Z The above exception was the direct cause of the following exception:
2024-12-18T01:39:53.9131602Z 
2024-12-18T01:39:53.9131712Z Traceback (most recent call last):
2024-12-18T01:39:53.9132182Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run
2024-12-18T01:39:53.9132672Z     ) = runner.load_model(
2024-12-18T01:39:53.9133163Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 372, in load_model
2024-12-18T01:39:53.9133723Z     self.validate_model(model, example_inputs)
2024-12-18T01:39:53.9134265Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2734, in validate_model
2024-12-18T01:39:53.9134830Z     raise RuntimeError("Eager run failed") from e
2024-12-18T01:39:53.9135176Z RuntimeError: Eager run failed
2024-12-18T01:39:53.9135359Z 
2024-12-18T01:39:53.9135459Z eager_fail_to_run
2024-12-18T01:39:58.1296890Z 
2024-12-18T01:40:01.3882785Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:40:01.3894008Z loading model: 0it [00:03, ?it/s]
2024-12-18T01:40:01.3894482Z cuda eval  detectron2_fasterrcnn_r_101_fpn    
2024-12-18T01:40:01.3898667Z Traceback (most recent call last):
2024-12-18T01:40:01.3899233Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2732, in validate_model
2024-12-18T01:40:01.3899838Z     self.model_iter_fn(model, example_inputs)
2024-12-18T01:40:01.3900772Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 444, in forward_pass
2024-12-18T01:40:01.3901326Z     return mod(*inputs)
2024-12-18T01:40:01.3902015Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:40:01.3902677Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:40:01.3903278Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:40:01.3903892Z     return forward_call(*args, **kwargs)
2024-12-18T01:40:01.3904521Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2024-12-18T01:40:01.3905171Z     return self.inference(batched_inputs)
2024-12-18T01:40:01.3905813Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2024-12-18T01:40:01.3906555Z     results, _ = self.roi_heads(images, features, proposals, None)
2024-12-18T01:40:01.3907266Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:40:01.3907905Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:40:01.3908493Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:40:01.3909090Z     return forward_call(*args, **kwargs)
2024-12-18T01:40:01.3909735Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2024-12-18T01:40:01.3910444Z     pred_instances = self._forward_box(features, proposals)
2024-12-18T01:40:01.3911168Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2024-12-18T01:40:01.3912206Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2024-12-18T01:40:01.3912985Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:40:01.3913619Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:40:01.3914215Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:40:01.3914811Z     return forward_call(*args, **kwargs)
2024-12-18T01:40:01.3915416Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2024-12-18T01:40:01.3916110Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2024-12-18T01:40:01.3916834Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:40:01.3917487Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:40:01.3918091Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:40:01.3918692Z     return forward_call(*args, **kwargs)
2024-12-18T01:40:01.3919286Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2024-12-18T01:40:01.3919860Z     return roi_align(
2024-12-18T01:40:01.3920396Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-12-18T01:40:01.3921201Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-12-18T01:40:01.3922006Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-12-18T01:40:01.3922780Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-12-18T01:40:01.3923609Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-12-18T01:40:01.3924401Z     v1 = masked_index(y_low, x_low)
2024-12-18T01:40:01.3924997Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-12-18T01:40:01.3925585Z     return input[
2024-12-18T01:40:01.3927551Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 19073.68 GiB. GPU 0 has a total capacity of 21.98 GiB of which 17.77 GiB is free. Process 227959 has 4.20 GiB memory in use. Of the allocated memory 3.79 GiB is allocated by PyTorch, and 112.62 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-12-18T01:40:01.3929456Z 
2024-12-18T01:40:01.3929667Z The above exception was the direct cause of the following exception:
2024-12-18T01:40:01.3930197Z 
2024-12-18T01:40:01.3930361Z Traceback (most recent call last):
2024-12-18T01:40:01.3930862Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run
2024-12-18T01:40:01.3931351Z     ) = runner.load_model(
2024-12-18T01:40:01.3932009Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 372, in load_model
2024-12-18T01:40:01.3932580Z     self.validate_model(model, example_inputs)
2024-12-18T01:40:01.3933129Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2734, in validate_model
2024-12-18T01:40:01.3933705Z     raise RuntimeError("Eager run failed") from e
2024-12-18T01:40:01.3934059Z RuntimeError: Eager run failed
2024-12-18T01:40:01.3934246Z 
2024-12-18T01:40:01.3934348Z eager_fail_to_run
2024-12-18T01:40:05.6100906Z 
2024-12-18T01:40:08.5825115Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:40:08.5825474Z loading model: 0it [00:02, ?it/s]
2024-12-18T01:40:08.5826095Z cuda eval  detectron2_fasterrcnn_r_50_c4      
2024-12-18T01:40:08.5839607Z Traceback (most recent call last):
2024-12-18T01:40:08.5840294Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2732, in validate_model
2024-12-18T01:40:08.5840879Z     self.model_iter_fn(model, example_inputs)
2024-12-18T01:40:08.5841451Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 444, in forward_pass
2024-12-18T01:40:08.5842009Z     return mod(*inputs)
2024-12-18T01:40:08.5842632Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:40:08.5843308Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:40:08.5843939Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:40:08.5844563Z     return forward_call(*args, **kwargs)
2024-12-18T01:40:08.5845207Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2024-12-18T01:40:08.5845874Z     return self.inference(batched_inputs)
2024-12-18T01:40:08.5846555Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2024-12-18T01:40:08.5847307Z     results, _ = self.roi_heads(images, features, proposals, None)
2024-12-18T01:40:08.5848050Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:40:08.5848706Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:40:08.5849321Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:40:08.5849936Z     return forward_call(*args, **kwargs)
2024-12-18T01:40:08.5850601Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2024-12-18T01:40:08.5851291Z     box_features = self._shared_roi_transform(
2024-12-18T01:40:08.5852254Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2024-12-18T01:40:08.5852979Z     x = self.pooler(features, boxes)
2024-12-18T01:40:08.5853619Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:40:08.5854274Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:40:08.5854883Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:40:08.5855495Z     return forward_call(*args, **kwargs)
2024-12-18T01:40:08.5856103Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2024-12-18T01:40:08.5856769Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2024-12-18T01:40:08.5857471Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:40:08.5858141Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:40:08.5858759Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:40:08.5859384Z     return forward_call(*args, **kwargs)
2024-12-18T01:40:08.5859991Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2024-12-18T01:40:08.5860584Z     return roi_align(
2024-12-18T01:40:08.5861148Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-12-18T01:40:08.5862056Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-12-18T01:40:08.5862899Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-12-18T01:40:08.5863712Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-12-18T01:40:08.5864704Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-12-18T01:40:08.5865383Z     v1 = masked_index(y_low, x_low)
2024-12-18T01:40:08.5865985Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-12-18T01:40:08.5866600Z     return input[
2024-12-18T01:40:08.5868624Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 27475.92 GiB. GPU 0 has a total capacity of 21.98 GiB of which 19.59 GiB is free. Process 228020 has 2.38 GiB memory in use. Of the allocated memory 1.86 GiB is allocated by PyTorch, and 233.53 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-12-18T01:40:08.5870649Z 
2024-12-18T01:40:08.5870868Z The above exception was the direct cause of the following exception:
2024-12-18T01:40:08.5871205Z 
2024-12-18T01:40:08.5871319Z Traceback (most recent call last):
2024-12-18T01:40:08.5871802Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run
2024-12-18T01:40:08.5872295Z     ) = runner.load_model(
2024-12-18T01:40:08.5872785Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 372, in load_model
2024-12-18T01:40:08.5873368Z     self.validate_model(model, example_inputs)
2024-12-18T01:40:08.5873929Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2734, in validate_model
2024-12-18T01:40:08.5874499Z     raise RuntimeError("Eager run failed") from e
2024-12-18T01:40:08.5874852Z RuntimeError: Eager run failed
2024-12-18T01:40:08.5875046Z 
2024-12-18T01:40:08.5875139Z eager_fail_to_run
2024-12-18T01:40:12.7768652Z 
2024-12-18T01:40:17.2837392Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:40:17.2837890Z loading model: 0it [00:04, ?it/s]
2024-12-18T01:40:17.2838338Z cuda eval  detectron2_fasterrcnn_r_50_dc5     
2024-12-18T01:40:17.2853877Z Traceback (most recent call last):
2024-12-18T01:40:17.2854493Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2732, in validate_model
2024-12-18T01:40:17.2855060Z     self.model_iter_fn(model, example_inputs)
2024-12-18T01:40:17.2855612Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 444, in forward_pass
2024-12-18T01:40:17.2856152Z     return mod(*inputs)
2024-12-18T01:40:17.2856808Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:40:17.2857454Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:40:17.2858049Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:40:17.2858671Z     return forward_call(*args, **kwargs)
2024-12-18T01:40:17.2859312Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2024-12-18T01:40:17.2859968Z     return self.inference(batched_inputs)
2024-12-18T01:40:17.2860607Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2024-12-18T01:40:17.2861331Z     results, _ = self.roi_heads(images, features, proposals, None)
2024-12-18T01:40:17.2862167Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:40:17.2862804Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:40:17.2863392Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:40:17.2863983Z     return forward_call(*args, **kwargs)
2024-12-18T01:40:17.2864616Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2024-12-18T01:40:17.2865527Z     pred_instances = self._forward_box(features, proposals)
2024-12-18T01:40:17.2866247Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2024-12-18T01:40:17.2867055Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2024-12-18T01:40:17.2867835Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:40:17.2868479Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:40:17.2869066Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:40:17.2869665Z     return forward_call(*args, **kwargs)
2024-12-18T01:40:17.2870273Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2024-12-18T01:40:17.2870949Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2024-12-18T01:40:17.2871631Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:40:17.2872268Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:40:17.2872856Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:40:17.2873456Z     return forward_call(*args, **kwargs)
2024-12-18T01:40:17.2874047Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2024-12-18T01:40:17.2874627Z     return roi_align(
2024-12-18T01:40:17.2875183Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-12-18T01:40:17.2875989Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-12-18T01:40:17.2876841Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-12-18T01:40:17.2877721Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-12-18T01:40:17.2878546Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-12-18T01:40:17.2879197Z     v1 = masked_index(y_low, x_low)
2024-12-18T01:40:17.2879795Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-12-18T01:40:17.2880390Z     return input[
2024-12-18T01:40:17.2882395Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 13541.47 GiB. GPU 0 has a total capacity of 21.98 GiB of which 18.77 GiB is free. Process 228081 has 3.20 GiB memory in use. Of the allocated memory 2.68 GiB is allocated by PyTorch, and 195.87 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-12-18T01:40:17.2884375Z 
2024-12-18T01:40:17.2884593Z The above exception was the direct cause of the following exception:
2024-12-18T01:40:17.2884906Z 
2024-12-18T01:40:17.2885018Z Traceback (most recent call last):
2024-12-18T01:40:17.2885491Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run
2024-12-18T01:40:17.2885975Z     ) = runner.load_model(
2024-12-18T01:40:17.2886468Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 372, in load_model
2024-12-18T01:40:17.2887031Z     self.validate_model(model, example_inputs)
2024-12-18T01:40:17.2887578Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2734, in validate_model
2024-12-18T01:40:17.2888143Z     raise RuntimeError("Eager run failed") from e
2024-12-18T01:40:17.2888488Z RuntimeError: Eager run failed
2024-12-18T01:40:17.2888764Z 
2024-12-18T01:40:17.2888857Z eager_fail_to_run
2024-12-18T01:40:21.4695795Z 
2024-12-18T01:40:24.3525078Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:40:24.3525438Z loading model: 0it [00:02, ?it/s]
2024-12-18T01:40:24.3525778Z cuda eval  detectron2_fasterrcnn_r_50_fpn     
2024-12-18T01:40:24.3542592Z Traceback (most recent call last):
2024-12-18T01:40:24.3543153Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2732, in validate_model
2024-12-18T01:40:24.3543878Z     self.model_iter_fn(model, example_inputs)
2024-12-18T01:40:24.3544468Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 444, in forward_pass
2024-12-18T01:40:24.3545011Z     return mod(*inputs)
2024-12-18T01:40:24.3545608Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:40:24.3546283Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:40:24.3546903Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:40:24.3547517Z     return forward_call(*args, **kwargs)
2024-12-18T01:40:24.3548154Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2024-12-18T01:40:24.3548803Z     return self.inference(batched_inputs)
2024-12-18T01:40:24.3549448Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2024-12-18T01:40:24.3550179Z     results, _ = self.roi_heads(images, features, proposals, None)
2024-12-18T01:40:24.3550897Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:40:24.3551552Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:40:24.3552162Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:40:24.3552778Z     return forward_call(*args, **kwargs)
2024-12-18T01:40:24.3553721Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2024-12-18T01:40:24.3554461Z     pred_instances = self._forward_box(features, proposals)
2024-12-18T01:40:24.3555185Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2024-12-18T01:40:24.3556002Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2024-12-18T01:40:24.3556778Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:40:24.3557437Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:40:24.3558026Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:40:24.3558663Z     return forward_call(*args, **kwargs)
2024-12-18T01:40:24.3559280Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2024-12-18T01:40:24.3559975Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2024-12-18T01:40:24.3562039Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:40:24.3562675Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:40:24.3563261Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:40:24.3563868Z     return forward_call(*args, **kwargs)
2024-12-18T01:40:24.3564452Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2024-12-18T01:40:24.3565035Z     return roi_align(
2024-12-18T01:40:24.3565597Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-12-18T01:40:24.3566603Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-12-18T01:40:24.3567410Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-12-18T01:40:24.3568193Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-12-18T01:40:24.3569017Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-12-18T01:40:24.3569674Z     v1 = masked_index(y_low, x_low)
2024-12-18T01:40:24.3570266Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-12-18T01:40:24.3570873Z     return input[
2024-12-18T01:40:24.3573025Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 21403.23 GiB. GPU 0 has a total capacity of 21.98 GiB of which 18.94 GiB is free. Process 228144 has 3.03 GiB memory in use. Of the allocated memory 2.65 GiB is allocated by PyTorch, and 93.33 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-12-18T01:40:24.3574973Z 
2024-12-18T01:40:24.3575193Z The above exception was the direct cause of the following exception:
2024-12-18T01:40:24.3575504Z 
2024-12-18T01:40:24.3575614Z Traceback (most recent call last):
2024-12-18T01:40:24.3576083Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run
2024-12-18T01:40:24.3576571Z     ) = runner.load_model(
2024-12-18T01:40:24.3577058Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 372, in load_model
2024-12-18T01:40:24.3577622Z     self.validate_model(model, example_inputs)
2024-12-18T01:40:24.3578177Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2734, in validate_model
2024-12-18T01:40:24.3578747Z     raise RuntimeError("Eager run failed") from e
2024-12-18T01:40:24.3579176Z RuntimeError: Eager run failed
2024-12-18T01:40:24.3579368Z 
2024-12-18T01:40:24.3579459Z eager_fail_to_run
2024-12-18T01:40:28.5355788Z 
2024-12-18T01:40:30.7443931Z loading model: 0it [00:00, ?it/s]WARNING:common:Model detectron2_fcos_r_50_fpn does not support bfloat16, running with amp instead
2024-12-18T01:40:31.4050120Z 
2024-12-18T01:40:31.4050614Z loading model: 0it [00:02, ?it/s]
2024-12-18T01:40:31.4051258Z WARNING:common:Model detectron2_fcos_r_50_fpn does not support bfloat16, running with amp instead
2024-12-18T01:40:31.4051903Z cuda eval  detectron2_fcos_r_50_fpn           
2024-12-18T01:40:43.8660270Z WARNING:common:Model detectron2_fcos_r_50_fpn does not support bfloat16, running with amp instead
2024-12-18T01:40:54.4400249Z skipping cudagraphs due to mutated inputs (1 instances). Found from : 
2024-12-18T01:40:54.4402663Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/structures/image_list.py", line 127, in torch_dynamo_resume_in_from_tensors_at_122
2024-12-18T01:40:54.4403578Z     batched_imgs[i, ..., : img.shape[-2], : img.shape[-1]].copy_(img)
2024-12-18T01:40:54.4403874Z 
2024-12-18T01:42:05.0398693Z W1218 01:42:05.038000 40617 site-packages/torch/_functorch/_aot_autograd/autograd_cache.py:856] [14/0] Bypassing autograd cache due to: Cannot cache a graph with functional tensor
2024-12-18T01:42:18.8829434Z W1218 01:42:18.881000 40617 site-packages/torch/_functorch/_aot_autograd/autograd_cache.py:856] [35/0] Bypassing autograd cache due to: Cannot cache a graph with functional tensor
2024-12-18T01:42:21.6485874Z skipping cudagraphs due to skipping cudagraphs due to cpu device (output)
2024-12-18T01:42:24.4161172Z skipping cudagraphs due to skipping cudagraphs due to cpu device (output)
2024-12-18T01:43:18.3404365Z pass
2024-12-18T01:43:18.3422389Z TIMING: entire_frame_compile:98.3284 _recursive_pre_grad_passes:0.04445 _recursive_joint_graph_passes:0.48122 inductor_compile:43.95478 backend_compile:68.56489 _recursive_post_grad_passes:0.56056 async_compile.precompile:0.36052 async_compile.wait:11.14857 code_gen:22.99578 total_wall_time:98.3284
2024-12-18T01:43:18.3425260Z STATS: call_* op count: 886 | FakeTensorMode.__torch_dispatch__:38261 | FakeTensor.__torch_dispatch__:4680 | ProxyTorchDispatchMode.__torch_dispatch__:9164 | attempt fast:1295 | fast is_contiguous:1285 | slow no contiguity match:10
2024-12-18T01:43:18.3426355Z Dynamo produced 38 graphs covering 886 ops with 22 graph breaks (4 unique)
2024-12-18T01:43:31.9628047Z 
2024-12-18T01:43:35.4307876Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:43:35.4308238Z loading model: 0it [00:03, ?it/s]
2024-12-18T01:43:35.4308573Z cuda eval  detectron2_maskrcnn_r_101_c4       
2024-12-18T01:43:35.4323441Z Traceback (most recent call last):
2024-12-18T01:43:35.4324333Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2732, in validate_model
2024-12-18T01:43:35.4325207Z     self.model_iter_fn(model, example_inputs)
2024-12-18T01:43:35.4326034Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 444, in forward_pass
2024-12-18T01:43:35.4326813Z     return mod(*inputs)
2024-12-18T01:43:35.4327735Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:43:35.4328757Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:43:35.4329647Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:43:35.4331036Z     return forward_call(*args, **kwargs)
2024-12-18T01:43:35.4332065Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2024-12-18T01:43:35.4333584Z     return self.inference(batched_inputs)
2024-12-18T01:43:35.4334641Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2024-12-18T01:43:35.4335728Z     results, _ = self.roi_heads(images, features, proposals, None)
2024-12-18T01:43:35.4337189Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:43:35.4338134Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:43:35.4339049Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:43:35.4339945Z     return forward_call(*args, **kwargs)
2024-12-18T01:43:35.4340947Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2024-12-18T01:43:35.4342126Z     box_features = self._shared_roi_transform(
2024-12-18T01:43:35.4343265Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2024-12-18T01:43:35.4344365Z     x = self.pooler(features, boxes)
2024-12-18T01:43:35.4345370Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:43:35.4346384Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:43:35.4347318Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:43:35.4348252Z     return forward_call(*args, **kwargs)
2024-12-18T01:43:35.4349193Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2024-12-18T01:43:35.4350242Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2024-12-18T01:43:35.4351382Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:43:35.4352463Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:43:35.4353514Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:43:35.4354725Z     return forward_call(*args, **kwargs)
2024-12-18T01:43:35.4355711Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2024-12-18T01:43:35.4356693Z     return roi_align(
2024-12-18T01:43:35.4357619Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-12-18T01:43:35.4359005Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-12-18T01:43:35.4360401Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-12-18T01:43:35.4361751Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-12-18T01:43:35.4363191Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-12-18T01:43:35.4364323Z     v1 = masked_index(y_low, x_low)
2024-12-18T01:43:35.4365323Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-12-18T01:43:35.4366348Z     return input[
2024-12-18T01:43:35.4369633Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 26943.76 GiB. GPU 0 has a total capacity of 21.98 GiB of which 18.47 GiB is free. Process 229017 has 3.50 GiB memory in use. Of the allocated memory 3.00 GiB is allocated by PyTorch, and 212.63 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-12-18T01:43:35.4372924Z 
2024-12-18T01:43:35.4373293Z The above exception was the direct cause of the following exception:
2024-12-18T01:43:35.4373848Z 
2024-12-18T01:43:35.4374045Z Traceback (most recent call last):
2024-12-18T01:43:35.4374928Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run
2024-12-18T01:43:35.4375903Z     ) = runner.load_model(
2024-12-18T01:43:35.4376672Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 372, in load_model
2024-12-18T01:43:35.4377634Z     self.validate_model(model, example_inputs)
2024-12-18T01:43:35.4378595Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2734, in validate_model
2024-12-18T01:43:35.4379558Z     raise RuntimeError("Eager run failed") from e
2024-12-18T01:43:35.4380154Z RuntimeError: Eager run failed
2024-12-18T01:43:35.4380473Z 
2024-12-18T01:43:35.4380639Z eager_fail_to_run
2024-12-18T01:43:39.6745985Z 
2024-12-18T01:43:42.9486469Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:43:42.9486895Z loading model: 0it [00:03, ?it/s]
2024-12-18T01:43:42.9487267Z cuda eval  detectron2_maskrcnn_r_101_fpn      
2024-12-18T01:43:42.9500985Z Traceback (most recent call last):
2024-12-18T01:43:42.9501666Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2732, in validate_model
2024-12-18T01:43:42.9502456Z     self.model_iter_fn(model, example_inputs)
2024-12-18T01:43:42.9503094Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 444, in forward_pass
2024-12-18T01:43:42.9503641Z     return mod(*inputs)
2024-12-18T01:43:42.9504304Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:43:42.9505036Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:43:42.9505705Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:43:42.9506342Z     return forward_call(*args, **kwargs)
2024-12-18T01:43:42.9507034Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2024-12-18T01:43:42.9507758Z     return self.inference(batched_inputs)
2024-12-18T01:43:42.9508792Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2024-12-18T01:43:42.9509602Z     results, _ = self.roi_heads(images, features, proposals, None)
2024-12-18T01:43:42.9510387Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:43:42.9511103Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:43:42.9511768Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:43:42.9512411Z     return forward_call(*args, **kwargs)
2024-12-18T01:43:42.9513093Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2024-12-18T01:43:42.9513890Z     pred_instances = self._forward_box(features, proposals)
2024-12-18T01:43:42.9514685Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2024-12-18T01:43:42.9515575Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2024-12-18T01:43:42.9516420Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:43:42.9517134Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:43:42.9517800Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:43:42.9518470Z     return forward_call(*args, **kwargs)
2024-12-18T01:43:42.9519074Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2024-12-18T01:43:42.9519847Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2024-12-18T01:43:42.9520651Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:43:42.9521378Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:43:42.9522215Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:43:42.9522912Z     return forward_call(*args, **kwargs)
2024-12-18T01:43:42.9523563Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2024-12-18T01:43:42.9524165Z     return roi_align(
2024-12-18T01:43:42.9524782Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-12-18T01:43:42.9525717Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-12-18T01:43:42.9526595Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-12-18T01:43:42.9527383Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-12-18T01:43:42.9528289Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-12-18T01:43:42.9528967Z     v1 = masked_index(y_low, x_low)
2024-12-18T01:43:42.9529563Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-12-18T01:43:42.9530541Z     return input[
2024-12-18T01:43:42.9532559Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 19183.01 GiB. GPU 0 has a total capacity of 21.98 GiB of which 17.75 GiB is free. Process 229083 has 4.22 GiB memory in use. Of the allocated memory 3.80 GiB is allocated by PyTorch, and 122.09 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-12-18T01:43:42.9534781Z 
2024-12-18T01:43:42.9535004Z The above exception was the direct cause of the following exception:
2024-12-18T01:43:42.9535325Z 
2024-12-18T01:43:42.9535440Z Traceback (most recent call last):
2024-12-18T01:43:42.9535922Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run
2024-12-18T01:43:42.9536417Z     ) = runner.load_model(
2024-12-18T01:43:42.9536902Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 372, in load_model
2024-12-18T01:43:42.9537460Z     self.validate_model(model, example_inputs)
2024-12-18T01:43:42.9538013Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2734, in validate_model
2024-12-18T01:43:42.9538566Z     raise RuntimeError("Eager run failed") from e
2024-12-18T01:43:42.9538909Z RuntimeError: Eager run failed
2024-12-18T01:43:42.9539101Z 
2024-12-18T01:43:42.9539191Z eager_fail_to_run
2024-12-18T01:43:47.1367829Z 
2024-12-18T01:43:50.1912444Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:43:50.1912777Z loading model: 0it [00:03, ?it/s]
2024-12-18T01:43:50.1913105Z cuda eval  detectron2_maskrcnn_r_50_c4        
2024-12-18T01:43:50.1927755Z Traceback (most recent call last):
2024-12-18T01:43:50.1928378Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2732, in validate_model
2024-12-18T01:43:50.1929002Z     self.model_iter_fn(model, example_inputs)
2024-12-18T01:43:50.1929556Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 444, in forward_pass
2024-12-18T01:43:50.1930536Z     return mod(*inputs)
2024-12-18T01:43:50.1931172Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:43:50.1931844Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:43:50.1932449Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:43:50.1933068Z     return forward_call(*args, **kwargs)
2024-12-18T01:43:50.1933710Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2024-12-18T01:43:50.1934634Z     return self.inference(batched_inputs)
2024-12-18T01:43:50.1935304Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2024-12-18T01:43:50.1936054Z     results, _ = self.roi_heads(images, features, proposals, None)
2024-12-18T01:43:50.1936784Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:43:50.1937438Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:43:50.1938045Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:43:50.1938702Z     return forward_call(*args, **kwargs)
2024-12-18T01:43:50.1939359Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2024-12-18T01:43:50.1940039Z     box_features = self._shared_roi_transform(
2024-12-18T01:43:50.1940778Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2024-12-18T01:43:50.1941498Z     x = self.pooler(features, boxes)
2024-12-18T01:43:50.1942187Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:43:50.1942838Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:43:50.1943444Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:43:50.1944052Z     return forward_call(*args, **kwargs)
2024-12-18T01:43:50.1944660Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2024-12-18T01:43:50.1945315Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2024-12-18T01:43:50.1946004Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:43:50.1947522Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:43:50.1948123Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:43:50.1948733Z     return forward_call(*args, **kwargs)
2024-12-18T01:43:50.1949322Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2024-12-18T01:43:50.1949907Z     return roi_align(
2024-12-18T01:43:50.1950453Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-12-18T01:43:50.1951275Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-12-18T01:43:50.1952086Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-12-18T01:43:50.1952892Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-12-18T01:43:50.1953725Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-12-18T01:43:50.1954388Z     v1 = masked_index(y_low, x_low)
2024-12-18T01:43:50.1954979Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-12-18T01:43:50.1955575Z     return input[
2024-12-18T01:43:50.1957590Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 27336.74 GiB. GPU 0 has a total capacity of 21.98 GiB of which 19.59 GiB is free. Process 229203 has 2.37 GiB memory in use. Of the allocated memory 1.86 GiB is allocated by PyTorch, and 222.09 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-12-18T01:43:50.1959608Z 
2024-12-18T01:43:50.1959906Z The above exception was the direct cause of the following exception:
2024-12-18T01:43:50.1960233Z 
2024-12-18T01:43:50.1960349Z Traceback (most recent call last):
2024-12-18T01:43:50.1960826Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run
2024-12-18T01:43:50.1961320Z     ) = runner.load_model(
2024-12-18T01:43:50.1961805Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 372, in load_model
2024-12-18T01:43:50.1962376Z     self.validate_model(model, example_inputs)
2024-12-18T01:43:50.1962926Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2734, in validate_model
2024-12-18T01:43:50.1963492Z     raise RuntimeError("Eager run failed") from e
2024-12-18T01:43:50.1963844Z RuntimeError: Eager run failed
2024-12-18T01:43:50.1964033Z 
2024-12-18T01:43:50.1964135Z eager_fail_to_run
2024-12-18T01:43:54.3690482Z 
2024-12-18T01:43:57.3035871Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:43:57.3036386Z loading model: 0it [00:02, ?it/s]
2024-12-18T01:43:57.3036904Z cuda eval  detectron2_maskrcnn_r_50_fpn       
2024-12-18T01:43:57.3051863Z Traceback (most recent call last):
2024-12-18T01:43:57.3052711Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2732, in validate_model
2024-12-18T01:43:57.3053520Z     self.model_iter_fn(model, example_inputs)
2024-12-18T01:43:57.3054334Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 444, in forward_pass
2024-12-18T01:43:57.3055132Z     return mod(*inputs)
2024-12-18T01:43:57.3055980Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:43:57.3057020Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:43:57.3057953Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:43:57.3059242Z     return forward_call(*args, **kwargs)
2024-12-18T01:43:57.3060304Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2024-12-18T01:43:57.3061306Z     return self.inference(batched_inputs)
2024-12-18T01:43:57.3062498Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2024-12-18T01:43:57.3063669Z     results, _ = self.roi_heads(images, features, proposals, None)
2024-12-18T01:43:57.3064728Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:43:57.3065673Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:43:57.3066583Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:43:57.3067522Z     return forward_call(*args, **kwargs)
2024-12-18T01:43:57.3068535Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2024-12-18T01:43:57.3069671Z     pred_instances = self._forward_box(features, proposals)
2024-12-18T01:43:57.3070790Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2024-12-18T01:43:57.3072049Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2024-12-18T01:43:57.3073275Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:43:57.3074285Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:43:57.3075118Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:43:57.3075989Z     return forward_call(*args, **kwargs)
2024-12-18T01:43:57.3076921Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2024-12-18T01:43:57.3078040Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2024-12-18T01:43:57.3079437Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:43:57.3080526Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:43:57.3081513Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:43:57.3082518Z     return forward_call(*args, **kwargs)
2024-12-18T01:43:57.3083451Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2024-12-18T01:43:57.3084431Z     return roi_align(
2024-12-18T01:43:57.3085373Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-12-18T01:43:57.3086757Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-12-18T01:43:57.3088166Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-12-18T01:43:57.3089518Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-12-18T01:43:57.3090959Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-12-18T01:43:57.3092094Z     v1 = masked_index(y_low, x_low)
2024-12-18T01:43:57.3093122Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-12-18T01:43:57.3094137Z     return input[
2024-12-18T01:43:57.3097517Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20915.45 GiB. GPU 0 has a total capacity of 21.98 GiB of which 18.88 GiB is free. Process 229265 has 3.09 GiB memory in use. Of the allocated memory 2.65 GiB is allocated by PyTorch, and 147.05 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-12-18T01:43:57.3100955Z 
2024-12-18T01:43:57.3101296Z The above exception was the direct cause of the following exception:
2024-12-18T01:43:57.3101790Z 
2024-12-18T01:43:57.3102076Z Traceback (most recent call last):
2024-12-18T01:43:57.3102804Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run
2024-12-18T01:43:57.3103589Z     ) = runner.load_model(
2024-12-18T01:43:57.3104363Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 372, in load_model
2024-12-18T01:43:57.3105257Z     self.validate_model(model, example_inputs)
2024-12-18T01:43:57.3106102Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2734, in validate_model
2024-12-18T01:43:57.3107005Z     raise RuntimeError("Eager run failed") from e
2024-12-18T01:43:57.3107594Z RuntimeError: Eager run failed
2024-12-18T01:43:57.3107913Z 
2024-12-18T01:43:57.3108066Z eager_fail_to_run
2024-12-18T01:44:01.5824382Z 
2024-12-18T01:44:10.5140591Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:44:10.5141044Z loading model: 0it [00:08, ?it/s]
2024-12-18T01:44:10.5146434Z cuda eval  dlrm                               
2024-12-18T01:44:15.8510503Z pass
2024-12-18T01:44:15.8602298Z TIMING: _recursive_pre_grad_passes:0.00589 _recursive_joint_graph_passes:1.47488 _recursive_post_grad_passes:0.05911 async_compile.wait:0.19521 code_gen:0.40225 inductor_compile:0.84118 backend_compile:3.3253 entire_frame_compile:3.89206 total_wall_time:3.89206
2024-12-18T01:44:15.8603794Z STATS: call_* op count: 72 | FakeTensorMode.__torch_dispatch__:2597 | ProxyTorchDispatchMode.__torch_dispatch__:935 | FakeTensor.__torch_dispatch__:165
2024-12-18T01:44:15.8604622Z Dynamo produced 1 graphs covering 72 ops with 0 graph breaks (0 unique)
2024-12-18T01:44:20.2416499Z 
2024-12-18T01:44:21.8040605Z loading model: 0it [00:00, ?it/s]Downloading https://doctr-static.mindee.com/models?id=v0.7.0/db_resnet50-79bd7d70.pt&src=0 to /var/lib/jenkins/.cache/doctr/models/db_resnet50-79bd7d70.pt
2024-12-18T01:44:22.2465287Z 
2024-12-18T01:44:22.2465473Z 
2024-12-18T01:44:22.3466771Z   0% 0/102021912 [00:00<?, ?it/s]
2024-12-18T01:44:22.3467007Z 
2024-12-18T01:44:22.4514807Z  24% 24767488/102021912 [00:00<00:00, 247673935.20it/s]
2024-12-18T01:44:22.4515069Z 
2024-12-18T01:44:22.5515218Z  49% 49534976/102021912 [00:00<00:00, 240807700.75it/s]
2024-12-18T01:44:22.5515482Z 
2024-12-18T01:44:22.7776761Z  75% 76843008/102021912 [00:00<00:00, 255313278.90it/s]
2024-12-18T01:44:22.7777168Z 102022144it [00:00, 192190468.83it/s]                  
2024-12-18T01:44:24.4725426Z Downloading https://doctr-static.mindee.com/models?id=v0.3.1/crnn_vgg16_bn-9762b0b0.pt&src=0 to /var/lib/jenkins/.cache/doctr/models/crnn_vgg16_bn-9762b0b0.pt
2024-12-18T01:44:24.7969995Z 
2024-12-18T01:44:24.7970034Z 
2024-12-18T01:44:24.8970963Z   0% 0/63286381 [00:00<?, ?it/s]
2024-12-18T01:44:24.8971180Z 
2024-12-18T01:44:25.0120258Z  45% 28620800/63286381 [00:00<00:00, 286200084.69it/s]
2024-12-18T01:44:25.0120515Z 
2024-12-18T01:44:25.0339297Z  90% 57241600/63286381 [00:00<00:00, 263143646.41it/s]
2024-12-18T01:44:25.0339693Z 63287296it [00:00, 267404360.90it/s]                  
2024-12-18T01:44:25.6276382Z WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2024-12-18T01:44:26.1332434Z 
2024-12-18T01:44:26.1332785Z loading model: 0it [00:05, ?it/s]
2024-12-18T01:44:26.1333297Z WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2024-12-18T01:44:26.1333832Z cuda eval  doctr_det_predictor                
2024-12-18T01:44:34.3211436Z WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2024-12-18T01:44:54.5876802Z W1218 01:44:54.586000 41732 site-packages/torch/_inductor/utils.py:1543] [2/0_1] DeviceCopy in input program
2024-12-18T01:44:59.3065480Z skipping cudagraphs due to skipping cudagraphs due to cpu device (device_put). Found from : 
2024-12-18T01:44:59.3067947Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 211, in forward
2024-12-18T01:44:59.3069071Z     for preds in self.postprocessor(prob_map.detach().cpu().permute((0, 2, 3, 1)).numpy())
2024-12-18T01:44:59.3069462Z 
2024-12-18T01:45:05.3382968Z W1218 01:45:05.337000 41732 site-packages/torch/_functorch/_aot_autograd/autograd_cache.py:856] [6/0_1] Bypassing autograd cache due to: Cannot cache a graph with functional tensor
2024-12-18T01:45:05.5907665Z pass
2024-12-18T01:45:05.6117766Z TIMING: entire_frame_compile:29.12686 _recursive_pre_grad_passes:0.01638 _recursive_joint_graph_passes:0.23853 inductor_compile:11.37075 backend_compile:16.7185 _recursive_post_grad_passes:0.30642 async_compile.wait:2.62793 code_gen:8.40801 total_wall_time:29.12686
2024-12-18T01:45:05.6119307Z STATS: call_* op count: 234 | FakeTensorMode.__torch_dispatch__:20861 | ProxyTorchDispatchMode.__torch_dispatch__:7814 | FakeTensor.__torch_dispatch__:3034
2024-12-18T01:45:05.6120150Z Dynamo produced 6 graphs covering 234 ops with 5 graph breaks (2 unique)
2024-12-18T01:45:10.8812223Z 
2024-12-18T01:45:14.0990212Z loading model: 0it [00:00, ?it/s]WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2024-12-18T01:45:14.4768823Z 
2024-12-18T01:45:14.4769593Z loading model: 0it [00:03, ?it/s]
2024-12-18T01:45:14.4770664Z WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2024-12-18T01:45:14.4771781Z cuda eval  doctr_reco_predictor               
2024-12-18T01:45:14.6421353Z WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2024-12-18T01:45:21.9729247Z W1218 01:45:21.971000 42166 site-packages/torch/fx/experimental/symbolic_shapes.py:6307] [3/0] failed during evaluate_expr(Ne(u0, 123), hint=None, size_oblivious=False, forcing_spec=False
2024-12-18T01:45:21.9731333Z E1218 01:45:21.972000 42166 site-packages/torch/fx/experimental/recording.py:299] [3/0] failed while running evaluate_expr(*(Ne(u0, 123), None), **{'fx_node': ne})
2024-12-18T01:45:22.0450975Z W1218 01:45:22.044000 42166 site-packages/torch/fx/experimental/symbolic_shapes.py:6307] [4/0] failed during evaluate_expr(Ne(u0, 123), hint=None, size_oblivious=False, forcing_spec=False
2024-12-18T01:45:22.0452424Z E1218 01:45:22.044000 42166 site-packages/torch/fx/experimental/recording.py:299] [4/0] failed while running evaluate_expr(*(Ne(u0, 123), None), **{'fx_node': ne})
2024-12-18T01:45:22.8558675Z pass
2024-12-18T01:45:22.8568691Z TIMING: entire_frame_compile:6.39079 _recursive_pre_grad_passes:0.00671 _recursive_joint_graph_passes:0.1444 inductor_compile:2.08643 backend_compile:3.9115 _recursive_post_grad_passes:0.0695 async_compile.wait:0.51055 code_gen:1.36183 total_wall_time:6.39079
2024-12-18T01:45:22.8570248Z STATS: call_* op count: 58 | FakeTensorMode.__torch_dispatch__:5039 | ProxyTorchDispatchMode.__torch_dispatch__:1547 | FakeTensor.__torch_dispatch__:794
2024-12-18T01:45:22.8571082Z Dynamo produced 3 graphs covering 58 ops with 4 graph breaks (4 unique)
2024-12-18T01:45:27.5188846Z 
2024-12-18T01:45:28.9367475Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:45:28.9367839Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:45:28.9372589Z cuda eval  drq                                
2024-12-18T01:45:33.6204367Z pass
2024-12-18T01:45:33.6206619Z TIMING: _recursive_pre_grad_passes:0.00494 _recursive_joint_graph_passes:0.82523 _recursive_post_grad_passes:0.0429 async_compile.wait:0.59019 code_gen:0.90902 inductor_compile:1.13606 backend_compile:2.82299 entire_frame_compile:3.32764 total_wall_time:3.32764
2024-12-18T01:45:33.6208155Z STATS: call_* op count: 56 | FakeTensorMode.__torch_dispatch__:1567 | FakeTensor.__torch_dispatch__:137 | ProxyTorchDispatchMode.__torch_dispatch__:525
2024-12-18T01:45:33.6209339Z Dynamo produced 1 graphs covering 56 ops with 0 graph breaks (0 unique)
2024-12-18T01:45:38.0969533Z 
2024-12-18T01:45:40.0296537Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:45:40.0297107Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:45:40.0351987Z cuda eval  fastNLP_Bert                       
2024-12-18T01:45:43.0982324Z W1218 01:45:43.097000 42582 site-packages/torch/_dynamo/variables/tensor.py:869] [0/0] Graph break from `Tensor.item()`, consider setting:
2024-12-18T01:45:43.0983503Z W1218 01:45:43.097000 42582 site-packages/torch/_dynamo/variables/tensor.py:869] [0/0]     torch._dynamo.config.capture_scalar_outputs = True
2024-12-18T01:45:43.0984473Z W1218 01:45:43.097000 42582 site-packages/torch/_dynamo/variables/tensor.py:869] [0/0] or:
2024-12-18T01:45:43.0985493Z W1218 01:45:43.097000 42582 site-packages/torch/_dynamo/variables/tensor.py:869] [0/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2024-12-18T01:45:43.0986451Z W1218 01:45:43.097000 42582 site-packages/torch/_dynamo/variables/tensor.py:869] [0/0] to include these operations in the captured graph.
2024-12-18T01:45:43.0987273Z W1218 01:45:43.097000 42582 site-packages/torch/_dynamo/variables/tensor.py:869] [0/0] 
2024-12-18T01:45:43.0988005Z W1218 01:45:43.097000 42582 site-packages/torch/_dynamo/variables/tensor.py:869] [0/0] Graph break: from user code at:
2024-12-18T01:45:43.0989095Z W1218 01:45:43.097000 42582 site-packages/torch/_dynamo/variables/tensor.py:869] [0/0]   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2770, in run_n_iterations
2024-12-18T01:45:43.0990278Z W1218 01:45:43.097000 42582 site-packages/torch/_dynamo/variables/tensor.py:869] [0/0]     self.model_iter_fn(mod, inputs, collect_outputs=False)
2024-12-18T01:45:43.0991445Z W1218 01:45:43.097000 42582 site-packages/torch/_dynamo/variables/tensor.py:869] [0/0]   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 444, in forward_pass
2024-12-18T01:45:43.0992502Z W1218 01:45:43.097000 42582 site-packages/torch/_dynamo/variables/tensor.py:869] [0/0]     return mod(*inputs)
2024-12-18T01:45:43.0993970Z W1218 01:45:43.097000 42582 site-packages/torch/_dynamo/variables/tensor.py:869] [0/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/models/bert.py", line 265, in forward
2024-12-18T01:45:43.0995141Z W1218 01:45:43.097000 42582 site-packages/torch/_dynamo/variables/tensor.py:869] [0/0]     sequence_output = self.bert(words)
2024-12-18T01:45:43.0996364Z W1218 01:45:43.097000 42582 site-packages/torch/_dynamo/variables/tensor.py:869] [0/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/embeddings/bert_embedding.py", line 137, in forward
2024-12-18T01:45:43.0997558Z W1218 01:45:43.097000 42582 site-packages/torch/_dynamo/variables/tensor.py:869] [0/0]     outputs = self.model(words)
2024-12-18T01:45:43.0998755Z W1218 01:45:43.097000 42582 site-packages/torch/_dynamo/variables/tensor.py:869] [0/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/embeddings/bert_embedding.py", line 445, in forward
2024-12-18T01:45:43.1002884Z W1218 01:45:43.097000 42582 site-packages/torch/_dynamo/variables/tensor.py:869] [0/0]     max_word_piece_length = batch_word_pieces_length.sum(dim=-1).max().item()  # 表示word piece的长度(包括padding)
2024-12-18T01:45:43.1003905Z W1218 01:45:43.097000 42582 site-packages/torch/_dynamo/variables/tensor.py:869] [0/0] 
2024-12-18T01:45:43.1004543Z W1218 01:45:43.097000 42582 site-packages/torch/_dynamo/variables/tensor.py:869] [0/0] 
2024-12-18T01:45:57.5033888Z pass
2024-12-18T01:45:57.5047777Z TIMING: entire_frame_compile:13.24352 _recursive_pre_grad_passes:0.01625 _recursive_joint_graph_passes:1.95301 _recursive_post_grad_passes:0.38912 async_compile.precompile:0.21102 async_compile.wait:0.36456 code_gen:1.8672 inductor_compile:3.92288 backend_compile:9.45799 total_wall_time:13.24352
2024-12-18T01:45:57.5049401Z STATS: call_* op count: 440 | FakeTensorMode.__torch_dispatch__:16463 | ProxyTorchDispatchMode.__torch_dispatch__:6925 | FakeTensor.__torch_dispatch__:1975
2024-12-18T01:45:57.5050532Z Dynamo produced 4 graphs covering 440 ops with 4 graph breaks (1 unique)
2024-12-18T01:46:02.2932180Z 
2024-12-18T01:46:03.2764540Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:46:03.2764884Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:46:03.2782357Z cuda eval  functorch_dp_cifar10               
2024-12-18T01:46:11.4308108Z pass
2024-12-18T01:46:11.4320170Z TIMING: _recursive_pre_grad_passes:0.00748 _recursive_joint_graph_passes:0.58237 _recursive_post_grad_passes:0.0801 async_compile.wait:0.33142 code_gen:2.01905 inductor_compile:3.02326 backend_compile:5.46994 entire_frame_compile:6.64378 total_wall_time:6.64378
2024-12-18T01:46:11.4321824Z STATS: call_* op count: 138 | FakeTensorMode.__torch_dispatch__:4942 | ProxyTorchDispatchMode.__torch_dispatch__:2467 | FakeTensor.__torch_dispatch__:866
2024-12-18T01:46:11.4322753Z Dynamo produced 1 graphs covering 138 ops with 0 graph breaks (0 unique)
2024-12-18T01:46:15.9826028Z 
2024-12-18T01:46:16.6717082Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:46:16.6717422Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:46:16.6723338Z cuda eval  functorch_maml_omniglot            
2024-12-18T01:46:20.8886046Z pass
2024-12-18T01:46:20.8888356Z TIMING: _recursive_pre_grad_passes:0.00415 _recursive_joint_graph_passes:0.35197 _recursive_post_grad_passes:0.17163 async_compile.wait:0.29709 code_gen:0.77622 inductor_compile:1.2094 backend_compile:2.47175 entire_frame_compile:2.83643 total_wall_time:2.83643
2024-12-18T01:46:20.8889867Z STATS: call_* op count: 28 | FakeTensorMode.__torch_dispatch__:1459 | ProxyTorchDispatchMode.__torch_dispatch__:477 | FakeTensor.__torch_dispatch__:137
2024-12-18T01:46:20.8890703Z Dynamo produced 1 graphs covering 28 ops with 0 graph breaks (0 unique)
2024-12-18T01:46:25.3131965Z 
2024-12-18T01:46:26.6296282Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:46:26.6297047Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:46:26.6303081Z cuda eval  hf_Albert                          
2024-12-18T01:46:41.4027390Z pass
2024-12-18T01:46:41.4169997Z TIMING: _recursive_pre_grad_passes:0.02184 _recursive_joint_graph_passes:0.56854 _recursive_post_grad_passes:0.39469 async_compile.wait:0.21324 code_gen:1.53281 inductor_compile:3.57551 backend_compile:8.72821 entire_frame_compile:11.22191 total_wall_time:11.22191
2024-12-18T01:46:41.4171537Z STATS: call_* op count: 872 | FakeTensorMode.__torch_dispatch__:23989 | FakeTensor.__torch_dispatch__:1721 | ProxyTorchDispatchMode.__torch_dispatch__:11505
2024-12-18T01:46:41.4172360Z Dynamo produced 1 graphs covering 872 ops with 0 graph breaks (0 unique)
2024-12-18T01:46:46.1577144Z 
2024-12-18T01:46:50.0533916Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:46:50.0534248Z loading model: 0it [00:03, ?it/s]
2024-12-18T01:46:50.0592192Z cuda eval  hf_Bart                            
2024-12-18T01:47:11.3761453Z pass
2024-12-18T01:47:11.4024123Z TIMING: _recursive_pre_grad_passes:0.02524 _recursive_joint_graph_passes:1.03985 _recursive_post_grad_passes:0.84337 async_compile.wait:0.34958 code_gen:2.56663 inductor_compile:5.46111 backend_compile:12.10624 entire_frame_compile:16.61333 total_wall_time:16.61333
2024-12-18T01:47:11.4025644Z STATS: call_* op count: 902 | FakeTensorMode.__torch_dispatch__:28137 | FakeTensor.__torch_dispatch__:2652 | ProxyTorchDispatchMode.__torch_dispatch__:13342
2024-12-18T01:47:11.4026475Z Dynamo produced 1 graphs covering 902 ops with 0 graph breaks (0 unique)
2024-12-18T01:47:16.3891777Z 
2024-12-18T01:47:19.0201006Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:47:19.0201353Z loading model: 0it [00:02, ?it/s]
2024-12-18T01:47:19.0252054Z cuda eval  hf_Bert                            
2024-12-18T01:47:37.5828103Z pass
2024-12-18T01:47:37.5972601Z TIMING: _recursive_pre_grad_passes:0.02141 _recursive_joint_graph_passes:1.06544 _recursive_post_grad_passes:0.39802 async_compile.wait:0.21553 code_gen:1.33562 inductor_compile:3.30662 backend_compile:9.61711 entire_frame_compile:14.6069 total_wall_time:14.6069
2024-12-18T01:47:37.5974468Z STATS: call_* op count: 734 | FakeTensorMode.__torch_dispatch__:24468 | FakeTensor.__torch_dispatch__:2022 | ProxyTorchDispatchMode.__torch_dispatch__:11783
2024-12-18T01:47:37.5975677Z Dynamo produced 1 graphs covering 734 ops with 0 graph breaks (0 unique)
2024-12-18T01:47:42.4537683Z 
2024-12-18T01:47:47.3932714Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:47:47.3933584Z loading model: 0it [00:04, ?it/s]
2024-12-18T01:47:47.4025395Z cuda eval  hf_Bert_large                      
2024-12-18T01:48:22.4823861Z pass
2024-12-18T01:48:22.5020990Z TIMING: _recursive_pre_grad_passes:0.03876 _recursive_joint_graph_passes:1.28877 _recursive_post_grad_passes:0.76152 async_compile.wait:0.21327 code_gen:2.60293 inductor_compile:6.29045 backend_compile:17.92221 entire_frame_compile:27.24637 total_wall_time:27.24637
2024-12-18T01:48:22.5023119Z STATS: call_* op count: 1430 | FakeTensorMode.__torch_dispatch__:47412 | FakeTensor.__torch_dispatch__:3918 | ProxyTorchDispatchMode.__torch_dispatch__:23075
2024-12-18T01:48:22.5024070Z Dynamo produced 1 graphs covering 1430 ops with 0 graph breaks (0 unique)
2024-12-18T01:48:27.8489923Z 
2024-12-18T01:48:31.0865474Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:48:31.0865917Z loading model: 0it [00:03, ?it/s]
2024-12-18T01:48:31.0914911Z cuda eval  hf_BigBird                         
2024-12-18T01:49:39.2121031Z W1218 01:49:39.211000 43651 site-packages/torch/_inductor/utils.py:1543] [0/0] DeviceCopy in input program
2024-12-18T01:49:39.7595031Z W1218 01:49:39.758000 43651 site-packages/torch/_inductor/utils.py:1543] [0/0] DeviceCopy in input program
2024-12-18T01:49:40.0511794Z W1218 01:49:40.050000 43651 site-packages/torch/_inductor/utils.py:1543] [0/0] DeviceCopy in input program
2024-12-18T01:49:40.3266821Z W1218 01:49:40.325000 43651 site-packages/torch/_inductor/utils.py:1543] [0/0] DeviceCopy in input program
2024-12-18T01:49:40.6005815Z W1218 01:49:40.599000 43651 site-packages/torch/_inductor/utils.py:1543] [0/0] DeviceCopy in input program
2024-12-18T01:49:40.8747531Z W1218 01:49:40.874000 43651 site-packages/torch/_inductor/utils.py:1543] [0/0] DeviceCopy in input program
2024-12-18T01:49:41.1493337Z W1218 01:49:41.148000 43651 site-packages/torch/_inductor/utils.py:1543] [0/0] DeviceCopy in input program
2024-12-18T01:49:41.4236688Z W1218 01:49:41.422000 43651 site-packages/torch/_inductor/utils.py:1543] [0/0] DeviceCopy in input program
2024-12-18T01:49:42.2132489Z W1218 01:49:42.212000 43651 site-packages/torch/_inductor/utils.py:1543] [0/0] DeviceCopy in input program
2024-12-18T01:49:42.4893399Z W1218 01:49:42.488000 43651 site-packages/torch/_inductor/utils.py:1543] [0/0] DeviceCopy in input program
2024-12-18T01:49:42.7665492Z W1218 01:49:42.765000 43651 site-packages/torch/_inductor/utils.py:1543] [0/0] DeviceCopy in input program
2024-12-18T01:49:43.0445913Z W1218 01:49:43.043000 43651 site-packages/torch/_inductor/utils.py:1543] [0/0] DeviceCopy in input program
2024-12-18T01:50:03.6403913Z skipping cudagraphs due to skipping cudagraphs due to cpu device (cat_170). Found from : 
2024-12-18T01:50:03.6406739Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2771, in run_n_iterations
2024-12-18T01:50:03.6407698Z     return self.model_iter_fn(mod, inputs, collect_outputs=True)
2024-12-18T01:50:03.6408622Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 444, in forward_pass
2024-12-18T01:50:03.6409464Z     return mod(*inputs)
2024-12-18T01:50:03.6410463Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2450, in forward
2024-12-18T01:50:03.6411514Z     outputs = self.bert(
2024-12-18T01:50:03.6412457Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2133, in forward
2024-12-18T01:50:03.6413428Z     encoder_outputs = self.encoder(
2024-12-18T01:50:03.6414373Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1631, in forward
2024-12-18T01:50:03.6415861Z     layer_outputs = layer_module(
2024-12-18T01:50:03.6416811Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1488, in forward
2024-12-18T01:50:03.6417819Z     self_attention_outputs = self.attention(
2024-12-18T01:50:03.6418807Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1401, in forward
2024-12-18T01:50:03.6419809Z     self_outputs = self.self(
2024-12-18T01:50:03.6420752Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 472, in forward
2024-12-18T01:50:03.6421908Z     context_layer, attention_probs = self.bigbird_block_sparse_attention(
2024-12-18T01:50:03.6423299Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 593, in bigbird_block_sparse_attention
2024-12-18T01:50:03.6424432Z     rand_attn = np.stack(rand_attn, axis=0)
2024-12-18T01:50:03.6424768Z 
2024-12-18T01:50:05.4722832Z fail_accuracy
2024-12-18T01:50:05.4905810Z TIMING: _recursive_pre_grad_passes:0.1771 _recursive_joint_graph_passes:3.02027 _recursive_post_grad_passes:1.86842 async_compile.wait:2.18159 code_gen:19.65632 inductor_compile:29.62498 backend_compile:57.1109 entire_frame_compile:69.72271 total_wall_time:69.72271
2024-12-18T01:50:05.4907642Z STATS: call_* op count: 6694 | FakeTensorMode.__torch_dispatch__:147926 | FakeTensor.__torch_dispatch__:7294 | ProxyTorchDispatchMode.__torch_dispatch__:59669
2024-12-18T01:50:05.4908487Z Dynamo produced 1 graphs covering 6694 ops with 0 graph breaks (0 unique)
2024-12-18T01:50:12.8693578Z 
2024-12-18T01:50:14.9468683Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:50:14.9469335Z loading model: 0it [00:02, ?it/s]
2024-12-18T01:50:14.9492841Z cuda eval  hf_DistilBert                      
2024-12-18T01:50:26.6184802Z pass
2024-12-18T01:50:26.6329667Z TIMING: _recursive_pre_grad_passes:0.01302 _recursive_joint_graph_passes:0.37705 _recursive_post_grad_passes:0.4752 async_compile.wait:0.40071 code_gen:1.47417 inductor_compile:3.08494 backend_compile:6.58599 entire_frame_compile:8.77953 total_wall_time:8.77953
2024-12-18T01:50:26.6331422Z STATS: call_* op count: 406 | FakeTensorMode.__torch_dispatch__:13806 | FakeTensor.__torch_dispatch__:1267 | ProxyTorchDispatchMode.__torch_dispatch__:6338
2024-12-18T01:50:26.6332290Z Dynamo produced 1 graphs covering 406 ops with 0 graph breaks (0 unique)
2024-12-18T01:50:31.3013798Z 
2024-12-18T01:50:34.5837902Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:50:34.5838237Z loading model: 0it [00:03, ?it/s]
2024-12-18T01:50:34.5875703Z cuda eval  hf_GPT2                            
2024-12-18T01:50:52.2091517Z pass
2024-12-18T01:50:52.2482302Z TIMING: _recursive_pre_grad_passes:0.03257 _recursive_joint_graph_passes:0.85427 _recursive_post_grad_passes:0.59908 async_compile.wait:0.36743 code_gen:1.59422 inductor_compile:3.63119 backend_compile:9.44328 entire_frame_compile:12.97905 total_wall_time:12.97905
2024-12-18T01:50:52.2485339Z STATS: call_* op count: 1244 | FakeTensorMode.__torch_dispatch__:23167 | FakeTensor.__torch_dispatch__:1808 | ProxyTorchDispatchMode.__torch_dispatch__:10793
2024-12-18T01:50:52.2487017Z Dynamo produced 1 graphs covering 1244 ops with 0 graph breaks (0 unique)
2024-12-18T01:50:57.1182738Z 
2024-12-18T01:51:09.8924849Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:51:09.8925209Z loading model: 0it [00:12, ?it/s]
2024-12-18T01:51:09.9037877Z cuda eval  hf_GPT2_large                      
2024-12-18T01:51:09.9274834Z pass_due_to_skip
2024-12-18T01:51:10.2596057Z TIMING: total_wall_time:0
2024-12-18T01:51:10.2596400Z STATS: call_* op count: 0
2024-12-18T01:51:10.2596800Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-12-18T01:51:14.3643996Z 
2024-12-18T01:51:15.6723460Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:51:15.6724074Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:51:15.6747286Z cuda eval  hf_Reformer                        
2024-12-18T01:51:23.3995787Z W1218 01:51:23.398000 44361 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0] Graph break from `Tensor.item()`, consider setting:
2024-12-18T01:51:23.3996837Z W1218 01:51:23.398000 44361 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0]     torch._dynamo.config.capture_scalar_outputs = True
2024-12-18T01:51:23.3997652Z W1218 01:51:23.398000 44361 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0] or:
2024-12-18T01:51:23.3998440Z W1218 01:51:23.398000 44361 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2024-12-18T01:51:23.3999382Z W1218 01:51:23.398000 44361 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0] to include these operations in the captured graph.
2024-12-18T01:51:23.4000169Z W1218 01:51:23.398000 44361 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0] 
2024-12-18T01:51:23.4000894Z W1218 01:51:23.398000 44361 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0] Graph break: from user code at:
2024-12-18T01:51:23.4002306Z W1218 01:51:23.398000 44361 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 2094, in torch_dynamo_resume_in_forward_at_2066
2024-12-18T01:51:23.4003748Z W1218 01:51:23.398000 44361 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0]     embedding_output = self.embeddings(
2024-12-18T01:51:23.4005010Z W1218 01:51:23.398000 44361 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 266, in forward
2024-12-18T01:51:23.4006374Z W1218 01:51:23.398000 44361 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0]     position_embeddings = self.position_embeddings(position_ids)
2024-12-18T01:51:23.4007732Z W1218 01:51:23.398000 44361 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 190, in forward
2024-12-18T01:51:23.4009379Z W1218 01:51:23.398000 44361 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0]     max_position_id = position_ids.max().item()
2024-12-18T01:51:23.4010169Z W1218 01:51:23.398000 44361 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0] 
2024-12-18T01:51:23.4010769Z W1218 01:51:23.398000 44361 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0] 
2024-12-18T01:51:24.3404301Z W1218 01:51:24.339000 44361 site-packages/torch/_functorch/_aot_autograd/autograd_cache.py:856] [7/0_1] Bypassing autograd cache due to: Cannot cache a graph with functional tensor
2024-12-18T01:51:33.0078209Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter.src Found from   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 2101, in torch_dynamo_resume_in_forward_at_2094
2024-12-18T01:51:33.0079529Z     encoder_outputs = self.encoder(
2024-12-18T01:51:33.0080245Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 1727, in forward
2024-12-18T01:51:33.0080989Z     hidden_states = _ReversibleFunction.apply(
2024-12-18T01:51:33.0081710Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 1615, in forward
2024-12-18T01:51:33.0082410Z     layer_outputs = layer(
2024-12-18T01:51:33.0083068Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 1480, in forward
2024-12-18T01:51:33.0083784Z     attn_outputs = self.attention(
2024-12-18T01:51:33.0084486Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 1313, in forward
2024-12-18T01:51:33.0085240Z     self_attention_outputs = self.self_attention(
2024-12-18T01:51:33.0086200Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 482, in forward
2024-12-18T01:51:33.0087112Z     sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx(
2024-12-18T01:51:33.0088167Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 702, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx
2024-12-18T01:51:33.0089116Z     undo_sorted_bucket_idx.scatter_(-1, sorted_bucket_idx, indices)
2024-12-18T01:51:33.0089421Z 
2024-12-18T01:51:33.0089426Z 
2024-12-18T01:51:35.0069485Z pass
2024-12-18T01:51:35.0131227Z TIMING: entire_frame_compile:15.24451 _recursive_pre_grad_passes:0.02107 _recursive_joint_graph_passes:0.39867 _recursive_post_grad_passes:0.22806 async_compile.wait:3.09794 code_gen:7.54375 inductor_compile:10.14871 backend_compile:12.70917 total_wall_time:15.24451
2024-12-18T01:51:35.0132838Z STATS: call_* op count: 482 | FakeTensorMode.__torch_dispatch__:12865 | ProxyTorchDispatchMode.__torch_dispatch__:4923 | FakeTensor.__torch_dispatch__:1353
2024-12-18T01:51:35.0133724Z Dynamo produced 7 graphs covering 482 ops with 5 graph breaks (2 unique)
2024-12-18T01:51:39.8373044Z 
2024-12-18T01:51:41.8854066Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:51:41.8854420Z loading model: 0it [00:02, ?it/s]
2024-12-18T01:51:41.8889609Z cuda eval  hf_T5                              
2024-12-18T01:52:13.6958454Z pass
2024-12-18T01:52:13.7498116Z TIMING: _recursive_pre_grad_passes:0.03866 _recursive_joint_graph_passes:0.66625 _recursive_post_grad_passes:0.18551 async_compile.wait:0.74235 code_gen:3.24322 inductor_compile:5.5408 backend_compile:13.07838 entire_frame_compile:20.45813 total_wall_time:20.45813
2024-12-18T01:52:13.7501270Z STATS: call_* op count: 1654 | FakeTensorMode.__torch_dispatch__:36345 | ProxyTorchDispatchMode.__torch_dispatch__:16080 | FakeTensor.__torch_dispatch__:2344
2024-12-18T01:52:13.7503041Z Dynamo produced 1 graphs covering 1654 ops with 0 graph breaks (0 unique)
2024-12-18T01:52:18.8009150Z 
2024-12-18T01:52:22.8848007Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:52:22.8849409Z loading model: 0it [00:04, ?it/s]
2024-12-18T01:52:22.8850046Z cuda eval  hf_T5_base                         
2024-12-18T01:52:22.8861542Z Traceback (most recent call last):
2024-12-18T01:52:22.8862168Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2732, in validate_model
2024-12-18T01:52:22.8862740Z     self.model_iter_fn(model, example_inputs)
2024-12-18T01:52:22.8863300Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 444, in forward_pass
2024-12-18T01:52:22.8863841Z     return mod(*inputs)
2024-12-18T01:52:22.8864446Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:52:22.8865278Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:52:22.8866144Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:52:22.8866951Z     return forward_call(*args, **kwargs)
2024-12-18T01:52:22.8867807Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 69, in forward
2024-12-18T01:52:22.8868671Z     return self.model(input_ids=input_ids, decoder_input_ids=decoder_input_ids)
2024-12-18T01:52:22.8869458Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:52:22.8870126Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:52:22.8870733Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:52:22.8871351Z     return forward_call(*args, **kwargs)
2024-12-18T01:52:22.8872080Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 69, in forward
2024-12-18T01:52:22.8873206Z     return self.model(input_ids=input_ids, decoder_input_ids=decoder_input_ids)
2024-12-18T01:52:22.8873999Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:52:22.8874669Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:52:22.8875277Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:52:22.8875895Z     return forward_call(*args, **kwargs)
2024-12-18T01:52:22.8876544Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1748, in forward
2024-12-18T01:52:22.8877201Z     decoder_outputs = self.decoder(
2024-12-18T01:52:22.8877830Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:52:22.8878492Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:52:22.8879098Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:52:22.8879732Z     return forward_call(*args, **kwargs)
2024-12-18T01:52:22.8880373Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1115, in forward
2024-12-18T01:52:22.8881030Z     layer_outputs = layer_module(
2024-12-18T01:52:22.8881665Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:52:22.8882325Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:52:22.8882949Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:52:22.8883569Z     return forward_call(*args, **kwargs)
2024-12-18T01:52:22.8884210Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 695, in forward
2024-12-18T01:52:22.8884872Z     self_attention_outputs = self.layer[0](
2024-12-18T01:52:22.8885614Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:52:22.8886270Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:52:22.8886872Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:52:22.8887490Z     return forward_call(*args, **kwargs)
2024-12-18T01:52:22.8888125Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward
2024-12-18T01:52:22.8888794Z     attention_output = self.SelfAttention(
2024-12-18T01:52:22.8889448Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T01:52:22.8890108Z     return self._call_impl(*args, **kwargs)
2024-12-18T01:52:22.8890715Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T01:52:22.8891330Z     return forward_call(*args, **kwargs)
2024-12-18T01:52:22.8891971Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 562, in forward
2024-12-18T01:52:22.8892732Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(
2024-12-18T01:52:22.8894937Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 768.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 58.44 MiB is free. Process 233007 has 21.91 GiB memory in use. Of the allocated memory 20.94 GiB is allocated by PyTorch, and 666.49 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-12-18T01:52:22.8896955Z 
2024-12-18T01:52:22.8897168Z The above exception was the direct cause of the following exception:
2024-12-18T01:52:22.8897581Z 
2024-12-18T01:52:22.8897692Z Traceback (most recent call last):
2024-12-18T01:52:22.8898170Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run
2024-12-18T01:52:22.8898653Z     ) = runner.load_model(
2024-12-18T01:52:22.8899137Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 372, in load_model
2024-12-18T01:52:22.8899694Z     self.validate_model(model, example_inputs)
2024-12-18T01:52:22.8900244Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2734, in validate_model
2024-12-18T01:52:22.8900809Z     raise RuntimeError("Eager run failed") from e
2024-12-18T01:52:22.8901154Z RuntimeError: Eager run failed
2024-12-18T01:52:22.8901337Z 
2024-12-18T01:52:22.8901434Z eager_fail_to_run
2024-12-18T01:52:27.0292222Z 
2024-12-18T01:52:34.2603518Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:52:34.2603871Z loading model: 0it [00:07, ?it/s]
2024-12-18T01:52:34.2638772Z cuda eval  hf_T5_generate                     
2024-12-18T01:53:03.4677279Z W1218 01:53:03.466000 45033 site-packages/torch/_dynamo/variables/tensor.py:869] [4/0] Graph break from `Tensor.item()`, consider setting:
2024-12-18T01:53:03.4678320Z W1218 01:53:03.466000 45033 site-packages/torch/_dynamo/variables/tensor.py:869] [4/0]     torch._dynamo.config.capture_scalar_outputs = True
2024-12-18T01:53:03.4679150Z W1218 01:53:03.466000 45033 site-packages/torch/_dynamo/variables/tensor.py:869] [4/0] or:
2024-12-18T01:53:03.4679931Z W1218 01:53:03.466000 45033 site-packages/torch/_dynamo/variables/tensor.py:869] [4/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2024-12-18T01:53:03.4680867Z W1218 01:53:03.466000 45033 site-packages/torch/_dynamo/variables/tensor.py:869] [4/0] to include these operations in the captured graph.
2024-12-18T01:53:03.4681645Z W1218 01:53:03.466000 45033 site-packages/torch/_dynamo/variables/tensor.py:869] [4/0] 
2024-12-18T01:53:03.4682365Z W1218 01:53:03.466000 45033 site-packages/torch/_dynamo/variables/tensor.py:869] [4/0] Graph break: from user code at:
2024-12-18T01:53:03.4684078Z W1218 01:53:03.466000 45033 site-packages/torch/_dynamo/variables/tensor.py:869] [4/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/generation/utils.py", line 1392, in torch_dynamo_resume_in_generate_at_1348
2024-12-18T01:53:03.4685590Z W1218 01:53:03.466000 45033 site-packages/torch/_dynamo/variables/tensor.py:869] [4/0]     model_kwargs["attention_mask"] = self._prepare_attention_mask_for_generation(
2024-12-18T01:53:03.4687041Z W1218 01:53:03.466000 45033 site-packages/torch/_dynamo/variables/tensor.py:869] [4/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/generation/utils.py", line 476, in _prepare_attention_mask_for_generation
2024-12-18T01:53:03.4688485Z W1218 01:53:03.466000 45033 site-packages/torch/_dynamo/variables/tensor.py:869] [4/0]     is_pad_token_in_inputs = (pad_token_id is not None) and (pad_token_id in inputs)
2024-12-18T01:53:03.4689380Z W1218 01:53:03.466000 45033 site-packages/torch/_dynamo/variables/tensor.py:869] [4/0] 
2024-12-18T01:53:03.4689991Z W1218 01:53:03.466000 45033 site-packages/torch/_dynamo/variables/tensor.py:869] [4/0] 
2024-12-18T01:53:56.2216920Z CUDAGraph supports dynamic shapes by recording a new graph for each distinct input size. Recording too many CUDAGraphs may lead to extra overhead. We have observed 51 distinct sizes. Please consider the following options for better performance: a) padding inputs to a few fixed number of shapes; or b) set torch._inductor.config.triton.cudagraph_skip_dynamic_graphs=True. Set torch._inductor.config.triton.cudagraph_dynamic_shape_warn_limit=None to silence this warning.
2024-12-18T01:59:17.9731304Z pass
2024-12-18T01:59:17.9764486Z TIMING: entire_frame_compile:51.71776 _recursive_pre_grad_passes:0.04599 _recursive_joint_graph_passes:3.1015 _recursive_post_grad_passes:0.71262 async_compile.wait:1.8544 code_gen:6.85657 inductor_compile:14.58543 backend_compile:30.46595 total_wall_time:51.71776
2024-12-18T01:59:17.9767460Z STATS: call_* op count: 1815 | FakeTensorMode.__torch_dispatch__:50821 | ProxyTorchDispatchMode.__torch_dispatch__:17788 | FakeTensor.__torch_dispatch__:5206 | attempt fast:109 | fast is_contiguous:79 | slow both tensors nontrivially broadcast:12 | slow no contiguity match:18
2024-12-18T01:59:17.9768719Z Dynamo produced 4 graphs covering 1815 ops with 5 graph breaks (3 unique)
2024-12-18T01:59:26.1027430Z 
2024-12-18T01:59:36.0379125Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:59:36.0379801Z loading model: 0it [00:09, ?it/s]
2024-12-18T01:59:36.0695576Z cuda eval  hf_T5_large                        
2024-12-18T01:59:36.0933390Z pass_due_to_skip
2024-12-18T01:59:36.3789791Z TIMING: total_wall_time:0
2024-12-18T01:59:36.3790211Z STATS: call_* op count: 0
2024-12-18T01:59:36.3790739Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-12-18T01:59:40.4902719Z 
2024-12-18T01:59:41.6321419Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:59:41.6322102Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:59:41.6339025Z cuda eval  hf_Whisper                         
2024-12-18T01:59:51.6180480Z pass
2024-12-18T01:59:51.6198339Z TIMING: _recursive_pre_grad_passes:0.00877 _recursive_joint_graph_passes:0.6254 _recursive_post_grad_passes:0.20202 async_compile.wait:1.55012 code_gen:2.37721 inductor_compile:3.32633 backend_compile:5.89888 entire_frame_compile:7.1565 total_wall_time:7.1565
2024-12-18T01:59:51.6199807Z STATS: call_* op count: 230 | FakeTensorMode.__torch_dispatch__:7791 | FakeTensor.__torch_dispatch__:766 | ProxyTorchDispatchMode.__torch_dispatch__:3569
2024-12-18T01:59:51.6200625Z Dynamo produced 1 graphs covering 230 ops with 0 graph breaks (0 unique)
2024-12-18T01:59:56.2441802Z 
2024-12-18T02:00:00.8789811Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:00:00.8790209Z loading model: 0it [00:04, ?it/s]
2024-12-18T02:00:00.8880453Z cuda eval  hf_distil_whisper                  
2024-12-18T02:00:51.0406608Z pass
2024-12-18T02:00:51.0565516Z TIMING: _recursive_pre_grad_passes:0.03443 _recursive_joint_graph_passes:0.71621 _recursive_post_grad_passes:1.35855 async_compile.wait:1.51435 code_gen:4.68015 inductor_compile:9.20835 backend_compile:18.52895 entire_frame_compile:24.96706 total_wall_time:24.96706
2024-12-18T02:00:51.0567296Z STATS: call_* op count: 1270 | FakeTensorMode.__torch_dispatch__:40709 | FakeTensor.__torch_dispatch__:4026 | ProxyTorchDispatchMode.__torch_dispatch__:19669
2024-12-18T02:00:51.0568135Z Dynamo produced 1 graphs covering 1270 ops with 0 graph breaks (0 unique)
2024-12-18T02:00:56.2987980Z 
2024-12-18T02:00:56.9036668Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:00:56.9037109Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:00:56.9039056Z cuda eval  lennard_jones                      
2024-12-18T02:01:00.4035513Z pass
2024-12-18T02:01:00.4039454Z TIMING: _recursive_pre_grad_passes:0.00366 _recursive_joint_graph_passes:0.81763 _recursive_post_grad_passes:0.03991 async_compile.precompile:0.08799 async_compile.wait:0.00025 code_gen:0.15482 inductor_compile:0.43311 backend_compile:1.99298 entire_frame_compile:2.19921 total_wall_time:2.19921
2024-12-18T02:01:00.4041171Z STATS: call_* op count: 18 | FakeTensorMode.__torch_dispatch__:976 | ProxyTorchDispatchMode.__torch_dispatch__:339 | FakeTensor.__torch_dispatch__:54
2024-12-18T02:01:00.4041989Z Dynamo produced 1 graphs covering 18 ops with 0 graph breaks (0 unique)
2024-12-18T02:01:01.6504048Z accuracy            pass_rate=66.67%
2024-12-18T02:01:01.6507544Z calls_captured      gmean=0.00x mean=578.146x
2024-12-18T02:01:01.6510982Z unique_graphs       gmean=0.00x mean=2.188x
2024-12-18T02:01:01.6514402Z graph_breaks        gmean=0.00x mean=1.208x
2024-12-18T02:01:01.6517777Z unique_graph_breaks gmean=0.00x mean=0.396x
2024-12-18T02:01:01.6521247Z autograd_captures   gmean=0.00x mean=0.000x
2024-12-18T02:01:01.6524497Z autograd_compiles   gmean=0.00x mean=0.000x
2024-12-18T02:01:01.6527860Z cudagraph_skips     gmean=0.00x mean=0.354x
2024-12-18T02:01:02.8066995Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_inference.csv
2024-12-18T02:01:03.1500187Z torchrec_dlrm                       XFAIL
2024-12-18T02:01:03.1503630Z BERT_pytorch                        PASS
2024-12-18T02:01:03.1508104Z Background_Matting                  XFAIL
2024-12-18T02:01:03.1512613Z LearningToPaint                     PASS
2024-12-18T02:01:03.1517426Z Super_SloMo                         PASS
2024-12-18T02:01:03.1522338Z alexnet                             PASS
2024-12-18T02:01:03.1527073Z basic_gnn_edgecnn                   PASS
2024-12-18T02:01:03.1532321Z basic_gnn_gcn                       PASS
2024-12-18T02:01:03.1537252Z basic_gnn_gin                       PASS
2024-12-18T02:01:03.1542165Z basic_gnn_sage                      PASS
2024-12-18T02:01:03.1547870Z cm3leon_generate                    PASS
2024-12-18T02:01:03.1552165Z dcgan                               PASS
2024-12-18T02:01:03.1556947Z demucs                              PASS
2024-12-18T02:01:03.1561688Z densenet121                         PASS
2024-12-18T02:01:03.1566618Z detectron2_fasterrcnn_r_101_c4      XFAIL
2024-12-18T02:01:03.1571507Z detectron2_fasterrcnn_r_101_dc5     XFAIL
2024-12-18T02:01:03.1576234Z detectron2_fasterrcnn_r_101_fpn     XFAIL
2024-12-18T02:01:03.1580932Z detectron2_fasterrcnn_r_50_c4       XFAIL
2024-12-18T02:01:03.1586142Z detectron2_fasterrcnn_r_50_dc5      XFAIL
2024-12-18T02:01:03.1590916Z detectron2_fasterrcnn_r_50_fpn      XFAIL
2024-12-18T02:01:03.1595864Z detectron2_fcos_r_50_fpn            PASS
2024-12-18T02:01:03.1600524Z detectron2_maskrcnn_r_101_c4        XFAIL
2024-12-18T02:01:03.1605375Z detectron2_maskrcnn_r_101_fpn       XFAIL
2024-12-18T02:01:03.1610202Z detectron2_maskrcnn_r_50_c4         XFAIL
2024-12-18T02:01:03.1615039Z detectron2_maskrcnn_r_50_fpn        XFAIL
2024-12-18T02:01:03.1619967Z dlrm                                PASS
2024-12-18T02:01:03.1625099Z doctr_det_predictor                 PASS
2024-12-18T02:01:03.1629916Z doctr_reco_predictor                PASS
2024-12-18T02:01:03.1635361Z drq                                 PASS
2024-12-18T02:01:03.1639796Z fastNLP_Bert                        PASS
2024-12-18T02:01:03.1644352Z functorch_dp_cifar10                PASS
2024-12-18T02:01:03.1648856Z functorch_maml_omniglot             PASS
2024-12-18T02:01:03.1653514Z hf_Albert                           PASS
2024-12-18T02:01:03.1658136Z hf_Bart                             PASS
2024-12-18T02:01:03.1662843Z hf_Bert                             PASS
2024-12-18T02:01:03.1667730Z hf_Bert_large                       PASS
2024-12-18T02:01:03.1672433Z hf_BigBird                          XFAIL
2024-12-18T02:01:03.1677030Z hf_DistilBert                       PASS
2024-12-18T02:01:03.1681634Z hf_GPT2                             PASS
2024-12-18T02:01:03.1686345Z hf_GPT2_large                       XFAIL
2024-12-18T02:01:03.1691022Z hf_Reformer                         PASS
2024-12-18T02:01:03.1695716Z hf_T5                               PASS
2024-12-18T02:01:03.1700389Z hf_T5_base                          XFAIL
2024-12-18T02:01:03.1705207Z hf_T5_generate                      PASS
2024-12-18T02:01:03.1709829Z hf_T5_large                         XFAIL
2024-12-18T02:01:03.1714539Z hf_Whisper                          PASS
2024-12-18T02:01:03.1719246Z hf_distil_whisper                   PASS
2024-12-18T02:01:03.1723952Z lennard_jones                       PASS
2024-12-18T02:01:03.2251229Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_inference.csv
2024-12-18T02:01:03.5629635Z torchrec_dlrm                       PASS
2024-12-18T02:01:03.5633191Z BERT_pytorch                        PASS
2024-12-18T02:01:03.5637904Z Background_Matting                  PASS
2024-12-18T02:01:03.5642203Z LearningToPaint                     PASS
2024-12-18T02:01:03.5646707Z Super_SloMo                         PASS
2024-12-18T02:01:03.5650900Z alexnet                             PASS
2024-12-18T02:01:03.5655315Z basic_gnn_edgecnn                   PASS
2024-12-18T02:01:03.5659460Z basic_gnn_gcn                       PASS
2024-12-18T02:01:03.5664139Z basic_gnn_gin                       PASS
2024-12-18T02:01:03.5668448Z basic_gnn_sage                      PASS
2024-12-18T02:01:03.5672909Z cm3leon_generate                    PASS
2024-12-18T02:01:03.5677229Z dcgan                               PASS
2024-12-18T02:01:03.5681533Z demucs                              PASS
2024-12-18T02:01:03.5685921Z densenet121                         PASS
2024-12-18T02:01:03.5690291Z detectron2_fasterrcnn_r_101_c4      PASS
2024-12-18T02:01:03.5694747Z detectron2_fasterrcnn_r_101_dc5     PASS
2024-12-18T02:01:03.5699010Z detectron2_fasterrcnn_r_101_fpn     PASS
2024-12-18T02:01:03.5703580Z detectron2_fasterrcnn_r_50_c4       PASS
2024-12-18T02:01:03.5707919Z detectron2_fasterrcnn_r_50_dc5      PASS
2024-12-18T02:01:03.5712300Z detectron2_fasterrcnn_r_50_fpn      PASS
2024-12-18T02:01:03.5716645Z detectron2_fcos_r_50_fpn            PASS
2024-12-18T02:01:03.5721046Z detectron2_maskrcnn_r_101_c4        PASS
2024-12-18T02:01:03.5725358Z detectron2_maskrcnn_r_101_fpn       PASS
2024-12-18T02:01:03.5729803Z detectron2_maskrcnn_r_50_c4         PASS
2024-12-18T02:01:03.5735260Z detectron2_maskrcnn_r_50_fpn        PASS
2024-12-18T02:01:03.5739406Z dlrm                                PASS
2024-12-18T02:01:03.5743907Z doctr_det_predictor                 PASS
2024-12-18T02:01:03.5748332Z doctr_reco_predictor                PASS
2024-12-18T02:01:03.5752531Z drq                                 PASS
2024-12-18T02:01:03.5756884Z fastNLP_Bert                        PASS
2024-12-18T02:01:03.5761211Z functorch_dp_cifar10                PASS
2024-12-18T02:01:03.5765524Z functorch_maml_omniglot             PASS
2024-12-18T02:01:03.5769937Z hf_Albert                           PASS
2024-12-18T02:01:03.5774218Z hf_Bart                             PASS
2024-12-18T02:01:03.5778632Z hf_Bert                             PASS
2024-12-18T02:01:03.5783343Z hf_Bert_large                       PASS
2024-12-18T02:01:03.5787563Z hf_BigBird                          PASS
2024-12-18T02:01:03.5791826Z hf_DistilBert                       PASS
2024-12-18T02:01:03.5796194Z hf_GPT2                             PASS
2024-12-18T02:01:03.5800562Z hf_GPT2_large                       PASS
2024-12-18T02:01:03.5805011Z hf_Reformer                         PASS
2024-12-18T02:01:03.5809364Z hf_T5                               PASS
2024-12-18T02:01:03.5813736Z hf_T5_base                          PASS
2024-12-18T02:01:03.5818112Z hf_T5_generate                      PASS
2024-12-18T02:01:03.5822582Z hf_T5_large                         PASS
2024-12-18T02:01:03.5826906Z hf_Whisper                          PASS
2024-12-18T02:01:03.5831857Z hf_distil_whisper                   PASS
2024-12-18T02:01:03.5836127Z lennard_jones                       PASS
2024-12-18T02:01:03.6355790Z + test_single_dynamo_benchmark training torchbench 0 --training --amp
2024-12-18T02:01:03.6359167Z ++ pwd
2024-12-18T02:01:03.6363273Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2024-12-18T02:01:03.6363774Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports
2024-12-18T02:01:03.6384213Z + local name=training
2024-12-18T02:01:03.6384574Z + shift
2024-12-18T02:01:03.6385065Z + local suite=torchbench
2024-12-18T02:01:03.6385409Z + shift
2024-12-18T02:01:03.6385611Z + local shard_id=0
2024-12-18T02:01:03.6385843Z + shift
2024-12-18T02:01:03.6386329Z + partition_flags=()
2024-12-18T02:01:03.6386632Z + local partition_flags
2024-12-18T02:01:03.6386876Z + [[ -n 2 ]]
2024-12-18T02:01:03.6387119Z + [[ -n 0 ]]
2024-12-18T02:01:03.6387582Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id")
2024-12-18T02:01:03.6388102Z + [[ inductor_torchbench == *perf_compare* ]]
2024-12-18T02:01:03.6388448Z + [[ inductor_torchbench == *perf* ]]
2024-12-18T02:01:03.6389047Z + [[ inductor_torchbench == *_avx2* ]]
2024-12-18T02:01:03.6389361Z + [[ inductor_torchbench == *_avx512* ]]
2024-12-18T02:01:03.6390418Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --inductor --device cuda --training --amp --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv
2024-12-18T02:01:09.9990536Z 
2024-12-18T02:01:11.2269514Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:01:11.2269868Z loading model: 0it [00:01, ?it/s]
2024-12-18T02:01:11.2270187Z cuda train torchrec_dlrm                      
2024-12-18T02:01:28.7211162Z pass
2024-12-18T02:01:28.8298421Z TIMING: entire_frame_compile:12.89805 _recursive_pre_grad_passes:0.00778 _recursive_joint_graph_passes:5.51279 _recursive_post_grad_passes:0.13075 async_compile.wait:1.81432 code_gen:3.73756 inductor_compile:5.28123 backend_compile:10.16606 entire_backward_compile:2.28197 total_wall_time:15.18003
2024-12-18T02:01:28.8300051Z STATS: call_* op count: 123 | FakeTensorMode.__torch_dispatch__:8225 | FakeTensor.__torch_dispatch__:1551 | ProxyTorchDispatchMode.__torch_dispatch__:3099
2024-12-18T02:01:28.8300904Z Dynamo produced 2 graphs covering 123 ops with 6 graph breaks (5 unique)
2024-12-18T02:01:29.9064935Z [rank0]:[W1218 02:01:29.762882557 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2024-12-18T02:01:33.5408371Z 
2024-12-18T02:01:35.9017804Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:01:35.9018157Z loading model: 0it [00:02, ?it/s]
2024-12-18T02:01:35.9019954Z cuda train BERT_pytorch                       
2024-12-18T02:02:10.5409938Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:02:10.5412308Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 453, in torch_dynamo_resume_in_forward_and_backward_pass_at_448
2024-12-18T02:02:10.5413134Z     pred = mod(*cloned_inputs)
2024-12-18T02:02:10.5414241Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch/bert_pytorch/model/bert.py", line 46, in forward
2024-12-18T02:02:10.5414966Z     x = self.embedding(x, segment_info)
2024-12-18T02:02:10.5415733Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch/bert_pytorch/model/embedding/bert.py", line 31, in forward
2024-12-18T02:02:10.5416621Z     x = self.token(sequence) + self.position(sequence) + self.segment(segment_label)
2024-12-18T02:02:10.5416982Z 
2024-12-18T02:02:10.9405889Z pass
2024-12-18T02:02:11.0104807Z TIMING: entire_frame_compile:22.04405 _recursive_pre_grad_passes:0.01635 _recursive_joint_graph_passes:1.22615 _recursive_post_grad_passes:1.48343 async_compile.wait:2.15434 code_gen:9.57924 inductor_compile:17.17812 backend_compile:16.63215 entire_backward_compile:9.44013 total_wall_time:31.48418
2024-12-18T02:02:11.0107439Z STATS: call_* op count: 542 | FakeTensorMode.__torch_dispatch__:42158 | ProxyTorchDispatchMode.__torch_dispatch__:20702 | FakeTensor.__torch_dispatch__:6647
2024-12-18T02:02:11.0108331Z Dynamo produced 2 graphs covering 542 ops with 6 graph breaks (5 unique)
2024-12-18T02:02:16.5603223Z 
2024-12-18T02:02:21.5695383Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:02:21.5695725Z loading model: 0it [00:05, ?it/s]
2024-12-18T02:02:21.5696052Z cuda train Background_Matting                 
2024-12-18T02:02:21.5927363Z pass_due_to_skip
2024-12-18T02:02:21.6183479Z TIMING: total_wall_time:0
2024-12-18T02:02:21.6184492Z STATS: call_* op count: 0
2024-12-18T02:02:21.6184954Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-12-18T02:02:25.7794469Z 
2024-12-18T02:02:29.1615549Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:02:29.1615895Z loading model: 0it [00:03, ?it/s]
2024-12-18T02:02:29.1616225Z cuda train LearningToPaint                    
2024-12-18T02:02:45.4585306Z pass
2024-12-18T02:02:45.4645146Z TIMING: entire_frame_compile:8.83612 _recursive_pre_grad_passes:0.00734 _recursive_joint_graph_passes:1.31095 _recursive_post_grad_passes:0.20059 async_compile.wait:1.78576 code_gen:4.71038 inductor_compile:7.38736 backend_compile:6.50165 entire_backward_compile:4.47873 total_wall_time:13.31485
2024-12-18T02:02:45.4646819Z STATS: call_* op count: 75 | FakeTensorMode.__torch_dispatch__:9310 | ProxyTorchDispatchMode.__torch_dispatch__:4406 | FakeTensor.__torch_dispatch__:2252
2024-12-18T02:02:45.4647658Z Dynamo produced 2 graphs covering 75 ops with 6 graph breaks (5 unique)
2024-12-18T02:02:50.5098802Z 
2024-12-18T02:02:53.2110354Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:02:53.2110703Z loading model: 0it [00:02, ?it/s]
2024-12-18T02:02:53.2111045Z cuda train Super_SloMo                        
2024-12-18T02:04:08.7636281Z W1218 02:04:08.762000 47733 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:04:33.8413232Z pass
2024-12-18T02:04:33.8672858Z TIMING: entire_frame_compile:50.1346 _recursive_pre_grad_passes:0.03959 _recursive_joint_graph_passes:0.54258 _recursive_post_grad_passes:0.67458 async_compile.wait:8.92758 code_gen:32.65548 inductor_compile:43.28523 backend_compile:40.59602 entire_backward_compile:14.88184 total_wall_time:65.01645
2024-12-18T02:04:33.8675269Z STATS: call_* op count: 857 | FakeTensorMode.__torch_dispatch__:39772 | ProxyTorchDispatchMode.__torch_dispatch__:17939 | FakeTensor.__torch_dispatch__:9604
2024-12-18T02:04:33.8676138Z Dynamo produced 3 graphs covering 857 ops with 7 graph breaks (5 unique)
2024-12-18T02:04:40.3555083Z 
2024-12-18T02:04:42.1332482Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:04:42.1332822Z loading model: 0it [00:01, ?it/s]
2024-12-18T02:04:42.1333142Z cuda train alexnet                            
2024-12-18T02:05:01.1254894Z pass
2024-12-18T02:05:01.1448108Z TIMING: entire_frame_compile:5.55509 _recursive_pre_grad_passes:0.00442 _recursive_joint_graph_passes:2.51812 _recursive_post_grad_passes:0.05778 async_compile.wait:1.70107 code_gen:8.66721 inductor_compile:9.45666 backend_compile:4.96919 entire_backward_compile:7.74415 total_wall_time:13.29924
2024-12-18T02:05:01.1449739Z STATS: call_* op count: 26 | FakeTensorMode.__torch_dispatch__:2623 | ProxyTorchDispatchMode.__torch_dispatch__:950 | FakeTensor.__torch_dispatch__:326
2024-12-18T02:05:01.1450571Z Dynamo produced 2 graphs covering 26 ops with 6 graph breaks (5 unique)
2024-12-18T02:05:05.8124043Z 
2024-12-18T02:05:07.9545539Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:05:07.9545904Z loading model: 0it [00:02, ?it/s]
2024-12-18T02:05:07.9546234Z cuda train basic_gnn_edgecnn                  
2024-12-18T02:05:11.5747464Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_reduce.two Found from   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 98, in torch_dynamo_resume_in_scatter_at_93
2024-12-18T02:05:11.5748702Z     return src.new_zeros(size).scatter_reduce_(
2024-12-18T02:05:11.5748948Z 
2024-12-18T02:05:11.5748953Z 
2024-12-18T02:05:13.7211970Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:05:13.7212524Z    File "/tmp/jenkins_pyg/tmpmi05hz7x.py", line 190, in propagate
2024-12-18T02:05:13.7212996Z     kwargs = self._collect(edge_index, the_size, in_kwargs)
2024-12-18T02:05:13.7213446Z   File "/tmp/jenkins_pyg/tmpmi05hz7x.py", line 128, in _collect
2024-12-18T02:05:13.7213843Z     x_i = self._lift(x_i, edge_def, i)
2024-12-18T02:05:13.7214207Z   File "/tmp/jenkins_pyg/tmpmi05hz7x.py", line 88, in _lift
2024-12-18T02:05:13.7214613Z     return src.index_select(self.node_dim, index)
2024-12-18T02:05:13.7214858Z 
2024-12-18T02:05:13.7272536Z W1218 02:05:13.726000 49599 site-packages/torch/_functorch/_aot_autograd/autograd_cache.py:856] [19/0_1] Bypassing autograd cache due to: Cannot cache a graph with functional tensor
2024-12-18T02:05:13.9195580Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:05:13.9196647Z    File "/tmp/jenkins_pyg/tmp2j8bvwb4.py", line 190, in propagate
2024-12-18T02:05:13.9197120Z     kwargs = self._collect(edge_index, the_size, in_kwargs)
2024-12-18T02:05:13.9197581Z   File "/tmp/jenkins_pyg/tmp2j8bvwb4.py", line 128, in _collect
2024-12-18T02:05:13.9197963Z     x_i = self._lift(x_i, edge_def, i)
2024-12-18T02:05:13.9198340Z   File "/tmp/jenkins_pyg/tmp2j8bvwb4.py", line 88, in _lift
2024-12-18T02:05:13.9198750Z     return src.index_select(self.node_dim, index)
2024-12-18T02:05:13.9198986Z 
2024-12-18T02:05:13.9247135Z W1218 02:05:13.924000 49599 site-packages/torch/_functorch/_aot_autograd/autograd_cache.py:856] [16/0_1] Bypassing autograd cache due to: Cannot cache a graph with functional tensor
2024-12-18T02:05:14.1836542Z W1218 02:05:14.182000 49599 site-packages/torch/_functorch/_aot_autograd/autograd_cache.py:856] [6/0_1] Bypassing autograd cache due to: Cannot cache a graph with functional tensor
2024-12-18T02:05:14.1957689Z W1218 02:05:14.195000 49599 site-packages/torch/_logging/_internal.py:1089] [24/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:05:14.3481865Z ('Grad tensors ["L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T02:05:19.2728632Z pass
2024-12-18T02:05:19.2739082Z TIMING: entire_frame_compile:7.99251 _recursive_pre_grad_passes:0.01194 _recursive_joint_graph_passes:0.18368 inductor_compile:4.99894 backend_compile:5.96497 _recursive_post_grad_passes:0.06715 async_compile.wait:3.11634 code_gen:4.0281 entire_backward_compile:0.95468 total_wall_time:8.94719
2024-12-18T02:05:19.2741922Z STATS: call_* op count: 128 | FakeTensorMode.__torch_dispatch__:4799 | ProxyTorchDispatchMode.__torch_dispatch__:1878 | FakeTensor.__torch_dispatch__:943
2024-12-18T02:05:19.2743266Z Dynamo produced 8 graphs covering 128 ops with 20 graph breaks (6 unique)
2024-12-18T02:05:23.9612218Z 
2024-12-18T02:05:26.1663588Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:05:26.1663932Z loading model: 0it [00:02, ?it/s]
2024-12-18T02:05:26.1664256Z cuda train basic_gnn_gcn                      
2024-12-18T02:05:29.0413766Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_add.default Found from   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/gcn_conv.py", line 100, in torch_dynamo_resume_in_gcn_norm_at_91
2024-12-18T02:05:29.0415089Z     deg = scatter(edge_weight, idx, dim=0, dim_size=num_nodes, reduce='sum')
2024-12-18T02:05:29.0415838Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 74, in scatter
2024-12-18T02:05:29.0416523Z     return src.new_zeros(size).scatter_add_(dim, index, src)
2024-12-18T02:05:29.0416808Z 
2024-12-18T02:05:29.0416812Z 
2024-12-18T02:05:29.6008200Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_add.default Found from   File "/tmp/jenkins_pyg/tmpffhjzjy0.py", line 243, in torch_dynamo_resume_in_forward_at_221
2024-12-18T02:05:29.6009234Z     out = self.propagate(edge_index, x=x, edge_weight=edge_weight,
2024-12-18T02:05:29.6009733Z   File "/tmp/jenkins_pyg/tmpffhjzjy0.py", line 188, in propagate
2024-12-18T02:05:29.6010309Z     out = self.aggregate(out, ptr=kwargs.ptr, dim_size=kwargs.dim_size, index=kwargs.index)
2024-12-18T02:05:29.6011150Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/message_passing.py", line 604, in aggregate
2024-12-18T02:05:29.6012230Z     return self.aggr_module(inputs, index, ptr=ptr, dim_size=dim_size,
2024-12-18T02:05:29.6012979Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 109, in __call__
2024-12-18T02:05:29.6013694Z     return super().__call__(x, index, ptr, dim_size, dim, **kwargs)
2024-12-18T02:05:29.6014367Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/basic.py", line 21, in forward
2024-12-18T02:05:29.6015052Z     return self.reduce(x, index, ptr, dim_size, dim, reduce='sum')
2024-12-18T02:05:29.6015711Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 155, in reduce
2024-12-18T02:05:29.6016326Z     return scatter(x, index, dim, dim_size, reduce)
2024-12-18T02:05:29.6016949Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 74, in scatter
2024-12-18T02:05:29.6017613Z     return src.new_zeros(size).scatter_add_(dim, index, src)
2024-12-18T02:05:29.6017895Z 
2024-12-18T02:05:29.6017900Z 
2024-12-18T02:05:30.0930921Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_add.default Found from   File "/tmp/jenkins_pyg/tmp5y33onao.py", line 243, in torch_dynamo_resume_in_forward_at_221
2024-12-18T02:05:30.0932000Z     out = self.propagate(edge_index, x=x, edge_weight=edge_weight,
2024-12-18T02:05:30.0932517Z   File "/tmp/jenkins_pyg/tmp5y33onao.py", line 188, in propagate
2024-12-18T02:05:30.0933111Z     out = self.aggregate(out, ptr=kwargs.ptr, dim_size=kwargs.dim_size, index=kwargs.index)
2024-12-18T02:05:30.0934011Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/message_passing.py", line 604, in aggregate
2024-12-18T02:05:30.0934794Z     return self.aggr_module(inputs, index, ptr=ptr, dim_size=dim_size,
2024-12-18T02:05:30.0935494Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 109, in __call__
2024-12-18T02:05:30.0936199Z     return super().__call__(x, index, ptr, dim_size, dim, **kwargs)
2024-12-18T02:05:30.0937145Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/basic.py", line 21, in forward
2024-12-18T02:05:30.0937840Z     return self.reduce(x, index, ptr, dim_size, dim, reduce='sum')
2024-12-18T02:05:30.0938511Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 155, in reduce
2024-12-18T02:05:30.0939146Z     return scatter(x, index, dim, dim_size, reduce)
2024-12-18T02:05:30.0939784Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 74, in scatter
2024-12-18T02:05:30.0940455Z     return src.new_zeros(size).scatter_add_(dim, index, src)
2024-12-18T02:05:30.0940726Z 
2024-12-18T02:05:30.0940731Z 
2024-12-18T02:05:30.3944075Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_add.default Found from   File "/tmp/jenkins_pyg/tmp5y33onao.py", line 243, in torch_dynamo_resume_in_forward_at_221
2024-12-18T02:05:30.3945137Z     out = self.propagate(edge_index, x=x, edge_weight=edge_weight,
2024-12-18T02:05:30.3945633Z   File "/tmp/jenkins_pyg/tmp5y33onao.py", line 188, in propagate
2024-12-18T02:05:30.3946197Z     out = self.aggregate(out, ptr=kwargs.ptr, dim_size=kwargs.dim_size, index=kwargs.index)
2024-12-18T02:05:30.3947026Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/message_passing.py", line 604, in aggregate
2024-12-18T02:05:30.3947774Z     return self.aggr_module(inputs, index, ptr=ptr, dim_size=dim_size,
2024-12-18T02:05:30.3948466Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 109, in __call__
2024-12-18T02:05:30.3949154Z     return super().__call__(x, index, ptr, dim_size, dim, **kwargs)
2024-12-18T02:05:30.3949814Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/basic.py", line 21, in forward
2024-12-18T02:05:30.3950847Z     return self.reduce(x, index, ptr, dim_size, dim, reduce='sum')
2024-12-18T02:05:30.3951518Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 155, in reduce
2024-12-18T02:05:30.3952131Z     return scatter(x, index, dim, dim_size, reduce)
2024-12-18T02:05:30.3952745Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 74, in scatter
2024-12-18T02:05:30.3953396Z     return src.new_zeros(size).scatter_add_(dim, index, src)
2024-12-18T02:05:30.3953671Z 
2024-12-18T02:05:30.3953676Z 
2024-12-18T02:05:31.0847337Z W1218 02:05:31.083000 49864 site-packages/torch/_logging/_internal.py:1089] [18/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:05:31.1913204Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T02:05:34.6474765Z pass
2024-12-18T02:05:34.6480131Z TIMING: entire_frame_compile:5.63611 _recursive_pre_grad_passes:0.01214 _recursive_joint_graph_passes:0.16357 inductor_compile:3.53473 backend_compile:4.5803 _recursive_post_grad_passes:0.04025 async_compile.wait:2.3043 code_gen:2.92172 entire_backward_compile:0.54133 total_wall_time:6.17744
2024-12-18T02:05:34.6481716Z STATS: call_* op count: 112 | FakeTensorMode.__torch_dispatch__:2891 | FakeTensor.__torch_dispatch__:482 | ProxyTorchDispatchMode.__torch_dispatch__:1051
2024-12-18T02:05:34.6482539Z Dynamo produced 10 graphs covering 112 ops with 13 graph breaks (6 unique)
2024-12-18T02:05:39.2661726Z 
2024-12-18T02:05:41.3897566Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:05:41.3897963Z loading model: 0it [00:02, ?it/s]
2024-12-18T02:05:41.3898286Z cuda train basic_gnn_gin                      
2024-12-18T02:05:44.3451353Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_add.default Found from   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 453, in torch_dynamo_resume_in_forward_and_backward_pass_at_448
2024-12-18T02:05:44.3452564Z     pred = mod(*cloned_inputs)
2024-12-18T02:05:44.3453178Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/models/basic_gnn.py", line 226, in forward
2024-12-18T02:05:44.3453803Z     x = self.convs[i](x, edge_index)
2024-12-18T02:05:44.3454190Z   File "/tmp/jenkins_pyg/tmpdaum9ibf.py", line 225, in forward
2024-12-18T02:05:44.3454623Z     out = self.propagate(edge_index, x=x, size=size)
2024-12-18T02:05:44.3455048Z   File "/tmp/jenkins_pyg/tmpdaum9ibf.py", line 182, in propagate
2024-12-18T02:05:44.3455620Z     out = self.aggregate(out, ptr=kwargs.ptr, index=kwargs.index, dim_size=kwargs.dim_size)
2024-12-18T02:05:44.3456458Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/message_passing.py", line 604, in aggregate
2024-12-18T02:05:44.3457197Z     return self.aggr_module(inputs, index, ptr=ptr, dim_size=dim_size,
2024-12-18T02:05:44.3457888Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 109, in __call__
2024-12-18T02:05:44.3458562Z     return super().__call__(x, index, ptr, dim_size, dim, **kwargs)
2024-12-18T02:05:44.3459226Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/basic.py", line 21, in forward
2024-12-18T02:05:44.3459899Z     return self.reduce(x, index, ptr, dim_size, dim, reduce='sum')
2024-12-18T02:05:44.3460570Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 155, in reduce
2024-12-18T02:05:44.3461179Z     return scatter(x, index, dim, dim_size, reduce)
2024-12-18T02:05:44.3462044Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 74, in scatter
2024-12-18T02:05:44.3462701Z     return src.new_zeros(size).scatter_add_(dim, index, src)
2024-12-18T02:05:44.3462978Z 
2024-12-18T02:05:44.3462983Z 
2024-12-18T02:05:45.3888052Z W1218 02:05:45.387000 50115 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:05:45.5420571Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T02:05:49.3701541Z pass
2024-12-18T02:05:49.3731363Z TIMING: entire_frame_compile:6.01978 _recursive_pre_grad_passes:0.01056 _recursive_joint_graph_passes:0.17943 _recursive_post_grad_passes:0.0907 async_compile.wait:1.92452 code_gen:2.85102 inductor_compile:4.40612 backend_compile:4.77038 entire_backward_compile:0.83029 total_wall_time:6.85007
2024-12-18T02:05:49.3733850Z STATS: call_* op count: 135 | FakeTensorMode.__torch_dispatch__:4198 | ProxyTorchDispatchMode.__torch_dispatch__:1842 | FakeTensor.__torch_dispatch__:839
2024-12-18T02:05:49.3735115Z Dynamo produced 3 graphs covering 135 ops with 7 graph breaks (5 unique)
2024-12-18T02:05:53.9953175Z 
2024-12-18T02:05:56.1302220Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:05:56.1302607Z loading model: 0it [00:02, ?it/s]
2024-12-18T02:05:56.1303029Z cuda train basic_gnn_sage                     
2024-12-18T02:05:58.9415434Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_add.default Found from   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 453, in torch_dynamo_resume_in_forward_and_backward_pass_at_448
2024-12-18T02:05:58.9416655Z     pred = mod(*cloned_inputs)
2024-12-18T02:05:58.9417275Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/models/basic_gnn.py", line 226, in forward
2024-12-18T02:05:58.9417912Z     x = self.convs[i](x, edge_index)
2024-12-18T02:05:58.9418296Z   File "/tmp/jenkins_pyg/tmpsfzux42q.py", line 228, in forward
2024-12-18T02:05:58.9418722Z     out = self.propagate(edge_index, x=x, size=size)
2024-12-18T02:05:58.9419159Z   File "/tmp/jenkins_pyg/tmpsfzux42q.py", line 182, in propagate
2024-12-18T02:05:58.9419728Z     out = self.aggregate(out, index=kwargs.index, ptr=kwargs.ptr, dim_size=kwargs.dim_size)
2024-12-18T02:05:58.9420575Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/message_passing.py", line 604, in aggregate
2024-12-18T02:05:58.9421322Z     return self.aggr_module(inputs, index, ptr=ptr, dim_size=dim_size,
2024-12-18T02:05:58.9422073Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 109, in __call__
2024-12-18T02:05:58.9422766Z     return super().__call__(x, index, ptr, dim_size, dim, **kwargs)
2024-12-18T02:05:58.9423439Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/basic.py", line 34, in forward
2024-12-18T02:05:58.9424127Z     return self.reduce(x, index, ptr, dim_size, dim, reduce='mean')
2024-12-18T02:05:58.9424793Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 155, in reduce
2024-12-18T02:05:58.9425428Z     return scatter(x, index, dim, dim_size, reduce)
2024-12-18T02:05:58.9426239Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 78, in scatter
2024-12-18T02:05:58.9426914Z     count.scatter_add_(0, index, src.new_ones(src.size(dim)))
2024-12-18T02:05:58.9427197Z 
2024-12-18T02:05:58.9427202Z 
2024-12-18T02:05:59.6374614Z W1218 02:05:59.636000 50300 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:05:59.9578952Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T02:06:03.2042553Z pass
2024-12-18T02:06:03.2069782Z TIMING: entire_frame_compile:5.44585 _recursive_pre_grad_passes:0.01021 _recursive_joint_graph_passes:0.17108 _recursive_post_grad_passes:0.06835 async_compile.wait:1.75258 code_gen:2.49434 inductor_compile:3.72568 backend_compile:4.22578 entire_backward_compile:0.51226 total_wall_time:5.95812
2024-12-18T02:06:03.2071565Z STATS: call_* op count: 126 | FakeTensorMode.__torch_dispatch__:3534 | ProxyTorchDispatchMode.__torch_dispatch__:1467 | FakeTensor.__torch_dispatch__:645
2024-12-18T02:06:03.2072466Z Dynamo produced 3 graphs covering 126 ops with 7 graph breaks (5 unique)
2024-12-18T02:06:07.8206502Z 
2024-12-18T02:06:08.5166996Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:06:08.5167342Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:06:08.5167657Z cuda train dcgan                              
2024-12-18T02:06:13.8650497Z pass
2024-12-18T02:06:13.8669709Z TIMING: entire_frame_compile:2.7341 _recursive_pre_grad_passes:0.00419 _recursive_joint_graph_passes:0.13503 _recursive_post_grad_passes:0.05982 async_compile.wait:0.46221 code_gen:1.06665 inductor_compile:2.14273 backend_compile:2.16077 entire_backward_compile:0.81553 total_wall_time:3.54962
2024-12-18T02:06:13.8671356Z STATS: call_* op count: 17 | FakeTensorMode.__torch_dispatch__:2062 | FakeTensor.__torch_dispatch__:447 | ProxyTorchDispatchMode.__torch_dispatch__:830
2024-12-18T02:06:13.8672162Z Dynamo produced 2 graphs covering 17 ops with 6 graph breaks (5 unique)
2024-12-18T02:06:18.3764143Z 
2024-12-18T02:06:22.2480812Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:06:22.2481171Z loading model: 0it [00:03, ?it/s]
2024-12-18T02:06:22.2481485Z cuda train demucs                             
2024-12-18T02:07:31.7356854Z pass
2024-12-18T02:07:31.7607322Z TIMING: entire_frame_compile:7.46663 _recursive_pre_grad_passes:0.00955 _recursive_joint_graph_passes:0.17164 inductor_compile:8.43527 backend_compile:5.55186 _recursive_post_grad_passes:0.29912 async_compile.wait:1.36476 code_gen:6.46236 entire_backward_compile:5.01905 total_wall_time:12.48568
2024-12-18T02:07:31.7609101Z STATS: call_* op count: 88 | FakeTensorMode.__torch_dispatch__:6249 | ProxyTorchDispatchMode.__torch_dispatch__:2363 | FakeTensor.__torch_dispatch__:1250
2024-12-18T02:07:31.7609964Z Dynamo produced 6 graphs covering 88 ops with 9 graph breaks (6 unique)
2024-12-18T02:07:36.6175600Z 
2024-12-18T02:07:39.1808258Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:07:39.1808745Z loading model: 0it [00:02, ?it/s]
2024-12-18T02:07:39.1809199Z cuda train densenet121                        
2024-12-18T02:09:26.5373361Z pass
2024-12-18T02:09:26.5454644Z TIMING: entire_frame_compile:51.23312 _recursive_pre_grad_passes:0.02784 _recursive_joint_graph_passes:2.06482 _recursive_post_grad_passes:1.26158 async_compile.wait:15.28445 code_gen:48.31073 inductor_compile:63.26448 backend_compile:36.97126 entire_backward_compile:46.26737 total_wall_time:97.5005
2024-12-18T02:09:26.5456765Z STATS: call_* op count: 435 | FakeTensorMode.__torch_dispatch__:59616 | ProxyTorchDispatchMode.__torch_dispatch__:30351 | FakeTensor.__torch_dispatch__:14945
2024-12-18T02:09:26.5457659Z Dynamo produced 2 graphs covering 435 ops with 6 graph breaks (5 unique)
2024-12-18T02:09:34.0629907Z 
2024-12-18T02:09:39.3626648Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:09:39.3627008Z loading model: 0it [00:05, ?it/s]
2024-12-18T02:09:39.3627340Z cuda train detectron2_maskrcnn_r_50_c4        
2024-12-18T02:09:39.3643747Z Traceback (most recent call last):
2024-12-18T02:09:39.3644441Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2732, in validate_model
2024-12-18T02:09:39.3645030Z     self.model_iter_fn(model, example_inputs)
2024-12-18T02:09:39.3645667Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 453, in forward_and_backward_pass
2024-12-18T02:09:39.3646344Z     pred = mod(*cloned_inputs)
2024-12-18T02:09:39.3647068Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T02:09:39.3647738Z     return self._call_impl(*args, **kwargs)
2024-12-18T02:09:39.3648353Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T02:09:39.3648960Z     return forward_call(*args, **kwargs)
2024-12-18T02:09:39.3649605Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2024-12-18T02:09:39.3650273Z     return self.inference(batched_inputs)
2024-12-18T02:09:39.3650933Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2024-12-18T02:09:39.3651678Z     results, _ = self.roi_heads(images, features, proposals, None)
2024-12-18T02:09:39.3652402Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T02:09:39.3653071Z     return self._call_impl(*args, **kwargs)
2024-12-18T02:09:39.3654832Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T02:09:39.3655440Z     return forward_call(*args, **kwargs)
2024-12-18T02:09:39.3656076Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2024-12-18T02:09:39.3656735Z     box_features = self._shared_roi_transform(
2024-12-18T02:09:39.3657445Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2024-12-18T02:09:39.3658141Z     x = self.pooler(features, boxes)
2024-12-18T02:09:39.3658754Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T02:09:39.3659409Z     return self._call_impl(*args, **kwargs)
2024-12-18T02:09:39.3660016Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T02:09:39.3660619Z     return forward_call(*args, **kwargs)
2024-12-18T02:09:39.3661215Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2024-12-18T02:09:39.3661867Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2024-12-18T02:09:39.3662601Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T02:09:39.3663236Z     return self._call_impl(*args, **kwargs)
2024-12-18T02:09:39.3663827Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T02:09:39.3664422Z     return forward_call(*args, **kwargs)
2024-12-18T02:09:39.3665004Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2024-12-18T02:09:39.3665745Z     return roi_align(
2024-12-18T02:09:39.3666285Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-12-18T02:09:39.3667104Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-12-18T02:09:39.3667918Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-12-18T02:09:39.3668689Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-12-18T02:09:39.3669508Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-12-18T02:09:39.3670174Z     v1 = masked_index(y_low, x_low)
2024-12-18T02:09:39.3670756Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-12-18T02:09:39.3671351Z     return input[
2024-12-18T02:09:39.3673320Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 21890.87 GiB. GPU 0 has a total capacity of 21.98 GiB of which 16.55 GiB is free. Process 242097 has 5.42 GiB memory in use. Of the allocated memory 4.58 GiB is allocated by PyTorch, and 558.89 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-12-18T02:09:39.3675488Z 
2024-12-18T02:09:39.3675700Z The above exception was the direct cause of the following exception:
2024-12-18T02:09:39.3676015Z 
2024-12-18T02:09:39.3676122Z Traceback (most recent call last):
2024-12-18T02:09:39.3676591Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run
2024-12-18T02:09:39.3677070Z     ) = runner.load_model(
2024-12-18T02:09:39.3677550Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 372, in load_model
2024-12-18T02:09:39.3678111Z     self.validate_model(model, example_inputs)
2024-12-18T02:09:39.3678795Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2734, in validate_model
2024-12-18T02:09:39.3679368Z     raise RuntimeError("Eager run failed") from e
2024-12-18T02:09:39.3679711Z RuntimeError: Eager run failed
2024-12-18T02:09:39.3679892Z 
2024-12-18T02:09:39.3679987Z eager_fail_to_run
2024-12-18T02:09:43.6184981Z 
2024-12-18T02:09:52.5504231Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:09:52.5504645Z loading model: 0it [00:08, ?it/s]
2024-12-18T02:09:52.5504985Z cuda train dlrm                               
2024-12-18T02:10:01.3727830Z skipping cudagraphs due to sparsity not handled. Please file issue for sparse inference weights. Found from : 
2024-12-18T02:10:01.3728903Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 453, in torch_dynamo_resume_in_forward_and_backward_pass_at_448
2024-12-18T02:10:01.3729633Z     pred = mod(*cloned_inputs)
2024-12-18T02:10:01.3730592Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 355, in forward
2024-12-18T02:10:01.3731328Z     return self.sequential_forward(dense_x, lS_o, lS_i)
2024-12-18T02:10:01.3732077Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 367, in sequential_forward
2024-12-18T02:10:01.3732810Z     ly = self.apply_emb(lS_o, lS_i, self.emb_l)
2024-12-18T02:10:01.3733493Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 308, in apply_emb
2024-12-18T02:10:01.3734215Z     V = E(sparse_index_group_batch, sparse_offset_group_batch)
2024-12-18T02:10:01.3734497Z 
2024-12-18T02:10:01.9081753Z pass
2024-12-18T02:10:01.9485888Z TIMING: entire_frame_compile:6.51268 _recursive_pre_grad_passes:0.00539 _recursive_joint_graph_passes:3.86414 _recursive_post_grad_passes:0.08527 async_compile.wait:0.39776 code_gen:1.04608 inductor_compile:1.84744 backend_compile:5.74662 entire_backward_compile:0.80685 total_wall_time:7.31953
2024-12-18T02:10:01.9487880Z STATS: call_* op count: 40 | FakeTensorMode.__torch_dispatch__:3481 | ProxyTorchDispatchMode.__torch_dispatch__:1379 | FakeTensor.__torch_dispatch__:375
2024-12-18T02:10:01.9488719Z Dynamo produced 2 graphs covering 40 ops with 6 graph breaks (5 unique)
2024-12-18T02:10:06.4957954Z 
2024-12-18T02:10:07.9455434Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:10:07.9455833Z loading model: 0it [00:01, ?it/s]
2024-12-18T02:10:07.9456155Z cuda train drq                                
2024-12-18T02:10:15.8230737Z W1218 02:10:15.821000 53792 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:10:24.1026780Z pass
2024-12-18T02:10:24.1058686Z TIMING: entire_frame_compile:12.92159 _recursive_pre_grad_passes:0.01372 _recursive_joint_graph_passes:2.6255 _recursive_post_grad_passes:0.28703 async_compile.wait:4.4121 code_gen:6.10431 inductor_compile:7.96513 backend_compile:10.99299 entire_backward_compile:1.69992 total_wall_time:14.6215
2024-12-18T02:10:24.1060551Z STATS: call_* op count: 174 | FakeTensorMode.__torch_dispatch__:7455 | FakeTensor.__torch_dispatch__:1658 | ProxyTorchDispatchMode.__torch_dispatch__:3427
2024-12-18T02:10:24.1061489Z Dynamo produced 5 graphs covering 174 ops with 7 graph breaks (5 unique)
2024-12-18T02:10:28.9798320Z 
2024-12-18T02:10:30.9447453Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:10:30.9447810Z loading model: 0it [00:01, ?it/s]
2024-12-18T02:10:30.9448211Z cuda train fastNLP_Bert                       
2024-12-18T02:10:37.6714403Z W1218 02:10:37.670000 54144 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0] Graph break from `Tensor.item()`, consider setting:
2024-12-18T02:10:37.6715457Z W1218 02:10:37.670000 54144 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0]     torch._dynamo.config.capture_scalar_outputs = True
2024-12-18T02:10:37.6716282Z W1218 02:10:37.670000 54144 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0] or:
2024-12-18T02:10:37.6717548Z W1218 02:10:37.670000 54144 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2024-12-18T02:10:37.6718505Z W1218 02:10:37.670000 54144 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0] to include these operations in the captured graph.
2024-12-18T02:10:37.6719299Z W1218 02:10:37.670000 54144 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0] 
2024-12-18T02:10:37.6720029Z W1218 02:10:37.670000 54144 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0] Graph break: from user code at:
2024-12-18T02:10:37.6721303Z W1218 02:10:37.670000 54144 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0]   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 453, in torch_dynamo_resume_in_forward_and_backward_pass_at_448
2024-12-18T02:10:37.6722567Z W1218 02:10:37.670000 54144 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0]     pred = mod(*cloned_inputs)
2024-12-18T02:10:37.6723706Z W1218 02:10:37.670000 54144 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/models/bert.py", line 265, in forward
2024-12-18T02:10:37.6724842Z W1218 02:10:37.670000 54144 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0]     sequence_output = self.bert(words)
2024-12-18T02:10:37.6726048Z W1218 02:10:37.670000 54144 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/embeddings/bert_embedding.py", line 137, in forward
2024-12-18T02:10:37.6727228Z W1218 02:10:37.670000 54144 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0]     outputs = self.model(words)
2024-12-18T02:10:37.6728421Z W1218 02:10:37.670000 54144 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/embeddings/bert_embedding.py", line 445, in forward
2024-12-18T02:10:37.6730702Z W1218 02:10:37.670000 54144 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0]     max_word_piece_length = batch_word_pieces_length.sum(dim=-1).max().item()  # 表示word piece的长度(包括padding)
2024-12-18T02:10:37.6733896Z W1218 02:10:37.670000 54144 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0] 
2024-12-18T02:10:37.6734522Z W1218 02:10:37.670000 54144 site-packages/torch/_dynamo/variables/tensor.py:869] [3/0] 
2024-12-18T02:11:02.3209886Z W1218 02:11:02.319000 54144 site-packages/torch/_functorch/_aot_autograd/autograd_cache.py:856] [10/0] Bypassing autograd cache due to: Cannot cache a graph with functional tensor
2024-12-18T02:11:11.4753039Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:11:11.4753848Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/modules/encoder/bert.py", line 512, in forward
2024-12-18T02:11:11.4754588Z     embedding_output = self.embeddings(input_ids, token_type_ids)
2024-12-18T02:11:11.4755349Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/modules/encoder/bert.py", line 236, in forward
2024-12-18T02:11:11.4756044Z     words_embeddings = self.word_embeddings(input_ids)
2024-12-18T02:11:11.4756313Z 
2024-12-18T02:11:14.3152725Z pass
2024-12-18T02:11:14.3814284Z TIMING: entire_frame_compile:24.03693 _recursive_pre_grad_passes:0.01705 _recursive_joint_graph_passes:4.01756 inductor_compile:17.38163 backend_compile:20.10526 _recursive_post_grad_passes:1.10853 async_compile.precompile:0.21372 async_compile.wait:1.88408 code_gen:10.3942 entire_backward_compile:9.432 total_wall_time:33.46892
2024-12-18T02:11:14.3816104Z STATS: call_* op count: 448 | FakeTensorMode.__torch_dispatch__:37266 | ProxyTorchDispatchMode.__torch_dispatch__:19311 | FakeTensor.__torch_dispatch__:5885
2024-12-18T02:11:14.3816965Z Dynamo produced 7 graphs covering 448 ops with 10 graph breaks (6 unique)
2024-12-18T02:11:20.1209531Z 
2024-12-18T02:11:21.2465731Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:11:21.2466437Z loading model: 0it [00:01, ?it/s]
2024-12-18T02:11:21.2467084Z cuda train functorch_dp_cifar10               
2024-12-18T02:11:42.4506097Z W1218 02:11:42.449000 54495 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:11:43.1641775Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][57].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T02:11:58.3996422Z pass
2024-12-18T02:11:58.4059754Z TIMING: entire_frame_compile:24.38233 _recursive_pre_grad_passes:0.02261 _recursive_joint_graph_passes:1.59716 _recursive_post_grad_passes:0.36193 async_compile.wait:4.87228 code_gen:15.37575 inductor_compile:21.43518 backend_compile:19.79396 entire_backward_compile:9.57732 total_wall_time:33.95964
2024-12-18T02:11:58.4062081Z STATS: call_* op count: 402 | FakeTensorMode.__torch_dispatch__:19021 | ProxyTorchDispatchMode.__torch_dispatch__:8433 | FakeTensor.__torch_dispatch__:6400
2024-12-18T02:11:58.4062966Z Dynamo produced 3 graphs covering 402 ops with 7 graph breaks (5 unique)
2024-12-18T02:12:03.9781412Z 
2024-12-18T02:12:04.8062069Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:12:04.8062420Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:12:04.8062772Z cuda train functorch_maml_omniglot            
2024-12-18T02:12:12.8331965Z W1218 02:12:12.832000 55078 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:12:18.2276505Z pass
2024-12-18T02:12:18.2277665Z TIMING: entire_frame_compile:8.45111 _recursive_pre_grad_passes:0.01092 _recursive_joint_graph_passes:1.27346 _recursive_post_grad_passes:0.06636 async_compile.wait:2.97245 code_gen:6.38608 inductor_compile:7.56982 backend_compile:7.32086 entire_backward_compile:2.87125 total_wall_time:11.32236
2024-12-18T02:12:18.2279277Z STATS: call_* op count: 107 | FakeTensorMode.__torch_dispatch__:4015 | ProxyTorchDispatchMode.__torch_dispatch__:1544 | FakeTensor.__torch_dispatch__:1109
2024-12-18T02:12:18.2280117Z Dynamo produced 3 graphs covering 107 ops with 7 graph breaks (5 unique)
2024-12-18T02:12:22.8964595Z 
2024-12-18T02:12:24.3559358Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:12:24.3559978Z loading model: 0it [00:01, ?it/s]
2024-12-18T02:12:24.3560499Z cuda train hf_Albert                          
2024-12-18T02:12:56.1346750Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:12:56.1350860Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 453, in torch_dynamo_resume_in_forward_and_backward_pass_at_448
2024-12-18T02:12:56.1351665Z     pred = mod(*cloned_inputs)
2024-12-18T02:12:56.1352343Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 986, in forward
2024-12-18T02:12:56.1353031Z     outputs = self.albert(
2024-12-18T02:12:56.1353675Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 729, in forward
2024-12-18T02:12:56.1354360Z     embedding_output = self.embeddings(
2024-12-18T02:12:56.1355292Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 250, in forward
2024-12-18T02:12:56.1356012Z     inputs_embeds = self.word_embeddings(input_ids)
2024-12-18T02:12:56.1356268Z 
2024-12-18T02:12:56.2866023Z W1218 02:12:56.285000 55378 site-packages/torch/_logging/_internal.py:1089] [5/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:12:56.5776539Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T02:13:04.6225887Z pass
2024-12-18T02:13:04.6380945Z TIMING: entire_frame_compile:22.84465 _recursive_pre_grad_passes:0.02236 _recursive_joint_graph_passes:2.07511 _recursive_post_grad_passes:1.07466 async_compile.wait:3.84118 code_gen:10.87976 inductor_compile:17.77054 backend_compile:18.94423 entire_backward_compile:8.43572 total_wall_time:31.28037
2024-12-18T02:13:04.6382732Z STATS: call_* op count: 599 | FakeTensorMode.__torch_dispatch__:43295 | FakeTensor.__torch_dispatch__:6951 | ProxyTorchDispatchMode.__torch_dispatch__:21739
2024-12-18T02:13:04.6383579Z Dynamo produced 3 graphs covering 599 ops with 6 graph breaks (5 unique)
2024-12-18T02:13:10.1874997Z 
2024-12-18T02:13:14.1758593Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:13:14.1758933Z loading model: 0it [00:03, ?it/s]
2024-12-18T02:13:14.1759256Z cuda train hf_Bart                            
2024-12-18T02:13:57.8042400Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:13:57.8044947Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 453, in torch_dynamo_resume_in_forward_and_backward_pass_at_448
2024-12-18T02:13:57.8045779Z     pred = mod(*cloned_inputs)
2024-12-18T02:13:57.8046495Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 69, in forward
2024-12-18T02:13:57.8047514Z     return self.model(input_ids=input_ids, decoder_input_ids=decoder_input_ids)
2024-12-18T02:13:57.8048488Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1731, in forward
2024-12-18T02:13:57.8049238Z     outputs = self.model(
2024-12-18T02:13:57.8049977Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1599, in forward
2024-12-18T02:13:57.8050704Z     encoder_outputs = self.encoder(
2024-12-18T02:13:57.8051385Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1152, in forward
2024-12-18T02:13:57.8052224Z     inputs_embeds = self.embed_tokens(input_ids) * self.embed_scale
2024-12-18T02:13:57.8052598Z 
2024-12-18T02:13:57.9633183Z W1218 02:13:57.962000 55728 site-packages/torch/_logging/_internal.py:1089] [5/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:14:00.1619520Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][163].grad", "L['self'].param_groups[0]['params'][164].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][166].grad", "L['self'].param_groups[0]['params'][167].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][169].grad", "L['self'].param_groups[0]['params'][170].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][172].grad", "L['self'].param_groups[0]['params'][173].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][175].grad", "L['self'].param_groups[0]['params'][176].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][178].grad", "L['self'].param_groups[0]['params'][179].grad", "L['self'].param_groups[0]['params'][180].grad", "L['self'].param_groups[0]['params'][181].grad", "L['self'].param_groups[0]['params'][182].grad", "L['self'].param_groups[0]['params'][183].grad", "L['self'].param_groups[0]['params'][184].grad", "L['self'].param_groups[0]['params'][185].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][187].grad", "L['self'].param_groups[0]['params'][188].grad", "L['self'].param_groups[0]['params'][189].grad", "L['self'].param_groups[0]['params'][190].grad", "L['self'].param_groups[0]['params'][191].grad", "L['self'].param_groups[0]['params'][192].grad", "L['self'].param_groups[0]['params'][193].grad", "L['self'].param_groups[0]['params'][194].grad", "L['self'].param_groups[0]['params'][195].grad", "L['self'].param_groups[0]['params'][196].grad", "L['self'].param_groups[0]['params'][197].grad", "L['self'].param_groups[0]['params'][198].grad", "L['self'].param_groups[0]['params'][199].grad", "L['self'].param_groups[0]['params'][200].grad", "L['self'].param_groups[0]['params'][201].grad", "L['self'].param_groups[0]['params'][202].grad", "L['self'].param_groups[0]['params'][203].grad", "L['self'].param_groups[0]['params'][204].grad", "L['self'].param_groups[0]['params'][205].grad", "L['self'].param_groups[0]['params'][206].grad", "L['self'].param_groups[0]['params'][207].grad", "L['self'].param_groups[0]['params'][208].grad", "L['self'].param_groups[0]['params'][209].grad", "L['self'].param_groups[0]['params'][210].grad", "L['self'].param_groups[0]['params'][211].grad", "L['self'].param_groups[0]['params'][212].grad", "L['self'].param_groups[0]['params'][213].grad", "L['self'].param_groups[0]['params'][214].grad", "L['self'].param_groups[0]['params'][215].grad", "L['self'].param_groups[0]['params'][216].grad", "L['self'].param_groups[0]['params'][217].grad", "L['self'].param_groups[0]['params'][218].grad", "L['self'].param_groups[0]['params'][219].grad", "L['self'].param_groups[0]['params'][220].grad", "L['self'].param_groups[0]['params'][221].grad", "L['self'].param_groups[0]['params'][222].grad", "L['self'].param_groups[0]['params'][223].grad", "L['self'].param_groups[0]['params'][224].grad", "L['self'].param_groups[0]['params'][225].grad", "L['self'].param_groups[0]['params'][226].grad", "L['self'].param_groups[0]['params'][227].grad", "L['self'].param_groups[0]['params'][228].grad", "L['self'].param_groups[0]['params'][229].grad", "L['self'].param_groups[0]['params'][230].grad", "L['self'].param_groups[0]['params'][231].grad", "L['self'].param_groups[0]['params'][232].grad", "L['self'].param_groups[0]['params'][233].grad", "L['self'].param_groups[0]['params'][234].grad", "L['self'].param_groups[0]['params'][235].grad", "L['self'].param_groups[0]['params'][236].grad", "L['self'].param_groups[0]['params'][237].grad", "L['self'].param_groups[0]['params'][238].grad", "L['self'].param_groups[0]['params'][239].grad", "L['self'].param_groups[0]['params'][240].grad", "L['self'].param_groups[0]['params'][241].grad", "L['self'].param_groups[0]['params'][242].grad", "L['self'].param_groups[0]['params'][243].grad", "L['self'].param_groups[0]['params'][244].grad", "L['self'].param_groups[0]['params'][245].grad", "L['self'].param_groups[0]['params'][246].grad", "L['self'].param_groups[0]['params'][247].grad", "L['self'].param_groups[0]['params'][248].grad", "L['self'].param_groups[0]['params'][249].grad", "L['self'].param_groups[0]['params'][250].grad", "L['self'].param_groups[0]['params'][251].grad", "L['self'].param_groups[0]['params'][252].grad", "L['self'].param_groups[0]['params'][253].grad", "L['self'].param_groups[0]['params'][254].grad", "L['self'].param_groups[0]['params'][255].grad", "L['self'].param_groups[0]['params'][256].grad", "L['self'].param_groups[0]['params'][257].grad", "L['self'].param_groups[0]['params'][258].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T02:14:49.1640638Z pass
2024-12-18T02:14:49.2155039Z TIMING: entire_frame_compile:71.49749 _recursive_pre_grad_passes:0.07672 _recursive_joint_graph_passes:2.68468 _recursive_post_grad_passes:1.47294 async_compile.wait:5.32168 code_gen:26.30528 inductor_compile:41.52456 backend_compile:55.11037 entire_backward_compile:9.45517 total_wall_time:80.95266
2024-12-18T02:14:49.2156664Z STATS: call_* op count: 1769 | FakeTensorMode.__torch_dispatch__:73938 | FakeTensor.__torch_dispatch__:18530 | ProxyTorchDispatchMode.__torch_dispatch__:34396
2024-12-18T02:14:49.2157499Z Dynamo produced 3 graphs covering 1769 ops with 6 graph breaks (5 unique)
2024-12-18T02:14:57.0792262Z 
2024-12-18T02:14:59.7197153Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:14:59.7197674Z loading model: 0it [00:02, ?it/s]
2024-12-18T02:14:59.7198161Z cuda train hf_Bert                            
2024-12-18T02:15:36.5550426Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:15:36.5551697Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 453, in torch_dynamo_resume_in_forward_and_backward_pass_at_448
2024-12-18T02:15:36.5553298Z     pred = mod(*cloned_inputs)
2024-12-18T02:15:36.5554282Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1360, in forward
2024-12-18T02:15:36.5555188Z     outputs = self.bert(
2024-12-18T02:15:36.5555810Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward
2024-12-18T02:15:36.5556489Z     embedding_output = self.embeddings(
2024-12-18T02:15:36.5557157Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward
2024-12-18T02:15:36.5557865Z     inputs_embeds = self.word_embeddings(input_ids)
2024-12-18T02:15:36.5558113Z 
2024-12-18T02:15:36.7148635Z W1218 02:15:36.714000 56087 site-packages/torch/_logging/_internal.py:1089] [5/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:15:38.4343410Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][163].grad", "L['self'].param_groups[0]['params'][164].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][166].grad", "L['self'].param_groups[0]['params'][167].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][169].grad", "L['self'].param_groups[0]['params'][170].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][172].grad", "L['self'].param_groups[0]['params'][173].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][175].grad", "L['self'].param_groups[0]['params'][176].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][178].grad", "L['self'].param_groups[0]['params'][179].grad", "L['self'].param_groups[0]['params'][180].grad", "L['self'].param_groups[0]['params'][181].grad", "L['self'].param_groups[0]['params'][182].grad", "L['self'].param_groups[0]['params'][183].grad", "L['self'].param_groups[0]['params'][184].grad", "L['self'].param_groups[0]['params'][185].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][187].grad", "L['self'].param_groups[0]['params'][188].grad", "L['self'].param_groups[0]['params'][189].grad", "L['self'].param_groups[0]['params'][190].grad", "L['self'].param_groups[0]['params'][191].grad", "L['self'].param_groups[0]['params'][192].grad", "L['self'].param_groups[0]['params'][193].grad", "L['self'].param_groups[0]['params'][194].grad", "L['self'].param_groups[0]['params'][195].grad", "L['self'].param_groups[0]['params'][196].grad", "L['self'].param_groups[0]['params'][197].grad", "L['self'].param_groups[0]['params'][198].grad", "L['self'].param_groups[0]['params'][199].grad", "L['self'].param_groups[0]['params'][200].grad", "L['self'].param_groups[0]['params'][201].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T02:16:16.3658510Z pass
2024-12-18T02:16:16.4081716Z TIMING: entire_frame_compile:58.22324 _recursive_pre_grad_passes:0.06131 _recursive_joint_graph_passes:3.19423 _recursive_post_grad_passes:1.05322 async_compile.wait:4.47078 code_gen:19.48675 inductor_compile:31.76626 backend_compile:44.28722 entire_backward_compile:7.25104 total_wall_time:65.47428
2024-12-18T02:16:16.4084055Z STATS: call_* op count: 1400 | FakeTensorMode.__torch_dispatch__:63336 | FakeTensor.__torch_dispatch__:14392 | ProxyTorchDispatchMode.__torch_dispatch__:29389
2024-12-18T02:16:16.4084932Z Dynamo produced 3 graphs covering 1400 ops with 6 graph breaks (5 unique)
2024-12-18T02:16:23.5479984Z 
2024-12-18T02:16:28.3845263Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:16:28.3845683Z loading model: 0it [00:04, ?it/s]
2024-12-18T02:16:28.3846087Z cuda train hf_Bert_large                      
2024-12-18T02:17:41.9510594Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:17:41.9512872Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 453, in torch_dynamo_resume_in_forward_and_backward_pass_at_448
2024-12-18T02:17:41.9514778Z     pred = mod(*cloned_inputs)
2024-12-18T02:17:41.9515752Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1360, in forward
2024-12-18T02:17:41.9516971Z     outputs = self.bert(
2024-12-18T02:17:41.9517608Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward
2024-12-18T02:17:41.9518286Z     embedding_output = self.embeddings(
2024-12-18T02:17:41.9518950Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward
2024-12-18T02:17:41.9519651Z     inputs_embeds = self.word_embeddings(input_ids)
2024-12-18T02:17:41.9519900Z 
2024-12-18T02:17:42.1761774Z W1218 02:17:42.175000 56387 site-packages/torch/_logging/_internal.py:1089] [5/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:17:45.5436986Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][163].grad", "L['self'].param_groups[0]['params'][164].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][166].grad", "L['self'].param_groups[0]['params'][167].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][169].grad", "L['self'].param_groups[0]['params'][170].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][172].grad", "L['self'].param_groups[0]['params'][173].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][175].grad", "L['self'].param_groups[0]['params'][176].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][178].grad", "L['self'].param_groups[0]['params'][179].grad", "L['self'].param_groups[0]['params'][180].grad", "L['self'].param_groups[0]['params'][181].grad", "L['self'].param_groups[0]['params'][182].grad", "L['self'].param_groups[0]['params'][183].grad", "L['self'].param_groups[0]['params'][184].grad", "L['self'].param_groups[0]['params'][185].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][187].grad", "L['self'].param_groups[0]['params'][188].grad", "L['self'].param_groups[0]['params'][189].grad", "L['self'].param_groups[0]['params'][190].grad", "L['self'].param_groups[0]['params'][191].grad", "L['self'].param_groups[0]['params'][192].grad", "L['self'].param_groups[0]['params'][193].grad", "L['self'].param_groups[0]['params'][194].grad", "L['self'].param_groups[0]['params'][195].grad", "L['self'].param_groups[0]['params'][196].grad", "L['self'].param_groups[0]['params'][197].grad", "L['self'].param_groups[0]['params'][198].grad", "L['self'].param_groups[0]['params'][199].grad", "L['self'].param_groups[0]['params'][200].grad", "L['self'].param_groups[0]['params'][201].grad", "L['self'].param_groups[0]['params'][202].grad", "L['self'].param_groups[0]['params'][203].grad", "L['self'].param_groups[0]['params'][204].grad", "L['self'].param_groups[0]['params'][205].grad", "L['self'].param_groups[0]['params'][206].grad", "L['self'].param_groups[0]['params'][207].grad", "L['self'].param_groups[0]['params'][208].grad", "L['self'].param_groups[0]['params'][209].grad", "L['self'].param_groups[0]['params'][210].grad", "L['self'].param_groups[0]['params'][211].grad", "L['self'].param_groups[0]['params'][212].grad", "L['self'].param_groups[0]['params'][213].grad", "L['self'].param_groups[0]['params'][214].grad", "L['self'].param_groups[0]['params'][215].grad", "L['self'].param_groups[0]['params'][216].grad", "L['self'].param_groups[0]['params'][217].grad", "L['self'].param_groups[0]['params'][218].grad", "L['self'].param_groups[0]['params'][219].grad", "L['self'].param_groups[0]['params'][220].grad", "L['self'].param_groups[0]['params'][221].grad", "L['self'].param_groups[0]['params'][222].grad", "L['self'].param_groups[0]['params'][223].grad", "L['self'].param_groups[0]['params'][224].grad", "L['self'].param_groups[0]['params'][225].grad", "L['self'].param_groups[0]['params'][226].grad", "L['self'].param_groups[0]['params'][227].grad", "L['self'].param_groups[0]['params'][228].grad", "L['self'].param_groups[0]['params'][229].grad", "L['self'].param_groups[0]['params'][230].grad", "L['self'].param_groups[0]['params'][231].grad", "L['self'].param_groups[0]['params'][232].grad", "L['self'].param_groups[0]['params'][233].grad", "L['self'].param_groups[0]['params'][234].grad", "L['self'].param_groups[0]['params'][235].grad", "L['self'].param_groups[0]['params'][236].grad", "L['self'].param_groups[0]['params'][237].grad", "L['self'].param_groups[0]['params'][238].grad", "L['self'].param_groups[0]['params'][239].grad", "L['self'].param_groups[0]['params'][240].grad", "L['self'].param_groups[0]['params'][241].grad", "L['self'].param_groups[0]['params'][242].grad", "L['self'].param_groups[0]['params'][243].grad", "L['self'].param_groups[0]['params'][244].grad", "L['self'].param_groups[0]['params'][245].grad", "L['self'].param_groups[0]['params'][246].grad", "L['self'].param_groups[0]['params'][247].grad", "L['self'].param_groups[0]['params'][248].grad", "L['self'].param_groups[0]['params'][249].grad", "L['self'].param_groups[0]['params'][250].grad", "L['self'].param_groups[0]['params'][251].grad", "L['self'].param_groups[0]['params'][252].grad", "L['self'].param_groups[0]['params'][253].grad", "L['self'].param_groups[0]['params'][254].grad", "L['self'].param_groups[0]['params'][255].grad", "L['self'].param_groups[0]['params'][256].grad", "L['self'].param_groups[0]['params'][257].grad", "L['self'].param_groups[0]['params'][258].grad", "L['self'].param_groups[0]['params'][259].grad", "L['self'].param_groups[0]['params'][260].grad", "L['self'].param_groups[0]['params'][261].grad", "L['self'].param_groups[0]['params'][262].grad", "L['self'].param_groups[0]['params'][263].grad", "L['self'].param_groups[0]['params'][264].grad", "L['self'].param_groups[0]['params'][265].grad", "L['self'].param_groups[0]['params'][266].grad", "L['self'].param_groups[0]['params'][267].grad", "L['self'].param_groups[0]['params'][268].grad", "L['self'].param_groups[0]['params'][269].grad", "L['self'].param_groups[0]['params'][270].grad", "L['self'].param_groups[0]['params'][271].grad", "L['self'].param_groups[0]['params'][272].grad", "L['self'].param_groups[0]['params'][273].grad", "L['self'].param_groups[0]['params'][274].grad", "L['self'].param_groups[0]['params'][275].grad", "L['self'].param_groups[0]['params'][276].grad", "L['self'].param_groups[0]['params'][277].grad", "L['self'].param_groups[0]['params'][278].grad", "L['self'].param_groups[0]['params'][279].grad", "L['self'].param_groups[0]['params'][280].grad", "L['self'].param_groups[0]['params'][281].grad", "L['self'].param_groups[0]['params'][282].grad", "L['self'].param_groups[0]['params'][283].grad", "L['self'].param_groups[0]['params'][284].grad", "L['self'].param_groups[0]['params'][285].grad", "L['self'].param_groups[0]['params'][286].grad", "L['self'].param_groups[0]['params'][287].grad", "L['self'].param_groups[0]['params'][288].grad", "L['self'].param_groups[0]['params'][289].grad", "L['self'].param_groups[0]['params'][290].grad", "L['self'].param_groups[0]['params'][291].grad", "L['self'].param_groups[0]['params'][292].grad", "L['self'].param_groups[0]['params'][293].grad", "L['self'].param_groups[0]['params'][294].grad", "L['self'].param_groups[0]['params'][295].grad", "L['self'].param_groups[0]['params'][296].grad", "L['self'].param_groups[0]['params'][297].grad", "L['self'].param_groups[0]['params'][298].grad", "L['self'].param_groups[0]['params'][299].grad", "L['self'].param_groups[0]['params'][300].grad", "L['self'].param_groups[0]['params'][301].grad", "L['self'].param_groups[0]['params'][302].grad", "L['self'].param_groups[0]['params'][303].grad", "L['self'].param_groups[0]['params'][304].grad", "L['self'].param_groups[0]['params'][305].grad", "L['self'].param_groups[0]['params'][306].grad", "L['self'].param_groups[0]['params'][307].grad", "L['self'].param_groups[0]['params'][308].grad", "L['self'].param_groups[0]['params'][309].grad", "L['self'].param_groups[0]['params'][310].grad", "L['self'].param_groups[0]['params'][311].grad", "L['self'].param_groups[0]['params'][312].grad", "L['self'].param_groups[0]['params'][313].grad", "L['self'].param_groups[0]['params'][314].grad", "L['self'].param_groups[0]['params'][315].grad", "L['self'].param_groups[0]['params'][316].grad", "L['self'].param_groups[0]['params'][317].grad", "L['self'].param_groups[0]['params'][318].grad", "L['self'].param_groups[0]['params'][319].grad", "L['self'].param_groups[0]['params'][320].grad", "L['self'].param_groups[0]['params'][321].grad", "L['self'].param_groups[0]['params'][322].grad", "L['self'].param_groups[0]['params'][323].grad", "L['self'].param_groups[0]['params'][324].grad", "L['self'].param_groups[0]['params'][325].grad", "L['self'].param_groups[0]['params'][326].grad", "L['self'].param_groups[0]['params'][327].grad", "L['self'].param_groups[0]['params'][328].grad", "L['self'].param_groups[0]['params'][329].grad", "L['self'].param_groups[0]['params'][330].grad", "L['self'].param_groups[0]['params'][331].grad", "L['self'].param_groups[0]['params'][332].grad", "L['self'].param_groups[0]['params'][333].grad", "L['self'].param_groups[0]['params'][334].grad", "L['self'].param_groups[0]['params'][335].grad", "L['self'].param_groups[0]['params'][336].grad", "L['self'].param_groups[0]['params'][337].grad", "L['self'].param_groups[0]['params'][338].grad", "L['self'].param_groups[0]['params'][339].grad", "L['self'].param_groups[0]['params'][340].grad", "L['self'].param_groups[0]['params'][341].grad", "L['self'].param_groups[0]['params'][342].grad", "L['self'].param_groups[0]['params'][343].grad", "L['self'].param_groups[0]['params'][344].grad", "L['self'].param_groups[0]['params'][345].grad", "L['self'].param_groups[0]['params'][346].grad", "L['self'].param_groups[0]['params'][347].grad", "L['self'].param_groups[0]['params'][348].grad", "L['self'].param_groups[0]['params'][349].grad", "L['self'].param_groups[0]['params'][350].grad", "L['self'].param_groups[0]['params'][351].grad", "L['self'].param_groups[0]['params'][352].grad", "L['self'].param_groups[0]['params'][353].grad", "L['self'].param_groups[0]['params'][354].grad", "L['self'].param_groups[0]['params'][355].grad", "L['self'].param_groups[0]['params'][356].grad", "L['self'].param_groups[0]['params'][357].grad", "L['self'].param_groups[0]['params'][358].grad", "L['self'].param_groups[0]['params'][359].grad", "L['self'].param_groups[0]['params'][360].grad", "L['self'].param_groups[0]['params'][361].grad", "L['self'].param_groups[0]['params'][362].grad", "L['self'].param_groups[0]['params'][363].grad", "L['self'].param_groups[0]['params'][364].grad", "L['self'].param_groups[0]['params'][365].grad", "L['self'].param_groups[0]['params'][366].grad", "L['self'].param_groups[0]['params'][367].grad", "L['self'].param_groups[0]['params'][368].grad", "L['self'].param_groups[0]['params'][369].grad", "L['self'].param_groups[0]['params'][370].grad", "L['self'].param_groups[0]['params'][371].grad", "L['self'].param_groups[0]['params'][372].grad", "L['self'].param_groups[0]['params'][373].grad", "L['self'].param_groups[0]['params'][374].grad", "L['self'].param_groups[0]['params'][375].grad", "L['self'].param_groups[0]['params'][376].grad", "L['self'].param_groups[0]['params'][377].grad", "L['self'].param_groups[0]['params'][378].grad", "L['self'].param_groups[0]['params'][379].grad", "L['self'].param_groups[0]['params'][380].grad", "L['self'].param_groups[0]['params'][381].grad", "L['self'].param_groups[0]['params'][382].grad", "L['self'].param_groups[0]['params'][383].grad", "L['self'].param_groups[0]['params'][384].grad", "L['self'].param_groups[0]['params'][385].grad", "L['self'].param_groups[0]['params'][386].grad", "L['self'].param_groups[0]['params'][387].grad", "L['self'].param_groups[0]['params'][388].grad", "L['self'].param_groups[0]['params'][389].grad", "L['self'].param_groups[0]['params'][390].grad", "L['self'].param_groups[0]['params'][391].grad", "L['self'].param_groups[0]['params'][392].grad", "L['self'].param_groups[0]['params'][393].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T02:19:01.4324702Z pass
2024-12-18T02:19:01.5477964Z TIMING: entire_frame_compile:113.73004 _recursive_pre_grad_passes:0.11494 _recursive_joint_graph_passes:5.10233 _recursive_post_grad_passes:2.15854 async_compile.wait:7.03695 code_gen:38.37969 inductor_compile:61.42778 backend_compile:85.58992 entire_backward_compile:13.59073 total_wall_time:127.32078
2024-12-18T02:19:01.5479954Z STATS: call_* op count: 2708 | FakeTensorMode.__torch_dispatch__:123730 | FakeTensor.__torch_dispatch__:27988 | ProxyTorchDispatchMode.__torch_dispatch__:57565
2024-12-18T02:19:01.5480826Z Dynamo produced 3 graphs covering 2708 ops with 6 graph breaks (5 unique)
2024-12-18T02:19:11.3516877Z 
2024-12-18T02:19:15.0016008Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:19:15.0016517Z loading model: 0it [00:03, ?it/s]
2024-12-18T02:19:15.0016985Z cuda train hf_BigBird                         
2024-12-18T02:19:26.3119628Z WARNING:common:fp64 golden ref were not generated for hf_BigBird. Setting accuracy check to cosine
2024-12-18T02:20:22.5533735Z W1218 02:20:22.552000 56709 site-packages/torch/_inductor/utils.py:1543] [3/0_1] DeviceCopy in input program
2024-12-18T02:20:46.5707290Z skipping cudagraphs due to skipping cudagraphs due to cpu device (cat_1). Found from : 
2024-12-18T02:20:46.5708260Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 453, in torch_dynamo_resume_in_forward_and_backward_pass_at_448
2024-12-18T02:20:46.5709015Z     pred = mod(*cloned_inputs)
2024-12-18T02:20:46.5709704Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2450, in forward
2024-12-18T02:20:46.5710384Z     outputs = self.bert(
2024-12-18T02:20:46.5711031Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2133, in forward
2024-12-18T02:20:46.5711730Z     encoder_outputs = self.encoder(
2024-12-18T02:20:46.5712402Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1631, in forward
2024-12-18T02:20:46.5713080Z     layer_outputs = layer_module(
2024-12-18T02:20:46.5713755Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1488, in forward
2024-12-18T02:20:46.5714457Z     self_attention_outputs = self.attention(
2024-12-18T02:20:46.5715461Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1401, in forward
2024-12-18T02:20:46.5716159Z     self_outputs = self.self(
2024-12-18T02:20:46.5716814Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 472, in forward
2024-12-18T02:20:46.5717592Z     context_layer, attention_probs = self.bigbird_block_sparse_attention(
2024-12-18T02:20:46.5718464Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 593, in bigbird_block_sparse_attention
2024-12-18T02:20:46.5719225Z     rand_attn = np.stack(rand_attn, axis=0)
2024-12-18T02:20:46.5719451Z 
2024-12-18T02:21:16.1173526Z W1218 02:21:16.116000 56709 site-packages/torch/_logging/_internal.py:1089] [5/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:21:17.8452220Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][163].grad", "L['self'].param_groups[0]['params'][164].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][166].grad", "L['self'].param_groups[0]['params'][167].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][169].grad", "L['self'].param_groups[0]['params'][170].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][172].grad", "L['self'].param_groups[0]['params'][173].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][175].grad", "L['self'].param_groups[0]['params'][176].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][178].grad", "L['self'].param_groups[0]['params'][179].grad", "L['self'].param_groups[0]['params'][180].grad", "L['self'].param_groups[0]['params'][181].grad", "L['self'].param_groups[0]['params'][182].grad", "L['self'].param_groups[0]['params'][183].grad", "L['self'].param_groups[0]['params'][184].grad", "L['self'].param_groups[0]['params'][185].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][187].grad", "L['self'].param_groups[0]['params'][188].grad", "L['self'].param_groups[0]['params'][189].grad", "L['self'].param_groups[0]['params'][190].grad", "L['self'].param_groups[0]['params'][191].grad", "L['self'].param_groups[0]['params'][192].grad", "L['self'].param_groups[0]['params'][193].grad", "L['self'].param_groups[0]['params'][194].grad", "L['self'].param_groups[0]['params'][195].grad", "L['self'].param_groups[0]['params'][196].grad", "L['self'].param_groups[0]['params'][199].grad", "L['self'].param_groups[0]['params'][200].grad", "L['self'].param_groups[0]['params'][201].grad", "L['self'].param_groups[0]['params'][202].grad", "L['self'].param_groups[0]['params'][203].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T02:21:58.2694252Z pass
2024-12-18T02:21:58.2976778Z TIMING: entire_frame_compile:116.53004 _recursive_pre_grad_passes:0.13949 _recursive_joint_graph_passes:6.71776 _recursive_post_grad_passes:4.19791 async_compile.wait:5.15316 code_gen:46.0867 inductor_compile:74.90901 backend_compile:91.83992 entire_backward_compile:28.11977 total_wall_time:144.64981
2024-12-18T02:21:58.2978558Z STATS: call_* op count: 4380 | FakeTensorMode.__torch_dispatch__:192534 | FakeTensor.__torch_dispatch__:24303 | ProxyTorchDispatchMode.__torch_dispatch__:85660
2024-12-18T02:21:58.2979415Z Dynamo produced 3 graphs covering 4380 ops with 6 graph breaks (5 unique)
2024-12-18T02:22:08.2221771Z 
2024-12-18T02:22:10.4129648Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:22:10.4130395Z loading model: 0it [00:02, ?it/s]
2024-12-18T02:22:10.4130728Z cuda train hf_DistilBert                      
2024-12-18T02:22:32.0913632Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:22:32.0914911Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 453, in torch_dynamo_resume_in_forward_and_backward_pass_at_448
2024-12-18T02:22:32.0916061Z     pred = mod(*cloned_inputs)
2024-12-18T02:22:32.0917544Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 905, in forward
2024-12-18T02:22:32.0918654Z     dlbrt_output = self.distilbert(
2024-12-18T02:22:32.0919379Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward
2024-12-18T02:22:32.0920246Z     embeddings = self.embeddings(input_ids, inputs_embeds)  # (bs, seq_length, dim)
2024-12-18T02:22:32.0921125Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 141, in forward
2024-12-18T02:22:32.0921971Z     input_embeds = self.word_embeddings(input_ids)  # (bs, max_seq_length, dim)
2024-12-18T02:22:32.0922319Z 
2024-12-18T02:22:32.2257294Z W1218 02:22:32.224000 57249 site-packages/torch/_logging/_internal.py:1089] [5/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:22:33.1419525Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T02:22:54.4450037Z pass
2024-12-18T02:22:54.4708005Z TIMING: entire_frame_compile:32.22093 _recursive_pre_grad_passes:0.03549 _recursive_joint_graph_passes:1.23557 _recursive_post_grad_passes:0.86944 async_compile.wait:3.96869 code_gen:12.7746 inductor_compile:20.07975 backend_compile:25.35173 entire_backward_compile:4.65753 total_wall_time:36.87847
2024-12-18T02:22:54.4711171Z STATS: call_* op count: 751 | FakeTensorMode.__torch_dispatch__:33137 | FakeTensor.__torch_dispatch__:7929 | ProxyTorchDispatchMode.__torch_dispatch__:14995
2024-12-18T02:22:54.4712816Z Dynamo produced 3 graphs covering 751 ops with 6 graph breaks (5 unique)
2024-12-18T02:23:00.2186650Z 
2024-12-18T02:23:03.5227329Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:23:03.5227739Z loading model: 0it [00:03, ?it/s]
2024-12-18T02:23:03.5228096Z cuda train hf_GPT2                            
2024-12-18T02:23:36.1522857Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:23:36.1523848Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 453, in torch_dynamo_resume_in_forward_and_backward_pass_at_448
2024-12-18T02:23:36.1524838Z     pred = mod(*cloned_inputs)
2024-12-18T02:23:36.1525727Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1074, in forward
2024-12-18T02:23:36.1526439Z     transformer_outputs = self.transformer(
2024-12-18T02:23:36.1527124Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 837, in forward
2024-12-18T02:23:36.1527804Z     inputs_embeds = self.wte(input_ids)
2024-12-18T02:23:36.1528029Z 
2024-12-18T02:23:36.2860212Z W1218 02:23:36.285000 57569 site-packages/torch/_logging/_internal.py:1089] [5/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:23:37.5244083Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T02:24:06.4965742Z pass
2024-12-18T02:24:06.5741780Z TIMING: entire_frame_compile:47.36145 _recursive_pre_grad_passes:0.05499 _recursive_joint_graph_passes:2.3043 _recursive_post_grad_passes:0.82611 async_compile.wait:4.3273 code_gen:18.34617 inductor_compile:28.48465 backend_compile:36.4591 entire_backward_compile:7.77195 total_wall_time:55.13341
2024-12-18T02:24:06.5743452Z STATS: call_* op count: 1385 | FakeTensorMode.__torch_dispatch__:50950 | FakeTensor.__torch_dispatch__:12350 | ProxyTorchDispatchMode.__torch_dispatch__:23100
2024-12-18T02:24:06.5744311Z Dynamo produced 3 graphs covering 1385 ops with 6 graph breaks (5 unique)
2024-12-18T02:24:13.1745825Z 
2024-12-18T02:24:26.1561597Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:24:26.1562035Z loading model: 0it [00:12, ?it/s]
2024-12-18T02:24:26.1562439Z cuda train hf_GPT2_large                      
2024-12-18T02:24:26.1793514Z pass_due_to_skip
2024-12-18T02:24:26.6868244Z TIMING: total_wall_time:0
2024-12-18T02:24:26.6868644Z STATS: call_* op count: 0
2024-12-18T02:24:26.6869295Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-12-18T02:24:30.8325943Z 
2024-12-18T02:24:32.5917518Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:24:32.5917992Z loading model: 0it [00:01, ?it/s]
2024-12-18T02:24:32.5920836Z cuda train hf_Reformer                        
2024-12-18T02:24:51.7975874Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter.src Found from   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 1480, in torch_dynamo_resume_in_forward_at_1478
2024-12-18T02:24:51.7977125Z     attn_outputs = self.attention(
2024-12-18T02:24:51.7977812Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 1313, in forward
2024-12-18T02:24:51.7978590Z     self_attention_outputs = self.self_attention(
2024-12-18T02:24:51.7979312Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 482, in forward
2024-12-18T02:24:51.7980207Z     sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx(
2024-12-18T02:24:51.7981262Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 702, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx
2024-12-18T02:24:51.7982265Z     undo_sorted_bucket_idx.scatter_(-1, sorted_bucket_idx, indices)
2024-12-18T02:24:51.7982574Z 
2024-12-18T02:24:51.7982584Z 
2024-12-18T02:24:54.7956856Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter.src Found from   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 1480, in torch_dynamo_resume_in_forward_at_1478
2024-12-18T02:24:54.7958639Z     attn_outputs = self.attention(
2024-12-18T02:24:54.7959341Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 1313, in forward
2024-12-18T02:24:54.7960078Z     self_attention_outputs = self.self_attention(
2024-12-18T02:24:54.7960804Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 482, in forward
2024-12-18T02:24:54.7961713Z     sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx(
2024-12-18T02:24:54.7962758Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 702, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx
2024-12-18T02:24:54.7963679Z     undo_sorted_bucket_idx.scatter_(-1, sorted_bucket_idx, indices)
2024-12-18T02:24:54.7963993Z 
2024-12-18T02:24:54.7964004Z 
2024-12-18T02:24:56.4724567Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter.src Found from   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 1480, in torch_dynamo_resume_in_forward_at_1478
2024-12-18T02:24:56.4725834Z     attn_outputs = self.attention(
2024-12-18T02:24:56.4726535Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 1313, in forward
2024-12-18T02:24:56.4727282Z     self_attention_outputs = self.self_attention(
2024-12-18T02:24:56.4728011Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 482, in forward
2024-12-18T02:24:56.4728926Z     sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx(
2024-12-18T02:24:56.4730152Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 702, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx
2024-12-18T02:24:56.4731361Z     undo_sorted_bucket_idx.scatter_(-1, sorted_bucket_idx, indices)
2024-12-18T02:24:56.4731662Z 
2024-12-18T02:24:56.4731667Z 
2024-12-18T02:24:59.0530844Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:24:59.0531821Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 2094, in torch_dynamo_resume_in_forward_at_2066
2024-12-18T02:24:59.0532662Z     embedding_output = self.embeddings(
2024-12-18T02:24:59.0533351Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 254, in forward
2024-12-18T02:24:59.0534089Z     inputs_embeds = self.word_embeddings(input_ids)
2024-12-18T02:24:59.0534351Z 
2024-12-18T02:24:59.0673433Z W1218 02:24:59.066000 57992 site-packages/torch/_logging/_internal.py:1089] [30/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:24:59.8037776Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T02:25:21.9534235Z pass
2024-12-18T02:25:21.9659685Z TIMING: entire_frame_compile:32.43334 _recursive_pre_grad_passes:0.0427 _recursive_joint_graph_passes:0.4703 inductor_compile:19.87091 backend_compile:25.32443 _recursive_post_grad_passes:0.17192 async_compile.wait:7.29483 code_gen:14.92227 entire_backward_compile:1.32094 total_wall_time:33.75427
2024-12-18T02:25:21.9661502Z STATS: call_* op count: 841 | FakeTensorMode.__torch_dispatch__:23236 | ProxyTorchDispatchMode.__torch_dispatch__:8959 | FakeTensor.__torch_dispatch__:4396 | attempt fast:21 | fast is_contiguous:21
2024-12-18T02:25:21.9662569Z Dynamo produced 17 graphs covering 841 ops with 23 graph breaks (8 unique)
2024-12-18T02:25:27.4487248Z 
2024-12-18T02:25:31.9075648Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:25:31.9076001Z loading model: 0it [00:04, ?it/s]
2024-12-18T02:25:31.9076322Z cuda train hf_T5_base                         
2024-12-18T02:25:35.0568949Z WARNING:common:fp64 golden ref were not generated for hf_T5_base. Setting accuracy check to cosine
2024-12-18T02:25:37.1529026Z ERROR:common:
2024-12-18T02:25:37.1529325Z Traceback (most recent call last):
2024-12-18T02:25:37.1529885Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2988, in check_accuracy
2024-12-18T02:25:37.1530630Z     correct_rerun_result = self.run_n_iterations(
2024-12-18T02:25:37.1531591Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2770, in run_n_iterations
2024-12-18T02:25:37.1532227Z     self.model_iter_fn(mod, inputs, collect_outputs=False)
2024-12-18T02:25:37.1532884Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 455, in forward_and_backward_pass
2024-12-18T02:25:37.1533513Z     self.grad_scaler.scale(loss).backward()
2024-12-18T02:25:37.1534089Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 626, in backward
2024-12-18T02:25:37.1534638Z     torch.autograd.backward(
2024-12-18T02:25:37.1535203Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward
2024-12-18T02:25:37.1535775Z     _engine_run_backward(
2024-12-18T02:25:37.1536353Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 823, in _engine_run_backward
2024-12-18T02:25:37.1537180Z     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2024-12-18T02:25:37.1539412Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 192.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 88.44 MiB is free. Process 247588 has 21.88 GiB memory in use. Of the allocated memory 20.95 GiB is allocated by PyTorch, and 595.70 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-12-18T02:25:37.2618360Z eager_2nd_run_OOM
2024-12-18T02:25:37.2652095Z TIMING: total_wall_time:0
2024-12-18T02:25:37.2652390Z STATS: call_* op count: 0
2024-12-18T02:25:37.2652788Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-12-18T02:25:41.4332662Z 
2024-12-18T02:25:51.7740243Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:25:51.7740589Z loading model: 0it [00:10, ?it/s]
2024-12-18T02:25:51.7740931Z cuda train hf_T5_large                        
2024-12-18T02:25:51.7976946Z pass_due_to_skip
2024-12-18T02:25:52.2460354Z TIMING: total_wall_time:0
2024-12-18T02:25:52.2460661Z STATS: call_* op count: 0
2024-12-18T02:25:52.2461060Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-12-18T02:25:56.3175126Z 
2024-12-18T02:25:57.5596200Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:25:57.5596650Z loading model: 0it [00:01, ?it/s]
2024-12-18T02:25:57.5597076Z cuda train hf_Whisper                         
2024-12-18T02:26:16.6065322Z W1218 02:26:16.605000 58526 site-packages/torch/_logging/_internal.py:1089] [5/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:26:33.1105861Z pass
2024-12-18T02:26:33.1171886Z TIMING: entire_frame_compile:25.21284 _recursive_pre_grad_passes:0.02539 _recursive_joint_graph_passes:1.58058 _recursive_post_grad_passes:0.38022 async_compile.wait:6.12693 code_gen:13.01172 inductor_compile:17.63387 backend_compile:20.51371 entire_backward_compile:4.28883 total_wall_time:29.50167
2024-12-18T02:26:33.1173910Z STATS: call_* op count: 488 | FakeTensorMode.__torch_dispatch__:20374 | FakeTensor.__torch_dispatch__:5402 | ProxyTorchDispatchMode.__torch_dispatch__:9193
2024-12-18T02:26:33.1174774Z Dynamo produced 3 graphs covering 488 ops with 6 graph breaks (5 unique)
2024-12-18T02:26:38.5269390Z 
2024-12-18T02:26:38.8842238Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:26:38.8842598Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:26:38.8842996Z cuda train hf_distil_whisper                  
2024-12-18T02:26:38.8859553Z Traceback (most recent call last):
2024-12-18T02:26:38.8860402Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run
2024-12-18T02:26:38.8860909Z     ) = runner.load_model(
2024-12-18T02:26:38.8861402Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 312, in load_model
2024-12-18T02:26:38.8861932Z     benchmark = benchmark_cls(
2024-12-18T02:26:38.8862943Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__
2024-12-18T02:26:38.8863699Z     obj = type.__call__(cls, *args, **kwargs)
2024-12-18T02:26:38.8864458Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper/__init__.py", line 12, in __init__
2024-12-18T02:26:38.8865208Z     raise NotImplementedError("Training is not implemented.")
2024-12-18T02:26:38.8865652Z NotImplementedError: Training is not implemented.
2024-12-18T02:26:38.8865910Z 
2024-12-18T02:26:38.8866007Z model_fail_to_load
2024-12-18T02:26:42.8400342Z 
2024-12-18T02:26:43.5032046Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:26:43.5032399Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:26:43.5032713Z cuda train lennard_jones                      
2024-12-18T02:26:48.9986078Z W1218 02:26:48.997000 58979 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:26:52.4532440Z pass
2024-12-18T02:26:52.4533760Z TIMING: entire_frame_compile:6.59975 _recursive_pre_grad_passes:0.0089 _recursive_joint_graph_passes:2.36241 _recursive_post_grad_passes:0.22494 async_compile.wait:1.5483 code_gen:2.15631 inductor_compile:3.07549 backend_compile:5.82826 entire_backward_compile:0.39894 total_wall_time:6.99869
2024-12-18T02:26:52.4535517Z STATS: call_* op count: 82 | FakeTensorMode.__torch_dispatch__:3022 | ProxyTorchDispatchMode.__torch_dispatch__:1300 | FakeTensor.__torch_dispatch__:726
2024-12-18T02:26:52.4536436Z Dynamo produced 3 graphs covering 82 ops with 7 graph breaks (5 unique)
2024-12-18T02:26:53.8752597Z accuracy            pass_rate=81.82%
2024-12-18T02:26:53.8753125Z calls_captured      gmean=0.00x mean=552.667x
2024-12-18T02:26:53.8756772Z unique_graphs       gmean=0.00x mean=3.303x
2024-12-18T02:26:53.8760429Z graph_breaks        gmean=0.00x mean=6.485x
2024-12-18T02:26:53.8763420Z unique_graph_breaks gmean=0.00x mean=4.303x
2024-12-18T02:26:53.8767000Z autograd_captures   gmean=0.00x mean=0.000x
2024-12-18T02:26:53.8770644Z autograd_compiles   gmean=0.00x mean=0.000x
2024-12-18T02:26:53.8775498Z cudagraph_skips     gmean=0.00x mean=0.727x
2024-12-18T02:26:55.4187445Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_training.csv
2024-12-18T02:26:55.7557235Z torchrec_dlrm                       PASS
2024-12-18T02:26:55.7561807Z BERT_pytorch                        PASS
2024-12-18T02:26:55.7565346Z Background_Matting                  XFAIL
2024-12-18T02:26:55.7568835Z LearningToPaint                     PASS
2024-12-18T02:26:55.7573235Z Super_SloMo                         PASS
2024-12-18T02:26:55.7577481Z alexnet                             PASS
2024-12-18T02:26:55.7581957Z basic_gnn_edgecnn                   PASS
2024-12-18T02:26:55.7586523Z basic_gnn_gcn                       PASS
2024-12-18T02:26:55.7590797Z basic_gnn_gin                       PASS
2024-12-18T02:26:55.7595076Z basic_gnn_sage                      PASS
2024-12-18T02:26:55.7599469Z dcgan                               PASS
2024-12-18T02:26:55.7603863Z demucs                              PASS
2024-12-18T02:26:55.7608198Z densenet121                         PASS
2024-12-18T02:26:55.7612546Z detectron2_maskrcnn_r_50_c4         XFAIL
2024-12-18T02:26:55.7616840Z dlrm                                PASS
2024-12-18T02:26:55.7621227Z drq                                 PASS
2024-12-18T02:26:55.7625673Z fastNLP_Bert                        PASS
2024-12-18T02:26:55.7630170Z functorch_dp_cifar10                PASS
2024-12-18T02:26:55.7634777Z functorch_maml_omniglot             PASS
2024-12-18T02:26:55.7639058Z hf_Albert                           PASS
2024-12-18T02:26:55.7643378Z hf_Bart                             PASS
2024-12-18T02:26:55.7647677Z hf_Bert                             PASS
2024-12-18T02:26:55.7651982Z hf_Bert_large                       PASS
2024-12-18T02:26:55.7656378Z hf_BigBird                          PASS
2024-12-18T02:26:55.7660669Z hf_DistilBert                       PASS
2024-12-18T02:26:55.7665264Z hf_GPT2                             PASS
2024-12-18T02:26:55.7669354Z hf_GPT2_large                       XFAIL
2024-12-18T02:26:55.7673741Z hf_Reformer                         PASS
2024-12-18T02:26:55.7678026Z hf_T5_base                          XFAIL
2024-12-18T02:26:55.7682353Z hf_T5_large                         XFAIL
2024-12-18T02:26:55.7686711Z hf_Whisper                          PASS
2024-12-18T02:26:55.7691027Z hf_distil_whisper                   XFAIL
2024-12-18T02:26:55.7695336Z lennard_jones                       PASS
2024-12-18T02:26:55.8208291Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_training.csv
2024-12-18T02:26:56.1565768Z torchrec_dlrm                       PASS
2024-12-18T02:26:56.1570216Z BERT_pytorch                        PASS
2024-12-18T02:26:56.1574789Z Background_Matting                  PASS
2024-12-18T02:26:56.1578769Z LearningToPaint                     PASS
2024-12-18T02:26:56.1583090Z Super_SloMo                         PASS
2024-12-18T02:26:56.1587700Z alexnet                             PASS
2024-12-18T02:26:56.1591976Z basic_gnn_edgecnn                   PASS
2024-12-18T02:26:56.1596349Z basic_gnn_gcn                       PASS
2024-12-18T02:26:56.1600689Z basic_gnn_gin                       PASS
2024-12-18T02:26:56.1605038Z basic_gnn_sage                      PASS
2024-12-18T02:26:56.1609475Z dcgan                               PASS
2024-12-18T02:26:56.1613741Z demucs                              PASS
2024-12-18T02:26:56.1618089Z densenet121                         PASS
2024-12-18T02:26:56.1622516Z detectron2_maskrcnn_r_50_c4         PASS
2024-12-18T02:26:56.1626961Z dlrm                                PASS
2024-12-18T02:26:56.1631615Z drq                                 PASS
2024-12-18T02:26:56.1637200Z fastNLP_Bert                        PASS
2024-12-18T02:26:56.1641706Z functorch_dp_cifar10                PASS
2024-12-18T02:26:56.1646305Z functorch_maml_omniglot             PASS
2024-12-18T02:26:56.1650453Z hf_Albert                           PASS
2024-12-18T02:26:56.1654888Z hf_Bart                             PASS
2024-12-18T02:26:56.1659194Z hf_Bert                             PASS
2024-12-18T02:26:56.1663668Z hf_Bert_large                       PASS
2024-12-18T02:26:56.1668011Z hf_BigBird                          PASS
2024-12-18T02:26:56.1672300Z hf_DistilBert                       PASS
2024-12-18T02:26:56.1676632Z hf_GPT2                             PASS
2024-12-18T02:26:56.1681030Z hf_GPT2_large                       PASS
2024-12-18T02:26:56.1685483Z hf_Reformer                         PASS
2024-12-18T02:26:56.1689771Z hf_T5_base                          PASS
2024-12-18T02:26:56.1694179Z hf_T5_large                         PASS
2024-12-18T02:26:56.1698638Z hf_Whisper                          PASS
2024-12-18T02:26:56.1703097Z hf_distil_whisper                   PASS
2024-12-18T02:26:56.1707501Z lennard_jones                       PASS
2024-12-18T02:26:56.2223635Z + cleanup_workspace
2024-12-18T02:26:56.2224163Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.'
2024-12-18T02:26:56.2224993Z sudo may print the following warning message that can be ignored. The chown command will still run.
2024-12-18T02:26:56.2225608Z + echo '    sudo: setrlimit(RLIMIT_STACK): Operation not permitted'
2024-12-18T02:26:56.2226076Z     sudo: setrlimit(RLIMIT_STACK): Operation not permitted
2024-12-18T02:26:56.2226608Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42'
2024-12-18T02:26:56.2227204Z For more details refer to https://github.com/sudo-project/sudo/issues/42
2024-12-18T02:26:56.2227677Z + sudo chown -R 1000 /var/lib/jenkins/workspace
2024-12-18T02:26:57.5004105Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@release/2.6
2024-12-18T02:26:57.5004827Z with:
2024-12-18T02:26:57.5005091Z   benchmark-results-dir: test/test-reports
2024-12-18T02:26:57.5005421Z   dry-run: false
2024-12-18T02:26:57.5005690Z   schema-version: v3
2024-12-18T02:26:57.5006116Z   github-token: ***
2024-12-18T02:26:57.5006354Z env:
2024-12-18T02:26:57.5006578Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:26:57.5006911Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:26:57.5007461Z   DOCKER_CONTAINER_ID: 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T02:26:57.5007945Z ##[endgroup]
2024-12-18T02:26:57.5034742Z ##[group]Run set -eux
2024-12-18T02:26:57.5035013Z set -eux
2024-12-18T02:26:57.5035275Z python3 -mpip install boto3==1.35.33
2024-12-18T02:26:57.5048971Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:26:57.5049347Z env:
2024-12-18T02:26:57.5049590Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:26:57.5049926Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:26:57.5050462Z   DOCKER_CONTAINER_ID: 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T02:26:57.5050944Z ##[endgroup]
2024-12-18T02:26:57.5088584Z + python3 -mpip install boto3==1.35.33
2024-12-18T02:26:57.8408156Z Defaulting to user installation because normal site-packages is not writeable
2024-12-18T02:26:57.8633320Z Requirement already satisfied: boto3==1.35.33 in /home/ec2-user/.local/lib/python3.9/site-packages (1.35.33)
2024-12-18T02:26:57.8681089Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.0)
2024-12-18T02:26:57.8684995Z Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.4)
2024-12-18T02:26:57.8688969Z Requirement already satisfied: botocore<1.36.0,>=1.35.33 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.33) (1.35.83)
2024-12-18T02:26:57.8740897Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.8.1)
2024-12-18T02:26:57.8751795Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.25.10)
2024-12-18T02:26:57.8790723Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.15.0)
2024-12-18T02:26:58.0010776Z ##[group]Run set -eux
2024-12-18T02:26:58.0011055Z set -eux
2024-12-18T02:26:58.0011282Z 
2024-12-18T02:26:58.0011525Z if [[ -z "${GITHUB_TOKEN}" ]]; then
2024-12-18T02:26:58.0011880Z   echo "Missing github-token input"
2024-12-18T02:26:58.0012198Z   exit 1
2024-12-18T02:26:58.0012425Z fi
2024-12-18T02:26:58.0021127Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:26:58.0021520Z env:
2024-12-18T02:26:58.0021740Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:26:58.0022163Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:26:58.0022714Z   DOCKER_CONTAINER_ID: 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T02:26:58.0023558Z   GITHUB_TOKEN: ***
2024-12-18T02:26:58.0023796Z ##[endgroup]
2024-12-18T02:26:58.0052473Z + [[ -z *** ]]
2024-12-18T02:26:58.0106227Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main
2024-12-18T02:26:58.0106666Z with:
2024-12-18T02:26:58.0107016Z   github-token: ***
2024-12-18T02:26:58.0107255Z env:
2024-12-18T02:26:58.0107479Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:26:58.0107824Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:26:58.0108375Z   DOCKER_CONTAINER_ID: 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T02:26:58.0108880Z ##[endgroup]
2024-12-18T02:26:58.0134359Z ##[group]Run set -eux
2024-12-18T02:26:58.0134815Z set -eux
2024-12-18T02:26:58.0135051Z 
2024-12-18T02:26:58.0135522Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2024-12-18T02:26:58.0143894Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:26:58.0144276Z env:
2024-12-18T02:26:58.0144503Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:26:58.0144842Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:26:58.0145391Z   DOCKER_CONTAINER_ID: 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T02:26:58.0146152Z   GITHUB_TOKEN: ***
2024-12-18T02:26:58.0146394Z ##[endgroup]
2024-12-18T02:26:58.0173028Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 12383255690 i-02d991818cc142eee
2024-12-18T02:26:59.2961261Z setting job-id=34567153284
2024-12-18T02:26:59.2961860Z setting job-name=cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T02:26:59.3070262Z ##[group]Run set -eux
2024-12-18T02:26:59.3070581Z set -eux
2024-12-18T02:26:59.3070809Z 
2024-12-18T02:26:59.3071192Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \
2024-12-18T02:26:59.3071704Z   --schema-version "${SCHEMA_VERSION}" \
2024-12-18T02:26:59.3072049Z   --repo "${REPO}" \
2024-12-18T02:26:59.3072349Z   --head-branch "${HEAD_BRANCH}" \
2024-12-18T02:26:59.3072683Z   --head-sha "${HEAD_SHA}" \
2024-12-18T02:26:59.3073015Z   --workflow-id "${WORKFLOW_RUN_ID}" \
2024-12-18T02:26:59.3073363Z   --run-attempt "${RUN_ATTEMPT}" \
2024-12-18T02:26:59.3073687Z   --job-id "${JOB_ID}" \
2024-12-18T02:26:59.3073984Z   --job-name "${JOB_NAME}"
2024-12-18T02:26:59.3082712Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:26:59.3083102Z env:
2024-12-18T02:26:59.3083321Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:26:59.3083649Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:26:59.3084200Z   DOCKER_CONTAINER_ID: 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T02:26:59.3084698Z   SCHEMA_VERSION: v3
2024-12-18T02:26:59.3084953Z   REPO: pytorch/pytorch
2024-12-18T02:26:59.3085228Z   HEAD_BRANCH: refs/heads/release/2.6
2024-12-18T02:26:59.3085618Z   HEAD_SHA: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T02:26:59.3085968Z   WORKFLOW_RUN_ID: 12383255690
2024-12-18T02:26:59.3086243Z   RUN_ATTEMPT: 1
2024-12-18T02:26:59.3086531Z   JOB_ID: 34567153284
2024-12-18T02:26:59.3087006Z   JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T02:26:59.3087523Z ##[endgroup]
2024-12-18T02:26:59.3118903Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/release/2.6/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/heads/release/2.6 --head-sha 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 --workflow-id 12383255690 --run-attempt 1 --job-id 34567153284 --job-name 'cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)'
2024-12-18T02:26:59.3450370Z ##[group]Run set -eux
2024-12-18T02:26:59.3450642Z set -eux
2024-12-18T02:26:59.3450863Z 
2024-12-18T02:26:59.3451111Z # TODO (huydhn): Implement this part
2024-12-18T02:26:59.3451474Z echo "runners=[]" >> "${GITHUB_OUTPUT}"
2024-12-18T02:26:59.3459677Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:26:59.3460048Z env:
2024-12-18T02:26:59.3460262Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:26:59.3460586Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:26:59.3461296Z   DOCKER_CONTAINER_ID: 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T02:26:59.3461926Z ##[endgroup]
2024-12-18T02:26:59.3490915Z + echo 'runners=[]'
2024-12-18T02:26:59.3522549Z ##[group]Run set -eux
2024-12-18T02:26:59.3522818Z set -eux
2024-12-18T02:26:59.3523040Z 
2024-12-18T02:26:59.3523293Z # TODO (huydhn): Implement this part
2024-12-18T02:26:59.3523674Z echo "dependencies={}" >> "${GITHUB_OUTPUT}"
2024-12-18T02:26:59.3533326Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:26:59.3533704Z env:
2024-12-18T02:26:59.3533919Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:26:59.3534247Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:26:59.3534793Z   DOCKER_CONTAINER_ID: 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T02:26:59.3535285Z ##[endgroup]
2024-12-18T02:26:59.3564532Z + echo 'dependencies={}'
2024-12-18T02:26:59.3594945Z ##[group]Run set -eux
2024-12-18T02:26:59.3595231Z set -eux
2024-12-18T02:26:59.3595479Z 
2024-12-18T02:26:59.3595755Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then
2024-12-18T02:26:59.3596193Z   echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping"
2024-12-18T02:26:59.3596662Z   # We don't want the job to fail if the directory doesn't exist
2024-12-18T02:26:59.3597048Z   exit 0
2024-12-18T02:26:59.3597273Z fi
2024-12-18T02:26:59.3597485Z 
2024-12-18T02:26:59.3597720Z if [[ "${DRY_RUN}" == "true" ]]; then
2024-12-18T02:26:59.3598178Z   python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \
2024-12-18T02:26:59.3598716Z     --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \
2024-12-18T02:26:59.3599134Z     --metadata "${BENCHMARK_METADATA}" \
2024-12-18T02:26:59.3599487Z     --runners "${RUNNER_INFO}" \
2024-12-18T02:26:59.3599828Z     --dependencies "${DEPENDENCIES}" \
2024-12-18T02:26:59.3600161Z     --dry-run
2024-12-18T02:26:59.3600408Z else
2024-12-18T02:26:59.3600784Z   python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \
2024-12-18T02:26:59.3601318Z     --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \
2024-12-18T02:26:59.3601732Z     --metadata "${BENCHMARK_METADATA}" \
2024-12-18T02:26:59.3602079Z     --runners "${RUNNER_INFO}" \
2024-12-18T02:26:59.3602420Z     --dependencies "${DEPENDENCIES}"
2024-12-18T02:26:59.3602732Z fi
2024-12-18T02:26:59.3610768Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:26:59.3611143Z env:
2024-12-18T02:26:59.3611362Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:26:59.3611695Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:26:59.3612245Z   DOCKER_CONTAINER_ID: 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T02:26:59.3612782Z   BENCHMARK_RESULTS_DIR: test/test-reports
2024-12-18T02:26:59.3613099Z   DRY_RUN: false
2024-12-18T02:26:59.3614459Z   BENCHMARK_METADATA: {"timestamp": 1734488819, "schema_version": "v3", "name": "cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/release/2.6", "head_sha": "0cdf8b1d09254cfda66191d1bd01e3041c3c76f7", "workflow_id": 12383255690, "run_attempt": 1, "job_id": 34567153284}
2024-12-18T02:26:59.3615875Z   RUNNER_INFO: []
2024-12-18T02:26:59.3616116Z   DEPENDENCIES: {}
2024-12-18T02:26:59.3616356Z ##[endgroup]
2024-12-18T02:26:59.3642589Z + [[ ! -d test/test-reports ]]
2024-12-18T02:26:59.3643002Z + [[ false == \t\r\u\e ]]
2024-12-18T02:26:59.3645739Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/release/2.6/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1734488819, "schema_version": "v3", "name": "cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/release/2.6", "head_sha": "0cdf8b1d09254cfda66191d1bd01e3041c3c76f7", "workflow_id": 12383255690, "run_attempt": 1, "job_id": 34567153284}' --runners '[]' --dependencies '{}'
2024-12-18T02:26:59.5789196Z INFO:root:Upload test/test-reports/inference_torchbench.json to s3://ossci-benchmarks/v3/pytorch/pytorch/12383255690/34567153284/inference_torchbench.json
2024-12-18T02:26:59.6196170Z INFO:botocore.credentials:Found credentials from IAM Role: gh-ci-github-action-runners-runner-role
2024-12-18T02:26:59.8767531Z INFO:root:Upload test/test-reports/inference_torchbench_graph_breaks.json to s3://ossci-benchmarks/v3/pytorch/pytorch/12383255690/34567153284/inference_torchbench_graph_breaks.json
2024-12-18T02:27:00.0244202Z INFO:root:Upload test/test-reports/training_torchbench.json to s3://ossci-benchmarks/v3/pytorch/pytorch/12383255690/34567153284/training_torchbench.json
2024-12-18T02:27:00.2137350Z INFO:root:Upload test/test-reports/training_torchbench_graph_breaks.json to s3://ossci-benchmarks/v3/pytorch/pytorch/12383255690/34567153284/training_torchbench_graph_breaks.json
2024-12-18T02:27:00.3938660Z ##[group]Run cat test/**/*_toprint.log || true
2024-12-18T02:27:00.3939058Z cat test/**/*_toprint.log || true
2024-12-18T02:27:00.3947996Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:27:00.3948370Z env:
2024-12-18T02:27:00.3948584Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:27:00.3948912Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:27:00.3949459Z   DOCKER_CONTAINER_ID: 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T02:27:00.3949937Z ##[endgroup]
2024-12-18T02:27:00.4039223Z cat: 'test/**/*_toprint.log': No such file or directory
2024-12-18T02:27:00.4076243Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2024-12-18T02:27:00.4076587Z kill "$MONITOR_SCRIPT_PID"
2024-12-18T02:27:00.4084466Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:27:00.4084837Z env:
2024-12-18T02:27:00.4085065Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:27:00.4085394Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:27:00.4085932Z   DOCKER_CONTAINER_ID: 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T02:27:00.4086455Z   MONITOR_SCRIPT_PID: 184603
2024-12-18T02:27:00.4086750Z ##[endgroup]
2024-12-18T02:27:00.4236779Z Prepare all required actions
2024-12-18T02:27:00.4237161Z Getting action download info
2024-12-18T02:27:00.5641406Z Download action repository 'actions/upload-artifact@v4' (SHA:6f51ac03b9356f520e9adb1b1b7802705f340c2b)
2024-12-18T02:27:00.9015445Z ##[group]Run ./.github/actions/upload-test-artifacts
2024-12-18T02:27:00.9015832Z with:
2024-12-18T02:27:00.9016269Z   file-suffix: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_34567153284
2024-12-18T02:27:00.9016777Z   s3-bucket: gha-artifacts
2024-12-18T02:27:00.9017041Z env:
2024-12-18T02:27:00.9017263Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:27:00.9017603Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:27:00.9018170Z   DOCKER_CONTAINER_ID: 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T02:27:00.9018649Z ##[endgroup]
2024-12-18T02:27:00.9047766Z ##[group]Run # Remove any previous test jsons if they exist
2024-12-18T02:27:00.9048246Z # Remove any previous test jsons if they exist
2024-12-18T02:27:00.9048614Z rm -f test-jsons-*.zip
2024-12-18T02:27:00.9049029Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json'
2024-12-18T02:27:00.9057819Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:27:00.9058193Z env:
2024-12-18T02:27:00.9058409Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:27:00.9058737Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:27:00.9059270Z   DOCKER_CONTAINER_ID: 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T02:27:00.9059936Z   FILE_SUFFIX: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_34567153284
2024-12-18T02:27:00.9060662Z ##[endgroup]
2024-12-18T02:27:00.9169642Z   adding: test/test-reports/inference_torchbench.json (deflated 99%)
2024-12-18T02:27:00.9207946Z   adding: test/test-reports/inference_torchbench_graph_breaks.json (deflated 98%)
2024-12-18T02:27:00.9249347Z   adding: test/test-reports/training_torchbench.json (deflated 99%)
2024-12-18T02:27:00.9365206Z   adding: test/test-reports/training_torchbench_graph_breaks.json (deflated 99%)
2024-12-18T02:27:00.9394680Z ##[group]Run # Remove any previous test reports if they exist
2024-12-18T02:27:00.9395153Z # Remove any previous test reports if they exist
2024-12-18T02:27:00.9395538Z rm -f test-reports-*.zip
2024-12-18T02:27:00.9396053Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv'
2024-12-18T02:27:00.9404340Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:27:00.9404715Z env:
2024-12-18T02:27:00.9404937Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:27:00.9405287Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:27:00.9405835Z   DOCKER_CONTAINER_ID: 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T02:27:00.9406507Z   FILE_SUFFIX: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_34567153284
2024-12-18T02:27:00.9406991Z ##[endgroup]
2024-12-18T02:27:00.9458681Z   adding: test/test-reports/inference_torchbench.csv (deflated 73%)
2024-12-18T02:27:00.9463714Z   adding: test/test-reports/inference_torchbench_graph_breaks.csv (deflated 92%)
2024-12-18T02:27:00.9465703Z   adding: test/test-reports/inference_torchbench_graph_break_deduped.csv (deflated 80%)
2024-12-18T02:27:00.9466307Z   adding: test/test-reports/training_torchbench.csv (deflated 63%)
2024-12-18T02:27:00.9473042Z   adding: test/test-reports/training_torchbench_graph_breaks.csv (deflated 97%)
2024-12-18T02:27:00.9474145Z   adding: test/test-reports/training_torchbench_graph_break_deduped.csv (deflated 77%)
2024-12-18T02:27:00.9503035Z ##[group]Run # Remove any previous usage logs if they exist
2024-12-18T02:27:00.9503487Z # Remove any previous usage logs if they exist
2024-12-18T02:27:00.9503854Z rm -f logs-*.zip
2024-12-18T02:27:00.9504313Z # this workflow is also run in bazel build test, but we dont generate usage reports for it
2024-12-18T02:27:00.9505051Z # so check to see if the file exists first
2024-12-18T02:27:00.9505417Z if [ -f 'usage_log.txt' ]; then
2024-12-18T02:27:00.9505785Z     zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt'
2024-12-18T02:27:00.9506125Z fi
2024-12-18T02:27:00.9506499Z if find "test/test-reports" -name "*.log" 2>/dev/null | grep -q .; then
2024-12-18T02:27:00.9507042Z     zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log'
2024-12-18T02:27:00.9507427Z fi
2024-12-18T02:27:00.9515415Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:27:00.9515789Z env:
2024-12-18T02:27:00.9516051Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:27:00.9516407Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:27:00.9516956Z   DOCKER_CONTAINER_ID: 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T02:27:00.9517626Z   FILE_SUFFIX: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_34567153284
2024-12-18T02:27:00.9518109Z ##[endgroup]
2024-12-18T02:27:00.9790138Z   adding: usage_log.txt (deflated 98%)
2024-12-18T02:27:00.9857569Z ##[group]Run # Remove any previous debugging artifacts if they exist
2024-12-18T02:27:00.9858080Z # Remove any previous debugging artifacts if they exist
2024-12-18T02:27:00.9858476Z rm -f debug-*.zip
2024-12-18T02:27:00.9858759Z if [ -d 'test/debug' ]; then
2024-12-18T02:27:00.9859109Z   zip -r "debug-${FILE_SUFFIX}.zip" test/debug
2024-12-18T02:27:00.9859430Z fi
2024-12-18T02:27:00.9867678Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:27:00.9868267Z env:
2024-12-18T02:27:00.9868500Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:27:00.9868852Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:27:00.9869399Z   DOCKER_CONTAINER_ID: 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T02:27:00.9870076Z   FILE_SUFFIX: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_34567153284
2024-12-18T02:27:00.9870557Z ##[endgroup]
2024-12-18T02:27:00.9956086Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-12-18T02:27:00.9956416Z with:
2024-12-18T02:27:00.9956645Z   s3-bucket: gha-artifacts
2024-12-18T02:27:00.9956957Z   s3-prefix: pytorch/pytorch/12383255690/1/artifact

2024-12-18T02:27:00.9957304Z   retention-days: 14
2024-12-18T02:27:00.9957559Z   if-no-files-found: warn
2024-12-18T02:27:00.9957833Z   path: test-jsons-*.zip
2024-12-18T02:27:00.9958089Z   name: artifact
2024-12-18T02:27:00.9958314Z   region: us-east-1
2024-12-18T02:27:00.9958543Z env:
2024-12-18T02:27:00.9958755Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:27:00.9959102Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:27:00.9959654Z   DOCKER_CONTAINER_ID: 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T02:27:00.9960130Z ##[endgroup]
2024-12-18T02:27:01.3589648Z NOTE: s3-prefix specified, ignoring name parameter
2024-12-18T02:27:01.3590122Z With the provided path, there will be 1 file uploaded
2024-12-18T02:27:01.3590572Z Uploading to s3 prefix: pytorch/pytorch/12383255690/1/artifact
2024-12-18T02:27:01.3649920Z Starting upload of test-jsons-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_34567153284.zip
2024-12-18T02:27:01.5491910Z Finished upload of test-jsons-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_34567153284.zip
2024-12-18T02:27:01.5784510Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-12-18T02:27:01.5784840Z with:
2024-12-18T02:27:01.5785056Z   s3-bucket: gha-artifacts
2024-12-18T02:27:01.5785377Z   s3-prefix: pytorch/pytorch/12383255690/1/artifact

2024-12-18T02:27:01.5785742Z   retention-days: 14
2024-12-18T02:27:01.5786001Z   if-no-files-found: error
2024-12-18T02:27:01.5786283Z   path: test-reports-*.zip
2024-12-18T02:27:01.5786542Z   name: artifact
2024-12-18T02:27:01.5786775Z   region: us-east-1
2024-12-18T02:27:01.5787004Z env:
2024-12-18T02:27:01.5787225Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:27:01.5787764Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:27:01.5788320Z   DOCKER_CONTAINER_ID: 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T02:27:01.5788802Z ##[endgroup]
2024-12-18T02:27:01.9121968Z NOTE: s3-prefix specified, ignoring name parameter
2024-12-18T02:27:01.9122587Z With the provided path, there will be 1 file uploaded
2024-12-18T02:27:01.9123196Z Uploading to s3 prefix: pytorch/pytorch/12383255690/1/artifact
2024-12-18T02:27:01.9181434Z Starting upload of test-reports-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_34567153284.zip
2024-12-18T02:27:02.1227707Z Finished upload of test-reports-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_34567153284.zip
2024-12-18T02:27:02.1519997Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-12-18T02:27:02.1520344Z with:
2024-12-18T02:27:02.1520578Z   s3-bucket: gha-artifacts
2024-12-18T02:27:02.1520903Z   s3-prefix: pytorch/pytorch/12383255690/1/artifact

2024-12-18T02:27:02.1521265Z   retention-days: 14
2024-12-18T02:27:02.1521518Z   if-no-files-found: ignore
2024-12-18T02:27:02.1521813Z   path: logs-*.zip
2024-12-18T02:27:02.1522051Z   name: artifact
2024-12-18T02:27:02.1522280Z   region: us-east-1
2024-12-18T02:27:02.1522513Z env:
2024-12-18T02:27:02.1522734Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:27:02.1523073Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:27:02.1523627Z   DOCKER_CONTAINER_ID: 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T02:27:02.1524118Z ##[endgroup]
2024-12-18T02:27:02.4814243Z NOTE: s3-prefix specified, ignoring name parameter
2024-12-18T02:27:02.4814814Z With the provided path, there will be 1 file uploaded
2024-12-18T02:27:02.4815649Z Uploading to s3 prefix: pytorch/pytorch/12383255690/1/artifact
2024-12-18T02:27:02.4874954Z Starting upload of logs-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_34567153284.zip
2024-12-18T02:27:02.6364117Z Finished upload of logs-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_34567153284.zip
2024-12-18T02:27:02.6660995Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-12-18T02:27:02.6661330Z with:
2024-12-18T02:27:02.6661555Z   s3-bucket: gha-artifacts
2024-12-18T02:27:02.6661875Z   s3-prefix: pytorch/pytorch/12383255690/1/artifact

2024-12-18T02:27:02.6662306Z   retention-days: 14
2024-12-18T02:27:02.6662564Z   if-no-files-found: ignore
2024-12-18T02:27:02.6662838Z   path: debug-*.zip
2024-12-18T02:27:02.6663065Z   name: artifact
2024-12-18T02:27:02.6663297Z   region: us-east-1
2024-12-18T02:27:02.6663523Z env:
2024-12-18T02:27:02.6663738Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:27:02.6664074Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:27:02.6664626Z   DOCKER_CONTAINER_ID: 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T02:27:02.6665114Z ##[endgroup]
2024-12-18T02:27:02.9901421Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded.
2024-12-18T02:27:03.0201578Z ##[group]Run # shellcheck disable=SC2156
2024-12-18T02:27:03.0201953Z # shellcheck disable=SC2156
2024-12-18T02:27:03.0202530Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2024-12-18T02:27:03.0212306Z shell: /usr/bin/bash -e {0}
2024-12-18T02:27:03.0212576Z env:
2024-12-18T02:27:03.0212791Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:27:03.0213126Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:27:03.0213669Z   DOCKER_CONTAINER_ID: 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T02:27:03.0214158Z ##[endgroup]
2024-12-18T02:27:03.2975053Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@release/2.6
2024-12-18T02:27:03.2975530Z with:
2024-12-18T02:27:03.2975744Z env:
2024-12-18T02:27:03.2976013Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:27:03.2976355Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:27:03.2976907Z   DOCKER_CONTAINER_ID: 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T02:27:03.2977401Z ##[endgroup]
2024-12-18T02:27:03.2998161Z ##[group]Run set -eou pipefail
2024-12-18T02:27:03.2998477Z set -eou pipefail
2024-12-18T02:27:03.2998745Z 
2024-12-18T02:27:03.2999105Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2024-12-18T02:27:03.2999571Z for _ in $(seq 1440); do
2024-12-18T02:27:03.2999901Z     # Break if no ssh session exists anymore
2024-12-18T02:27:03.3000248Z     if [ "$(who)" = "" ]; then
2024-12-18T02:27:03.3000544Z       break
2024-12-18T02:27:03.3000814Z     fi
2024-12-18T02:27:03.3001045Z     echo "."
2024-12-18T02:27:03.3001284Z     sleep 5
2024-12-18T02:27:03.3001525Z done
2024-12-18T02:27:03.3010195Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:27:03.3010572Z env:
2024-12-18T02:27:03.3010792Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:27:03.3011130Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:27:03.3011685Z   DOCKER_CONTAINER_ID: 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T02:27:03.3012175Z ##[endgroup]
2024-12-18T02:27:03.3042222Z Holding runner for 2 hours until all ssh sessions have logged out
2024-12-18T02:27:03.3117660Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2024-12-18T02:27:03.3118222Z # ignore expansion of "docker ps -q" since it could be empty
2024-12-18T02:27:03.3118646Z # shellcheck disable=SC2046
2024-12-18T02:27:03.3118990Z docker stop $(docker ps -q) || true
2024-12-18T02:27:03.3119528Z # Prune all of the docker images
2024-12-18T02:27:03.3119860Z docker system prune -af
2024-12-18T02:27:03.3129029Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:27:03.3129414Z env:
2024-12-18T02:27:03.3129630Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:27:03.3130286Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:27:03.3130873Z   DOCKER_CONTAINER_ID: 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T02:27:03.3131368Z ##[endgroup]
2024-12-18T02:27:04.1031219Z 5a407e2ba2ce
2024-12-18T02:27:08.2609049Z Deleted Containers:
2024-12-18T02:27:08.2609573Z 5a407e2ba2cee73e5e098b81ff050fa6a43e9c7919a5c419d096d7f4e0a70a8f
2024-12-18T02:27:08.2609899Z 
2024-12-18T02:27:17.7920190Z Deleted Images:
2024-12-18T02:27:17.7921227Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T02:27:17.7922829Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks@sha256:ff80d4c93d18e8ab7a2501f0e62b7c40cf97aa83fc9702e88b9c2d02cdcbecdb
2024-12-18T02:27:17.7923987Z deleted: sha256:67e93a8badaf799ceb664d7c453077c6c6785d56dc10c720be9358edeb91759b
2024-12-18T02:27:17.7924612Z deleted: sha256:7958cbbed07d0e7e404de5150d4669da383984a17de74e455563b7ecc409b2e8
2024-12-18T02:27:17.7925230Z deleted: sha256:dc6a8b10e3a8452ea47655ddad3b52816f12bcbe35298b3aa96aae8a0da9bc12
2024-12-18T02:27:17.7925874Z deleted: sha256:c92bf6e7cfbdecd0dfe86d452eac5dbe5aa5b2a6aa7ef2eaeffb21af2870cda7
2024-12-18T02:27:17.7926502Z deleted: sha256:f9597cb89d67c0c899e266adc5e5f04817474bad200712b3c256ed33c346b3e8
2024-12-18T02:27:17.7927113Z deleted: sha256:eb062308fcca86ab5a341555bf03af093d21f1d44dbe3dce6f9b7781b650b2f1
2024-12-18T02:27:17.7928173Z deleted: sha256:96f28e0a6369023dd4e75125d2454531ff686a14467f3e06d3b5c6da89893472
2024-12-18T02:27:17.7928803Z deleted: sha256:3e6e64b255956d1bf66c97393ae4325f7fc67414cd710e284c17190cb262c92d
2024-12-18T02:27:17.7929414Z deleted: sha256:a3c0a58b4f23e46b45dd1b385363731800042b3f04ca66f8867b99f3c9311f11
2024-12-18T02:27:17.7930375Z deleted: sha256:435e519973ffb9c85330ae8db1d1fb1fa75d673e2f27640c2ca7277c6352092b
2024-12-18T02:27:17.7931023Z deleted: sha256:69800681a8b6382d520ac4b8b8828bd5540a297961ddeb004e415bc46b33f80d
2024-12-18T02:27:17.7931642Z deleted: sha256:1e837ad662894f46afc56a73aa5ccb04064e4a974c9e3cad86ce9e5546b1d63d
2024-12-18T02:27:17.7932251Z deleted: sha256:688eccf2945022bd75f9f2782e2397ea54f1032c2435aa8f7dffa23cbcf2ae99
2024-12-18T02:27:17.7932859Z deleted: sha256:4855651bf4c3c8acad23029fda7d17f43bf32a69e8d575bb43f4491f0d653c1f
2024-12-18T02:27:17.7933474Z deleted: sha256:ca2d39907b7e10e3c7ae753b8bd230fac77c4e4b15980befd71173da8a36da93
2024-12-18T02:27:17.7934092Z deleted: sha256:d4f04a8342f272455eb816a413dc2dfcc75b9ebb9ef7f6fab32436621412ba6b
2024-12-18T02:27:17.7934708Z deleted: sha256:63ae01c22dc840b763ddc08bf4e145c14a04e0ff8a833b357ed45cb0962a8116
2024-12-18T02:27:17.7935340Z deleted: sha256:cf91e4695bf05d3607b7fa68e3ba3d6cc8762e6fac8773483f8116812f3d52c0
2024-12-18T02:27:17.7935943Z deleted: sha256:71baa7b72633376bb45367512dcb676c81b1e09964085a3151381530df7cdc28
2024-12-18T02:27:17.7936532Z deleted: sha256:68855433b69809d035da2a30cb22864ed1cd2795434571a0c77532de19344497
2024-12-18T02:27:17.7937116Z deleted: sha256:929280209b1f4e29860f386a519df32161cb608338d5338f24735201939dc91f
2024-12-18T02:27:17.7937716Z deleted: sha256:25133b63282efce5897b0e8fbf64d8b2a2625e15792dad95bdada277b78c777d
2024-12-18T02:27:17.7938322Z deleted: sha256:11fddfe713ea9ddd5acd8285319a606609b5c428111f15a019c8286e7164b9b4
2024-12-18T02:27:17.7939009Z deleted: sha256:b1464f3888e3b0a1694aa960a726584d01ae6b62df53fe8743d197d2d5051036
2024-12-18T02:27:17.7939603Z deleted: sha256:9627c2925d7a3936b439a6f519083e650d160bc2522d2a9244d6a1dfda02bc83
2024-12-18T02:27:17.7940204Z deleted: sha256:33880827ded8236cc91531c2f928d6de5c7cfb95a241da2653b7e8a8d1ea4881
2024-12-18T02:27:17.7941028Z deleted: sha256:6851b39610a99eb7ec6939a15e35657e144eb3e628a3856afa0a0adff49b8c2f
2024-12-18T02:27:17.7941646Z deleted: sha256:1a76c7f0af1e05689638a3792e8b9d4418648384aed1f56d3ccde5b0a9681d29
2024-12-18T02:27:17.7942307Z deleted: sha256:072f501c97aae0be55d1457426bc5f8238b0bb86eac9e92889b01924352d79a5
2024-12-18T02:27:17.7942903Z deleted: sha256:8277868848759cc86b200280dee8701d7e8328a811bf75330ed24aeaf490e69e
2024-12-18T02:27:17.7943503Z deleted: sha256:1da994d575d08f659512ea580648bfddd62020acc1aef2f7d4c6008d36a6f83f
2024-12-18T02:27:17.7944128Z deleted: sha256:ddf6d49d71559688ae65aeecccbbc58dd555887d4b3306d02a87d049114dd2f9
2024-12-18T02:27:17.7944746Z deleted: sha256:648d99cd03665c7b88aae7e3291f27daa65f496f7ab2c37f0ee62c72a4459151
2024-12-18T02:27:17.7945346Z deleted: sha256:6e57cd88598cddad4070d39467d8e1d4c387388f89d2970a55391653f9243e1a
2024-12-18T02:27:17.7945960Z deleted: sha256:9b0c5db353ac5d4e0995c2a70ae3c28390676afe60dbd11e4ade07c792eac4ce
2024-12-18T02:27:17.7946578Z deleted: sha256:8fa5311fe41b6caf697456e9678220a152bbf6a7645f4b7ff89ea9317ae04444
2024-12-18T02:27:17.7947181Z deleted: sha256:dc438d6465050b81bff4a8c5925e70c2167712dbab174c38f79b6a5d7af407af
2024-12-18T02:27:17.7947776Z deleted: sha256:29b8f0a42a1f3b944f2c54348348d173d322522b578e3d485428d67996f6adde
2024-12-18T02:27:17.7948365Z deleted: sha256:26231024334c7a704f9acba86430547a013e959014bfd7a31e26c6ed78521479
2024-12-18T02:27:17.7948964Z deleted: sha256:4166e9dd7564ae2f46ea2c3dcb8a8ef6bef9e06f4faf1742f4c6f21861967193
2024-12-18T02:27:17.7949579Z deleted: sha256:80a3113f256f146f0a2528ca69aa9244a1f9ba1ecdf54c3ef8ba0fb21507cbc4
2024-12-18T02:27:17.7950194Z deleted: sha256:e90bbba5d7c08e01c5bedb0a855cc5cdd371688e23994155674704963591296f
2024-12-18T02:27:17.7950806Z deleted: sha256:d3b3df431f230849965cc370c2f1853a81d78dd6eb3f7ac3bc3b4ca64aa9818a
2024-12-18T02:27:17.7951415Z deleted: sha256:7a33f9ca143b08c9f5d8c53974b2d6fe840739b5d2cd30694ba1c86db9393908
2024-12-18T02:27:17.7952201Z deleted: sha256:bad1d7dae968956b4a81369dedfbcf539fd5b61dd3051e2eccb5457174fe788c
2024-12-18T02:27:17.7952833Z deleted: sha256:08a420245ca6eed4aa5662d0fec04e7892165cdb3fd2f5b99428adbb4d6199c0
2024-12-18T02:27:17.7953450Z deleted: sha256:c3fda6b9a4d08a345824508dc5625f52e4d23163843d0caf337c60feda68a830
2024-12-18T02:27:17.7954071Z deleted: sha256:c74a8a3ca5e31670eed41e4bbb4a89f25c7c05dacabdf93cbd03984ead0330e2
2024-12-18T02:27:17.7954692Z deleted: sha256:4171d4c54045bd3035546906a7dd3b63c9ca748e0a34eeba27fe918f52cdcf85
2024-12-18T02:27:17.7955302Z deleted: sha256:f6a1d112657a4aeaca96ac2792a10ea09463dc3d633cb8572314814d5abf1ddb
2024-12-18T02:27:17.7955919Z deleted: sha256:af5d54d84b9c302254498eed8b0bbce1cdd705351bf5cc210d3f11c0deb269b3
2024-12-18T02:27:17.7956537Z deleted: sha256:45ae227353c6db8230b86fc1c4b8c50def31bd64895b879d91ea6e600669045d
2024-12-18T02:27:17.7957139Z deleted: sha256:6032ea27ad90071e3b75290e03fd0763a74155f513f356d21e56663a878b0ed9
2024-12-18T02:27:17.7957744Z deleted: sha256:d5aef8ace96787c48070530c09b68e4bcb23bc92a9fb0978220204f64bfbd97f
2024-12-18T02:27:17.7958363Z deleted: sha256:e3116153021ece2a3d29eccdce1f7e0f092316e889c1d534257789f6b5743d51
2024-12-18T02:27:17.7958975Z deleted: sha256:fb0f11460e5fa2e4895230ae84c97fe0552e9509dfbae60abf94e1b8a6563fb9
2024-12-18T02:27:17.7959936Z deleted: sha256:e262415e0abb361b284df9e05790ae27cc3a1ea7b8d4618e7eb3eca17cf1d18e
2024-12-18T02:27:17.7960640Z deleted: sha256:7f810eab57a6121f1a2aae664f7a5940d12f789661445ce3500ecd3b3688b46b
2024-12-18T02:27:17.7961346Z deleted: sha256:6ed71f55ff78520154c43468df2f05a92eaa28ee2aa2e28b8ea860b9ffb258ef
2024-12-18T02:27:17.7962111Z deleted: sha256:92a10d8d42a56a8516d0e9e14fec53263302fe6e6438ced263a3845680456dd7
2024-12-18T02:27:17.7962839Z deleted: sha256:ec67f912ff53e9b7164f48250da3bb4c1ffc94c3eb2c1c56c534a94fe55e3845
2024-12-18T02:27:17.7963536Z deleted: sha256:c85033e4f60ace56eaad80a8151b1bdd1619865b10a9ccee5638878b0a027c23
2024-12-18T02:27:17.7964471Z deleted: sha256:0e8680489f0a37851beb2cb5e42f0df129247a68b755b08a6367ea42180b3e7a
2024-12-18T02:27:17.7965312Z deleted: sha256:3a767fb38f6b1124e5fe9a230fdd383ccb188c38fe1d6ec252260676efe255c8
2024-12-18T02:27:17.7966022Z deleted: sha256:2cce59d2a43773cb7c6b83d239dab3a53659767f96bc0d47a70ee0d51cfa096f
2024-12-18T02:27:17.7966919Z deleted: sha256:51e50e75789ab5b4e40a66bb2f284e2da947e37c0e757c74d49fdeea5da9cbec
2024-12-18T02:27:17.7967628Z deleted: sha256:29e55e4feccba0660950ed92941d309527fe5f41bea609662d0e703b9c76b3a1
2024-12-18T02:27:17.7968400Z deleted: sha256:d15bee8fc455fcb45d16b0abc67f18c99ce1cf2c9e998ef5a100bf8373f2f8ab
2024-12-18T02:27:17.7969128Z deleted: sha256:363936db6a8869d44b6f0bee97a786b168093848e9c08c8042830fc7fccd956b
2024-12-18T02:27:17.7969827Z deleted: sha256:0f54d1f89ceaf71ded488007635a618fac1528bf3603d9415a593fe4926f8b2a
2024-12-18T02:27:17.7970551Z deleted: sha256:2f5796e9879cab7dac02bdbb178aec5301ac00029b38be8613333ac639c4dc3a
2024-12-18T02:27:17.7971336Z deleted: sha256:e68525d57f9da8f6c74b456d92869c4e60e451b88cb403addbc4ea917d8ab3ac
2024-12-18T02:27:17.7972069Z deleted: sha256:3386e25983944ce3b2ebdde10d840832d581a7dfafd1df66568822a18979e268
2024-12-18T02:27:17.7972834Z deleted: sha256:d3a036e6bd36f014b3a1540d5b848655ecdf72a20dc81c0da5f80430700cfa9a
2024-12-18T02:27:17.7973543Z deleted: sha256:96d0556708ff4ec0d5d727d5947d6cb29a9d6c16e1a1ee76a92dc85aa3ea8f25
2024-12-18T02:27:17.7974261Z deleted: sha256:f3c2fb3abccb3a33dbf204d3f0d8857537348e6c1d3cc87a431afe866435d41b
2024-12-18T02:27:17.7975071Z deleted: sha256:45cd6c9a41f12991ee60a89e775abb604b0892ef3936526c18184bbbf4de57da
2024-12-18T02:27:17.7975789Z deleted: sha256:b3cacd0508bbfd2fd668f30c539033e7c6b606bf669258c8e8a3cbf80bf050c6
2024-12-18T02:27:17.7976504Z deleted: sha256:de72ce930361a96c151a8a4725f8360f33df6c812b6ed6d3dd5b6e6710a5d2c0
2024-12-18T02:27:17.7977301Z deleted: sha256:2e5d223d2bfa0f47e628b631a540d20c284dabb380ad9ee4caec65bf8d0ecde2
2024-12-18T02:27:17.7978034Z deleted: sha256:14a85e604a0401093323d356b3bb7f75a8473abd712b4bf1a2ef01fa432b344c
2024-12-18T02:27:17.7978874Z deleted: sha256:15a95a1e8f47d7c18dcbf9a6e2fc530cddcb161c0093c663181f3e73dc9f599e
2024-12-18T02:27:17.7979576Z deleted: sha256:106e8431b412f51ccd75ea46a2d5cb4343b23273cbcf50188377cb93aa9a6d82
2024-12-18T02:27:17.7980031Z 
2024-12-18T02:27:17.7980181Z Total reclaimed space: 52.78GB
2024-12-18T02:27:17.8067828Z Post job cleanup.
2024-12-18T02:27:17.8127266Z Post job cleanup.
2024-12-18T02:27:17.8974457Z [command]/usr/bin/git version
2024-12-18T02:27:17.9042613Z git version 2.40.1
2024-12-18T02:27:17.9083302Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/f7151e00-414d-4ecd-b197-3916724ed3d4' before making global git config changes
2024-12-18T02:27:17.9084228Z Adding repository directory to the temporary git global config as a safe directory
2024-12-18T02:27:17.9087894Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-12-18T02:27:17.9120294Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2024-12-18T02:27:17.9150679Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2024-12-18T02:27:17.9517916Z Entering 'android/libs/fbjni'
2024-12-18T02:27:17.9588124Z Entering 'third_party/FP16'
2024-12-18T02:27:17.9658811Z Entering 'third_party/FXdiv'
2024-12-18T02:27:17.9733732Z Entering 'third_party/NNPACK'
2024-12-18T02:27:17.9805614Z Entering 'third_party/NVTX'
2024-12-18T02:27:17.9876280Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T02:27:17.9946899Z Entering 'third_party/XNNPACK'
2024-12-18T02:27:18.0031078Z Entering 'third_party/benchmark'
2024-12-18T02:27:18.0099681Z Entering 'third_party/composable_kernel'
2024-12-18T02:27:18.0174872Z Entering 'third_party/cpp-httplib'
2024-12-18T02:27:18.0243627Z Entering 'third_party/cpuinfo'
2024-12-18T02:27:18.0311880Z Entering 'third_party/cudnn_frontend'
2024-12-18T02:27:18.0382839Z Entering 'third_party/cutlass'
2024-12-18T02:27:18.0458453Z Entering 'third_party/eigen'
2024-12-18T02:27:18.0528963Z Entering 'third_party/fbgemm'
2024-12-18T02:27:18.0601283Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T02:27:18.0671015Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T02:27:18.0744113Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T02:27:18.0817940Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T02:27:18.0883868Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T02:27:18.0951302Z Entering 'third_party/flatbuffers'
2024-12-18T02:27:18.1024114Z Entering 'third_party/fmt'
2024-12-18T02:27:18.1094981Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T02:27:18.1164454Z Entering 'third_party/gloo'
2024-12-18T02:27:18.1234516Z Entering 'third_party/googletest'
2024-12-18T02:27:18.1303662Z Entering 'third_party/ideep'
2024-12-18T02:27:18.1371989Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T02:27:18.1448765Z Entering 'third_party/ittapi'
2024-12-18T02:27:18.1517435Z Entering 'third_party/kineto'
2024-12-18T02:27:18.1586402Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T02:27:18.1653484Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T02:27:18.1723106Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T02:27:18.1790210Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T02:27:18.1857667Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T02:27:18.1923327Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T02:27:18.1995120Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T02:27:18.2063527Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T02:27:18.2129159Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T02:27:18.2197374Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T02:27:18.2267902Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T02:27:18.2334151Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T02:27:18.2405940Z Entering 'third_party/mimalloc'
2024-12-18T02:27:18.2476443Z Entering 'third_party/nccl/nccl'
2024-12-18T02:27:18.2545420Z Entering 'third_party/nlohmann'
2024-12-18T02:27:18.2615308Z Entering 'third_party/onnx'
2024-12-18T02:27:18.2698532Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T02:27:18.2769461Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T02:27:18.2837835Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T02:27:18.2903291Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T02:27:18.2970975Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T02:27:18.3038465Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T02:27:18.3107404Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T02:27:18.3175375Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T02:27:18.3242957Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T02:27:18.3307056Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T02:27:18.3376026Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T02:27:18.3448756Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T02:27:18.3537709Z Entering 'third_party/pocketfft'
2024-12-18T02:27:18.3607463Z Entering 'third_party/protobuf'
2024-12-18T02:27:18.3678086Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T02:27:18.3745806Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T02:27:18.3816963Z Entering 'third_party/psimd'
2024-12-18T02:27:18.3888205Z Entering 'third_party/pthreadpool'
2024-12-18T02:27:18.3958386Z Entering 'third_party/pybind11'
2024-12-18T02:27:18.4028274Z Entering 'third_party/python-peachpy'
2024-12-18T02:27:18.4099269Z Entering 'third_party/sleef'
2024-12-18T02:27:18.4169847Z Entering 'third_party/tensorpipe'
2024-12-18T02:27:18.4242334Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T02:27:18.4308311Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T02:27:18.4375926Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T02:27:18.4446478Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T02:27:18.4511609Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T02:27:18.4605686Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2024-12-18T02:27:18.4627101Z http.https://github.com/.extraheader
2024-12-18T02:27:18.4637877Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2024-12-18T02:27:18.4669762Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2024-12-18T02:27:18.5023071Z Entering 'android/libs/fbjni'
2024-12-18T02:27:18.5067537Z http.https://github.com/.extraheader
2024-12-18T02:27:18.5111118Z Entering 'third_party/FP16'
2024-12-18T02:27:18.5155829Z http.https://github.com/.extraheader
2024-12-18T02:27:18.5199481Z Entering 'third_party/FXdiv'
2024-12-18T02:27:18.5248411Z http.https://github.com/.extraheader
2024-12-18T02:27:18.5291819Z Entering 'third_party/NNPACK'
2024-12-18T02:27:18.5340653Z http.https://github.com/.extraheader
2024-12-18T02:27:18.5383151Z Entering 'third_party/NVTX'
2024-12-18T02:27:18.5432322Z http.https://github.com/.extraheader
2024-12-18T02:27:18.5477172Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T02:27:18.5521281Z http.https://github.com/.extraheader
2024-12-18T02:27:18.5565991Z Entering 'third_party/XNNPACK'
2024-12-18T02:27:18.5609541Z http.https://github.com/.extraheader
2024-12-18T02:27:18.5668821Z Entering 'third_party/benchmark'
2024-12-18T02:27:18.5713673Z http.https://github.com/.extraheader
2024-12-18T02:27:18.5758866Z Entering 'third_party/composable_kernel'
2024-12-18T02:27:18.5803026Z http.https://github.com/.extraheader
2024-12-18T02:27:18.5853297Z Entering 'third_party/cpp-httplib'
2024-12-18T02:27:18.5900165Z http.https://github.com/.extraheader
2024-12-18T02:27:18.5942907Z Entering 'third_party/cpuinfo'
2024-12-18T02:27:18.5987089Z http.https://github.com/.extraheader
2024-12-18T02:27:18.6031008Z Entering 'third_party/cudnn_frontend'
2024-12-18T02:27:18.6078819Z http.https://github.com/.extraheader
2024-12-18T02:27:18.6122815Z Entering 'third_party/cutlass'
2024-12-18T02:27:18.6170400Z http.https://github.com/.extraheader
2024-12-18T02:27:18.6221951Z Entering 'third_party/eigen'
2024-12-18T02:27:18.6268275Z http.https://github.com/.extraheader
2024-12-18T02:27:18.6313436Z Entering 'third_party/fbgemm'
2024-12-18T02:27:18.6357158Z http.https://github.com/.extraheader
2024-12-18T02:27:18.6399961Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T02:27:18.6442808Z http.https://github.com/.extraheader
2024-12-18T02:27:18.6487658Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T02:27:18.6530614Z http.https://github.com/.extraheader
2024-12-18T02:27:18.6573030Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T02:27:18.6615092Z http.https://github.com/.extraheader
2024-12-18T02:27:18.6667334Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T02:27:18.6710843Z http.https://github.com/.extraheader
2024-12-18T02:27:18.6753318Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T02:27:18.6796321Z http.https://github.com/.extraheader
2024-12-18T02:27:18.6842151Z Entering 'third_party/flatbuffers'
2024-12-18T02:27:18.6887431Z http.https://github.com/.extraheader
2024-12-18T02:27:18.6935987Z Entering 'third_party/fmt'
2024-12-18T02:27:18.6980919Z http.https://github.com/.extraheader
2024-12-18T02:27:18.7025089Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T02:27:18.7070283Z http.https://github.com/.extraheader
2024-12-18T02:27:18.7115213Z Entering 'third_party/gloo'
2024-12-18T02:27:18.7160881Z http.https://github.com/.extraheader
2024-12-18T02:27:18.7205464Z Entering 'third_party/googletest'
2024-12-18T02:27:18.7250847Z http.https://github.com/.extraheader
2024-12-18T02:27:18.7295676Z Entering 'third_party/ideep'
2024-12-18T02:27:18.7343117Z http.https://github.com/.extraheader
2024-12-18T02:27:18.7385746Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T02:27:18.7427160Z http.https://github.com/.extraheader
2024-12-18T02:27:18.7478000Z Entering 'third_party/ittapi'
2024-12-18T02:27:18.7522155Z http.https://github.com/.extraheader
2024-12-18T02:27:18.7566372Z Entering 'third_party/kineto'
2024-12-18T02:27:18.7610199Z http.https://github.com/.extraheader
2024-12-18T02:27:18.7654298Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T02:27:18.7700658Z http.https://github.com/.extraheader
2024-12-18T02:27:18.7743857Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T02:27:18.7786626Z http.https://github.com/.extraheader
2024-12-18T02:27:18.7831119Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T02:27:18.7873851Z http.https://github.com/.extraheader
2024-12-18T02:27:18.7916652Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T02:27:18.7959078Z http.https://github.com/.extraheader
2024-12-18T02:27:18.8009529Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T02:27:18.8053892Z http.https://github.com/.extraheader
2024-12-18T02:27:18.8097191Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T02:27:18.8141511Z http.https://github.com/.extraheader
2024-12-18T02:27:18.8188901Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T02:27:18.8234387Z http.https://github.com/.extraheader
2024-12-18T02:27:18.8277768Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T02:27:18.8321969Z http.https://github.com/.extraheader
2024-12-18T02:27:18.8366945Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T02:27:18.8410768Z http.https://github.com/.extraheader
2024-12-18T02:27:18.8456090Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T02:27:18.8499228Z http.https://github.com/.extraheader
2024-12-18T02:27:18.8547741Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T02:27:18.8590910Z http.https://github.com/.extraheader
2024-12-18T02:27:18.8634785Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T02:27:18.8677923Z http.https://github.com/.extraheader
2024-12-18T02:27:18.8723986Z Entering 'third_party/mimalloc'
2024-12-18T02:27:18.8769009Z http.https://github.com/.extraheader
2024-12-18T02:27:18.8813604Z Entering 'third_party/nccl/nccl'
2024-12-18T02:27:18.8860155Z http.https://github.com/.extraheader
2024-12-18T02:27:18.8903325Z Entering 'third_party/nlohmann'
2024-12-18T02:27:18.8952282Z http.https://github.com/.extraheader
2024-12-18T02:27:18.8997338Z Entering 'third_party/onnx'
2024-12-18T02:27:18.9041421Z http.https://github.com/.extraheader
2024-12-18T02:27:18.9099845Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T02:27:18.9143491Z http.https://github.com/.extraheader
2024-12-18T02:27:18.9191244Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T02:27:18.9236243Z http.https://github.com/.extraheader
2024-12-18T02:27:18.9281910Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T02:27:18.9324473Z http.https://github.com/.extraheader
2024-12-18T02:27:18.9367696Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T02:27:18.9409969Z http.https://github.com/.extraheader
2024-12-18T02:27:18.9453054Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T02:27:18.9495481Z http.https://github.com/.extraheader
2024-12-18T02:27:18.9538820Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T02:27:18.9580763Z http.https://github.com/.extraheader
2024-12-18T02:27:18.9625528Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T02:27:18.9669625Z http.https://github.com/.extraheader
2024-12-18T02:27:18.9714244Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T02:27:18.9757227Z http.https://github.com/.extraheader
2024-12-18T02:27:18.9799552Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T02:27:18.9842715Z http.https://github.com/.extraheader
2024-12-18T02:27:18.9885003Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T02:27:18.9927258Z http.https://github.com/.extraheader
2024-12-18T02:27:18.9972690Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T02:27:19.0016558Z http.https://github.com/.extraheader
2024-12-18T02:27:19.0064640Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T02:27:19.0108366Z http.https://github.com/.extraheader
2024-12-18T02:27:19.0175570Z Entering 'third_party/pocketfft'
2024-12-18T02:27:19.0223171Z http.https://github.com/.extraheader
2024-12-18T02:27:19.0267578Z Entering 'third_party/protobuf'
2024-12-18T02:27:19.0312538Z http.https://github.com/.extraheader
2024-12-18T02:27:19.0359923Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T02:27:19.0403471Z http.https://github.com/.extraheader
2024-12-18T02:27:19.0447020Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T02:27:19.0489815Z http.https://github.com/.extraheader
2024-12-18T02:27:19.0539032Z Entering 'third_party/psimd'
2024-12-18T02:27:19.0583660Z http.https://github.com/.extraheader
2024-12-18T02:27:19.0627932Z Entering 'third_party/pthreadpool'
2024-12-18T02:27:19.0673570Z http.https://github.com/.extraheader
2024-12-18T02:27:19.0716828Z Entering 'third_party/pybind11'
2024-12-18T02:27:19.0761862Z http.https://github.com/.extraheader
2024-12-18T02:27:19.0807698Z Entering 'third_party/python-peachpy'
2024-12-18T02:27:19.0852414Z http.https://github.com/.extraheader
2024-12-18T02:27:19.0895292Z Entering 'third_party/sleef'
2024-12-18T02:27:19.0939882Z http.https://github.com/.extraheader
2024-12-18T02:27:19.0984760Z Entering 'third_party/tensorpipe'
2024-12-18T02:27:19.1028612Z http.https://github.com/.extraheader
2024-12-18T02:27:19.1073066Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T02:27:19.1116383Z http.https://github.com/.extraheader
2024-12-18T02:27:19.1160057Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T02:27:19.1202882Z http.https://github.com/.extraheader
2024-12-18T02:27:19.1248744Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T02:27:19.1291316Z http.https://github.com/.extraheader
2024-12-18T02:27:19.1334643Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T02:27:19.1377252Z http.https://github.com/.extraheader
2024-12-18T02:27:19.1419060Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T02:27:19.1469177Z http.https://github.com/.extraheader
2024-12-18T02:27:19.1631918Z A job completed hook has been configured by the self-hosted runner administrator
2024-12-18T02:27:19.1661152Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh'
2024-12-18T02:27:19.1669091Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:27:19.1669475Z ##[endgroup]
2024-12-18T02:27:26.5342857Z Cleaning up orphan processes