2024-12-18T00:18:00.2632814Z Current runner version: '2.321.0'
2024-12-18T00:18:00.2640000Z Runner name: 'i-029fac8e3f5b7f275'
2024-12-18T00:18:00.2641096Z Runner group name: 'Default'
2024-12-18T00:18:00.2642312Z Machine name: 'ip-10-0-71-248'
2024-12-18T00:18:00.2647756Z ##[group]GITHUB_TOKEN Permissions
2024-12-18T00:18:00.2650481Z Actions: read
2024-12-18T00:18:00.2651336Z Attestations: read
2024-12-18T00:18:00.2652052Z Checks: read
2024-12-18T00:18:00.2652750Z Contents: read
2024-12-18T00:18:00.2653612Z Deployments: read
2024-12-18T00:18:00.2654354Z Discussions: read
2024-12-18T00:18:00.2655007Z Issues: read
2024-12-18T00:18:00.2655779Z Metadata: read
2024-12-18T00:18:00.2656497Z Packages: read
2024-12-18T00:18:00.2657230Z Pages: read
2024-12-18T00:18:00.2657958Z PullRequests: read
2024-12-18T00:18:00.2658657Z RepositoryProjects: read
2024-12-18T00:18:00.2659493Z SecurityEvents: read
2024-12-18T00:18:00.2660274Z Statuses: read
2024-12-18T00:18:00.2660965Z ##[endgroup]
2024-12-18T00:18:00.2664443Z Secret source: Actions
2024-12-18T00:18:00.2665599Z Prepare workflow directory
2024-12-18T00:18:00.3522916Z Prepare all required actions
2024-12-18T00:18:00.3562947Z Getting action download info
2024-12-18T00:18:00.5722384Z Download action repository 'pytorch/test-infra@release/2.6' (SHA:eb0adf5a84668865394af69e26428b32c8105c1c)
2024-12-18T00:18:02.8284197Z Download action repository 'pytorch/pytorch@release/2.6' (SHA:0cdf8b1d09254cfda66191d1bd01e3041c3c76f7)
2024-12-18T00:18:20.3743762Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0)
2024-12-18T00:18:20.6222637Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2024-12-18T00:18:20.9348980Z Getting action download info
2024-12-18T00:18:21.0807307Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c)
2024-12-18T00:18:21.3536484Z Getting action download info
2024-12-18T00:18:21.4770866Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e)
2024-12-18T00:18:21.6904605Z Getting action download info
2024-12-18T00:18:21.8220832Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482)
2024-12-18T00:18:22.0171396Z Getting action download info
2024-12-18T00:18:22.1577793Z Download action repository 'pytorch/test-infra@main' (SHA:a07505a74641a4ff5123d635defac481ef28ef1e)
2024-12-18T00:18:24.0205329Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/release/2.6 (0cdf8b1d09254cfda66191d1bd01e3041c3c76f7)
2024-12-18T00:18:24.0207243Z ##[group] Inputs
2024-12-18T00:18:24.0207618Z   build-environment: linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-12-18T00:18:24.0209575Z   test-matrix: {"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}
2024-12-18T00:18:24.0211977Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:18:24.0212873Z   sync-tag: 
2024-12-18T00:18:24.0213600Z   timeout-minutes: 240
2024-12-18T00:18:24.0213864Z   use-gha: 
2024-12-18T00:18:24.0214102Z   dashboard-tag: 
2024-12-18T00:18:24.0214363Z   s3-bucket: gha-artifacts
2024-12-18T00:18:24.0214649Z   aws-role-to-assume: 
2024-12-18T00:18:24.0215264Z   disable-monitor: false
2024-12-18T00:18:24.0215548Z ##[endgroup]
2024-12-18T00:18:24.0216301Z Complete job name: cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:18:24.0751008Z A job started hook has been configured by the self-hosted runner administrator
2024-12-18T00:18:24.0851522Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh'
2024-12-18T00:18:24.0862762Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:18:24.0863376Z ##[endgroup]
2024-12-18T00:18:25.1483867Z Runner Type: linux.g5.4xlarge.nvidia.gpu
2024-12-18T00:18:25.1484360Z Instance Type: g5.4xlarge
2024-12-18T00:18:25.1484635Z AMI Name: unknown
2024-12-18T00:18:25.1522170Z AMI ID: ami-0fff1b9a61dec8a5f
2024-12-18T00:18:30.9290817Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@release/2.6
2024-12-18T00:18:30.9291262Z with:
2024-12-18T00:18:30.9291865Z   github-secret: ***
2024-12-18T00:18:30.9292562Z   instructions: All testing is done inside the container, to start an interactive session run:
  docker exec -it $(docker container ps --format '{{.ID}}') bash

2024-12-18T00:18:30.9293308Z   activate-with-label: false
2024-12-18T00:18:30.9293608Z   label: with-ssh
2024-12-18T00:18:30.9293865Z   remove-existing-keys: true
2024-12-18T00:18:30.9294154Z   fail-silently: true
2024-12-18T00:18:30.9294418Z env:
2024-12-18T00:18:30.9294640Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:18:30.9294914Z ##[endgroup]
2024-12-18T00:18:31.0514949Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info.
2024-12-18T00:18:31.0516505Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys
2024-12-18T00:18:31.0688516Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@release/2.6
2024-12-18T00:18:31.0688979Z with:
2024-12-18T00:18:31.0689204Z   no-sudo: true
2024-12-18T00:18:31.0689456Z   submodules: recursive
2024-12-18T00:18:31.0689733Z   fetch-depth: 0
2024-12-18T00:18:31.0689961Z env:
2024-12-18T00:18:31.0690192Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:18:31.0690471Z ##[endgroup]
2024-12-18T00:18:31.0768687Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T00:18:31.0769655Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T00:18:31.0781527Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:18:31.0781922Z env:
2024-12-18T00:18:31.0782145Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:18:31.0782415Z ##[endgroup]
2024-12-18T00:18:31.1392461Z ##[group]Run retry () {
2024-12-18T00:18:31.1392790Z retry () {
2024-12-18T00:18:31.1393162Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2024-12-18T00:18:31.1393613Z }
2024-12-18T00:18:31.1393859Z echo "${GITHUB_WORKSPACE}"
2024-12-18T00:18:31.1394201Z if [ -z "${NO_SUDO}" ]; then
2024-12-18T00:18:31.1394563Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2024-12-18T00:18:31.1394899Z else
2024-12-18T00:18:31.1395181Z   retry rm -rf "${GITHUB_WORKSPACE}"
2024-12-18T00:18:31.1395509Z fi
2024-12-18T00:18:31.1395757Z mkdir "${GITHUB_WORKSPACE}"
2024-12-18T00:18:31.1404873Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:18:31.1405255Z env:
2024-12-18T00:18:31.1405479Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:18:31.1405751Z   NO_SUDO: true
2024-12-18T00:18:31.1405991Z ##[endgroup]
2024-12-18T00:18:31.1437285Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-12-18T00:18:31.1532023Z ##[group]Run malfet/checkout@silent-checkout
2024-12-18T00:18:31.1532411Z with:
2024-12-18T00:18:31.1532677Z   ref: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:18:31.1533026Z   fetch-depth: 0
2024-12-18T00:18:31.1533287Z   submodules: recursive
2024-12-18T00:18:31.1533558Z   quiet-checkout: true
2024-12-18T00:18:31.1533844Z   repository: pytorch/pytorch
2024-12-18T00:18:31.1534227Z   token: ***
2024-12-18T00:18:31.1534465Z   ssh-strict: true
2024-12-18T00:18:31.1534942Z   persist-credentials: true
2024-12-18T00:18:31.1535234Z   clean: true
2024-12-18T00:18:31.1535490Z   sparse-checkout-cone-mode: true
2024-12-18T00:18:31.1535804Z   lfs: false
2024-12-18T00:18:31.1536048Z   set-safe-directory: true
2024-12-18T00:18:31.1536324Z env:
2024-12-18T00:18:31.1536553Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:18:31.1536830Z ##[endgroup]
2024-12-18T00:18:31.2452645Z Syncing repository: pytorch/pytorch
2024-12-18T00:18:31.2453907Z ##[group]Getting Git version info
2024-12-18T00:18:31.2454423Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2024-12-18T00:18:31.2455115Z [command]/usr/bin/git version
2024-12-18T00:18:31.2455422Z git version 2.40.1
2024-12-18T00:18:31.2475726Z ##[endgroup]
2024-12-18T00:18:31.2485556Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/05cf2692-8fbd-4d16-a126-4e4904b483ef/.gitconfig'
2024-12-18T00:18:31.2494749Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/05cf2692-8fbd-4d16-a126-4e4904b483ef' before making global git config changes
2024-12-18T00:18:31.2495690Z Adding repository directory to the temporary git global config as a safe directory
2024-12-18T00:18:31.2499386Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-12-18T00:18:31.2536202Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2024-12-18T00:18:31.2539699Z ##[group]Initializing the repository
2024-12-18T00:18:31.2542456Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-12-18T00:18:31.2572525Z hint: Using 'master' as the name for the initial branch. This default branch name
2024-12-18T00:18:31.2573189Z hint: is subject to change. To configure the initial branch name to use in all
2024-12-18T00:18:31.2573737Z hint: of your new repositories, which will suppress this warning, call:
2024-12-18T00:18:31.2574149Z hint: 
2024-12-18T00:18:31.2574455Z hint: 	git config --global init.defaultBranch <name>
2024-12-18T00:18:31.2574818Z hint: 
2024-12-18T00:18:31.2575154Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2024-12-18T00:18:31.2575719Z hint: 'development'. The just-created branch can be renamed via this command:
2024-12-18T00:18:31.2576154Z hint: 
2024-12-18T00:18:31.2576384Z hint: 	git branch -m <name>
2024-12-18T00:18:31.2578613Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2024-12-18T00:18:31.2586898Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2024-12-18T00:18:31.2615699Z ##[endgroup]
2024-12-18T00:18:31.2616148Z ##[group]Disabling automatic garbage collection
2024-12-18T00:18:31.2618673Z [command]/usr/bin/git config --local gc.auto 0
2024-12-18T00:18:31.2646458Z ##[endgroup]
2024-12-18T00:18:31.2646869Z ##[group]Setting up auth
2024-12-18T00:18:31.2651224Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2024-12-18T00:18:31.2679291Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2024-12-18T00:18:31.3019451Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2024-12-18T00:18:31.3047732Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2024-12-18T00:18:31.3390866Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2024-12-18T00:18:31.3438134Z ##[endgroup]
2024-12-18T00:18:31.3438871Z ##[group]Fetching the repository
2024-12-18T00:18:31.3444351Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2024-12-18T00:18:34.4028047Z remote: Enumerating objects: 1056571        
2024-12-18T00:18:34.4028788Z remote: Enumerating objects: 1057277, done.        
2024-12-18T00:18:34.4030589Z remote: Counting objects:   0% (1/706)        
2024-12-18T00:18:34.4031069Z remote: Counting objects:   1% (8/706)        
2024-12-18T00:18:34.4031738Z remote: Counting objects:   2% (15/706)        
2024-12-18T00:18:34.4032301Z remote: Counting objects:   3% (22/706)        
2024-12-18T00:18:34.4033720Z remote: Counting objects:   4% (29/706)        
2024-12-18T00:18:34.4034374Z remote: Counting objects:   5% (36/706)        
2024-12-18T00:18:34.4034867Z remote: Counting objects:   6% (43/706)        
2024-12-18T00:18:34.4035438Z remote: Counting objects:   7% (50/706)        
2024-12-18T00:18:34.4036018Z remote: Counting objects:   8% (57/706)        
2024-12-18T00:18:34.4036544Z remote: Counting objects:   9% (64/706)        
2024-12-18T00:18:34.4037078Z remote: Counting objects:  10% (71/706)        
2024-12-18T00:18:34.4037464Z remote: Counting objects:  11% (78/706)        
2024-12-18T00:18:34.4037840Z remote: Counting objects:  12% (85/706)        
2024-12-18T00:18:34.4038219Z remote: Counting objects:  13% (92/706)        
2024-12-18T00:18:34.4038570Z remote: Counting objects:  14% (99/706)        
2024-12-18T00:18:34.4038933Z remote: Counting objects:  15% (106/706)        
2024-12-18T00:18:34.4039353Z remote: Counting objects:  16% (113/706)        
2024-12-18T00:18:34.4039719Z remote: Counting objects:  17% (121/706)        
2024-12-18T00:18:34.4040349Z remote: Counting objects:  18% (128/706)        
2024-12-18T00:18:34.4040719Z remote: Counting objects:  19% (135/706)        
2024-12-18T00:18:34.4041083Z remote: Counting objects:  20% (142/706)        
2024-12-18T00:18:34.4041542Z remote: Counting objects:  21% (149/706)        
2024-12-18T00:18:34.4041908Z remote: Counting objects:  22% (156/706)        
2024-12-18T00:18:34.4042269Z remote: Counting objects:  23% (163/706)        
2024-12-18T00:18:34.4042631Z remote: Counting objects:  24% (170/706)        
2024-12-18T00:18:34.4042994Z remote: Counting objects:  25% (177/706)        
2024-12-18T00:18:34.4043356Z remote: Counting objects:  26% (184/706)        
2024-12-18T00:18:34.4043832Z remote: Counting objects:  27% (191/706)        
2024-12-18T00:18:34.4044194Z remote: Counting objects:  28% (198/706)        
2024-12-18T00:18:34.4044556Z remote: Counting objects:  29% (205/706)        
2024-12-18T00:18:34.4044915Z remote: Counting objects:  30% (212/706)        
2024-12-18T00:18:34.4045276Z remote: Counting objects:  31% (219/706)        
2024-12-18T00:18:34.4045637Z remote: Counting objects:  32% (226/706)        
2024-12-18T00:18:34.4045998Z remote: Counting objects:  33% (233/706)        
2024-12-18T00:18:34.4046359Z remote: Counting objects:  34% (241/706)        
2024-12-18T00:18:34.4046721Z remote: Counting objects:  35% (248/706)        
2024-12-18T00:18:34.4047081Z remote: Counting objects:  36% (255/706)        
2024-12-18T00:18:34.4047434Z remote: Counting objects:  37% (262/706)        
2024-12-18T00:18:34.4047803Z remote: Counting objects:  38% (269/706)        
2024-12-18T00:18:34.4048164Z remote: Counting objects:  39% (276/706)        
2024-12-18T00:18:34.4048528Z remote: Counting objects:  40% (283/706)        
2024-12-18T00:18:34.4048892Z remote: Counting objects:  41% (290/706)        
2024-12-18T00:18:34.4049247Z remote: Counting objects:  42% (297/706)        
2024-12-18T00:18:34.4049610Z remote: Counting objects:  43% (304/706)        
2024-12-18T00:18:34.4049977Z remote: Counting objects:  44% (311/706)        
2024-12-18T00:18:34.4050346Z remote: Counting objects:  45% (318/706)        
2024-12-18T00:18:34.4050710Z remote: Counting objects:  46% (325/706)        
2024-12-18T00:18:34.4051064Z remote: Counting objects:  47% (332/706)        
2024-12-18T00:18:34.4051426Z remote: Counting objects:  48% (339/706)        
2024-12-18T00:18:34.4051786Z remote: Counting objects:  49% (346/706)        
2024-12-18T00:18:34.4052149Z remote: Counting objects:  50% (353/706)        
2024-12-18T00:18:34.4052618Z remote: Counting objects:  51% (361/706)        
2024-12-18T00:18:34.4052981Z remote: Counting objects:  52% (368/706)        
2024-12-18T00:18:34.4053335Z remote: Counting objects:  53% (375/706)        
2024-12-18T00:18:34.4053703Z remote: Counting objects:  54% (382/706)        
2024-12-18T00:18:34.4054062Z remote: Counting objects:  55% (389/706)        
2024-12-18T00:18:34.4054427Z remote: Counting objects:  56% (396/706)        
2024-12-18T00:18:34.4054791Z remote: Counting objects:  57% (403/706)        
2024-12-18T00:18:34.4055146Z remote: Counting objects:  58% (410/706)        
2024-12-18T00:18:34.4055508Z remote: Counting objects:  59% (417/706)        
2024-12-18T00:18:34.4055871Z remote: Counting objects:  60% (424/706)        
2024-12-18T00:18:34.4056230Z remote: Counting objects:  61% (431/706)        
2024-12-18T00:18:34.4056588Z remote: Counting objects:  62% (438/706)        
2024-12-18T00:18:34.4056967Z remote: Counting objects:  63% (445/706)        
2024-12-18T00:18:34.4057338Z remote: Counting objects:  64% (452/706)        
2024-12-18T00:18:34.4057692Z remote: Counting objects:  65% (459/706)        
2024-12-18T00:18:34.4058058Z remote: Counting objects:  66% (466/706)        
2024-12-18T00:18:34.4058415Z remote: Counting objects:  67% (474/706)        
2024-12-18T00:18:34.4058775Z remote: Counting objects:  68% (481/706)        
2024-12-18T00:18:34.4059139Z remote: Counting objects:  69% (488/706)        
2024-12-18T00:18:34.4059493Z remote: Counting objects:  70% (495/706)        
2024-12-18T00:18:34.4059954Z remote: Counting objects:  71% (502/706)        
2024-12-18T00:18:34.4060316Z remote: Counting objects:  72% (509/706)        
2024-12-18T00:18:34.4060676Z remote: Counting objects:  73% (516/706)        
2024-12-18T00:18:34.4061037Z remote: Counting objects:  74% (523/706)        
2024-12-18T00:18:34.4061448Z remote: Counting objects:  75% (530/706)        
2024-12-18T00:18:34.4061806Z remote: Counting objects:  76% (537/706)        
2024-12-18T00:18:34.4062163Z remote: Counting objects:  77% (544/706)        
2024-12-18T00:18:34.4062529Z remote: Counting objects:  78% (551/706)        
2024-12-18T00:18:34.4062889Z remote: Counting objects:  79% (558/706)        
2024-12-18T00:18:34.4063247Z remote: Counting objects:  80% (565/706)        
2024-12-18T00:18:34.4063610Z remote: Counting objects:  81% (572/706)        
2024-12-18T00:18:34.4063971Z remote: Counting objects:  82% (579/706)        
2024-12-18T00:18:34.4064331Z remote: Counting objects:  83% (586/706)        
2024-12-18T00:18:34.4064697Z remote: Counting objects:  84% (594/706)        
2024-12-18T00:18:34.4065061Z remote: Counting objects:  85% (601/706)        
2024-12-18T00:18:34.4065417Z remote: Counting objects:  86% (608/706)        
2024-12-18T00:18:34.4065779Z remote: Counting objects:  87% (615/706)        
2024-12-18T00:18:34.4066143Z remote: Counting objects:  88% (622/706)        
2024-12-18T00:18:34.4066509Z remote: Counting objects:  89% (629/706)        
2024-12-18T00:18:34.4067288Z remote: Counting objects:  90% (636/706)        
2024-12-18T00:18:34.4067827Z remote: Counting objects:  91% (643/706)        
2024-12-18T00:18:34.4068194Z remote: Counting objects:  92% (650/706)        
2024-12-18T00:18:34.4068561Z remote: Counting objects:  93% (657/706)        
2024-12-18T00:18:34.4068928Z remote: Counting objects:  94% (664/706)        
2024-12-18T00:18:34.4069352Z remote: Counting objects:  95% (671/706)        
2024-12-18T00:18:34.4069715Z remote: Counting objects:  96% (678/706)        
2024-12-18T00:18:34.4070081Z remote: Counting objects:  97% (685/706)        
2024-12-18T00:18:34.4070447Z remote: Counting objects:  98% (692/706)        
2024-12-18T00:18:34.4070810Z remote: Counting objects:  99% (699/706)        
2024-12-18T00:18:34.4071172Z remote: Counting objects: 100% (706/706)        
2024-12-18T00:18:34.4071563Z remote: Counting objects: 100% (706/706), done.        
2024-12-18T00:18:34.4192816Z remote: Compressing objects:   0% (1/300)        
2024-12-18T00:18:34.4285616Z remote: Compressing objects:   1% (3/300)        
2024-12-18T00:18:34.4338401Z remote: Compressing objects:   2% (6/300)        
2024-12-18T00:18:34.4387101Z remote: Compressing objects:   3% (9/300)        
2024-12-18T00:18:34.4462906Z remote: Compressing objects:   4% (12/300)        
2024-12-18T00:18:34.4872657Z remote: Compressing objects:   5% (15/300)        
2024-12-18T00:18:34.5354402Z remote: Compressing objects:   6% (18/300)        
2024-12-18T00:18:34.5901083Z remote: Compressing objects:   7% (21/300)        
2024-12-18T00:18:34.6572545Z remote: Compressing objects:   8% (24/300)        
2024-12-18T00:18:34.7075812Z remote: Compressing objects:   9% (27/300)        
2024-12-18T00:18:34.7375046Z remote: Compressing objects:  10% (30/300)        
2024-12-18T00:18:34.7519616Z remote: Compressing objects:  11% (33/300)        
2024-12-18T00:18:34.7597180Z remote: Compressing objects:  12% (36/300)        
2024-12-18T00:18:34.7598244Z remote: Compressing objects:  13% (39/300)        
2024-12-18T00:18:34.7601456Z remote: Compressing objects:  14% (42/300)        
2024-12-18T00:18:34.7604080Z remote: Compressing objects:  15% (45/300)        
2024-12-18T00:18:34.7605863Z remote: Compressing objects:  16% (48/300)        
2024-12-18T00:18:34.7613420Z remote: Compressing objects:  17% (51/300)        
2024-12-18T00:18:34.7614976Z remote: Compressing objects:  18% (54/300)        
2024-12-18T00:18:34.7616705Z remote: Compressing objects:  19% (57/300)        
2024-12-18T00:18:34.7618155Z remote: Compressing objects:  20% (60/300)        
2024-12-18T00:18:34.7622697Z remote: Compressing objects:  21% (63/300)        
2024-12-18T00:18:34.7625261Z remote: Compressing objects:  22% (66/300)        
2024-12-18T00:18:34.7630345Z remote: Compressing objects:  23% (69/300)        
2024-12-18T00:18:34.7638229Z remote: Compressing objects:  24% (72/300)        
2024-12-18T00:18:34.7642566Z remote: Compressing objects:  25% (75/300)        
2024-12-18T00:18:34.7645704Z remote: Compressing objects:  26% (78/300)        
2024-12-18T00:18:34.7647028Z remote: Compressing objects:  27% (81/300)        
2024-12-18T00:18:34.7649755Z remote: Compressing objects:  28% (84/300)        
2024-12-18T00:18:34.7653311Z remote: Compressing objects:  29% (87/300)        
2024-12-18T00:18:34.7662541Z remote: Compressing objects:  30% (90/300)        
2024-12-18T00:18:34.7664140Z remote: Compressing objects:  31% (93/300)        
2024-12-18T00:18:34.7666860Z remote: Compressing objects:  32% (96/300)        
2024-12-18T00:18:34.7669767Z remote: Compressing objects:  33% (99/300)        
2024-12-18T00:18:34.7676158Z remote: Compressing objects:  34% (102/300)        
2024-12-18T00:18:34.7683316Z remote: Compressing objects:  35% (105/300)        
2024-12-18T00:18:34.7687080Z remote: Compressing objects:  36% (108/300)        
2024-12-18T00:18:34.7689404Z remote: Compressing objects:  37% (111/300)        
2024-12-18T00:18:34.7693390Z remote: Compressing objects:  38% (114/300)        
2024-12-18T00:18:34.7697568Z remote: Compressing objects:  39% (117/300)        
2024-12-18T00:18:34.7699204Z remote: Compressing objects:  40% (120/300)        
2024-12-18T00:18:34.7701496Z remote: Compressing objects:  41% (123/300)        
2024-12-18T00:18:34.7703931Z remote: Compressing objects:  42% (126/300)        
2024-12-18T00:18:34.7709454Z remote: Compressing objects:  43% (129/300)        
2024-12-18T00:18:34.7713679Z remote: Compressing objects:  44% (132/300)        
2024-12-18T00:18:34.7714363Z remote: Compressing objects:  45% (135/300)        
2024-12-18T00:18:34.7719094Z remote: Compressing objects:  46% (138/300)        
2024-12-18T00:18:34.7722920Z remote: Compressing objects:  47% (141/300)        
2024-12-18T00:18:34.7727575Z remote: Compressing objects:  48% (144/300)        
2024-12-18T00:18:34.7728470Z remote: Compressing objects:  49% (147/300)        
2024-12-18T00:18:34.7733316Z remote: Compressing objects:  50% (150/300)        
2024-12-18T00:18:34.7736122Z remote: Compressing objects:  51% (153/300)        
2024-12-18T00:18:34.7740720Z remote: Compressing objects:  52% (156/300)        
2024-12-18T00:18:34.7741343Z remote: Compressing objects:  53% (159/300)        
2024-12-18T00:18:34.7743411Z remote: Compressing objects:  54% (162/300)        
2024-12-18T00:18:34.7746454Z remote: Compressing objects:  55% (165/300)        
2024-12-18T00:18:34.7749011Z remote: Compressing objects:  56% (168/300)        
2024-12-18T00:18:34.7751932Z remote: Compressing objects:  57% (171/300)        
2024-12-18T00:18:34.7753356Z remote: Compressing objects:  58% (174/300)        
2024-12-18T00:18:34.7757837Z remote: Compressing objects:  59% (177/300)        
2024-12-18T00:18:34.7760153Z remote: Compressing objects:  60% (180/300)        
2024-12-18T00:18:34.7762862Z remote: Compressing objects:  61% (183/300)        
2024-12-18T00:18:34.7764450Z remote: Compressing objects:  62% (186/300)        
2024-12-18T00:18:34.7767880Z remote: Compressing objects:  63% (189/300)        
2024-12-18T00:18:34.7771100Z remote: Compressing objects:  64% (192/300)        
2024-12-18T00:18:34.7772474Z remote: Compressing objects:  65% (195/300)        
2024-12-18T00:18:34.7774096Z remote: Compressing objects:  66% (198/300)        
2024-12-18T00:18:34.7775404Z remote: Compressing objects:  67% (201/300)        
2024-12-18T00:18:34.7782176Z remote: Compressing objects:  68% (204/300)        
2024-12-18T00:18:34.7786267Z remote: Compressing objects:  69% (207/300)        
2024-12-18T00:18:34.7789295Z remote: Compressing objects:  70% (210/300)        
2024-12-18T00:18:34.7791949Z remote: Compressing objects:  71% (213/300)        
2024-12-18T00:18:34.7793404Z remote: Compressing objects:  72% (216/300)        
2024-12-18T00:18:34.7796709Z remote: Compressing objects:  73% (219/300)        
2024-12-18T00:18:34.7799552Z remote: Compressing objects:  74% (222/300)        
2024-12-18T00:18:34.7802348Z remote: Compressing objects:  75% (225/300)        
2024-12-18T00:18:34.7804092Z remote: Compressing objects:  76% (228/300)        
2024-12-18T00:18:34.7805735Z remote: Compressing objects:  77% (231/300)        
2024-12-18T00:18:34.7806923Z remote: Compressing objects:  78% (234/300)        
2024-12-18T00:18:34.7808639Z remote: Compressing objects:  79% (237/300)        
2024-12-18T00:18:34.7809019Z remote: Compressing objects:  80% (240/300)        
2024-12-18T00:18:34.7810841Z remote: Compressing objects:  81% (243/300)        
2024-12-18T00:18:34.7812791Z remote: Compressing objects:  82% (246/300)        
2024-12-18T00:18:34.7815196Z remote: Compressing objects:  83% (249/300)        
2024-12-18T00:18:34.7818147Z remote: Compressing objects:  84% (252/300)        
2024-12-18T00:18:34.7820474Z remote: Compressing objects:  85% (255/300)        
2024-12-18T00:18:34.7822142Z remote: Compressing objects:  86% (258/300)        
2024-12-18T00:18:34.7823943Z remote: Compressing objects:  87% (261/300)        
2024-12-18T00:18:34.7825733Z remote: Compressing objects:  88% (264/300)        
2024-12-18T00:18:34.7827098Z remote: Compressing objects:  89% (267/300)        
2024-12-18T00:18:34.7828774Z remote: Compressing objects:  90% (270/300)        
2024-12-18T00:18:34.7830224Z remote: Compressing objects:  91% (273/300)        
2024-12-18T00:18:34.7831223Z remote: Compressing objects:  92% (276/300)        
2024-12-18T00:18:34.7832534Z remote: Compressing objects:  93% (279/300)        
2024-12-18T00:18:34.7833544Z remote: Compressing objects:  94% (282/300)        
2024-12-18T00:18:34.7834887Z remote: Compressing objects:  95% (285/300)        
2024-12-18T00:18:34.7836486Z remote: Compressing objects:  96% (288/300)        
2024-12-18T00:18:34.7837108Z remote: Compressing objects:  97% (291/300)        
2024-12-18T00:18:34.7838935Z remote: Compressing objects:  98% (294/300)        
2024-12-18T00:18:34.7839878Z remote: Compressing objects:  99% (297/300)        
2024-12-18T00:18:34.7840329Z remote: Compressing objects: 100% (300/300)        
2024-12-18T00:18:34.7840735Z remote: Compressing objects: 100% (300/300), done.        
2024-12-18T00:18:58.2844676Z remote: Total 1057277 (delta 567), reused 410 (delta 406), pack-reused 1056571 (from 3)        
2024-12-18T00:19:20.5737414Z [command]/usr/bin/git rev-parse --verify --quiet 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7^{object}
2024-12-18T00:19:20.5760952Z 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:19:20.5766226Z ##[endgroup]
2024-12-18T00:19:20.5766866Z ##[group]Determining the checkout info
2024-12-18T00:19:20.5768062Z ##[endgroup]
2024-12-18T00:19:20.5768472Z ##[group]Checking out the ref
2024-12-18T00:19:20.5771716Z [command]/usr/bin/git checkout --quiet --force 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:19:22.3083899Z ##[endgroup]
2024-12-18T00:19:22.3084355Z ##[group]Setting up auth for fetching submodules
2024-12-18T00:19:22.3088061Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2024-12-18T00:19:22.3136749Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2024-12-18T00:19:22.3164056Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2024-12-18T00:19:22.3194515Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2024-12-18T00:19:22.3224743Z ##[endgroup]
2024-12-18T00:19:22.3225152Z ##[group]Fetching submodules
2024-12-18T00:19:22.3227295Z [command]/usr/bin/git submodule sync --recursive
2024-12-18T00:19:22.3593626Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2024-12-18T00:19:22.3945777Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2024-12-18T00:19:22.3946988Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2024-12-18T00:19:22.3950390Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2024-12-18T00:19:22.3953448Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2024-12-18T00:19:22.3956560Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX'
2024-12-18T00:19:22.3960378Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2024-12-18T00:19:22.3963643Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2024-12-18T00:19:22.3967212Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2024-12-18T00:19:22.3971368Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel'
2024-12-18T00:19:22.3974949Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib'
2024-12-18T00:19:22.3978811Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2024-12-18T00:19:22.3982535Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2024-12-18T00:19:22.3986605Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2024-12-18T00:19:22.3990613Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2024-12-18T00:19:22.3994511Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2024-12-18T00:19:22.3999656Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2024-12-18T00:19:22.4007378Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2024-12-18T00:19:22.4011668Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2024-12-18T00:19:22.4015944Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2024-12-18T00:19:22.4020299Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2024-12-18T00:19:22.4024594Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2024-12-18T00:19:22.4029465Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2024-12-18T00:19:22.4034006Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2024-12-18T00:19:22.4038648Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc'
2024-12-18T00:19:22.4043269Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2024-12-18T00:19:22.4048052Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2024-12-18T00:19:22.4052900Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2024-12-18T00:19:22.4058255Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp'
2024-12-18T00:19:22.4063275Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2024-12-18T00:19:22.4070383Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2024-12-18T00:19:22.4074151Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2024-12-18T00:19:22.4079251Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2024-12-18T00:19:22.4084551Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2024-12-18T00:19:22.4093103Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2024-12-18T00:19:22.4098412Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2024-12-18T00:19:22.4103903Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2024-12-18T00:19:22.4139093Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2024-12-18T00:19:22.7212553Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2024-12-18T00:19:22.9124106Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2024-12-18T00:19:23.1036292Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2024-12-18T00:19:23.3957572Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'...
2024-12-18T00:19:23.7375090Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2024-12-18T00:19:25.8580200Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2024-12-18T00:19:37.3548364Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2024-12-18T00:19:37.8072360Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'...
2024-12-18T00:19:40.0767780Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'...
2024-12-18T00:19:40.6293859Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2024-12-18T00:19:41.2675705Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2024-12-18T00:19:42.6083859Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2024-12-18T00:19:44.9609319Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2024-12-18T00:19:51.2248854Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2024-12-18T00:19:52.9445431Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2024-12-18T00:19:54.2052413Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2024-12-18T00:19:55.5904747Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2024-12-18T00:19:56.0269763Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2024-12-18T00:19:56.4164303Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2024-12-18T00:19:57.6046350Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2024-12-18T00:19:58.0263539Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2024-12-18T00:19:58.3475316Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2024-12-18T00:20:00.1817541Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'...
2024-12-18T00:20:01.1462486Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2024-12-18T00:20:01.5600286Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2024-12-18T00:20:09.0519197Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2024-12-18T00:20:11.3458235Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'...
2024-12-18T00:20:16.4060216Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2024-12-18T00:20:16.7145647Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2024-12-18T00:20:26.6176802Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2024-12-18T00:20:26.8228555Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2024-12-18T00:20:27.1110687Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2024-12-18T00:20:28.2536925Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2024-12-18T00:20:28.5846497Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2024-12-18T00:20:29.2314163Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2024-12-18T00:20:29.6813176Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2024-12-18T00:20:29.6960821Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2024-12-18T00:20:29.7079898Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2024-12-18T00:20:29.7391311Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2024-12-18T00:20:29.7838665Z Submodule path 'third_party/NVTX': checked out 'e170594ac7cf1dac584da473d4ca9301087090c1'
2024-12-18T00:20:29.8292897Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2024-12-18T00:20:30.7436010Z Submodule path 'third_party/XNNPACK': checked out '4ea82e595b36106653175dcb04b2aa532660d0d8'
2024-12-18T00:20:30.7722661Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2024-12-18T00:20:31.0724348Z Submodule path 'third_party/composable_kernel': checked out '50ee4267e27b875d149e642f4cebd47be1dc3b57'
2024-12-18T00:20:31.1282314Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006'
2024-12-18T00:20:31.2411988Z Submodule path 'third_party/cpuinfo': checked out '1e83a2fdd3102f65c6f1fb602c1b320486218a99'
2024-12-18T00:20:31.2820304Z Submodule path 'third_party/cudnn_frontend': checked out '936021bfed8c91dc416af1588b2c4eca631a9e45'
2024-12-18T00:20:31.9019200Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49'
2024-12-18T00:20:32.1913552Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2024-12-18T00:20:32.2986698Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e'
2024-12-18T00:20:32.3011440Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2024-12-18T00:20:32.3014871Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T00:20:32.3018378Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass'
2024-12-18T00:20:32.3021979Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2024-12-18T00:20:32.3025930Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T00:20:32.3058130Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2024-12-18T00:20:33.3179745Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2024-12-18T00:20:33.9966033Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'...
2024-12-18T00:20:36.2808393Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2024-12-18T00:20:37.3751722Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2024-12-18T00:20:37.8155911Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2024-12-18T00:20:37.9268025Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2024-12-18T00:20:38.4348273Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6'
2024-12-18T00:20:38.5056138Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2024-12-18T00:20:38.5209159Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1'
2024-12-18T00:20:38.6855161Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b'
2024-12-18T00:20:38.7298058Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592'
2024-12-18T00:20:38.7754207Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2024-12-18T00:20:38.8085047Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509'
2024-12-18T00:20:38.8594042Z Submodule path 'third_party/googletest': checked out 'b514bdc898e2951020cbdca1304b75f5950d1f59'
2024-12-18T00:20:38.8754237Z Submodule path 'third_party/ideep': checked out 'c7ccd5bdbe5434ba156f4e856dcef0601637334b'
2024-12-18T00:20:38.8776888Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2024-12-18T00:20:38.8808237Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2024-12-18T00:20:53.9467641Z Submodule path 'third_party/ideep/mkl-dnn': checked out '66f0cb9eb66affd2da3bf5f8d897376f04aae6af'
2024-12-18T00:20:53.9690435Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2024-12-18T00:20:54.0685878Z Submodule path 'third_party/kineto': checked out '338140f58a28d599da3434ced4fd2d75dd1a213d'
2024-12-18T00:20:54.0711221Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T00:20:54.0714394Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T00:20:54.0717946Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T00:20:54.0749661Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'...
2024-12-18T00:20:54.9050899Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2024-12-18T00:20:56.2047761Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2024-12-18T00:20:57.3788167Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e'
2024-12-18T00:20:57.3808925Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T00:20:57.3812340Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T00:20:57.3815775Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T00:20:57.3819471Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T00:20:57.3823225Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T00:20:57.3827212Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T00:20:57.3831012Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T00:20:57.3834915Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T00:20:57.3867432Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'...
2024-12-18T00:20:58.3668664Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'...
2024-12-18T00:20:58.7571970Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'...
2024-12-18T00:21:00.0640758Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'...
2024-12-18T00:21:00.4312994Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'...
2024-12-18T00:21:00.9497106Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'...
2024-12-18T00:21:02.0743942Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'...
2024-12-18T00:21:09.6339402Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'...
2024-12-18T00:21:10.0640668Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9'
2024-12-18T00:21:10.0877569Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400'
2024-12-18T00:21:10.1311198Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2024-12-18T00:21:10.1481815Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067'
2024-12-18T00:21:10.1501436Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T00:21:10.1532798Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'...
2024-12-18T00:21:10.5025732Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4'
2024-12-18T00:21:10.5253860Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446'
2024-12-18T00:21:10.5735824Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850'
2024-12-18T00:21:10.6899019Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5'
2024-12-18T00:21:10.7103217Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150'
2024-12-18T00:21:10.7550186Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164'
2024-12-18T00:21:10.8219676Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2024-12-18T00:21:10.8654444Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78'
2024-12-18T00:21:10.8999434Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290'
2024-12-18T00:21:11.0316540Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2024-12-18T00:21:11.5262025Z Submodule path 'third_party/onnx': checked out 'b8baa8446686496da4cc8fda09f2b6fe65c2a02c'
2024-12-18T00:21:11.5302962Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2024-12-18T00:21:11.5334838Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2024-12-18T00:21:12.8664855Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b'
2024-12-18T00:21:12.9573386Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878'
2024-12-18T00:21:12.9598868Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T00:21:12.9602411Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T00:21:12.9605799Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T00:21:12.9609385Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T00:21:12.9613218Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T00:21:12.9616797Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T00:21:12.9620653Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T00:21:12.9624172Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T00:21:12.9658091Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'...
2024-12-18T00:21:13.4340936Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'...
2024-12-18T00:21:14.5526333Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'...
2024-12-18T00:21:14.8953967Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'...
2024-12-18T00:21:22.4176408Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'...
2024-12-18T00:21:22.7007612Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'...
2024-12-18T00:21:22.9101117Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'...
2024-12-18T00:21:23.2559569Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'...
2024-12-18T00:21:30.4090191Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2'
2024-12-18T00:21:30.4562128Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1'
2024-12-18T00:21:30.4753938Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa'
2024-12-18T00:21:30.6084616Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d'
2024-12-18T00:21:30.6248895Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce'
2024-12-18T00:21:30.6437376Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5'
2024-12-18T00:21:30.6639194Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d'
2024-12-18T00:21:30.6658790Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T00:21:30.6662182Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T00:21:30.6693748Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'...
2024-12-18T00:21:32.6570783Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'...
2024-12-18T00:21:34.0428160Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4'
2024-12-18T00:21:34.0967201Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2024-12-18T00:21:34.7519401Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50'
2024-12-18T00:21:34.7663809Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe'
2024-12-18T00:21:35.0945769Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2024-12-18T00:21:35.0972688Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2024-12-18T00:21:35.0976024Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2024-12-18T00:21:35.1007341Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2024-12-18T00:21:35.6135731Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2024-12-18T00:21:36.7213021Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2024-12-18T00:21:36.8026116Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2024-12-18T00:21:36.8146172Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2024-12-18T00:21:36.8304079Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8'
2024-12-18T00:21:36.8742726Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4'
2024-12-18T00:21:36.9086863Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2024-12-18T00:21:36.9585151Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b'
2024-12-18T00:21:36.9929294Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2024-12-18T00:21:36.9950826Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2024-12-18T00:21:36.9954200Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2024-12-18T00:21:36.9957585Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2024-12-18T00:21:36.9961243Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T00:21:36.9992869Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2024-12-18T00:21:38.0719107Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2024-12-18T00:21:38.3292324Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2024-12-18T00:21:40.6486030Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2024-12-18T00:21:41.8263860Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2024-12-18T00:21:41.8457151Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2024-12-18T00:21:41.9162751Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2024-12-18T00:21:41.9501730Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2024-12-18T00:21:41.9521409Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T00:21:41.9554383Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2024-12-18T00:21:42.2160561Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2024-12-18T00:21:42.2205608Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2024-12-18T00:21:42.2565468Z Entering 'android/libs/fbjni'
2024-12-18T00:21:42.2621610Z Entering 'third_party/FP16'
2024-12-18T00:21:42.2673922Z Entering 'third_party/FXdiv'
2024-12-18T00:21:42.2726191Z Entering 'third_party/NNPACK'
2024-12-18T00:21:42.2778463Z Entering 'third_party/NVTX'
2024-12-18T00:21:42.2829786Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T00:21:42.2882152Z Entering 'third_party/XNNPACK'
2024-12-18T00:21:42.2951789Z Entering 'third_party/benchmark'
2024-12-18T00:21:42.3005803Z Entering 'third_party/composable_kernel'
2024-12-18T00:21:42.3063405Z Entering 'third_party/cpp-httplib'
2024-12-18T00:21:42.3115538Z Entering 'third_party/cpuinfo'
2024-12-18T00:21:42.3167714Z Entering 'third_party/cudnn_frontend'
2024-12-18T00:21:42.3219995Z Entering 'third_party/cutlass'
2024-12-18T00:21:42.3279732Z Entering 'third_party/eigen'
2024-12-18T00:21:42.3333468Z Entering 'third_party/fbgemm'
2024-12-18T00:21:42.3385074Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T00:21:42.3435231Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T00:21:42.3491602Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T00:21:42.3547173Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T00:21:42.3601253Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T00:21:42.3653565Z Entering 'third_party/flatbuffers'
2024-12-18T00:21:42.3708981Z Entering 'third_party/fmt'
2024-12-18T00:21:42.3759960Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T00:21:42.3812070Z Entering 'third_party/gloo'
2024-12-18T00:21:42.3863369Z Entering 'third_party/googletest'
2024-12-18T00:21:42.3916108Z Entering 'third_party/ideep'
2024-12-18T00:21:42.3965463Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T00:21:42.4027416Z Entering 'third_party/ittapi'
2024-12-18T00:21:42.4079325Z Entering 'third_party/kineto'
2024-12-18T00:21:42.4130039Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T00:21:42.4179107Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T00:21:42.4232203Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T00:21:42.4283674Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T00:21:42.4334919Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T00:21:42.4384690Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T00:21:42.4439507Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T00:21:42.4490377Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T00:21:42.4540560Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T00:21:42.4593019Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T00:21:42.4647061Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T00:21:42.4698230Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T00:21:42.4752146Z Entering 'third_party/mimalloc'
2024-12-18T00:21:42.4804280Z Entering 'third_party/nccl/nccl'
2024-12-18T00:21:42.4855861Z Entering 'third_party/nlohmann'
2024-12-18T00:21:42.4910068Z Entering 'third_party/onnx'
2024-12-18T00:21:42.4975092Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T00:21:42.5031379Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T00:21:42.5088848Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T00:21:42.5138332Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T00:21:42.5188658Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T00:21:42.5238574Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T00:21:42.5290000Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T00:21:42.5339034Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T00:21:42.5388945Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T00:21:42.5437476Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T00:21:42.5492818Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T00:21:42.5545876Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T00:21:42.5617440Z Entering 'third_party/pocketfft'
2024-12-18T00:21:42.5669657Z Entering 'third_party/protobuf'
2024-12-18T00:21:42.5723116Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T00:21:42.5774678Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T00:21:42.5830504Z Entering 'third_party/psimd'
2024-12-18T00:21:42.5883877Z Entering 'third_party/pthreadpool'
2024-12-18T00:21:42.5935800Z Entering 'third_party/pybind11'
2024-12-18T00:21:42.5991892Z Entering 'third_party/python-peachpy'
2024-12-18T00:21:42.6043233Z Entering 'third_party/sleef'
2024-12-18T00:21:42.6095528Z Entering 'third_party/tensorpipe'
2024-12-18T00:21:42.6146585Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T00:21:42.6199562Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T00:21:42.6249060Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T00:21:42.6300559Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T00:21:42.6348067Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T00:21:42.6421635Z ##[endgroup]
2024-12-18T00:21:42.6425425Z ##[group]Persisting credentials for submodules
2024-12-18T00:21:42.6426936Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :"
2024-12-18T00:21:42.6782501Z Entering 'android/libs/fbjni'
2024-12-18T00:21:42.6849938Z Entering 'third_party/FP16'
2024-12-18T00:21:42.6920048Z Entering 'third_party/FXdiv'
2024-12-18T00:21:42.6992315Z Entering 'third_party/NNPACK'
2024-12-18T00:21:42.7058713Z Entering 'third_party/NVTX'
2024-12-18T00:21:42.7127607Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T00:21:42.7200573Z Entering 'third_party/XNNPACK'
2024-12-18T00:21:42.7285771Z Entering 'third_party/benchmark'
2024-12-18T00:21:42.7352564Z Entering 'third_party/composable_kernel'
2024-12-18T00:21:42.7426732Z Entering 'third_party/cpp-httplib'
2024-12-18T00:21:42.7497652Z Entering 'third_party/cpuinfo'
2024-12-18T00:21:42.7564249Z Entering 'third_party/cudnn_frontend'
2024-12-18T00:21:42.7632674Z Entering 'third_party/cutlass'
2024-12-18T00:21:42.7716661Z Entering 'third_party/eigen'
2024-12-18T00:21:42.7792276Z Entering 'third_party/fbgemm'
2024-12-18T00:21:42.7857761Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T00:21:42.7929907Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T00:21:42.7997254Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T00:21:42.8070647Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T00:21:42.8136534Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T00:21:42.8205905Z Entering 'third_party/flatbuffers'
2024-12-18T00:21:42.8280919Z Entering 'third_party/fmt'
2024-12-18T00:21:42.8347697Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T00:21:42.8417359Z Entering 'third_party/gloo'
2024-12-18T00:21:42.8490668Z Entering 'third_party/googletest'
2024-12-18T00:21:42.8561703Z Entering 'third_party/ideep'
2024-12-18T00:21:42.8628115Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T00:21:42.8702647Z Entering 'third_party/ittapi'
2024-12-18T00:21:42.8774119Z Entering 'third_party/kineto'
2024-12-18T00:21:42.8840348Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T00:21:42.8909808Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T00:21:42.8979319Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T00:21:42.9046493Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T00:21:42.9113132Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T00:21:42.9178187Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T00:21:42.9249649Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T00:21:42.9318828Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T00:21:42.9385884Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T00:21:42.9453879Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T00:21:42.9524670Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T00:21:42.9592106Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T00:21:42.9660756Z Entering 'third_party/mimalloc'
2024-12-18T00:21:42.9731156Z Entering 'third_party/nccl/nccl'
2024-12-18T00:21:42.9798476Z Entering 'third_party/nlohmann'
2024-12-18T00:21:42.9867862Z Entering 'third_party/onnx'
2024-12-18T00:21:42.9950760Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T00:21:43.0024234Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T00:21:43.0096078Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T00:21:43.0162230Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T00:21:43.0228958Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T00:21:43.0294587Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T00:21:43.0362012Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T00:21:43.0428401Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T00:21:43.0494036Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T00:21:43.0558107Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T00:21:43.0631690Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T00:21:43.0701658Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T00:21:43.0789253Z Entering 'third_party/pocketfft'
2024-12-18T00:21:43.0855960Z Entering 'third_party/protobuf'
2024-12-18T00:21:43.0926272Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T00:21:43.0992887Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T00:21:43.1062636Z Entering 'third_party/psimd'
2024-12-18T00:21:43.1131753Z Entering 'third_party/pthreadpool'
2024-12-18T00:21:43.1199890Z Entering 'third_party/pybind11'
2024-12-18T00:21:43.1266366Z Entering 'third_party/python-peachpy'
2024-12-18T00:21:43.1337036Z Entering 'third_party/sleef'
2024-12-18T00:21:43.1404182Z Entering 'third_party/tensorpipe'
2024-12-18T00:21:43.1470029Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T00:21:43.1535491Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T00:21:43.1601432Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T00:21:43.1667556Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T00:21:43.1730165Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T00:21:43.1822076Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url"
2024-12-18T00:21:43.2168661Z Entering 'android/libs/fbjni'
2024-12-18T00:21:43.2231102Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2024-12-18T00:21:43.2253035Z Entering 'third_party/FP16'
2024-12-18T00:21:43.2315738Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2024-12-18T00:21:43.2338168Z Entering 'third_party/FXdiv'
2024-12-18T00:21:43.2402263Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2024-12-18T00:21:43.2424705Z Entering 'third_party/NNPACK'
2024-12-18T00:21:43.2487783Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2024-12-18T00:21:43.2509981Z Entering 'third_party/NVTX'
2024-12-18T00:21:43.2572434Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config	remote.origin.url
2024-12-18T00:21:43.2594973Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T00:21:43.2657005Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2024-12-18T00:21:43.2679632Z Entering 'third_party/XNNPACK'
2024-12-18T00:21:43.2741663Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2024-12-18T00:21:43.2778513Z Entering 'third_party/benchmark'
2024-12-18T00:21:43.2841135Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2024-12-18T00:21:43.2863211Z Entering 'third_party/composable_kernel'
2024-12-18T00:21:43.2925033Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config	remote.origin.url
2024-12-18T00:21:43.2952987Z Entering 'third_party/cpp-httplib'
2024-12-18T00:21:43.3019913Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config	remote.origin.url
2024-12-18T00:21:43.3041859Z Entering 'third_party/cpuinfo'
2024-12-18T00:21:43.3105214Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2024-12-18T00:21:43.3128355Z Entering 'third_party/cudnn_frontend'
2024-12-18T00:21:43.3191366Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2024-12-18T00:21:43.3213600Z Entering 'third_party/cutlass'
2024-12-18T00:21:43.3275713Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2024-12-18T00:21:43.3305164Z Entering 'third_party/eigen'
2024-12-18T00:21:43.3366050Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2024-12-18T00:21:43.3390663Z Entering 'third_party/fbgemm'
2024-12-18T00:21:43.3451946Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2024-12-18T00:21:43.3472448Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T00:21:43.3533546Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2024-12-18T00:21:43.3554211Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T00:21:43.3620654Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2024-12-18T00:21:43.3642340Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T00:21:43.3706373Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config	remote.origin.url
2024-12-18T00:21:43.3737731Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T00:21:43.3799179Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:43.3819913Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T00:21:43.3880845Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2024-12-18T00:21:43.3904979Z Entering 'third_party/flatbuffers'
2024-12-18T00:21:43.3966228Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2024-12-18T00:21:43.3991813Z Entering 'third_party/fmt'
2024-12-18T00:21:43.4053478Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2024-12-18T00:21:43.4075333Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T00:21:43.4137237Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2024-12-18T00:21:43.4159264Z Entering 'third_party/gloo'
2024-12-18T00:21:43.4221563Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2024-12-18T00:21:43.4243497Z Entering 'third_party/googletest'
2024-12-18T00:21:43.4307110Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:43.4327883Z Entering 'third_party/ideep'
2024-12-18T00:21:43.4390832Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2024-12-18T00:21:43.4410219Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T00:21:43.4475375Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2024-12-18T00:21:43.4503577Z Entering 'third_party/ittapi'
2024-12-18T00:21:43.4565868Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2024-12-18T00:21:43.4587952Z Entering 'third_party/kineto'
2024-12-18T00:21:43.4650709Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2024-12-18T00:21:43.4670843Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T00:21:43.4733630Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config	remote.origin.url
2024-12-18T00:21:43.4753278Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T00:21:43.4815783Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config	remote.origin.url
2024-12-18T00:21:43.4838377Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T00:21:43.4899933Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config	remote.origin.url
2024-12-18T00:21:43.4920735Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T00:21:43.4982539Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config	remote.origin.url
2024-12-18T00:21:43.5003616Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T00:21:43.5065354Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config	remote.origin.url
2024-12-18T00:21:43.5084695Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T00:21:43.5147393Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config	remote.origin.url
2024-12-18T00:21:43.5172000Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T00:21:43.5233249Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config	remote.origin.url
2024-12-18T00:21:43.5254134Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T00:21:43.5316498Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:43.5337698Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T00:21:43.5399331Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config	remote.origin.url
2024-12-18T00:21:43.5421553Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T00:21:43.5482761Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config	remote.origin.url
2024-12-18T00:21:43.5508038Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T00:21:43.5572991Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2024-12-18T00:21:43.5593929Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T00:21:43.5660225Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:43.5684916Z Entering 'third_party/mimalloc'
2024-12-18T00:21:43.5745873Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config	remote.origin.url
2024-12-18T00:21:43.5767896Z Entering 'third_party/nccl/nccl'
2024-12-18T00:21:43.5829292Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2024-12-18T00:21:43.5850015Z Entering 'third_party/nlohmann'
2024-12-18T00:21:43.5911611Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2024-12-18T00:21:43.5934067Z Entering 'third_party/onnx'
2024-12-18T00:21:43.5996273Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2024-12-18T00:21:43.6030923Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T00:21:43.6092699Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2024-12-18T00:21:43.6118771Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T00:21:43.6183591Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config	remote.origin.url
2024-12-18T00:21:43.6204648Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T00:21:43.6268337Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config	remote.origin.url
2024-12-18T00:21:43.6290371Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T00:21:43.6352135Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:43.6373611Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T00:21:43.6435045Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config	remote.origin.url
2024-12-18T00:21:43.6455089Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T00:21:43.6517003Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config	remote.origin.url
2024-12-18T00:21:43.6539752Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T00:21:43.6601486Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config	remote.origin.url
2024-12-18T00:21:43.6622169Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T00:21:43.6683794Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config	remote.origin.url
2024-12-18T00:21:43.6704312Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T00:21:43.6770964Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config	remote.origin.url
2024-12-18T00:21:43.6789602Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T00:21:43.6851112Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2024-12-18T00:21:43.6873722Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T00:21:43.6934529Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2024-12-18T00:21:43.6958273Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T00:21:43.7022762Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config	remote.origin.url
2024-12-18T00:21:43.7064507Z Entering 'third_party/pocketfft'
2024-12-18T00:21:43.7125752Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2024-12-18T00:21:43.7146837Z Entering 'third_party/protobuf'
2024-12-18T00:21:43.7208149Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2024-12-18T00:21:43.7230799Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T00:21:43.7294156Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2024-12-18T00:21:43.7314964Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T00:21:43.7378268Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:43.7402720Z Entering 'third_party/psimd'
2024-12-18T00:21:43.7463359Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2024-12-18T00:21:43.7485347Z Entering 'third_party/pthreadpool'
2024-12-18T00:21:43.7546193Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2024-12-18T00:21:43.7567814Z Entering 'third_party/pybind11'
2024-12-18T00:21:43.7627296Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2024-12-18T00:21:43.7648959Z Entering 'third_party/python-peachpy'
2024-12-18T00:21:43.7710050Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2024-12-18T00:21:43.7731474Z Entering 'third_party/sleef'
2024-12-18T00:21:43.7795150Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2024-12-18T00:21:43.7817527Z Entering 'third_party/tensorpipe'
2024-12-18T00:21:43.7880806Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2024-12-18T00:21:43.7901079Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T00:21:43.7962151Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2024-12-18T00:21:43.7983876Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T00:21:43.8044473Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2024-12-18T00:21:43.8065152Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T00:21:43.8126804Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2024-12-18T00:21:43.8147642Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T00:21:43.8212950Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2024-12-18T00:21:43.8231625Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T00:21:43.8294597Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2024-12-18T00:21:43.9585945Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2024-12-18T00:21:43.9939909Z Entering 'android/libs/fbjni'
2024-12-18T00:21:43.9993423Z Entering 'third_party/FP16'
2024-12-18T00:21:44.0048128Z Entering 'third_party/FXdiv'
2024-12-18T00:21:44.0100808Z Entering 'third_party/NNPACK'
2024-12-18T00:21:44.0153480Z Entering 'third_party/NVTX'
2024-12-18T00:21:44.0206811Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T00:21:44.0259240Z Entering 'third_party/XNNPACK'
2024-12-18T00:21:44.0326979Z Entering 'third_party/benchmark'
2024-12-18T00:21:44.0379922Z Entering 'third_party/composable_kernel'
2024-12-18T00:21:44.0439511Z Entering 'third_party/cpp-httplib'
2024-12-18T00:21:44.0491380Z Entering 'third_party/cpuinfo'
2024-12-18T00:21:44.0546898Z Entering 'third_party/cudnn_frontend'
2024-12-18T00:21:44.0599983Z Entering 'third_party/cutlass'
2024-12-18T00:21:44.0659551Z Entering 'third_party/eigen'
2024-12-18T00:21:44.0714047Z Entering 'third_party/fbgemm'
2024-12-18T00:21:44.0765337Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T00:21:44.0817145Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T00:21:44.0868544Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T00:21:44.0924396Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T00:21:44.0975950Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T00:21:44.1028916Z Entering 'third_party/flatbuffers'
2024-12-18T00:21:44.1084352Z Entering 'third_party/fmt'
2024-12-18T00:21:44.1135342Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T00:21:44.1188374Z Entering 'third_party/gloo'
2024-12-18T00:21:44.1239563Z Entering 'third_party/googletest'
2024-12-18T00:21:44.1291258Z Entering 'third_party/ideep'
2024-12-18T00:21:44.1341193Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T00:21:44.1399702Z Entering 'third_party/ittapi'
2024-12-18T00:21:44.1450937Z Entering 'third_party/kineto'
2024-12-18T00:21:44.1501561Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T00:21:44.1549977Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T00:21:44.1605009Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T00:21:44.1656116Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T00:21:44.1709475Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T00:21:44.1759194Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T00:21:44.1815826Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T00:21:44.1868427Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T00:21:44.1919526Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T00:21:44.1970535Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T00:21:44.2024503Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T00:21:44.2080687Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T00:21:44.2135484Z Entering 'third_party/mimalloc'
2024-12-18T00:21:44.2188406Z Entering 'third_party/nccl/nccl'
2024-12-18T00:21:44.2241365Z Entering 'third_party/nlohmann'
2024-12-18T00:21:44.2295591Z Entering 'third_party/onnx'
2024-12-18T00:21:44.2361525Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T00:21:44.2420476Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T00:21:44.2474438Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T00:21:44.2527084Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T00:21:44.2576680Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T00:21:44.2625588Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T00:21:44.2678281Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T00:21:44.2729001Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T00:21:44.2782661Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T00:21:44.2831826Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T00:21:44.2886369Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T00:21:44.2940691Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T00:21:44.3012728Z Entering 'third_party/pocketfft'
2024-12-18T00:21:44.3064830Z Entering 'third_party/protobuf'
2024-12-18T00:21:44.3119643Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T00:21:44.3176411Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T00:21:44.3231545Z Entering 'third_party/psimd'
2024-12-18T00:21:44.3285826Z Entering 'third_party/pthreadpool'
2024-12-18T00:21:44.3337978Z Entering 'third_party/pybind11'
2024-12-18T00:21:44.3391316Z Entering 'third_party/python-peachpy'
2024-12-18T00:21:44.3445667Z Entering 'third_party/sleef'
2024-12-18T00:21:44.3498516Z Entering 'third_party/tensorpipe'
2024-12-18T00:21:44.3549962Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T00:21:44.3600039Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T00:21:44.3649206Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T00:21:44.3699628Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T00:21:44.3747754Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T00:21:44.3827527Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2024-12-18T00:21:44.4193069Z Entering 'android/libs/fbjni'
2024-12-18T00:21:44.4246693Z Entering 'third_party/FP16'
2024-12-18T00:21:44.4299170Z Entering 'third_party/FXdiv'
2024-12-18T00:21:44.4351361Z Entering 'third_party/NNPACK'
2024-12-18T00:21:44.4402514Z Entering 'third_party/NVTX'
2024-12-18T00:21:44.4453006Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T00:21:44.4505300Z Entering 'third_party/XNNPACK'
2024-12-18T00:21:44.4573170Z Entering 'third_party/benchmark'
2024-12-18T00:21:44.4626275Z Entering 'third_party/composable_kernel'
2024-12-18T00:21:44.4686418Z Entering 'third_party/cpp-httplib'
2024-12-18T00:21:44.4738782Z Entering 'third_party/cpuinfo'
2024-12-18T00:21:44.4790967Z Entering 'third_party/cudnn_frontend'
2024-12-18T00:21:44.4842933Z Entering 'third_party/cutlass'
2024-12-18T00:21:44.4904875Z Entering 'third_party/eigen'
2024-12-18T00:21:44.4958966Z Entering 'third_party/fbgemm'
2024-12-18T00:21:44.5010147Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T00:21:44.5059370Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T00:21:44.5110548Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T00:21:44.5172612Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T00:21:44.5223643Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T00:21:44.5280800Z Entering 'third_party/flatbuffers'
2024-12-18T00:21:44.5335783Z Entering 'third_party/fmt'
2024-12-18T00:21:44.5388490Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T00:21:44.5441401Z Entering 'third_party/gloo'
2024-12-18T00:21:44.5496939Z Entering 'third_party/googletest'
2024-12-18T00:21:44.5547586Z Entering 'third_party/ideep'
2024-12-18T00:21:44.5603166Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T00:21:44.5661892Z Entering 'third_party/ittapi'
2024-12-18T00:21:44.5714532Z Entering 'third_party/kineto'
2024-12-18T00:21:44.5772349Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T00:21:44.5822831Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T00:21:44.5876349Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T00:21:44.5927090Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T00:21:44.5979001Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T00:21:44.6028172Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T00:21:44.6083928Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T00:21:44.6136358Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T00:21:44.6188322Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T00:21:44.6239814Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T00:21:44.6294293Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T00:21:44.6345303Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T00:21:44.6403738Z Entering 'third_party/mimalloc'
2024-12-18T00:21:44.6456115Z Entering 'third_party/nccl/nccl'
2024-12-18T00:21:44.6508464Z Entering 'third_party/nlohmann'
2024-12-18T00:21:44.6559346Z Entering 'third_party/onnx'
2024-12-18T00:21:44.6628042Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T00:21:44.6684316Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T00:21:44.6738066Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T00:21:44.6791409Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T00:21:44.6842604Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T00:21:44.6893018Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T00:21:44.6944204Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T00:21:44.6995075Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T00:21:44.7044900Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T00:21:44.7093859Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T00:21:44.7146212Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T00:21:44.7199609Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T00:21:44.7269808Z Entering 'third_party/pocketfft'
2024-12-18T00:21:44.7323732Z Entering 'third_party/protobuf'
2024-12-18T00:21:44.7381548Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T00:21:44.7431445Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T00:21:44.7486871Z Entering 'third_party/psimd'
2024-12-18T00:21:44.7539085Z Entering 'third_party/pthreadpool'
2024-12-18T00:21:44.7592643Z Entering 'third_party/pybind11'
2024-12-18T00:21:44.7644794Z Entering 'third_party/python-peachpy'
2024-12-18T00:21:44.7697548Z Entering 'third_party/sleef'
2024-12-18T00:21:44.7750925Z Entering 'third_party/tensorpipe'
2024-12-18T00:21:44.7802722Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T00:21:44.7856559Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T00:21:44.7901846Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T00:21:44.7950488Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T00:21:44.7998204Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T00:21:44.8071985Z ##[endgroup]
2024-12-18T00:21:44.8107826Z [command]/usr/bin/git log -1 --format='%H'
2024-12-18T00:21:44.8131651Z '0cdf8b1d09254cfda66191d1bd01e3041c3c76f7'
2024-12-18T00:21:44.8300166Z Prepare all required actions
2024-12-18T00:21:44.8300656Z Getting action download info
2024-12-18T00:21:45.0146444Z ##[group]Run ./.github/actions/setup-linux
2024-12-18T00:21:45.0146979Z env:
2024-12-18T00:21:45.0147198Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:45.0147461Z ##[endgroup]
2024-12-18T00:21:45.0200024Z ##[group]Run set -euo pipefail
2024-12-18T00:21:45.0200369Z set -euo pipefail
2024-12-18T00:21:45.0200660Z function get_ec2_metadata() {
2024-12-18T00:21:45.0201060Z   # Pulled from instance metadata endpoint for EC2
2024-12-18T00:21:45.0201810Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2024-12-18T00:21:45.0202366Z   category=$1
2024-12-18T00:21:45.0202785Z   # If it is GCP runner (runner name contains gcp), do not run this
2024-12-18T00:21:45.0203221Z   runner_name_str=i-029fac8e3f5b7f275
2024-12-18T00:21:45.0203601Z   if [[ -f /.inarc ]]; then
2024-12-18T00:21:45.0203951Z     echo "ARC Runner, no info on ec2 metadata"
2024-12-18T00:21:45.0204332Z   elif [[ $runner_name_str == *"gcp"* ]]; then
2024-12-18T00:21:45.0204789Z     echo "Runner is from Google Cloud Platform, No info on ec2 metadata"
2024-12-18T00:21:45.0205268Z   else
2024-12-18T00:21:45.0206113Z     curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2024-12-18T00:21:45.0207069Z   fi
2024-12-18T00:21:45.0207300Z }
2024-12-18T00:21:45.0207572Z echo "ami-id: $(get_ec2_metadata ami-id)"
2024-12-18T00:21:45.0207997Z echo "instance-id: $(get_ec2_metadata instance-id)"
2024-12-18T00:21:45.0208528Z echo "instance-type: $(get_ec2_metadata instance-type)"
2024-12-18T00:21:45.0208944Z echo "system info $(uname -a)"
2024-12-18T00:21:45.0220124Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:45.0220511Z env:
2024-12-18T00:21:45.0220728Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:45.0220994Z ##[endgroup]
2024-12-18T00:21:45.0365112Z ami-id: ami-0fff1b9a61dec8a5f
2024-12-18T00:21:45.0483507Z instance-id: i-029fac8e3f5b7f275
2024-12-18T00:21:45.0595023Z instance-type: g5.4xlarge
2024-12-18T00:21:45.0608101Z system info Linux ip-10-0-71-248.ec2.internal 6.1.109-118.189.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 10 08:59:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
2024-12-18T00:21:45.0644843Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T00:21:45.0645868Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T00:21:45.0655380Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:45.0655783Z env:
2024-12-18T00:21:45.0656046Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:45.0656422Z ##[endgroup]
2024-12-18T00:21:45.0727674Z ##[group]Run if systemctl is-active --quiet docker; then
2024-12-18T00:21:45.0728103Z if systemctl is-active --quiet docker; then
2024-12-18T00:21:45.0728480Z     echo "Docker daemon is running...";
2024-12-18T00:21:45.0728801Z else
2024-12-18T00:21:45.0729147Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2024-12-18T00:21:45.0729935Z fi
2024-12-18T00:21:45.0738149Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:45.0738518Z env:
2024-12-18T00:21:45.0738737Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:45.0738997Z ##[endgroup]
2024-12-18T00:21:45.0832720Z Docker daemon is running...
2024-12-18T00:21:45.0888855Z ##[group]Run nick-fields/retry@v3.0.0
2024-12-18T00:21:45.0889155Z with:
2024-12-18T00:21:45.0889366Z   shell: bash
2024-12-18T00:21:45.0889802Z   timeout_minutes: 5
2024-12-18T00:21:45.0890058Z   max_attempts: 3
2024-12-18T00:21:45.0890294Z   retry_wait_seconds: 30
2024-12-18T00:21:45.0892453Z   command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \
    --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"

# For LF Runners we need to make sure we also login to Meta's ECR docker registry too.
META_AWS_ACCOUNT_ID=308535385114
if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then
    aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \
        --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
fi

2024-12-18T00:21:45.0894617Z   polling_interval_seconds: 1
2024-12-18T00:21:45.0894901Z   warning_on_retry: true
2024-12-18T00:21:45.0895163Z   continue_on_error: false
2024-12-18T00:21:45.0895418Z env:
2024-12-18T00:21:45.0895640Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:45.0895910Z   AWS_RETRY_MODE: standard
2024-12-18T00:21:45.0896171Z   AWS_MAX_ATTEMPTS: 5
2024-12-18T00:21:45.0896426Z   AWS_DEFAULT_REGION: us-east-1
2024-12-18T00:21:45.0896709Z ##[endgroup]
2024-12-18T00:21:46.9048120Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-12-18T00:21:46.9048728Z Configure a credential helper to remove this warning. See
2024-12-18T00:21:46.9049303Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-12-18T00:21:46.9049712Z 
2024-12-18T00:21:46.9049819Z Login Succeeded
2024-12-18T00:21:47.2400601Z Command completed after 1 attempt(s).
2024-12-18T00:21:47.2482955Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2024-12-18T00:21:47.2483479Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2024-12-18T00:21:47.2483932Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2024-12-18T00:21:47.2494153Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:47.2494526Z env:
2024-12-18T00:21:47.2494748Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:47.2495016Z ##[endgroup]
2024-12-18T00:21:47.2599442Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2024-12-18T00:21:47.2599991Z # ignore expansion of "docker ps -q" since it could be empty
2024-12-18T00:21:47.2600406Z # shellcheck disable=SC2046
2024-12-18T00:21:47.2600733Z docker stop $(docker ps -q) || true
2024-12-18T00:21:47.2601091Z # Prune all of the docker images
2024-12-18T00:21:47.2601509Z docker system prune -af
2024-12-18T00:21:47.2609600Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:47.2609968Z env:
2024-12-18T00:21:47.2610187Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:47.2610444Z ##[endgroup]
2024-12-18T00:21:47.2943305Z "docker stop" requires at least 1 argument.
2024-12-18T00:21:47.2943684Z See 'docker stop --help'.
2024-12-18T00:21:47.2943866Z 
2024-12-18T00:21:47.2944057Z Usage:  docker stop [OPTIONS] CONTAINER [CONTAINER...]
2024-12-18T00:21:47.2944328Z 
2024-12-18T00:21:47.2944442Z Stop one or more running containers
2024-12-18T00:21:47.3557632Z Total reclaimed space: 0B
2024-12-18T00:21:47.3618952Z ##[group]Run set +e
2024-12-18T00:21:47.3619243Z set +e
2024-12-18T00:21:47.3619480Z set -x
2024-12-18T00:21:47.3619707Z 
2024-12-18T00:21:47.3619959Z PT_DOMAIN=download.pytorch.org
2024-12-18T00:21:47.3620698Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400,
2024-12-18T00:21:47.3621426Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last
2024-12-18T00:21:47.3621940Z # one is returned at random
2024-12-18T00:21:47.3622331Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1)
2024-12-18T00:21:47.3622701Z 
2024-12-18T00:21:47.3623106Z if [ -z "${RESOLVED_IP}" ]; then
2024-12-18T00:21:47.3623532Z   echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..."
2024-12-18T00:21:47.3624044Z   RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1)
2024-12-18T00:21:47.3624431Z 
2024-12-18T00:21:47.3624676Z   if [ -z "${RESOLVED_IP}" ]; then
2024-12-18T00:21:47.3625061Z     echo "Couldn't resolve ${PT_DOMAIN}, exiting..."
2024-12-18T00:21:47.3625417Z     exit 1
2024-12-18T00:21:47.3625648Z   fi
2024-12-18T00:21:47.3625875Z fi
2024-12-18T00:21:47.3626091Z 
2024-12-18T00:21:47.3626356Z if grep -r "${PT_DOMAIN}" /etc/hosts; then
2024-12-18T00:21:47.3626720Z   # Clean up any old records first
2024-12-18T00:21:47.3627074Z   sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts
2024-12-18T00:21:47.3627399Z fi
2024-12-18T00:21:47.3627613Z 
2024-12-18T00:21:47.3627929Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts
2024-12-18T00:21:47.3628321Z cat /etc/hosts
2024-12-18T00:21:47.3637272Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:47.3637670Z env:
2024-12-18T00:21:47.3637896Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:47.3638175Z ##[endgroup]
2024-12-18T00:21:47.3668734Z + PT_DOMAIN=download.pytorch.org
2024-12-18T00:21:47.3674553Z ++ dig -4 +short download.pytorch.org
2024-12-18T00:21:47.3675055Z ++ tail -n1
2024-12-18T00:21:47.4200568Z + RESOLVED_IP=18.160.18.37
2024-12-18T00:21:47.4200896Z + '[' -z 18.160.18.37 ']'
2024-12-18T00:21:47.4201191Z + grep -r download.pytorch.org /etc/hosts
2024-12-18T00:21:47.4219937Z + echo '18.160.18.37 download.pytorch.org'
2024-12-18T00:21:47.4220865Z + sudo tee -a /etc/hosts
2024-12-18T00:21:47.5337074Z 18.160.18.37 download.pytorch.org
2024-12-18T00:21:47.5359004Z + cat /etc/hosts
2024-12-18T00:21:47.5370238Z 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
2024-12-18T00:21:47.5375568Z ::1         localhost6 localhost6.localdomain6
2024-12-18T00:21:47.5375915Z 18.160.18.37 download.pytorch.org
2024-12-18T00:21:47.5533935Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@release/2.6
2024-12-18T00:21:47.5534416Z with:
2024-12-18T00:21:47.5535334Z   docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:47.5536384Z   docker-build-dir: .ci/docker
2024-12-18T00:21:47.5536728Z   working-directory: .
2024-12-18T00:21:47.5537103Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:47.5537535Z   force-push: false
2024-12-18T00:21:47.5537770Z env:
2024-12-18T00:21:47.5537987Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:47.5538249Z ##[endgroup]
2024-12-18T00:21:47.5572074Z ##[group]Run set -ex
2024-12-18T00:21:47.5572363Z set -ex
2024-12-18T00:21:47.5572596Z 
2024-12-18T00:21:47.5572981Z # If the docker build directory or the build script doesn't exist, the action will
2024-12-18T00:21:47.5573654Z # gracefully return the docker image name as it is.  Pulling docker image in Linux
2024-12-18T00:21:47.5574201Z # job could then download the pre-built image as usual
2024-12-18T00:21:47.5574702Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then
2024-12-18T00:21:47.5575169Z   echo "skip=true" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:47.5575793Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:47.5576199Z 
2024-12-18T00:21:47.5576552Z   echo "There is no Docker build script in ${REPO_NAME} repo, skipping..."
2024-12-18T00:21:47.5576984Z   exit 0
2024-12-18T00:21:47.5577214Z else
2024-12-18T00:21:47.5577483Z   echo "skip=false" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:47.5577809Z fi
2024-12-18T00:21:47.5578024Z 
2024-12-18T00:21:47.5578356Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then
2024-12-18T00:21:47.5578934Z   # The docker image name already includes the ECR prefix and tag, so we can just
2024-12-18T00:21:47.5579450Z   # use it as it is, but first let's extract the tag
2024-12-18T00:21:47.5579921Z   DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}')
2024-12-18T00:21:47.5580417Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:47.5580903Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:47.5581301Z else
2024-12-18T00:21:47.5581606Z   DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}")
2024-12-18T00:21:47.5582059Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:47.5582675Z   echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:47.5583220Z fi
2024-12-18T00:21:47.5592418Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:47.5592789Z env:
2024-12-18T00:21:47.5593002Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:47.5593271Z   REPO_NAME: pytorch
2024-12-18T00:21:47.5594107Z   DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:47.5595002Z   DOCKER_BUILD_DIR: .ci/docker
2024-12-18T00:21:47.5595374Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:47.5595758Z ##[endgroup]
2024-12-18T00:21:47.5623720Z + [[ ! -d .ci/docker ]]
2024-12-18T00:21:47.5623996Z + [[ ! -f .ci/docker/build.sh ]]
2024-12-18T00:21:47.5624273Z + echo skip=false
2024-12-18T00:21:47.5625551Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]]
2024-12-18T00:21:47.5631538Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:47.5632413Z ++ awk -F '[:,]' '{print $2}'
2024-12-18T00:21:47.5655776Z + DOCKER_TAG=45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:47.5656286Z + echo docker-tag=45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:47.5657242Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:47.5698793Z ##[group]Run set +e
2024-12-18T00:21:47.5699102Z set +e
2024-12-18T00:21:47.5699340Z set -x
2024-12-18T00:21:47.5699574Z 
2024-12-18T00:21:47.5699787Z login() {
2024-12-18T00:21:47.5700274Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2024-12-18T00:21:47.5700788Z }
2024-12-18T00:21:47.5701007Z 
2024-12-18T00:21:47.5701245Z retry () {
2024-12-18T00:21:47.5701529Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2024-12-18T00:21:47.5701849Z }
2024-12-18T00:21:47.5702070Z 
2024-12-18T00:21:47.5702304Z retry login "${DOCKER_REGISTRY}"
2024-12-18T00:21:47.5702613Z 
2024-12-18T00:21:47.5702986Z START_TIME=$(date +%s)
2024-12-18T00:21:47.5703283Z # Wait up to 90 minutes
2024-12-18T00:21:47.5703648Z while [[ $(( $(date +%s) - 5400 )) -lt $START_TIME ]]; do
2024-12-18T00:21:47.5704129Z   # Check if image already exists, if it does then skip building it
2024-12-18T00:21:47.5704611Z   if docker manifest inspect "${DOCKER_IMAGE}"; then
2024-12-18T00:21:47.5704978Z     exit 0
2024-12-18T00:21:47.5705222Z   fi
2024-12-18T00:21:47.5705445Z 
2024-12-18T00:21:47.5705824Z   # NB: This flag is used by Docker build workflow to push the image to ECR, so we can
2024-12-18T00:21:47.5706462Z   # use this to differentiate between the Docker build and regular build jobs. For the
2024-12-18T00:21:47.5707103Z   # latter, it will wait for the Docker images to become available before continuing
2024-12-18T00:21:47.5707614Z   if [ "${DOCKER_PUSH:-false}" == "true" ]; then
2024-12-18T00:21:47.5708016Z     # It's a Docker build job, let's build the image
2024-12-18T00:21:47.5708360Z     break
2024-12-18T00:21:47.5708594Z   else
2024-12-18T00:21:47.5708933Z     # It's a regular build job, wait for the image to become available
2024-12-18T00:21:47.5709340Z     sleep 300
2024-12-18T00:21:47.5709592Z   fi
2024-12-18T00:21:47.5709817Z done
2024-12-18T00:21:47.5710034Z 
2024-12-18T00:21:47.5710388Z # NB: This part requires a full checkout. Otherwise, the merge base will
2024-12-18T00:21:47.5710945Z # be empty.  The default action would be to continue rebuild the image
2024-12-18T00:21:47.5711452Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then
2024-12-18T00:21:47.5711899Z   # if we're on the base branch then use the parent commit
2024-12-18T00:21:47.5712298Z   MERGE_BASE=$(git rev-parse HEAD~)
2024-12-18T00:21:47.5712614Z else
2024-12-18T00:21:47.5712940Z   # otherwise we're on a PR, so use the most recent base commit
2024-12-18T00:21:47.5713414Z   MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION")
2024-12-18T00:21:47.5713777Z fi
2024-12-18T00:21:47.5714001Z 
2024-12-18T00:21:47.5714247Z if [[ -z "${MERGE_BASE}" ]]; then
2024-12-18T00:21:47.5714599Z   echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:47.5714928Z 
2024-12-18T00:21:47.5715547Z   echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..."
2024-12-18T00:21:47.5716089Z   exit 0
2024-12-18T00:21:47.5716326Z fi
2024-12-18T00:21:47.5716565Z 
2024-12-18T00:21:47.5716898Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then
2024-12-18T00:21:47.5717567Z   echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit"
2024-12-18T00:21:47.5718135Z   exit 1
2024-12-18T00:21:47.5718375Z fi
2024-12-18T00:21:47.5718588Z 
2024-12-18T00:21:47.5718949Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}")
2024-12-18T00:21:47.5719591Z # If no image exists but the hash is the same as the previous hash then we should error out here
2024-12-18T00:21:47.5720168Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then
2024-12-18T00:21:47.5720834Z   echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch"
2024-12-18T00:21:47.5721697Z   echo "         Will re-build docker image to store in local cache, TTS may be longer"
2024-12-18T00:21:47.5722145Z fi
2024-12-18T00:21:47.5722358Z 
2024-12-18T00:21:47.5722623Z echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2024-12-18T00:21:47.5730688Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:47.5731156Z env:
2024-12-18T00:21:47.5731377Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:47.5731648Z   DOCKER_BUILD_DIR: .ci/docker
2024-12-18T00:21:47.5731997Z   BASE_REVISION: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:21:47.5732932Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:47.5733839Z   DOCKER_TAG: 45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:47.5734281Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:47.5734669Z   DOCKER_PUSH: 
2024-12-18T00:21:47.5734897Z ##[endgroup]
2024-12-18T00:21:47.5761294Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:47.5761734Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:47.5764476Z + aws ecr get-login-password --region us-east-1
2024-12-18T00:21:47.5766142Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:48.1145505Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-12-18T00:21:48.1146254Z Configure a credential helper to remove this warning. See
2024-12-18T00:21:48.1146995Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-12-18T00:21:48.1147475Z 
2024-12-18T00:21:48.1147613Z Login Succeeded
2024-12-18T00:21:48.1173178Z ++ date +%s
2024-12-18T00:21:48.1185044Z + START_TIME=1734481308
2024-12-18T00:21:48.1189616Z ++ date +%s
2024-12-18T00:21:48.1201539Z + [[ 1734475908 -lt 1734481308 ]]
2024-12-18T00:21:48.1202463Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:48.3866966Z {
2024-12-18T00:21:48.3867259Z 	"schemaVersion": 2,
2024-12-18T00:21:48.3867740Z 	"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
2024-12-18T00:21:48.3868201Z 	"config": {
2024-12-18T00:21:48.3868537Z 		"mediaType": "application/vnd.docker.container.image.v1+json",
2024-12-18T00:21:48.3868943Z 		"size": 52680,
2024-12-18T00:21:48.3869354Z 		"digest": "sha256:67e93a8badaf799ceb664d7c453077c6c6785d56dc10c720be9358edeb91759b"
2024-12-18T00:21:48.3869828Z 	},
2024-12-18T00:21:48.3870031Z 	"layers": [
2024-12-18T00:21:48.3870244Z 		{
2024-12-18T00:21:48.3870564Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3871048Z 			"size": 28584506,
2024-12-18T00:21:48.3871805Z 			"digest": "sha256:80888bc6716fcbb8874e75ac88898d3e38e6f1bc55678f0e97ca9d706b7f3733"
2024-12-18T00:21:48.3872443Z 		},
2024-12-18T00:21:48.3872702Z 		{
2024-12-18T00:21:48.3873068Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3873480Z 			"size": 7944698,
2024-12-18T00:21:48.3873904Z 			"digest": "sha256:fbcd35dc5bc3a7bda41926aadd083020f942b001ebac6f1d30480f0f065394c0"
2024-12-18T00:21:48.3874392Z 		},
2024-12-18T00:21:48.3874585Z 		{
2024-12-18T00:21:48.3874909Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3875313Z 			"size": 57593527,
2024-12-18T00:21:48.3875739Z 			"digest": "sha256:c7232af9ae05f7de83f8d6171bd0c35a4dd0a85ebafb15b950dbc08f89ea5fb5"
2024-12-18T00:21:48.3876211Z 		},
2024-12-18T00:21:48.3876404Z 		{
2024-12-18T00:21:48.3876725Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3877131Z 			"size": 185,
2024-12-18T00:21:48.3877550Z 			"digest": "sha256:db6cdef1932a0d9ca6ef9a539e08d491f66d1b1ed81926ae1525375bdd8100cc"
2024-12-18T00:21:48.3878024Z 		},
2024-12-18T00:21:48.3878218Z 		{
2024-12-18T00:21:48.3878538Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3878943Z 			"size": 6886,
2024-12-18T00:21:48.3879345Z 			"digest": "sha256:56dc8550293751a1604e97ac949cfae82ba20cb2a28e034737bafd7382559609"
2024-12-18T00:21:48.3879802Z 		},
2024-12-18T00:21:48.3880164Z 		{
2024-12-18T00:21:48.3880477Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3880893Z 			"size": 1374287670,
2024-12-18T00:21:48.3881401Z 			"digest": "sha256:30c0ea6140d07e2a8deb70d780f277c63cf61836ff33d66eef944728a4bef6bd"
2024-12-18T00:21:48.3881869Z 		},
2024-12-18T00:21:48.3882061Z 		{
2024-12-18T00:21:48.3882375Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3882788Z 			"size": 62641,
2024-12-18T00:21:48.3883197Z 			"digest": "sha256:71bdb1a72c2d6dc97bbdbca82383f0260c4ee87556701e8e606c08a6bb0f0da5"
2024-12-18T00:21:48.3883669Z 		},
2024-12-18T00:21:48.3883877Z 		{
2024-12-18T00:21:48.3884190Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3884601Z 			"size": 1685,
2024-12-18T00:21:48.3885007Z 			"digest": "sha256:4829486be7c30f19f4136fa56adbb3de206ed0bbf0705b59fb2147406778ce38"
2024-12-18T00:21:48.3885475Z 		},
2024-12-18T00:21:48.3885671Z 		{
2024-12-18T00:21:48.3885985Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3886409Z 			"size": 1523,
2024-12-18T00:21:48.3886818Z 			"digest": "sha256:1f4e68d7b5e4224ba1da78ef461ff7f01e8d59c09d39281277521384105a9441"
2024-12-18T00:21:48.3887282Z 		},
2024-12-18T00:21:48.3887478Z 		{
2024-12-18T00:21:48.3887791Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3888208Z 			"size": 2626741021,
2024-12-18T00:21:48.3888627Z 			"digest": "sha256:7c373e2d9b7e82a6878d4a31293dd857915a0fe47d07dce541cea03b043d57fc"
2024-12-18T00:21:48.3889087Z 		},
2024-12-18T00:21:48.3889286Z 		{
2024-12-18T00:21:48.3889604Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3890021Z 			"size": 86545,
2024-12-18T00:21:48.3890421Z 			"digest": "sha256:6223811417458a3c93b84ee3b65f8b08d9e2828b926f0aed863041610d7d95d4"
2024-12-18T00:21:48.3890865Z 		},
2024-12-18T00:21:48.3891057Z 		{
2024-12-18T00:21:48.3891374Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3891793Z 			"size": 1824,
2024-12-18T00:21:48.3892210Z 			"digest": "sha256:f003d065e31e5e8c0cea490c88a912dd331bddcbd2f5ff22ccc692a5343bcbeb"
2024-12-18T00:21:48.3892682Z 		},
2024-12-18T00:21:48.3892871Z 		{
2024-12-18T00:21:48.3893187Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3893603Z 			"size": 246420139,
2024-12-18T00:21:48.3894020Z 			"digest": "sha256:cd6ce71168153e485b99493ce07032bbd9fd5cefceb68ef33ce088244306a653"
2024-12-18T00:21:48.3894483Z 		},
2024-12-18T00:21:48.3894675Z 		{
2024-12-18T00:21:48.3895098Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3895514Z 			"size": 703,
2024-12-18T00:21:48.3895940Z 			"digest": "sha256:e843066a8f68042266cfe814f1729975619fb8f733f371be115e8b2fe81cbc9d"
2024-12-18T00:21:48.3896485Z 		},
2024-12-18T00:21:48.3896683Z 		{
2024-12-18T00:21:48.3897001Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3897424Z 			"size": 1256,
2024-12-18T00:21:48.3897840Z 			"digest": "sha256:bac50ca2930bc4b1afa8a2f1b4b0d1d499d63b7922b784af690c3b31ec0c2f70"
2024-12-18T00:21:48.3898319Z 		},
2024-12-18T00:21:48.3898512Z 		{
2024-12-18T00:21:48.3898832Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3899248Z 			"size": 484,
2024-12-18T00:21:48.3899651Z 			"digest": "sha256:57e52c6751f4d865ada4cae919859fe563fe5dcbb93ca82cb4746f98caccbd3a"
2024-12-18T00:21:48.3900121Z 		},
2024-12-18T00:21:48.3900313Z 		{
2024-12-18T00:21:48.3900631Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3901047Z 			"size": 91715265,
2024-12-18T00:21:48.3901456Z 			"digest": "sha256:a9c5c14d77c035c7501566b1247024dfd241bd6bc12fbe2bca28ec7d90b26069"
2024-12-18T00:21:48.3901920Z 		},
2024-12-18T00:21:48.3902113Z 		{
2024-12-18T00:21:48.3902427Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3902845Z 			"size": 3706,
2024-12-18T00:21:48.3903341Z 			"digest": "sha256:b86099b8619ad88e8fe6c8e693180d52e0f2fbeac5927f0b403c8a9ef76d7f6b"
2024-12-18T00:21:48.3903802Z 		},
2024-12-18T00:21:48.3903998Z 		{
2024-12-18T00:21:48.3904311Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3904726Z 			"size": 1860,
2024-12-18T00:21:48.3905138Z 			"digest": "sha256:3c8ea8a600fff6cce4c1b36acc5f3949ecd664e55b5eaf4a50fe5499351574a6"
2024-12-18T00:21:48.3905616Z 		},
2024-12-18T00:21:48.3905814Z 		{
2024-12-18T00:21:48.3906129Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3906588Z 			"size": 701,
2024-12-18T00:21:48.3907011Z 			"digest": "sha256:bebd539a1d598fe86afc0da77bf16ae4c7ebfb9f031f21ed7634150fbe23f996"
2024-12-18T00:21:48.3907492Z 		},
2024-12-18T00:21:48.3907687Z 		{
2024-12-18T00:21:48.3908002Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3908418Z 			"size": 478,
2024-12-18T00:21:48.3908808Z 			"digest": "sha256:292b18ec2dab9bd60d00192387c1269d39b0b4886b782585d2c879808218fc66"
2024-12-18T00:21:48.3909269Z 		},
2024-12-18T00:21:48.3909458Z 		{
2024-12-18T00:21:48.3909772Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3910187Z 			"size": 2941869535,
2024-12-18T00:21:48.3910601Z 			"digest": "sha256:76e3817b4b4cec0a5b005161867dc257681b792271ba9d23c9b79b3a6801dcd9"
2024-12-18T00:21:48.3911057Z 		},
2024-12-18T00:21:48.3911247Z 		{
2024-12-18T00:21:48.3911558Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3911980Z 			"size": 380,
2024-12-18T00:21:48.3912383Z 			"digest": "sha256:50b1f245e9f5be8670d8ba5790e3b96450eee3428dcac7060367db48ae00234e"
2024-12-18T00:21:48.3912846Z 		},
2024-12-18T00:21:48.3913039Z 		{
2024-12-18T00:21:48.3913352Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3913764Z 			"size": 12901,
2024-12-18T00:21:48.3914178Z 			"digest": "sha256:ad93cf5205f24ada86b5f7eab5b63ed97ae7858368f6fdbce373343d2568d66e"
2024-12-18T00:21:48.3914653Z 		},
2024-12-18T00:21:48.3914848Z 		{
2024-12-18T00:21:48.3915163Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3915574Z 			"size": 863,
2024-12-18T00:21:48.3915981Z 			"digest": "sha256:8d17c59eaf7d84d49e3660380debcc70734b2a178e6cab86f0d4fd3e7a8100d1"
2024-12-18T00:21:48.3916448Z 		},
2024-12-18T00:21:48.3916643Z 		{
2024-12-18T00:21:48.3916959Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3917375Z 			"size": 106,
2024-12-18T00:21:48.3917889Z 			"digest": "sha256:b36a8ccf5c8c8ef0bf3e168ec9f568a0ada36c49103c9cacfef6dd072ddd681a"
2024-12-18T00:21:48.3918365Z 		},
2024-12-18T00:21:48.3918560Z 		{
2024-12-18T00:21:48.3918873Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3919288Z 			"size": 504,
2024-12-18T00:21:48.3919690Z 			"digest": "sha256:aa0a3b87e9e34ab15903206018bfd9fb1eb54fd21f05a61ef254557703890732"
2024-12-18T00:21:48.3920162Z 		},
2024-12-18T00:21:48.3920354Z 		{
2024-12-18T00:21:48.3920667Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3921081Z 			"size": 121477314,
2024-12-18T00:21:48.3921563Z 			"digest": "sha256:72a632052c5be4900c60ce09e32f8a0fd3b637ab0b7685dc5b07bda4776ddd15"
2024-12-18T00:21:48.3922022Z 		},
2024-12-18T00:21:48.3922213Z 		{
2024-12-18T00:21:48.3922526Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3922940Z 			"size": 109,
2024-12-18T00:21:48.3923352Z 			"digest": "sha256:ceed6316bc249ffffee9563cb7505023b3e70fba306f187bd612fece63508263"
2024-12-18T00:21:48.3923816Z 		},
2024-12-18T00:21:48.3924008Z 		{
2024-12-18T00:21:48.3924318Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3924728Z 			"size": 489,
2024-12-18T00:21:48.3925132Z 			"digest": "sha256:d0ca0f491f809a9f1dedd391f5b25cc3f59d8e95e7f76299f18ec3c931e56ed1"
2024-12-18T00:21:48.3925601Z 		},
2024-12-18T00:21:48.3925901Z 		{
2024-12-18T00:21:48.3926258Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3926665Z 			"size": 296,
2024-12-18T00:21:48.3927057Z 			"digest": "sha256:cf02c706cd567c781eeb567c90960801dc1ef78872b73fbc047df898951ecbb8"
2024-12-18T00:21:48.3927515Z 		},
2024-12-18T00:21:48.3927697Z 		{
2024-12-18T00:21:48.3928014Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3928426Z 			"size": 103,
2024-12-18T00:21:48.3928823Z 			"digest": "sha256:16073d3f107dab837d7edb5e6044449660fec7e2ded263ca81d5dcfea5beb486"
2024-12-18T00:21:48.3929291Z 		},
2024-12-18T00:21:48.3929475Z 		{
2024-12-18T00:21:48.3929788Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3930202Z 			"size": 1473,
2024-12-18T00:21:48.3930605Z 			"digest": "sha256:d234faf2f1bfb6cb9d4ba14103226d20d540e568b47690b67c2a2b2c5a49645f"
2024-12-18T00:21:48.3931071Z 		},
2024-12-18T00:21:48.3931253Z 		{
2024-12-18T00:21:48.3931572Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3931990Z 			"size": 430258758,
2024-12-18T00:21:48.3932409Z 			"digest": "sha256:3f0ade86af1a5bc72c9088735bb1a7ab9819fa20fdd96437fc080b2c711dfd3d"
2024-12-18T00:21:48.3932870Z 		},
2024-12-18T00:21:48.3933054Z 		{
2024-12-18T00:21:48.3933375Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3933785Z 			"size": 164,
2024-12-18T00:21:48.3934185Z 			"digest": "sha256:034b6a03f5f751ed12f7505175ce3443eb3ed7b34d02d1bb25de0ee52d93a44b"
2024-12-18T00:21:48.3934647Z 		},
2024-12-18T00:21:48.3949967Z 		{
2024-12-18T00:21:48.3950332Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3950761Z 			"size": 564,
2024-12-18T00:21:48.3951170Z 			"digest": "sha256:21f8f066852dd484ccbf79470452fe8005cde08773dd4fedc07fed783a0b9bf8"
2024-12-18T00:21:48.3951636Z 		},
2024-12-18T00:21:48.3951823Z 		{
2024-12-18T00:21:48.3952146Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3952571Z 			"size": 35865656,
2024-12-18T00:21:48.3952983Z 			"digest": "sha256:9dc1f315a979ec860a882d7df33584e02c1d18f86293521839ce42e21082e639"
2024-12-18T00:21:48.3953450Z 		},
2024-12-18T00:21:48.3953638Z 		{
2024-12-18T00:21:48.3953959Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3954376Z 			"size": 104,
2024-12-18T00:21:48.3954770Z 			"digest": "sha256:cd6a6063178b945c150fe21a31df6e33203295dd814802e0658835e810fb2bf9"
2024-12-18T00:21:48.3955230Z 		},
2024-12-18T00:21:48.3955416Z 		{
2024-12-18T00:21:48.3955864Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3956286Z 			"size": 425,
2024-12-18T00:21:48.3956676Z 			"digest": "sha256:2116738f4ea9299766a55629d39b881c4d33fc4fc0ce915965be1a850fcd7aaa"
2024-12-18T00:21:48.3957137Z 		},
2024-12-18T00:21:48.3957321Z 		{
2024-12-18T00:21:48.3957642Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3958114Z 			"size": 20262160,
2024-12-18T00:21:48.3958537Z 			"digest": "sha256:d7fa134316d91d2ec39bceb121619d44cdfdf1c7ac30c72768a9fd48f9763bd1"
2024-12-18T00:21:48.3958998Z 		},
2024-12-18T00:21:48.3959189Z 		{
2024-12-18T00:21:48.3959507Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3959919Z 			"size": 643,
2024-12-18T00:21:48.3960321Z 			"digest": "sha256:af03dc3fd15ea8f5f1fa41d514400f6a1d963985cf046b09d789f7f0359b2aba"
2024-12-18T00:21:48.3960784Z 		},
2024-12-18T00:21:48.3960975Z 		{
2024-12-18T00:21:48.3961365Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3961779Z 			"size": 701,
2024-12-18T00:21:48.3962193Z 			"digest": "sha256:bebd539a1d598fe86afc0da77bf16ae4c7ebfb9f031f21ed7634150fbe23f996"
2024-12-18T00:21:48.3962667Z 		},
2024-12-18T00:21:48.3962861Z 		{
2024-12-18T00:21:48.3963182Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3963594Z 			"size": 142,
2024-12-18T00:21:48.3964120Z 			"digest": "sha256:fd9de3dc425828e7b8984d4ced05f4102762137b81a5dc67104fc6b0805c570d"
2024-12-18T00:21:48.3964576Z 		},
2024-12-18T00:21:48.3964768Z 		{
2024-12-18T00:21:48.3965086Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3965504Z 			"size": 135,
2024-12-18T00:21:48.3965904Z 			"digest": "sha256:b85093661f7768ac691f4b3ee5f553b9ab14cb2a6877c8edc6133d74caa88f69"
2024-12-18T00:21:48.3966367Z 		},
2024-12-18T00:21:48.3966557Z 		{
2024-12-18T00:21:48.3967225Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3967652Z 			"size": 4995482131,
2024-12-18T00:21:48.3968078Z 			"digest": "sha256:ab6ef6c2ac4d23af7a3249cee1d4761f63a89513796ba720da7cc425c8afa6d2"
2024-12-18T00:21:48.3968543Z 		},
2024-12-18T00:21:48.3968734Z 		{
2024-12-18T00:21:48.3969054Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3969467Z 			"size": 196,
2024-12-18T00:21:48.3969864Z 			"digest": "sha256:56c896077120aaa4cbcc2203662d13b4ea14cf998e9657e91b252535ceae5348"
2024-12-18T00:21:48.3970323Z 		},
2024-12-18T00:21:48.3970517Z 		{
2024-12-18T00:21:48.3970837Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3971259Z 			"size": 564,
2024-12-18T00:21:48.3971662Z 			"digest": "sha256:d88e9c5e7c70fe36bd1bd2e94d43d4192d94a1f28e323234a73a75a2d1da48f1"
2024-12-18T00:21:48.3972122Z 		},
2024-12-18T00:21:48.3972314Z 		{
2024-12-18T00:21:48.3972636Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3973052Z 			"size": 43164612,
2024-12-18T00:21:48.3973468Z 			"digest": "sha256:53617ee83302d7f97eb56f6756d78d18f8dc41bf252bf2d04cb643198f400f6f"
2024-12-18T00:21:48.3973925Z 		},
2024-12-18T00:21:48.3974117Z 		{
2024-12-18T00:21:48.3974438Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3974852Z 			"size": 106,
2024-12-18T00:21:48.3975261Z 			"digest": "sha256:3edbde10d4043bf2ebf32fddfa55c8d0923435465637df7dc8f4addcc03a0b84"
2024-12-18T00:21:48.3975724Z 		},
2024-12-18T00:21:48.3975914Z 		{
2024-12-18T00:21:48.3976233Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3976672Z 			"size": 1402,
2024-12-18T00:21:48.3977121Z 			"digest": "sha256:f4038f7524c3ed705e7560db6dd6efe76f548363f16bc7a93cfe360d5bc6dcaa"
2024-12-18T00:21:48.3977580Z 		},
2024-12-18T00:21:48.3977775Z 		{
2024-12-18T00:21:48.3978093Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3978506Z 			"size": 701,
2024-12-18T00:21:48.3979097Z 			"digest": "sha256:bebd539a1d598fe86afc0da77bf16ae4c7ebfb9f031f21ed7634150fbe23f996"
2024-12-18T00:21:48.3979574Z 		},
2024-12-18T00:21:48.3979771Z 		{
2024-12-18T00:21:48.3980089Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3980501Z 			"size": 137,
2024-12-18T00:21:48.3980897Z 			"digest": "sha256:4a51173f77576ab68fba2f862dd4d12b9ff1f250145c090520b05f88f5c81391"
2024-12-18T00:21:48.3981358Z 		},
2024-12-18T00:21:48.3981552Z 		{
2024-12-18T00:21:48.3981869Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3982278Z 			"size": 120,
2024-12-18T00:21:48.3982674Z 			"digest": "sha256:ed6a71d41274e5b1eea4025d24a4851d6880b902b08f7d4dd0618ea81154f575"
2024-12-18T00:21:48.3983125Z 		},
2024-12-18T00:21:48.3983318Z 		{
2024-12-18T00:21:48.3983636Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3984048Z 			"size": 4886226234,
2024-12-18T00:21:48.3984472Z 			"digest": "sha256:389dc939e2a414ef26b642b1e8657a710358afa6a6be3ebc4380b01ffacbbacc"
2024-12-18T00:21:48.3984931Z 		},
2024-12-18T00:21:48.3985121Z 		{
2024-12-18T00:21:48.3985439Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3985856Z 			"size": 174,
2024-12-18T00:21:48.3986313Z 			"digest": "sha256:0341eaeed33df462f8bd28003bb9b09739b0786abba58ccc17ba4cec535ee4ef"
2024-12-18T00:21:48.3986771Z 		},
2024-12-18T00:21:48.3987100Z 		{
2024-12-18T00:21:48.3987420Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3987831Z 			"size": 908,
2024-12-18T00:21:48.3988234Z 			"digest": "sha256:537cdb2d31685c0dbaf612d24633539a141cfcb69d90c5ce932923a6e4ab4474"
2024-12-18T00:21:48.3988694Z 		},
2024-12-18T00:21:48.3988887Z 		{
2024-12-18T00:21:48.3989206Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3989621Z 			"size": 701,
2024-12-18T00:21:48.3990042Z 			"digest": "sha256:bebd539a1d598fe86afc0da77bf16ae4c7ebfb9f031f21ed7634150fbe23f996"
2024-12-18T00:21:48.3990523Z 		},
2024-12-18T00:21:48.3990716Z 		{
2024-12-18T00:21:48.3991035Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3991448Z 			"size": 135,
2024-12-18T00:21:48.3991837Z 			"digest": "sha256:76d4a07024975dd727ff7b1c8873c513043bc9a5d310b469e51c3b110fdb6084"
2024-12-18T00:21:48.3992290Z 		},
2024-12-18T00:21:48.3992485Z 		{
2024-12-18T00:21:48.3992802Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3993218Z 			"size": 32,
2024-12-18T00:21:48.3993618Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-12-18T00:21:48.3994082Z 		},
2024-12-18T00:21:48.3994276Z 		{
2024-12-18T00:21:48.3994597Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3995010Z 			"size": 159,
2024-12-18T00:21:48.3995399Z 			"digest": "sha256:3287a539d61e3913b93941ecd49e55887b5122d63403e0fe9e7a3a63b6ed03c6"
2024-12-18T00:21:48.3995848Z 		},
2024-12-18T00:21:48.3996042Z 		{
2024-12-18T00:21:48.3996370Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3996826Z 			"size": 1932,
2024-12-18T00:21:48.3997216Z 			"digest": "sha256:9b763e24a7d42c731eb337815675c9478832146e73305559d0db1c88a0731492"
2024-12-18T00:21:48.3997663Z 		},
2024-12-18T00:21:48.3997853Z 		{
2024-12-18T00:21:48.3998173Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.3998595Z 			"size": 196810750,
2024-12-18T00:21:48.3999008Z 			"digest": "sha256:f6ef3d4c814e330a230273579dc978970563774df75f73fba78181ddcf6ab98e"
2024-12-18T00:21:48.3999462Z 		},
2024-12-18T00:21:48.3999659Z 		{
2024-12-18T00:21:48.3999977Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.4000387Z 			"size": 165,
2024-12-18T00:21:48.4000784Z 			"digest": "sha256:06914b00eb49b8005245cba4a631c34d0a2aaf8bf6291bc6477705206de11a06"
2024-12-18T00:21:48.4001301Z 		},
2024-12-18T00:21:48.4001492Z 		{
2024-12-18T00:21:48.4001905Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.4002323Z 			"size": 7943,
2024-12-18T00:21:48.4002711Z 			"digest": "sha256:724553524224508e03de9422ca47b1a1c047e53cd305c2a1344147bb0d82d1d0"
2024-12-18T00:21:48.4003167Z 		},
2024-12-18T00:21:48.4003359Z 		{
2024-12-18T00:21:48.4003678Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.4004092Z 			"size": 8070,
2024-12-18T00:21:48.4004493Z 			"digest": "sha256:47d9fb6dae26902b0fdb744916689adbd42add22a476d1c5ec89ae4e30b91e39"
2024-12-18T00:21:48.4004963Z 		},
2024-12-18T00:21:48.4005154Z 		{
2024-12-18T00:21:48.4005468Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.4005881Z 			"size": 303,
2024-12-18T00:21:48.4006287Z 			"digest": "sha256:acce7b690dcb2fde77ba3ce65df70452fc7c98de73a42fe72652e477f86fc62d"
2024-12-18T00:21:48.4006813Z 		},
2024-12-18T00:21:48.4007003Z 		{
2024-12-18T00:21:48.4007326Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.4007742Z 			"size": 7633763,
2024-12-18T00:21:48.4008148Z 			"digest": "sha256:58dc9f64def75bcb2abecdf39b798c3800e837d37da971e766a05731ea5537c2"
2024-12-18T00:21:48.4008625Z 		},
2024-12-18T00:21:48.4008817Z 		{
2024-12-18T00:21:48.4009133Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.4009544Z 			"size": 108,
2024-12-18T00:21:48.4010029Z 			"digest": "sha256:94dfe5ef1b7e91ebd6a1a5acc1d7732e051334d5102d8a69c509c1d2f6e95598"
2024-12-18T00:21:48.4010501Z 		},
2024-12-18T00:21:48.4010691Z 		{
2024-12-18T00:21:48.4011007Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.4011418Z 			"size": 54145665,
2024-12-18T00:21:48.4011813Z 			"digest": "sha256:40a2c9135b0683a7e2432748a9a6ff19926f6a2a49722d844c07e307320f7362"
2024-12-18T00:21:48.4012277Z 		},
2024-12-18T00:21:48.4012472Z 		{
2024-12-18T00:21:48.4012790Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.4013202Z 			"size": 484,
2024-12-18T00:21:48.4013603Z 			"digest": "sha256:8839d6b0c8218eccb426367e5bfe066b172b8c6059bae8473e915e11c4e582a9"
2024-12-18T00:21:48.4014068Z 		},
2024-12-18T00:21:48.4014263Z 		{
2024-12-18T00:21:48.4014583Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.4014989Z 			"size": 1374859271,
2024-12-18T00:21:48.4015400Z 			"digest": "sha256:1c588f0a9e97f3e192252a179d987d1ba1e5201ba4b6e74c429074e9e34df1b2"
2024-12-18T00:21:48.4015881Z 		},
2024-12-18T00:21:48.4016104Z 		{
2024-12-18T00:21:48.4016432Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.4016840Z 			"size": 106,
2024-12-18T00:21:48.4017243Z 			"digest": "sha256:6727e7e279360b8b66f0137a15b357efede06e69bfd0800ebad6ed7c8ed71df4"
2024-12-18T00:21:48.4017706Z 		},
2024-12-18T00:21:48.4017894Z 		{
2024-12-18T00:21:48.4018210Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.4018618Z 			"size": 568,
2024-12-18T00:21:48.4019016Z 			"digest": "sha256:3a55100d1957b103c43e314deb55697c459aa6fa225f91e7b1c69bc4c01dca4e"
2024-12-18T00:21:48.4019462Z 		},
2024-12-18T00:21:48.4019653Z 		{
2024-12-18T00:21:48.4019972Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.4020380Z 			"size": 303706071,
2024-12-18T00:21:48.4020786Z 			"digest": "sha256:3d6f160d335161c5321e13ce2823ec68379a6a3401b07bd7378602a69c85b80c"
2024-12-18T00:21:48.4021237Z 		},
2024-12-18T00:21:48.4021428Z 		{
2024-12-18T00:21:48.4021745Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.4022150Z 			"size": 111,
2024-12-18T00:21:48.4022550Z 			"digest": "sha256:b2a8bb947a87562144fe67a39c280d4dee50cbe9d8ff349140f5db6c377517e6"
2024-12-18T00:21:48.4023015Z 		},
2024-12-18T00:21:48.4023204Z 		{
2024-12-18T00:21:48.4023522Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.4023926Z 			"size": 529,
2024-12-18T00:21:48.4024419Z 			"digest": "sha256:69a74b2360f4ede95654274ce7ef6cc1597a7234cc17e86d5cdf4c98286d97c3"
2024-12-18T00:21:48.4024886Z 		},
2024-12-18T00:21:48.4025076Z 		{
2024-12-18T00:21:48.4025391Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.4025796Z 			"size": 48289052,
2024-12-18T00:21:48.4026208Z 			"digest": "sha256:f9e12a253d89c16863b9d4d90138ffa673447c9b7f27afcba54a2a679e58b5ab"
2024-12-18T00:21:48.4026730Z 		},
2024-12-18T00:21:48.4026919Z 		{
2024-12-18T00:21:48.4027234Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.4027636Z 			"size": 106,
2024-12-18T00:21:48.4028035Z 			"digest": "sha256:8fdb06d51b96d538045b9566a381de3dc25e59d1adb9d0a4e567afcb3213f676"
2024-12-18T00:21:48.4028503Z 		},
2024-12-18T00:21:48.4028691Z 		{
2024-12-18T00:21:48.4029011Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.4029415Z 			"size": 32,
2024-12-18T00:21:48.4029823Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-12-18T00:21:48.4030296Z 		},
2024-12-18T00:21:48.4030483Z 		{
2024-12-18T00:21:48.4030800Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.4031205Z 			"size": 32,
2024-12-18T00:21:48.4031607Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-12-18T00:21:48.4032078Z 		},
2024-12-18T00:21:48.4032270Z 		{
2024-12-18T00:21:48.4032672Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.4033077Z 			"size": 32,
2024-12-18T00:21:48.4033479Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-12-18T00:21:48.4033947Z 		},
2024-12-18T00:21:48.4034136Z 		{
2024-12-18T00:21:48.4034453Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-12-18T00:21:48.4034856Z 			"size": 32,
2024-12-18T00:21:48.4035251Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-12-18T00:21:48.4035715Z 		}
2024-12-18T00:21:48.4035907Z 	]
2024-12-18T00:21:48.4036099Z }
2024-12-18T00:21:48.4036296Z + exit 0
2024-12-18T00:21:48.4076310Z ##[group]Run set -eux
2024-12-18T00:21:48.4076597Z set -eux
2024-12-18T00:21:48.4077418Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin
2024-12-18T00:21:48.4086975Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:48.4087352Z env:
2024-12-18T00:21:48.4087576Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:48.4087845Z ##[endgroup]
2024-12-18T00:21:48.4119375Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token
2024-12-18T00:21:48.4120263Z + jq --raw-output .SecretString
2024-12-18T00:21:48.4122882Z + docker login --username pytorchbot --password-stdin
2024-12-18T00:21:48.4123345Z + jq -r .docker_hub_readonly_token
2024-12-18T00:21:49.0458139Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-12-18T00:21:49.0459356Z Configure a credential helper to remove this warning. See
2024-12-18T00:21:49.0460481Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-12-18T00:21:49.0461244Z 
2024-12-18T00:21:49.0461454Z Login Succeeded
2024-12-18T00:21:49.0558998Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/}
2024-12-18T00:21:49.0559379Z tag=${ECR_DOCKER_IMAGE##*/}
2024-12-18T00:21:49.0559773Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}"
2024-12-18T00:21:49.0569196Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:49.0569570Z env:
2024-12-18T00:21:49.0569794Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:49.0570646Z   ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:49.0571516Z ##[endgroup]
2024-12-18T00:21:49.0601394Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks-45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:49.0654736Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@release/2.6
2024-12-18T00:21:49.0655200Z with:
2024-12-18T00:21:49.0656004Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:49.0657024Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:49.0657405Z env:
2024-12-18T00:21:49.0657631Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:49.0657897Z ##[endgroup]
2024-12-18T00:21:49.0721319Z ##[group]Run set -x
2024-12-18T00:21:49.0721595Z set -x
2024-12-18T00:21:49.0721825Z set +e
2024-12-18T00:21:49.0722045Z 
2024-12-18T00:21:49.0722261Z login() {
2024-12-18T00:21:49.0722725Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2024-12-18T00:21:49.0723232Z }
2024-12-18T00:21:49.0723444Z 
2024-12-18T00:21:49.0723678Z retry () {
2024-12-18T00:21:49.0723953Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2024-12-18T00:21:49.0724258Z }
2024-12-18T00:21:49.0724470Z 
2024-12-18T00:21:49.0724704Z retry login "${DOCKER_REGISTRY}"
2024-12-18T00:21:49.0725179Z 
2024-12-18T00:21:49.0725385Z set -e
2024-12-18T00:21:49.0725716Z # ignore output since only exit code is used for conditional
2024-12-18T00:21:49.0726196Z # only pull docker image if it's not available locally
2024-12-18T00:21:49.0726770Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2024-12-18T00:21:49.0727263Z   retry docker pull "${DOCKER_IMAGE}"
2024-12-18T00:21:49.0727576Z fi
2024-12-18T00:21:49.0736476Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:21:49.0736843Z env:
2024-12-18T00:21:49.0737059Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:21:49.0737900Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:49.0738874Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:49.0739265Z ##[endgroup]
2024-12-18T00:21:49.0767749Z + set +e
2024-12-18T00:21:49.0768143Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:49.0768581Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:49.0772042Z + aws ecr get-login-password --region us-east-1
2024-12-18T00:21:49.0772819Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-12-18T00:21:49.6106007Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-12-18T00:21:49.6106616Z Configure a credential helper to remove this warning. See
2024-12-18T00:21:49.6107191Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-12-18T00:21:49.6107604Z 
2024-12-18T00:21:49.6107738Z Login Succeeded
2024-12-18T00:21:49.6128875Z + set -e
2024-12-18T00:21:49.6129751Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:49.6287704Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:49.6289231Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:21:49.9025588Z 45e1356b47a284893081276eff3000b7b534f3b1: Pulling from pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks
2024-12-18T00:21:49.9026598Z 80888bc6716f: Pulling fs layer
2024-12-18T00:21:49.9026958Z fbcd35dc5bc3: Pulling fs layer
2024-12-18T00:21:49.9027677Z c7232af9ae05: Pulling fs layer
2024-12-18T00:21:49.9028106Z db6cdef1932a: Pulling fs layer
2024-12-18T00:21:49.9028417Z 56dc85502937: Pulling fs layer
2024-12-18T00:21:49.9028792Z 30c0ea6140d0: Pulling fs layer
2024-12-18T00:21:49.9029174Z 71bdb1a72c2d: Pulling fs layer
2024-12-18T00:21:49.9029580Z 4829486be7c3: Pulling fs layer
2024-12-18T00:21:49.9029992Z 1f4e68d7b5e4: Pulling fs layer
2024-12-18T00:21:49.9030394Z 7c373e2d9b7e: Pulling fs layer
2024-12-18T00:21:49.9030697Z 622381141745: Pulling fs layer
2024-12-18T00:21:49.9030971Z f003d065e31e: Pulling fs layer
2024-12-18T00:21:49.9031248Z cd6ce7116815: Pulling fs layer
2024-12-18T00:21:49.9031517Z e843066a8f68: Pulling fs layer
2024-12-18T00:21:49.9031858Z 56dc85502937: Waiting
2024-12-18T00:21:49.9032134Z bac50ca2930b: Pulling fs layer
2024-12-18T00:21:49.9032412Z 57e52c6751f4: Pulling fs layer
2024-12-18T00:21:49.9032776Z a9c5c14d77c0: Pulling fs layer
2024-12-18T00:21:49.9033077Z b86099b8619a: Pulling fs layer
2024-12-18T00:21:49.9033369Z 3c8ea8a600ff: Pulling fs layer
2024-12-18T00:21:49.9033768Z bebd539a1d59: Pulling fs layer
2024-12-18T00:21:49.9034108Z 292b18ec2dab: Pulling fs layer
2024-12-18T00:21:49.9034399Z 76e3817b4b4c: Pulling fs layer
2024-12-18T00:21:49.9034773Z 50b1f245e9f5: Pulling fs layer
2024-12-18T00:21:49.9035437Z ad93cf5205f2: Pulling fs layer
2024-12-18T00:21:49.9035808Z 8d17c59eaf7d: Pulling fs layer
2024-12-18T00:21:49.9036151Z b36a8ccf5c8c: Pulling fs layer
2024-12-18T00:21:49.9036511Z aa0a3b87e9e3: Pulling fs layer
2024-12-18T00:21:49.9036796Z 72a632052c5b: Pulling fs layer
2024-12-18T00:21:49.9037074Z ceed6316bc24: Pulling fs layer
2024-12-18T00:21:49.9037435Z d0ca0f491f80: Pulling fs layer
2024-12-18T00:21:49.9037718Z 71bdb1a72c2d: Waiting
2024-12-18T00:21:49.9038070Z cf02c706cd56: Pulling fs layer
2024-12-18T00:21:49.9038450Z f003d065e31e: Waiting
2024-12-18T00:21:49.9038850Z 16073d3f107d: Pulling fs layer
2024-12-18T00:21:49.9039249Z d234faf2f1bf: Pulling fs layer
2024-12-18T00:21:49.9039621Z e843066a8f68: Waiting
2024-12-18T00:21:49.9039893Z bac50ca2930b: Waiting
2024-12-18T00:21:49.9077485Z 3f0ade86af1a: Pulling fs layer
2024-12-18T00:21:49.9077918Z cd6ce7116815: Waiting
2024-12-18T00:21:49.9078281Z 034b6a03f5f7: Pulling fs layer
2024-12-18T00:21:49.9078654Z 21f8f066852d: Pulling fs layer
2024-12-18T00:21:49.9079067Z 9dc1f315a979: Pulling fs layer
2024-12-18T00:21:49.9079425Z cd6a6063178b: Pulling fs layer
2024-12-18T00:21:49.9079696Z 2116738f4ea9: Pulling fs layer
2024-12-18T00:21:49.9079976Z d7fa134316d9: Pulling fs layer
2024-12-18T00:21:49.9080232Z 622381141745: Waiting
2024-12-18T00:21:49.9080470Z 1f4e68d7b5e4: Waiting
2024-12-18T00:21:49.9080723Z af03dc3fd15e: Pulling fs layer
2024-12-18T00:21:49.9081006Z fd9de3dc4258: Pulling fs layer
2024-12-18T00:21:49.9081369Z b85093661f77: Pulling fs layer
2024-12-18T00:21:49.9081632Z ad93cf5205f2: Waiting
2024-12-18T00:21:49.9082178Z ab6ef6c2ac4d: Pulling fs layer
2024-12-18T00:21:49.9082463Z 56c896077120: Pulling fs layer
2024-12-18T00:21:49.9082759Z d88e9c5e7c70: Pulling fs layer
2024-12-18T00:21:49.9083039Z 53617ee83302: Pulling fs layer
2024-12-18T00:21:49.9083304Z 50b1f245e9f5: Waiting
2024-12-18T00:21:49.9083550Z 8d17c59eaf7d: Waiting
2024-12-18T00:21:49.9083791Z b36a8ccf5c8c: Waiting
2024-12-18T00:21:49.9084055Z 30c0ea6140d0: Waiting
2024-12-18T00:21:49.9084470Z 76e3817b4b4c: Waiting
2024-12-18T00:21:49.9084711Z db6cdef1932a: Waiting
2024-12-18T00:21:49.9084949Z d7fa134316d9: Waiting
2024-12-18T00:21:49.9085182Z aa0a3b87e9e3: Waiting
2024-12-18T00:21:49.9085416Z fd9de3dc4258: Waiting
2024-12-18T00:21:49.9085645Z af03dc3fd15e: Waiting
2024-12-18T00:21:49.9085901Z 3edbde10d404: Pulling fs layer
2024-12-18T00:21:49.9086177Z b85093661f77: Waiting
2024-12-18T00:21:49.9086453Z f4038f7524c3: Pulling fs layer
2024-12-18T00:21:49.9086744Z 4a51173f7757: Pulling fs layer
2024-12-18T00:21:49.9087012Z ed6a71d41274: Pulling fs layer
2024-12-18T00:21:49.9087287Z 389dc939e2a4: Pulling fs layer
2024-12-18T00:21:49.9087558Z bebd539a1d59: Waiting
2024-12-18T00:21:49.9088191Z a9c5c14d77c0: Waiting
2024-12-18T00:21:49.9088450Z 0341eaeed33d: Pulling fs layer
2024-12-18T00:21:49.9088719Z 537cdb2d3168: Pulling fs layer
2024-12-18T00:21:49.9088981Z 72a632052c5b: Waiting
2024-12-18T00:21:49.9089220Z d234faf2f1bf: Waiting
2024-12-18T00:21:49.9089465Z 76d4a0702497: Pulling fs layer
2024-12-18T00:21:49.9089739Z ceed6316bc24: Waiting
2024-12-18T00:21:49.9089968Z 034b6a03f5f7: Waiting
2024-12-18T00:21:49.9090263Z 4f4fb700ef54: Pulling fs layer
2024-12-18T00:21:49.9090526Z d0ca0f491f80: Waiting
2024-12-18T00:21:49.9090760Z 21f8f066852d: Waiting
2024-12-18T00:21:49.9090994Z 3287a539d61e: Pulling fs layer
2024-12-18T00:21:49.9091262Z 3f0ade86af1a: Waiting
2024-12-18T00:21:49.9091510Z 9b763e24a7d4: Pulling fs layer
2024-12-18T00:21:49.9091789Z f6ef3d4c814e: Pulling fs layer
2024-12-18T00:21:49.9092060Z 9dc1f315a979: Waiting
2024-12-18T00:21:49.9092289Z cf02c706cd56: Waiting
2024-12-18T00:21:49.9092566Z 06914b00eb49: Pulling fs layer
2024-12-18T00:21:49.9092840Z 724553524224: Pulling fs layer
2024-12-18T00:21:49.9093112Z cd6a6063178b: Waiting
2024-12-18T00:21:49.9093365Z 47d9fb6dae26: Pulling fs layer
2024-12-18T00:21:49.9093647Z acce7b690dcb: Pulling fs layer
2024-12-18T00:21:49.9093922Z 58dc9f64def7: Pulling fs layer
2024-12-18T00:21:49.9094201Z 94dfe5ef1b7e: Pulling fs layer
2024-12-18T00:21:49.9094631Z 56c896077120: Waiting
2024-12-18T00:21:49.9094871Z 292b18ec2dab: Waiting
2024-12-18T00:21:49.9095113Z d88e9c5e7c70: Waiting
2024-12-18T00:21:49.9095351Z 40a2c9135b06: Pulling fs layer
2024-12-18T00:21:49.9095619Z 53617ee83302: Waiting
2024-12-18T00:21:49.9095854Z 2116738f4ea9: Waiting
2024-12-18T00:21:49.9096086Z 16073d3f107d: Waiting
2024-12-18T00:21:49.9096311Z 389dc939e2a4: Waiting
2024-12-18T00:21:49.9096541Z 4829486be7c3: Waiting
2024-12-18T00:21:49.9096775Z f4038f7524c3: Waiting
2024-12-18T00:21:49.9097006Z 57e52c6751f4: Waiting
2024-12-18T00:21:49.9097254Z 8839d6b0c821: Pulling fs layer
2024-12-18T00:21:49.9097514Z 3edbde10d404: Waiting
2024-12-18T00:21:49.9097753Z 4a51173f7757: Waiting
2024-12-18T00:21:49.9097991Z 0341eaeed33d: Waiting
2024-12-18T00:21:49.9098227Z 537cdb2d3168: Waiting
2024-12-18T00:21:49.9098458Z 4f4fb700ef54: Waiting
2024-12-18T00:21:49.9098698Z 1c588f0a9e97: Pulling fs layer
2024-12-18T00:21:49.9098971Z 6727e7e27936: Pulling fs layer
2024-12-18T00:21:49.9099234Z 9b763e24a7d4: Waiting
2024-12-18T00:21:49.9099467Z 3287a539d61e: Waiting
2024-12-18T00:21:49.9099697Z 06914b00eb49: Waiting
2024-12-18T00:21:49.9099935Z 3a55100d1957: Pulling fs layer
2024-12-18T00:21:49.9100195Z acce7b690dcb: Waiting
2024-12-18T00:21:49.9100433Z 47d9fb6dae26: Waiting
2024-12-18T00:21:49.9100677Z 3d6f160d3351: Pulling fs layer
2024-12-18T00:21:49.9100953Z b2a8bb947a87: Pulling fs layer
2024-12-18T00:21:49.9101225Z 69a74b2360f4: Pulling fs layer
2024-12-18T00:21:49.9101480Z 6727e7e27936: Waiting
2024-12-18T00:21:49.9101715Z 76d4a0702497: Waiting
2024-12-18T00:21:49.9101949Z 40a2c9135b06: Waiting
2024-12-18T00:21:49.9102179Z 3a55100d1957: Waiting
2024-12-18T00:21:49.9102412Z 94dfe5ef1b7e: Waiting
2024-12-18T00:21:49.9102649Z f6ef3d4c814e: Waiting
2024-12-18T00:21:49.9102885Z 1c588f0a9e97: Waiting
2024-12-18T00:21:49.9103116Z 3c8ea8a600ff: Waiting
2024-12-18T00:21:49.9103348Z b86099b8619a: Waiting
2024-12-18T00:21:49.9103579Z ab6ef6c2ac4d: Waiting
2024-12-18T00:21:49.9103814Z 3d6f160d3351: Waiting
2024-12-18T00:21:49.9104044Z b2a8bb947a87: Waiting
2024-12-18T00:21:49.9104274Z 69a74b2360f4: Waiting
2024-12-18T00:21:49.9104516Z f9e12a253d89: Pulling fs layer
2024-12-18T00:21:49.9104786Z 8fdb06d51b96: Pulling fs layer
2024-12-18T00:21:49.9105046Z f9e12a253d89: Waiting
2024-12-18T00:21:49.9105276Z 8fdb06d51b96: Waiting
2024-12-18T00:21:49.9105509Z 58dc9f64def7: Waiting
2024-12-18T00:21:50.1211004Z fbcd35dc5bc3: Verifying Checksum
2024-12-18T00:21:50.1212062Z fbcd35dc5bc3: Download complete
2024-12-18T00:21:50.2109011Z db6cdef1932a: Verifying Checksum
2024-12-18T00:21:50.2109478Z db6cdef1932a: Download complete
2024-12-18T00:21:50.2387940Z 80888bc6716f: Verifying Checksum
2024-12-18T00:21:50.2388681Z 80888bc6716f: Download complete
2024-12-18T00:21:50.2884349Z 56dc85502937: Verifying Checksum
2024-12-18T00:21:50.2884800Z 56dc85502937: Download complete
2024-12-18T00:21:50.3686034Z 71bdb1a72c2d: Verifying Checksum
2024-12-18T00:21:50.3686656Z 71bdb1a72c2d: Download complete
2024-12-18T00:21:50.4574293Z 4829486be7c3: Verifying Checksum
2024-12-18T00:21:50.4574751Z 4829486be7c3: Download complete
2024-12-18T00:21:50.5458145Z c7232af9ae05: Verifying Checksum
2024-12-18T00:21:50.5458637Z c7232af9ae05: Download complete
2024-12-18T00:21:50.5676519Z 1f4e68d7b5e4: Verifying Checksum
2024-12-18T00:21:50.5676979Z 1f4e68d7b5e4: Download complete
2024-12-18T00:21:50.6801090Z 622381141745: Verifying Checksum
2024-12-18T00:21:50.7787667Z f003d065e31e: Verifying Checksum
2024-12-18T00:21:50.7788149Z f003d065e31e: Download complete
2024-12-18T00:21:51.3338892Z 80888bc6716f: Pull complete
2024-12-18T00:21:51.6319354Z fbcd35dc5bc3: Pull complete
2024-12-18T00:21:52.4216930Z c7232af9ae05: Pull complete
2024-12-18T00:21:52.4517752Z db6cdef1932a: Pull complete
2024-12-18T00:21:52.4775192Z 56dc85502937: Pull complete
2024-12-18T00:21:53.3086651Z cd6ce7116815: Verifying Checksum
2024-12-18T00:21:53.3087024Z cd6ce7116815: Download complete
2024-12-18T00:21:53.4001873Z e843066a8f68: Verifying Checksum
2024-12-18T00:21:53.4002559Z e843066a8f68: Download complete
2024-12-18T00:21:53.5057912Z bac50ca2930b: Verifying Checksum
2024-12-18T00:21:53.5058401Z bac50ca2930b: Download complete
2024-12-18T00:21:53.6022532Z 57e52c6751f4: Download complete
2024-12-18T00:21:54.6053561Z a9c5c14d77c0: Verifying Checksum
2024-12-18T00:21:54.6053925Z a9c5c14d77c0: Download complete
2024-12-18T00:21:54.7035894Z b86099b8619a: Verifying Checksum
2024-12-18T00:21:54.7036247Z b86099b8619a: Download complete
2024-12-18T00:21:54.7979036Z 3c8ea8a600ff: Verifying Checksum
2024-12-18T00:21:54.7979357Z 3c8ea8a600ff: Download complete
2024-12-18T00:21:54.8760665Z bebd539a1d59: Verifying Checksum
2024-12-18T00:21:54.8760995Z bebd539a1d59: Download complete
2024-12-18T00:21:54.9724499Z 292b18ec2dab: Download complete
2024-12-18T00:22:04.0256587Z 30c0ea6140d0: Verifying Checksum
2024-12-18T00:22:04.0259016Z 30c0ea6140d0: Download complete
2024-12-18T00:22:04.1104655Z 50b1f245e9f5: Verifying Checksum
2024-12-18T00:22:04.1105087Z 50b1f245e9f5: Download complete
2024-12-18T00:22:04.2175332Z ad93cf5205f2: Download complete
2024-12-18T00:22:04.2957562Z 8d17c59eaf7d: Download complete
2024-12-18T00:22:04.3997567Z b36a8ccf5c8c: Download complete
2024-12-18T00:22:04.4989374Z aa0a3b87e9e3: Verifying Checksum
2024-12-18T00:22:04.4989840Z aa0a3b87e9e3: Download complete
2024-12-18T00:22:05.7739845Z 72a632052c5b: Verifying Checksum
2024-12-18T00:22:05.7740205Z 72a632052c5b: Download complete
2024-12-18T00:22:05.8729578Z ceed6316bc24: Verifying Checksum
2024-12-18T00:22:05.8729915Z ceed6316bc24: Download complete
2024-12-18T00:22:05.9783775Z d0ca0f491f80: Verifying Checksum
2024-12-18T00:22:05.9784201Z d0ca0f491f80: Download complete
2024-12-18T00:22:06.0645102Z cf02c706cd56: Download complete
2024-12-18T00:22:06.1414339Z 16073d3f107d: Verifying Checksum
2024-12-18T00:22:06.1414739Z 16073d3f107d: Download complete
2024-12-18T00:22:06.2498203Z d234faf2f1bf: Download complete
2024-12-18T00:22:10.6241449Z 3f0ade86af1a: Verifying Checksum
2024-12-18T00:22:10.6241859Z 3f0ade86af1a: Download complete
2024-12-18T00:22:10.7228425Z 034b6a03f5f7: Download complete
2024-12-18T00:22:10.7994101Z 21f8f066852d: Verifying Checksum
2024-12-18T00:22:10.7994523Z 21f8f066852d: Download complete
2024-12-18T00:22:11.2170317Z 9dc1f315a979: Verifying Checksum
2024-12-18T00:22:11.2170673Z 9dc1f315a979: Download complete
2024-12-18T00:22:11.3165254Z cd6a6063178b: Verifying Checksum
2024-12-18T00:22:11.3165689Z cd6a6063178b: Download complete
2024-12-18T00:22:11.4306231Z 2116738f4ea9: Download complete
2024-12-18T00:22:11.6900623Z d7fa134316d9: Verifying Checksum
2024-12-18T00:22:11.6900976Z d7fa134316d9: Download complete
2024-12-18T00:22:11.7868665Z af03dc3fd15e: Verifying Checksum
2024-12-18T00:22:11.7869881Z af03dc3fd15e: Download complete
2024-12-18T00:22:11.8671047Z fd9de3dc4258: Download complete
2024-12-18T00:22:11.9535378Z b85093661f77: Download complete
2024-12-18T00:22:16.0078972Z 30c0ea6140d0: Pull complete
2024-12-18T00:22:16.2045563Z 71bdb1a72c2d: Pull complete
2024-12-18T00:22:16.3326918Z 4829486be7c3: Pull complete
2024-12-18T00:22:16.5091447Z 1f4e68d7b5e4: Pull complete
2024-12-18T00:22:16.8670268Z 7c373e2d9b7e: Verifying Checksum
2024-12-18T00:22:16.8670710Z 7c373e2d9b7e: Download complete
2024-12-18T00:22:16.9538030Z 56c896077120: Verifying Checksum
2024-12-18T00:22:16.9538470Z 56c896077120: Download complete
2024-12-18T00:22:17.0392721Z d88e9c5e7c70: Download complete
2024-12-18T00:22:17.5488423Z 53617ee83302: Verifying Checksum
2024-12-18T00:22:17.5488811Z 53617ee83302: Download complete
2024-12-18T00:22:17.6190011Z 3edbde10d404: Verifying Checksum
2024-12-18T00:22:17.6190489Z 3edbde10d404: Download complete
2024-12-18T00:22:17.7960521Z f4038f7524c3: Download complete
2024-12-18T00:22:17.8895442Z 4a51173f7757: Verifying Checksum
2024-12-18T00:22:17.8895830Z 4a51173f7757: Download complete
2024-12-18T00:22:17.9802321Z ed6a71d41274: Verifying Checksum
2024-12-18T00:22:24.4636419Z ed6a71d41274: Download complete
2024-12-18T00:22:24.4636816Z 76e3817b4b4c: Verifying Checksum
2024-12-18T00:22:24.4637511Z 76e3817b4b4c: Download complete
2024-12-18T00:22:24.5668129Z 0341eaeed33d: Verifying Checksum
2024-12-18T00:22:24.5668634Z 0341eaeed33d: Download complete
2024-12-18T00:22:24.6509615Z 537cdb2d3168: Verifying Checksum
2024-12-18T00:22:24.6510087Z 537cdb2d3168: Download complete
2024-12-18T00:22:24.7480264Z 76d4a0702497: Verifying Checksum
2024-12-18T00:22:24.7480744Z 76d4a0702497: Download complete
2024-12-18T00:22:24.7549347Z 4f4fb700ef54: Verifying Checksum
2024-12-18T00:22:24.7549783Z 4f4fb700ef54: Download complete
2024-12-18T00:22:24.8301656Z 3287a539d61e: Verifying Checksum
2024-12-18T00:22:24.8302121Z 3287a539d61e: Download complete
2024-12-18T00:22:24.9191349Z 9b763e24a7d4: Verifying Checksum
2024-12-18T00:22:24.9191873Z 9b763e24a7d4: Download complete
2024-12-18T00:22:26.9553313Z f6ef3d4c814e: Verifying Checksum
2024-12-18T00:22:26.9553746Z f6ef3d4c814e: Download complete
2024-12-18T00:22:27.0447122Z 06914b00eb49: Download complete
2024-12-18T00:22:27.1424795Z 724553524224: Verifying Checksum
2024-12-18T00:22:27.1425381Z 724553524224: Download complete
2024-12-18T00:22:27.2409109Z 47d9fb6dae26: Download complete
2024-12-18T00:22:27.3197782Z acce7b690dcb: Download complete
2024-12-18T00:22:27.4509978Z 58dc9f64def7: Verifying Checksum
2024-12-18T00:22:27.4510368Z 58dc9f64def7: Download complete
2024-12-18T00:22:27.5514990Z 94dfe5ef1b7e: Verifying Checksum
2024-12-18T00:22:27.5515473Z 94dfe5ef1b7e: Download complete
2024-12-18T00:22:28.1534469Z 40a2c9135b06: Verifying Checksum
2024-12-18T00:22:28.1534994Z 40a2c9135b06: Download complete
2024-12-18T00:22:28.2453073Z 8839d6b0c821: Verifying Checksum
2024-12-18T00:22:28.2453690Z 8839d6b0c821: Download complete
2024-12-18T00:23:04.4388328Z 1c588f0a9e97: Verifying Checksum
2024-12-18T00:23:04.4388709Z 1c588f0a9e97: Download complete
2024-12-18T00:23:04.5242794Z 6727e7e27936: Verifying Checksum
2024-12-18T00:23:04.5243206Z 6727e7e27936: Download complete
2024-12-18T00:23:04.6477806Z 3a55100d1957: Verifying Checksum
2024-12-18T00:23:04.6478141Z 3a55100d1957: Download complete
2024-12-18T00:23:13.2282643Z 3d6f160d3351: Verifying Checksum
2024-12-18T00:23:13.2283118Z 3d6f160d3351: Download complete
2024-12-18T00:23:13.3322240Z b2a8bb947a87: Verifying Checksum
2024-12-18T00:23:13.3322575Z b2a8bb947a87: Download complete
2024-12-18T00:23:13.4269533Z 69a74b2360f4: Verifying Checksum
2024-12-18T00:23:13.4270012Z 69a74b2360f4: Download complete
2024-12-18T00:23:14.7040577Z f9e12a253d89: Verifying Checksum
2024-12-18T00:23:14.7040925Z f9e12a253d89: Download complete
2024-12-18T00:23:14.7915371Z 8fdb06d51b96: Verifying Checksum
2024-12-18T00:23:14.7915694Z 8fdb06d51b96: Download complete
2024-12-18T00:23:42.7589419Z ab6ef6c2ac4d: Verifying Checksum
2024-12-18T00:23:42.7592634Z ab6ef6c2ac4d: Download complete
2024-12-18T00:23:47.4529370Z 389dc939e2a4: Verifying Checksum
2024-12-18T00:23:47.4529732Z 389dc939e2a4: Download complete
2024-12-18T00:24:00.2572873Z 7c373e2d9b7e: Pull complete
2024-12-18T00:24:00.4787039Z 622381141745: Pull complete
2024-12-18T00:24:00.6922492Z f003d065e31e: Pull complete
2024-12-18T00:24:08.3608621Z cd6ce7116815: Pull complete
2024-12-18T00:24:08.5017797Z e843066a8f68: Pull complete
2024-12-18T00:24:08.6277182Z bac50ca2930b: Pull complete
2024-12-18T00:24:08.8082250Z 57e52c6751f4: Pull complete
2024-12-18T00:24:11.0941173Z a9c5c14d77c0: Pull complete
2024-12-18T00:24:11.2673342Z b86099b8619a: Pull complete
2024-12-18T00:24:11.4444860Z 3c8ea8a600ff: Pull complete
2024-12-18T00:24:11.6349588Z bebd539a1d59: Pull complete
2024-12-18T00:24:11.7826659Z 292b18ec2dab: Pull complete
2024-12-18T00:25:04.0385899Z 76e3817b4b4c: Pull complete
2024-12-18T00:25:04.1862434Z 50b1f245e9f5: Pull complete
2024-12-18T00:25:04.3517503Z ad93cf5205f2: Pull complete
2024-12-18T00:25:04.5050661Z 8d17c59eaf7d: Pull complete
2024-12-18T00:25:04.6453084Z b36a8ccf5c8c: Pull complete
2024-12-18T00:25:04.7818170Z aa0a3b87e9e3: Pull complete
2024-12-18T00:25:07.1902166Z 72a632052c5b: Pull complete
2024-12-18T00:25:07.4131914Z ceed6316bc24: Pull complete
2024-12-18T00:25:07.6427126Z d0ca0f491f80: Pull complete
2024-12-18T00:25:07.8570838Z cf02c706cd56: Pull complete
2024-12-18T00:25:08.0660228Z 16073d3f107d: Pull complete
2024-12-18T00:25:08.2752715Z d234faf2f1bf: Pull complete
2024-12-18T00:25:16.6658538Z 3f0ade86af1a: Pull complete
2024-12-18T00:25:16.8754015Z 034b6a03f5f7: Pull complete
2024-12-18T00:25:17.0943351Z 21f8f066852d: Pull complete
2024-12-18T00:25:17.8676516Z 9dc1f315a979: Pull complete
2024-12-18T00:25:18.0861281Z cd6a6063178b: Pull complete
2024-12-18T00:25:18.3050796Z 2116738f4ea9: Pull complete
2024-12-18T00:25:18.7353433Z d7fa134316d9: Pull complete
2024-12-18T00:25:18.9574722Z af03dc3fd15e: Pull complete
2024-12-18T00:25:19.3505018Z fd9de3dc4258: Pull complete
2024-12-18T00:25:19.5629178Z b85093661f77: Pull complete
2024-12-18T00:25:50.1987319Z ab6ef6c2ac4d: Pull complete
2024-12-18T00:25:50.4170020Z 56c896077120: Pull complete
2024-12-18T00:25:50.6300207Z d88e9c5e7c70: Pull complete
2024-12-18T00:25:53.2002197Z 53617ee83302: Pull complete
2024-12-18T00:25:53.4216918Z 3edbde10d404: Pull complete
2024-12-18T00:25:53.6267190Z f4038f7524c3: Pull complete
2024-12-18T00:25:54.0294200Z 4a51173f7757: Pull complete
2024-12-18T00:25:54.2500139Z ed6a71d41274: Pull complete
2024-12-18T00:27:27.3342819Z 389dc939e2a4: Pull complete
2024-12-18T00:27:27.5047260Z 0341eaeed33d: Pull complete
2024-12-18T00:27:27.6896090Z 537cdb2d3168: Pull complete
2024-12-18T00:27:28.0250146Z 76d4a0702497: Pull complete
2024-12-18T00:27:28.1152927Z 4f4fb700ef54: Pull complete
2024-12-18T00:27:28.1667129Z 3287a539d61e: Pull complete
2024-12-18T00:27:28.2831255Z 9b763e24a7d4: Pull complete
2024-12-18T00:27:39.1299354Z f6ef3d4c814e: Pull complete
2024-12-18T00:27:39.3185420Z 06914b00eb49: Pull complete
2024-12-18T00:27:39.4061570Z 724553524224: Pull complete
2024-12-18T00:27:39.4566974Z 47d9fb6dae26: Pull complete
2024-12-18T00:27:39.5630759Z acce7b690dcb: Pull complete
2024-12-18T00:27:40.8880668Z 58dc9f64def7: Pull complete
2024-12-18T00:27:40.9420931Z 94dfe5ef1b7e: Pull complete
2024-12-18T00:27:43.2324714Z 40a2c9135b06: Pull complete
2024-12-18T00:27:43.2803459Z 8839d6b0c821: Pull complete
2024-12-18T00:28:28.9093540Z 1c588f0a9e97: Pull complete
2024-12-18T00:28:29.0221028Z 6727e7e27936: Pull complete
2024-12-18T00:28:29.0945222Z 3a55100d1957: Pull complete
2024-12-18T00:28:38.1507005Z 3d6f160d3351: Pull complete
2024-12-18T00:28:38.3720253Z b2a8bb947a87: Pull complete
2024-12-18T00:28:38.5812923Z 69a74b2360f4: Pull complete
2024-12-18T00:28:39.2079668Z f9e12a253d89: Pull complete
2024-12-18T00:28:39.4267782Z 8fdb06d51b96: Pull complete
2024-12-18T00:28:40.3290899Z Digest: sha256:ff80d4c93d18e8ab7a2501f0e62b7c40cf97aa83fc9702e88b9c2d02cdcbecdb
2024-12-18T00:28:40.3706033Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:28:40.3959285Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:28:40.4016526Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T00:28:40.4017434Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2024-12-18T00:28:40.4027607Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:28:40.4027979Z env:
2024-12-18T00:28:40.4028202Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:28:40.4028468Z ##[endgroup]
2024-12-18T00:28:40.4312229Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@release/2.6
2024-12-18T00:28:40.4312901Z with:
2024-12-18T00:28:40.4313241Z   driver-version: 550.54.15
2024-12-18T00:28:40.4313636Z env:
2024-12-18T00:28:40.4313923Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:28:40.4314298Z ##[endgroup]
2024-12-18T00:28:40.4421792Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2024-12-18T00:28:40.4422510Z with:
2024-12-18T00:28:40.4422837Z   timeout_minutes: 10
2024-12-18T00:28:40.4423201Z   max_attempts: 3
2024-12-18T00:28:40.4448932Z   command: # Is it disgusting to have a full shell script here in this github action? Sure
# But is it the best way to make it so that this action relies on nothing else? Absolutely
set -eou pipefail

DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID)
DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run"

install_nvidia_docker2_amzn2() {
    (
        set -x
        # Needed for yum-config-manager
        sudo yum install -y yum-utils
        if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then
          YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo"
        else
          # Amazon Linux 2
          YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo"
        fi

        sudo yum-config-manager --add-repo "${YUM_REPO_URL}"
        sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2
        sudo systemctl restart docker
    )
}

install_nvidia_docker2_ubuntu20() {
    (
        set -x
        # Install nvidia-driver package if not installed
        status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)"
        if [ ! $? = 0 ] || [ ! "$status" = installed ]; then
          sudo apt-get install -y nvidia-docker2 nvidia-container-toolkit-1.16.2
          sudo systemctl restart docker
        fi
    )
}

pre_install_nvidia_driver_amzn2() {
    (
        # Purge any nvidia driver installed from RHEL repo
        sudo yum remove -y nvidia-driver-latest-dkms
    )
}

install_nvidia_driver_common() {
    (
        # Try to gather more information about the runner and its existing NVIDIA driver if any
        echo "Before installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        HAS_NVIDIA_DRIVER=0
        # Check if NVIDIA driver has already been installed
        if [ -x "$(command -v nvidia-smi)" ]; then
            set +e
            # The driver exists, check its version next. Also check only the first GPU if there are more than one of them
            # so that the same driver version is not print over multiple lines
            INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
            NVIDIA_SMI_STATUS=$?

            if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing"
            elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing"
            else
                HAS_NVIDIA_DRIVER=1
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation"
            fi
            set -e
        fi

        if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then
            # CAUTION: this may need to be updated in future
            if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then
                  sudo yum groupinstall -y "Development Tools"
                  # ensure our kernel install is the same as our underlying kernel,
                  # groupinstall "Development Tools" has a habit of mismatching kernel headers
                  sudo yum install -y "kernel-devel-uname-r == $(uname -r)"
                  sudo modprobe backlight
            fi
            sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN"

            set +e
            sudo /bin/bash /tmp/nvidia_driver -s --no-drm
            NVIDIA_INSTALLATION_STATUS=$?

            RESET_GPU=0
            if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then
                sudo cat /var/log/nvidia-installer.log
                # Fail to install NVIDIA driver, try to reset the GPU
                RESET_GPU=1
            elif [ -x "$(command -v nvidia-smi)" ]; then
                # Check again if nvidia-smi works even if the driver installation completes successfully
                INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
                NVIDIA_SMI_STATUS=$?

                if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                    RESET_GPU=1
                fi
            fi

            if [ "$RESET_GPU" -eq 1 ]; then
                NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1)
                # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this
                # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388
                for PCI_ID in $NVIDIA_DEVICES; do
                    DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable)

                    echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)"
                    # This requires sudo permission of course
                    echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset
                    sleep 1
                done
            fi

            sudo rm -fv /tmp/nvidia_driver
            set -e
        fi
    )
}

post_install_nvidia_driver_common() {
    (
        sudo modprobe nvidia || true
        echo "After installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        (
            set +e

            nvidia-smi
            # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in
            # the case where the driver has already crashed as it still can get the driver version
            # and some basic information like the bus ID.  However, the rest of the information
            # would be missing (ERR!), for example:
            #
            # +-----------------------------------------------------------------------------+
            # | NVIDIA-SMI 525.89.02    Driver Version: 525.89.02    CUDA Version: 12.0     |
            # |-------------------------------+----------------------+----------------------+
            # | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
            # | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
            # |                               |                      |               MIG M. |
            # |===============================+======================+======================|
            # |   0  ERR!                Off  | 00000000:00:1E.0 Off |                 ERR! |
            # |ERR!  ERR! ERR!    ERR! / ERR! |   4184MiB / 23028MiB |    ERR!      Default |
            # |                               |                      |                 ERR! |
            # +-------------------------------+----------------------+----------------------+
            #
            # +-----------------------------------------------------------------------------+
            # | Processes:                                                                  |
            # |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
            # |        ID   ID                                                   Usage      |
            # |=============================================================================|
            # +-----------------------------------------------------------------------------+
            #
            # This should be reported as a failure instead as it will guarantee to fail when
            # Docker tries to run with --gpus all
            #
            # So, the correct check here is to query one of the missing piece of info like
            # GPU name, so that the command can fail accordingly
            nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0
            NVIDIA_SMI_STATUS=$?

            # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285
            if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then
                echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}"
            else
                echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}"
                exit ${NVIDIA_SMI_STATUS}
            fi
            set -e
        )
    )
}

install_nvidia_driver_amzn2() {
    (
        set -x
        pre_install_nvidia_driver_amzn2
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

install_nvidia_driver_ubuntu20() {
    (
        set -x
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

echo "== Installing nvidia driver ${DRIVER_FN} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_driver_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_driver_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac

# Install container toolkit based on distribution
echo "== Installing nvidia container toolkit for ${DISTRIBUTION} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_docker2_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_docker2_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac
echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}"

# Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with
# more than one GPUs. This just needs to be run once. The command fails
# on subsequent runs and complains that the mode is already on, but that's
# ok
sudo nvidia-persistenced || true
# This should show persistence mode ON
nvidia-smi

2024-12-18T00:28:40.4474374Z   retry_wait_seconds: 10
2024-12-18T00:28:40.4474839Z   polling_interval_seconds: 1
2024-12-18T00:28:40.4475193Z   warning_on_retry: true
2024-12-18T00:28:40.4475627Z   continue_on_error: false
2024-12-18T00:28:40.4476050Z env:
2024-12-18T00:28:40.4476377Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:28:40.4476756Z   DRIVER_VERSION: 550.54.15
2024-12-18T00:28:40.4477132Z ##[endgroup]
2024-12-18T00:28:41.5962383Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run ==
2024-12-18T00:28:41.5963311Z + pre_install_nvidia_driver_amzn2
2024-12-18T00:28:41.5966258Z + sudo yum remove -y nvidia-driver-latest-dkms
2024-12-18T00:28:43.2817881Z No match for argument: nvidia-driver-latest-dkms
2024-12-18T00:28:43.2819563Z No packages marked for removal.
2024-12-18T00:28:43.3241256Z Dependencies resolved.
2024-12-18T00:28:43.3252292Z Nothing to do.
2024-12-18T00:28:43.3252690Z Complete!
2024-12-18T00:28:43.4101603Z + install_nvidia_driver_common
2024-12-18T00:28:43.4104738Z + echo 'Before installing NVIDIA driver'
2024-12-18T00:28:43.4105061Z + lspci
2024-12-18T00:28:43.4105718Z Before installing NVIDIA driver
2024-12-18T00:28:43.4225553Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma]
2024-12-18T00:28:43.4226091Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2024-12-18T00:28:43.4226700Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
2024-12-18T00:28:43.4227264Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111
2024-12-18T00:28:43.4227788Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller
2024-12-18T00:28:43.4228353Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2024-12-18T00:28:43.4228874Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1)
2024-12-18T00:28:43.4229394Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller
2024-12-18T00:28:43.4230183Z + lsmod
2024-12-18T00:28:43.4514958Z Module                  Size  Used by
2024-12-18T00:28:43.4515601Z nvidia_modeset       1351680  0
2024-12-18T00:28:43.4516050Z video                  65536  1 nvidia_modeset
2024-12-18T00:28:43.4516495Z wmi                    36864  1 video
2024-12-18T00:28:43.4516895Z nvidia_uvm           4706304  0
2024-12-18T00:28:43.4517337Z nvidia              54071296  7 nvidia_uvm,nvidia_modeset
2024-12-18T00:28:43.4518053Z drm                   602112  1 nvidia
2024-12-18T00:28:43.4518449Z drm_panel_orientation_quirks    32768  1 drm
2024-12-18T00:28:43.4518836Z backlight              24576  3 video,drm,nvidia_modeset
2024-12-18T00:28:43.4519204Z i2c_core              106496  2 nvidia,drm
2024-12-18T00:28:43.4519514Z xt_conntrack           16384  1
2024-12-18T00:28:43.4519802Z nft_chain_nat          16384  3
2024-12-18T00:28:43.4520181Z xt_MASQUERADE          20480  1
2024-12-18T00:28:43.4521110Z nf_nat                 57344  2 nft_chain_nat,xt_MASQUERADE
2024-12-18T00:28:43.4521573Z nf_conntrack_netlink    57344  0
2024-12-18T00:28:43.4522147Z nf_conntrack          184320  4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE
2024-12-18T00:28:43.4522770Z nf_defrag_ipv6         24576  1 nf_conntrack
2024-12-18T00:28:43.4523213Z nf_defrag_ipv4         16384  1 nf_conntrack
2024-12-18T00:28:43.4523621Z xfrm_user              57344  1
2024-12-18T00:28:43.4523905Z xfrm_algo              16384  1 xfrm_user
2024-12-18T00:28:43.4524222Z xt_addrtype            16384  2
2024-12-18T00:28:43.4524498Z nft_compat             20480  4
2024-12-18T00:28:43.4524823Z nf_tables             311296  57 nft_compat,nft_chain_nat
2024-12-18T00:28:43.4525250Z nfnetlink              20480  4 nft_compat,nf_conntrack_netlink,nf_tables
2024-12-18T00:28:43.4525668Z br_netfilter           36864  0
2024-12-18T00:28:43.4525963Z bridge                323584  1 br_netfilter
2024-12-18T00:28:43.4526285Z stp                    16384  1 bridge
2024-12-18T00:28:43.4526631Z llc                    16384  2 bridge,stp
2024-12-18T00:28:43.4527029Z overlay               167936  0
2024-12-18T00:28:43.4527362Z tls                   114688  0
2024-12-18T00:28:43.4527715Z nls_ascii              16384  1
2024-12-18T00:28:43.4528065Z nls_cp437              20480  1
2024-12-18T00:28:43.4528414Z vfat                   24576  1
2024-12-18T00:28:43.4528781Z sunrpc                692224  1
2024-12-18T00:28:43.4529144Z fat                    86016  1 vfat
2024-12-18T00:28:43.4529549Z ena                   167936  0
2024-12-18T00:28:43.4529905Z i8042                  45056  0
2024-12-18T00:28:43.4530297Z serio                  28672  3 i8042
2024-12-18T00:28:43.4530722Z ghash_clmulni_intel    16384  0
2024-12-18T00:28:43.4531123Z button                 24576  0
2024-12-18T00:28:43.4531535Z sch_fq_codel           20480  17
2024-12-18T00:28:43.4531943Z dm_mod                188416  0
2024-12-18T00:28:43.4532333Z fuse                  163840  1
2024-12-18T00:28:43.4532735Z loop                   36864  0
2024-12-18T00:28:43.4533141Z configfs               57344  1
2024-12-18T00:28:43.4533567Z dax                    45056  1 dm_mod
2024-12-18T00:28:43.4534008Z dmi_sysfs              20480  0
2024-12-18T00:28:43.4534416Z crc32_pclmul           16384  0
2024-12-18T00:28:43.4534836Z crc32c_intel           24576  0
2024-12-18T00:28:43.4535234Z efivarfs               24576  1
2024-12-18T00:28:43.4535636Z + modinfo nvidia
2024-12-18T00:28:43.4536328Z filename:       /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko
2024-12-18T00:28:43.4537072Z alias:          char-major-195-*
2024-12-18T00:28:43.4537515Z version:        550.54.15
2024-12-18T00:28:43.4537917Z supported:      external
2024-12-18T00:28:43.4538298Z license:        NVIDIA
2024-12-18T00:28:43.4538716Z firmware:       nvidia/550.54.15/gsp_tu10x.bin
2024-12-18T00:28:43.4539270Z firmware:       nvidia/550.54.15/gsp_ga10x.bin
2024-12-18T00:28:43.4539804Z srcversion:     833721318DA517F0C2FEC97
2024-12-18T00:28:43.4540547Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2024-12-18T00:28:43.4541097Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2024-12-18T00:28:43.4541651Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2024-12-18T00:28:43.4542142Z depends:        i2c-core,drm
2024-12-18T00:28:43.4542557Z retpoline:      Y
2024-12-18T00:28:43.4542915Z name:           nvidia
2024-12-18T00:28:43.4543486Z vermagic:       6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 
2024-12-18T00:28:43.4544396Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2024-12-18T00:28:43.4545112Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2024-12-18T00:28:43.4545564Z parm:           NVreg_ResmanDebugLevel:int
2024-12-18T00:28:43.4545900Z parm:           NVreg_RmLogonRC:int
2024-12-18T00:28:43.4546217Z parm:           NVreg_ModifyDeviceFiles:int
2024-12-18T00:28:43.4546553Z parm:           NVreg_DeviceFileUID:int
2024-12-18T00:28:43.4546877Z parm:           NVreg_DeviceFileGID:int
2024-12-18T00:28:43.4547211Z parm:           NVreg_DeviceFileMode:int
2024-12-18T00:28:43.4547596Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2024-12-18T00:28:43.4548002Z parm:           NVreg_UsePageAttributeTable:int
2024-12-18T00:28:43.4548359Z parm:           NVreg_EnablePCIeGen3:int
2024-12-18T00:28:43.4548679Z parm:           NVreg_EnableMSI:int
2024-12-18T00:28:43.4548989Z parm:           NVreg_TCEBypassMode:int
2024-12-18T00:28:43.4549332Z parm:           NVreg_EnableStreamMemOPs:int
2024-12-18T00:28:43.4549711Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2024-12-18T00:28:43.4550135Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2024-12-18T00:28:43.4550538Z parm:           NVreg_EnableS0ixPowerManagement:int
2024-12-18T00:28:43.4550973Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2024-12-18T00:28:43.4551406Z parm:           NVreg_DynamicPowerManagement:int
2024-12-18T00:28:43.4551848Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2024-12-18T00:28:43.4552284Z parm:           NVreg_EnableGpuFirmware:int
2024-12-18T00:28:43.4552641Z parm:           NVreg_EnableGpuFirmwareLogs:int
2024-12-18T00:28:43.4553029Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2024-12-18T00:28:43.4553421Z parm:           NVreg_EnableUserNUMAManagement:int
2024-12-18T00:28:43.4553781Z parm:           NVreg_MemoryPoolSize:int
2024-12-18T00:28:43.4554116Z parm:           NVreg_KMallocHeapMaxSize:int
2024-12-18T00:28:43.4554471Z parm:           NVreg_VMallocHeapMaxSize:int
2024-12-18T00:28:43.4554814Z parm:           NVreg_IgnoreMMIOCheck:int
2024-12-18T00:28:43.4555143Z parm:           NVreg_NvLinkDisable:int
2024-12-18T00:28:43.4555513Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2024-12-18T00:28:43.4555889Z parm:           NVreg_RegisterPCIDriver:int
2024-12-18T00:28:43.4556237Z parm:           NVreg_EnableResizableBar:int
2024-12-18T00:28:43.4556591Z parm:           NVreg_EnableDbgBreakpoint:int
2024-12-18T00:28:43.4556955Z parm:           NVreg_EnableNonblockingOpen:int
2024-12-18T00:28:43.4557318Z parm:           NVreg_RegistryDwords:charp
2024-12-18T00:28:43.4557674Z parm:           NVreg_RegistryDwordsPerDevice:charp
2024-12-18T00:28:43.4558023Z parm:           NVreg_RmMsg:charp
2024-12-18T00:28:43.4558325Z parm:           NVreg_GpuBlacklist:charp
2024-12-18T00:28:43.4558667Z parm:           NVreg_TemporaryFilePath:charp
2024-12-18T00:28:43.4559010Z parm:           NVreg_ExcludedGpus:charp
2024-12-18T00:28:43.4559343Z parm:           NVreg_DmaRemapPeerMmio:int
2024-12-18T00:28:43.4559692Z parm:           NVreg_RmNvlinkBandwidth:charp
2024-12-18T00:28:43.4560040Z parm:           NVreg_ImexChannelCount:int
2024-12-18T00:28:43.4560370Z parm:           rm_firmware_active:charp
2024-12-18T00:28:43.4560689Z + HAS_NVIDIA_DRIVER=0
2024-12-18T00:28:43.4560946Z ++ command -v nvidia-smi
2024-12-18T00:28:43.4561320Z + '[' -x /usr/bin/nvidia-smi ']'
2024-12-18T00:28:43.4561598Z + set +e
2024-12-18T00:28:43.4561937Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0
2024-12-18T00:28:43.4823780Z + INSTALLED_DRIVER_VERSION=550.54.15
2024-12-18T00:28:43.4824107Z + NVIDIA_SMI_STATUS=0
2024-12-18T00:28:43.4824361Z + '[' 0 -ne 0 ']'
2024-12-18T00:28:43.4824600Z + '[' 550.54.15 '!=' 550.54.15 ']'
2024-12-18T00:28:43.4824882Z + HAS_NVIDIA_DRIVER=1
2024-12-18T00:28:43.4825332Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation'
2024-12-18T00:28:43.4825961Z + set -e
2024-12-18T00:28:43.4826168Z + '[' 1 -eq 0 ']'
2024-12-18T00:28:43.4826570Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation
2024-12-18T00:28:43.4827055Z + post_install_nvidia_driver_common
2024-12-18T00:28:43.4831245Z + sudo modprobe nvidia
2024-12-18T00:28:43.6162800Z + echo 'After installing NVIDIA driver'
2024-12-18T00:28:43.6163129Z + lspci
2024-12-18T00:28:43.6163353Z After installing NVIDIA driver
2024-12-18T00:28:43.6279612Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma]
2024-12-18T00:28:43.6280151Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2024-12-18T00:28:43.6280733Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
2024-12-18T00:28:43.6281334Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111
2024-12-18T00:28:43.6281828Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller
2024-12-18T00:28:43.6282388Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2024-12-18T00:28:43.6282889Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1)
2024-12-18T00:28:43.6283391Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller
2024-12-18T00:28:43.6283804Z + lsmod
2024-12-18T00:28:43.6313633Z Module                  Size  Used by
2024-12-18T00:28:43.6313957Z nvidia_modeset       1351680  0
2024-12-18T00:28:43.6314253Z video                  65536  1 nvidia_modeset
2024-12-18T00:28:43.6314573Z wmi                    36864  1 video
2024-12-18T00:28:43.6314867Z nvidia_uvm           4706304  0
2024-12-18T00:28:43.6315225Z nvidia              54071296  7 nvidia_uvm,nvidia_modeset
2024-12-18T00:28:43.6315583Z drm                   602112  1 nvidia
2024-12-18T00:28:43.6315900Z drm_panel_orientation_quirks    32768  1 drm
2024-12-18T00:28:43.6316283Z backlight              24576  3 video,drm,nvidia_modeset
2024-12-18T00:28:43.6316649Z i2c_core              106496  2 nvidia,drm
2024-12-18T00:28:43.6316962Z xt_conntrack           16384  1
2024-12-18T00:28:43.6317239Z nft_chain_nat          16384  3
2024-12-18T00:28:43.6317512Z xt_MASQUERADE          20480  1
2024-12-18T00:28:43.6317830Z nf_nat                 57344  2 nft_chain_nat,xt_MASQUERADE
2024-12-18T00:28:43.6318177Z nf_conntrack_netlink    57344  0
2024-12-18T00:28:43.6318593Z nf_conntrack          184320  4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE
2024-12-18T00:28:43.6319051Z nf_defrag_ipv6         24576  1 nf_conntrack
2024-12-18T00:28:43.6319375Z nf_defrag_ipv4         16384  1 nf_conntrack
2024-12-18T00:28:43.6319691Z xfrm_user              57344  1
2024-12-18T00:28:43.6319977Z xfrm_algo              16384  1 xfrm_user
2024-12-18T00:28:43.6320285Z xt_addrtype            16384  2
2024-12-18T00:28:43.6320562Z nft_compat             20480  4
2024-12-18T00:28:43.6320884Z nf_tables             311296  57 nft_compat,nft_chain_nat
2024-12-18T00:28:43.6321396Z nfnetlink              20480  4 nft_compat,nf_conntrack_netlink,nf_tables
2024-12-18T00:28:43.6321795Z br_netfilter           36864  0
2024-12-18T00:28:43.6322088Z bridge                323584  1 br_netfilter
2024-12-18T00:28:43.6322402Z stp                    16384  1 bridge
2024-12-18T00:28:43.6322706Z llc                    16384  2 bridge,stp
2024-12-18T00:28:43.6322999Z overlay               167936  0
2024-12-18T00:28:43.6323270Z tls                   114688  0
2024-12-18T00:28:43.6323537Z nls_ascii              16384  1
2024-12-18T00:28:43.6323806Z nls_cp437              20480  1
2024-12-18T00:28:43.6324067Z vfat                   24576  1
2024-12-18T00:28:43.6324568Z sunrpc                692224  1
2024-12-18T00:28:43.6324850Z fat                    86016  1 vfat
2024-12-18T00:28:43.6325176Z ena                   167936  0
2024-12-18T00:28:43.6325444Z i8042                  45056  0
2024-12-18T00:28:43.6325709Z serio                  28672  3 i8042
2024-12-18T00:28:43.6326008Z ghash_clmulni_intel    16384  0
2024-12-18T00:28:43.6326286Z button                 24576  0
2024-12-18T00:28:43.6326690Z sch_fq_codel           20480  17
2024-12-18T00:28:43.6326964Z dm_mod                188416  0
2024-12-18T00:28:43.6327221Z fuse                  163840  1
2024-12-18T00:28:43.6327486Z loop                   36864  0
2024-12-18T00:28:43.6327752Z configfs               57344  1
2024-12-18T00:28:43.6328025Z dax                    45056  1 dm_mod
2024-12-18T00:28:43.6328316Z dmi_sysfs              20480  0
2024-12-18T00:28:43.6328584Z crc32_pclmul           16384  0
2024-12-18T00:28:43.6328853Z crc32c_intel           24576  0
2024-12-18T00:28:43.6329132Z efivarfs               24576  1
2024-12-18T00:28:43.6329402Z + modinfo nvidia
2024-12-18T00:28:43.6332710Z filename:       /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko
2024-12-18T00:28:43.6333218Z alias:          char-major-195-*
2024-12-18T00:28:43.6333499Z version:        550.54.15
2024-12-18T00:28:43.6333764Z supported:      external
2024-12-18T00:28:43.6334022Z license:        NVIDIA
2024-12-18T00:28:43.6334312Z firmware:       nvidia/550.54.15/gsp_tu10x.bin
2024-12-18T00:28:43.6334665Z firmware:       nvidia/550.54.15/gsp_ga10x.bin
2024-12-18T00:28:43.6334997Z srcversion:     833721318DA517F0C2FEC97
2024-12-18T00:28:43.6335333Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2024-12-18T00:28:43.6335691Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2024-12-18T00:28:43.6336044Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2024-12-18T00:28:43.6336377Z depends:        i2c-core,drm
2024-12-18T00:28:43.6336644Z retpoline:      Y
2024-12-18T00:28:43.6336885Z name:           nvidia
2024-12-18T00:28:43.6337263Z vermagic:       6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 
2024-12-18T00:28:43.6337778Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2024-12-18T00:28:43.6338247Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2024-12-18T00:28:43.6338685Z parm:           NVreg_ResmanDebugLevel:int
2024-12-18T00:28:43.6339012Z parm:           NVreg_RmLogonRC:int
2024-12-18T00:28:43.6339333Z parm:           NVreg_ModifyDeviceFiles:int
2024-12-18T00:28:43.6339667Z parm:           NVreg_DeviceFileUID:int
2024-12-18T00:28:43.6339989Z parm:           NVreg_DeviceFileGID:int
2024-12-18T00:28:43.6340312Z parm:           NVreg_DeviceFileMode:int
2024-12-18T00:28:43.6340689Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2024-12-18T00:28:43.6341101Z parm:           NVreg_UsePageAttributeTable:int
2024-12-18T00:28:43.6341452Z parm:           NVreg_EnablePCIeGen3:int
2024-12-18T00:28:43.6341776Z parm:           NVreg_EnableMSI:int
2024-12-18T00:28:43.6342087Z parm:           NVreg_TCEBypassMode:int
2024-12-18T00:28:43.6342416Z parm:           NVreg_EnableStreamMemOPs:int
2024-12-18T00:28:43.6342801Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2024-12-18T00:28:43.6343221Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2024-12-18T00:28:43.6343624Z parm:           NVreg_EnableS0ixPowerManagement:int
2024-12-18T00:28:43.6344060Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2024-12-18T00:28:43.6344480Z parm:           NVreg_DynamicPowerManagement:int
2024-12-18T00:28:43.6344918Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2024-12-18T00:28:43.6345352Z parm:           NVreg_EnableGpuFirmware:int
2024-12-18T00:28:43.6345761Z parm:           NVreg_EnableGpuFirmwareLogs:int
2024-12-18T00:28:43.6346153Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2024-12-18T00:28:43.6346540Z parm:           NVreg_EnableUserNUMAManagement:int
2024-12-18T00:28:43.6347010Z parm:           NVreg_MemoryPoolSize:int
2024-12-18T00:28:43.6347352Z parm:           NVreg_KMallocHeapMaxSize:int
2024-12-18T00:28:43.6347701Z parm:           NVreg_VMallocHeapMaxSize:int
2024-12-18T00:28:43.6348042Z parm:           NVreg_IgnoreMMIOCheck:int
2024-12-18T00:28:43.6348364Z parm:           NVreg_NvLinkDisable:int
2024-12-18T00:28:43.6348727Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2024-12-18T00:28:43.6349186Z parm:           NVreg_RegisterPCIDriver:int
2024-12-18T00:28:43.6349533Z parm:           NVreg_EnableResizableBar:int
2024-12-18T00:28:43.6349886Z parm:           NVreg_EnableDbgBreakpoint:int
2024-12-18T00:28:43.6350247Z parm:           NVreg_EnableNonblockingOpen:int
2024-12-18T00:28:43.6350595Z parm:           NVreg_RegistryDwords:charp
2024-12-18T00:28:43.6350954Z parm:           NVreg_RegistryDwordsPerDevice:charp
2024-12-18T00:28:43.6351303Z parm:           NVreg_RmMsg:charp
2024-12-18T00:28:43.6351612Z parm:           NVreg_GpuBlacklist:charp
2024-12-18T00:28:43.6351957Z parm:           NVreg_TemporaryFilePath:charp
2024-12-18T00:28:43.6352294Z parm:           NVreg_ExcludedGpus:charp
2024-12-18T00:28:43.6352626Z parm:           NVreg_DmaRemapPeerMmio:int
2024-12-18T00:28:43.6352973Z parm:           NVreg_RmNvlinkBandwidth:charp
2024-12-18T00:28:43.6353321Z parm:           NVreg_ImexChannelCount:int
2024-12-18T00:28:43.6353651Z parm:           rm_firmware_active:charp
2024-12-18T00:28:43.6353950Z + set +e
2024-12-18T00:28:43.6354165Z + nvidia-smi
2024-12-18T00:28:43.6546537Z Wed Dec 18 00:28:43 2024       
2024-12-18T00:28:43.6546939Z +-----------------------------------------------------------------------------------------+
2024-12-18T00:28:43.6547475Z | NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
2024-12-18T00:28:43.6548000Z |-----------------------------------------+------------------------+----------------------+
2024-12-18T00:28:43.6548525Z | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
2024-12-18T00:28:43.6549098Z | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
2024-12-18T00:28:43.6549567Z |                                         |                        |               MIG M. |
2024-12-18T00:28:43.6549927Z |=========================================+========================+======================|
2024-12-18T00:28:43.6805107Z |   0  NVIDIA A10G                    On  |   00000000:00:1E.0 Off |                    0 |
2024-12-18T00:28:43.6805632Z |  0%   22C    P8             16W /  300W |       0MiB /  23028MiB |      0%      Default |
2024-12-18T00:28:43.6806041Z |                                         |                        |                  N/A |
2024-12-18T00:28:43.6806464Z +-----------------------------------------+------------------------+----------------------+
2024-12-18T00:28:43.6809826Z                                                                                          
2024-12-18T00:28:43.6810266Z +-----------------------------------------------------------------------------------------+
2024-12-18T00:28:43.6810727Z | Processes:                                                                              |
2024-12-18T00:28:43.6811219Z |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
2024-12-18T00:28:43.6811680Z |        ID   ID                                                               Usage      |
2024-12-18T00:28:43.6812052Z |=========================================================================================|
2024-12-18T00:28:43.6815153Z |  No running processes found                                                             |
2024-12-18T00:28:43.6815666Z +-----------------------------------------------------------------------------------------+
2024-12-18T00:28:43.9740785Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0
2024-12-18T00:28:43.9926105Z NVIDIA A10G
2024-12-18T00:28:43.9988042Z + NVIDIA_SMI_STATUS=0
2024-12-18T00:28:43.9988315Z + '[' 0 -eq 0 ']'
2024-12-18T00:28:43.9988846Z + echo 'INFO: Ignoring allowed status 0'
2024-12-18T00:28:43.9989163Z + set -e
2024-12-18T00:28:43.9989499Z INFO: Ignoring allowed status 0
2024-12-18T00:28:43.9997919Z == Installing nvidia container toolkit for amzn2023 ==
2024-12-18T00:28:44.0001834Z + sudo yum install -y yum-utils
2024-12-18T00:28:44.6541473Z Last metadata expiration check: 0:31:12 ago on Tue Dec 17 23:57:32 2024.
2024-12-18T00:28:44.6760216Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed.
2024-12-18T00:28:44.7094573Z Dependencies resolved.
2024-12-18T00:28:44.7232593Z Nothing to do.
2024-12-18T00:28:44.7233976Z Complete!
2024-12-18T00:28:44.8038672Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]]
2024-12-18T00:28:44.8039296Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
2024-12-18T00:28:44.8040205Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
2024-12-18T00:28:45.0578437Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
2024-12-18T00:28:45.1328987Z + sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2
2024-12-18T00:28:45.6400176Z nvidia-container-toolkit                         11 kB/s | 833  B     00:00    
2024-12-18T00:28:45.6625587Z Package nvidia-docker2-2.14.0-1.noarch is already installed.
2024-12-18T00:28:45.6961428Z Dependencies resolved.
2024-12-18T00:28:45.7215748Z ================================================================================
2024-12-18T00:28:45.7216193Z  Package                       Arch   Version    Repository                Size
2024-12-18T00:28:45.7216618Z ================================================================================
2024-12-18T00:28:45.7216948Z Downgrading:
2024-12-18T00:28:45.7217339Z  nvidia-container-toolkit      x86_64 1.16.2-1   nvidia-container-toolkit 1.2 M
2024-12-18T00:28:45.7217967Z  nvidia-container-toolkit-base x86_64 1.16.2-1   nvidia-container-toolkit 5.6 M
2024-12-18T00:28:45.7218348Z 
2024-12-18T00:28:45.7218451Z Transaction Summary
2024-12-18T00:28:45.7218722Z ================================================================================
2024-12-18T00:28:45.7219057Z Downgrade  2 Packages
2024-12-18T00:28:45.7219216Z 
2024-12-18T00:28:45.7219357Z Total download size: 6.8 M
2024-12-18T00:28:45.7220941Z Downloading Packages:
2024-12-18T00:28:45.8126899Z (1/2): nvidia-container-toolkit-1.16.2-1.x86_64  14 MB/s | 1.2 MB     00:00    
2024-12-18T00:28:45.9724905Z (2/2): nvidia-container-toolkit-base-1.16.2-1.x  22 MB/s | 5.6 MB     00:00    
2024-12-18T00:28:45.9732948Z --------------------------------------------------------------------------------
2024-12-18T00:28:45.9735862Z Total                                            27 MB/s | 6.8 MB     00:00     
2024-12-18T00:28:45.9738334Z Running transaction check
2024-12-18T00:28:45.9839566Z Transaction check succeeded.
2024-12-18T00:28:45.9839872Z Running transaction test
2024-12-18T00:28:46.0222007Z Transaction test succeeded.
2024-12-18T00:28:46.0224526Z Running transaction
2024-12-18T00:28:46.6656337Z   Preparing        :                                                        1/1 
2024-12-18T00:28:46.8147301Z   Downgrading      : nvidia-container-toolkit-base-1.16.2-1.x86_64          1/4 
2024-12-18T00:28:46.8413402Z   Downgrading      : nvidia-container-toolkit-1.16.2-1.x86_64               2/4 
2024-12-18T00:28:46.9115789Z   Running scriptlet: nvidia-container-toolkit-1.16.2-1.x86_64               2/4 
2024-12-18T00:28:46.9116425Z   Cleanup          : nvidia-container-toolkit-1.17.3-1.x86_64               3/4 
2024-12-18T00:28:46.9463564Z   Running scriptlet: nvidia-container-toolkit-1.17.3-1.x86_64               3/4 
2024-12-18T00:28:46.9745509Z   Cleanup          : nvidia-container-toolkit-base-1.17.3-1.x86_64          4/4 
2024-12-18T00:29:11.4889127Z   Running scriptlet: nvidia-container-toolkit-1.16.2-1.x86_64               4/4 
2024-12-18T00:29:11.4892816Z   Verifying        : nvidia-container-toolkit-1.16.2-1.x86_64               1/4 
2024-12-18T00:29:11.4893435Z   Verifying        : nvidia-container-toolkit-1.17.3-1.x86_64               2/4 
2024-12-18T00:29:11.4894001Z   Verifying        : nvidia-container-toolkit-base-1.16.2-1.x86_64          3/4 
2024-12-18T00:29:11.6424958Z   Verifying        : nvidia-container-toolkit-base-1.17.3-1.x86_64          4/4================================================================================
2024-12-18T00:29:11.6425876Z WARNING:
2024-12-18T00:29:11.6426143Z   A newer release of "Amazon Linux" is available.
2024-12-18T00:29:11.6426388Z 
2024-12-18T00:29:11.6426493Z   Available Versions:
2024-12-18T00:29:11.6426659Z 
2024-12-18T00:29:11.6426759Z   Version 2023.6.20241010:
2024-12-18T00:29:11.6427096Z     Run the following command to upgrade to 2023.6.20241010:
2024-12-18T00:29:11.6429538Z 
2024-12-18T00:29:11.6430221Z       dnf upgrade --releasever=2023.6.20241010
2024-12-18T00:29:11.6430721Z 
2024-12-18T00:29:11.6430936Z     Release notes:
2024-12-18T00:29:11.6431693Z      https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241010.html
2024-12-18T00:29:11.6432085Z 
2024-12-18T00:29:11.6432202Z   Version 2023.6.20241028:
2024-12-18T00:29:11.6432539Z     Run the following command to upgrade to 2023.6.20241028:
2024-12-18T00:29:11.6432810Z 
2024-12-18T00:29:11.6432937Z       dnf upgrade --releasever=2023.6.20241028
2024-12-18T00:29:11.6433178Z 
2024-12-18T00:29:11.6433274Z     Release notes:
2024-12-18T00:29:11.6433695Z      https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241028.html
2024-12-18T00:29:11.6434085Z 
2024-12-18T00:29:11.6434182Z   Version 2023.6.20241031:
2024-12-18T00:29:11.6434513Z     Run the following command to upgrade to 2023.6.20241031:
2024-12-18T00:29:11.6434784Z 
2024-12-18T00:29:11.6434915Z       dnf upgrade --releasever=2023.6.20241031
2024-12-18T00:29:11.6435136Z 
2024-12-18T00:29:11.6435234Z     Release notes:
2024-12-18T00:29:11.6435653Z      https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241031.html
2024-12-18T00:29:11.6436045Z 
2024-12-18T00:29:11.6436144Z   Version 2023.6.20241111:
2024-12-18T00:29:11.6436478Z     Run the following command to upgrade to 2023.6.20241111:
2024-12-18T00:29:11.6436744Z 
2024-12-18T00:29:11.6436878Z       dnf upgrade --releasever=2023.6.20241111
2024-12-18T00:29:11.6437097Z 
2024-12-18T00:29:11.6437201Z     Release notes:
2024-12-18T00:29:11.6437625Z      https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241111.html
2024-12-18T00:29:11.6438015Z 
2024-12-18T00:29:11.6438115Z   Version 2023.6.20241121:
2024-12-18T00:29:11.6438447Z     Run the following command to upgrade to 2023.6.20241121:
2024-12-18T00:29:11.6438718Z 
2024-12-18T00:29:11.6438843Z       dnf upgrade --releasever=2023.6.20241121
2024-12-18T00:29:11.6439062Z 
2024-12-18T00:29:11.6439169Z     Release notes:
2024-12-18T00:29:11.6439589Z      https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241121.html
2024-12-18T00:29:11.6439974Z 
2024-12-18T00:29:11.6440071Z   Version 2023.6.20241212:
2024-12-18T00:29:11.6440399Z     Run the following command to upgrade to 2023.6.20241212:
2024-12-18T00:29:11.6440666Z 
2024-12-18T00:29:11.6440790Z       dnf upgrade --releasever=2023.6.20241212
2024-12-18T00:29:11.6441015Z 
2024-12-18T00:29:11.6441107Z     Release notes:
2024-12-18T00:29:11.6441660Z      https://docs.aws.amazon.com/linux/al2023/release-notes/relnotes-2023.6.20241212.html
2024-12-18T00:29:11.6442050Z 
2024-12-18T00:29:11.6442175Z ================================================================================
2024-12-18T00:29:11.6698288Z  
2024-12-18T00:29:11.6698446Z 
2024-12-18T00:29:11.6698546Z Downgraded:
2024-12-18T00:29:11.6698952Z   nvidia-container-toolkit-1.16.2-1.x86_64                                      
2024-12-18T00:29:11.6699551Z   nvidia-container-toolkit-base-1.16.2-1.x86_64                                 
2024-12-18T00:29:11.6699911Z 
2024-12-18T00:29:11.6700004Z Complete!
2024-12-18T00:29:11.7205271Z + sudo systemctl restart docker
2024-12-18T00:29:15.7838606Z nvidia-persistenced failed to initialize. Check syslog for more details.
2024-12-18T00:29:15.8070564Z Wed Dec 18 00:29:15 2024       
2024-12-18T00:29:15.8071020Z +-----------------------------------------------------------------------------------------+
2024-12-18T00:29:15.8071544Z | NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
2024-12-18T00:29:15.8072351Z |-----------------------------------------+------------------------+----------------------+
2024-12-18T00:29:15.8072917Z | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
2024-12-18T00:29:15.8073481Z | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
2024-12-18T00:29:15.8073940Z |                                         |                        |               MIG M. |
2024-12-18T00:29:15.8074312Z |=========================================+========================+======================|
2024-12-18T00:29:15.8284380Z |   0  NVIDIA A10G                    On  |   00000000:00:1E.0 Off |                    0 |
2024-12-18T00:29:15.8284854Z |  0%   22C    P8             16W /  300W |       0MiB /  23028MiB |      0%      Default |
2024-12-18T00:29:15.8285259Z |                                         |                        |                  N/A |
2024-12-18T00:29:15.8285674Z +-----------------------------------------+------------------------+----------------------+
2024-12-18T00:29:15.8289612Z                                                                                          
2024-12-18T00:29:15.8290028Z +-----------------------------------------------------------------------------------------+
2024-12-18T00:29:15.8290482Z | Processes:                                                                              |
2024-12-18T00:29:15.8290968Z |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
2024-12-18T00:29:15.8291438Z |        ID   ID                                                               Usage      |
2024-12-18T00:29:15.8291808Z |=========================================================================================|
2024-12-18T00:29:15.8294921Z |  No running processes found                                                             |
2024-12-18T00:29:15.8295415Z +-----------------------------------------------------------------------------------------+
2024-12-18T00:29:16.5881777Z Command completed after 1 attempt(s).
2024-12-18T00:29:16.5977813Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84
2024-12-18T00:29:16.5978399Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84
2024-12-18T00:29:16.5978916Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2024-12-18T00:29:16.5979389Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}"
2024-12-18T00:29:16.5993525Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:16.5993906Z env:
2024-12-18T00:29:16.5994134Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:16.5994483Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:16.5994846Z ##[endgroup]
2024-12-18T00:29:17.0944242Z Defaulting to user installation because normal site-packages is not writeable
2024-12-18T00:29:17.5344857Z Collecting psutil==5.9.1
2024-12-18T00:29:17.5717970Z   Downloading psutil-5.9.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB)
2024-12-18T00:29:17.6234798Z Collecting nvidia-ml-py==11.525.84
2024-12-18T00:29:17.6271204Z   Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB)
2024-12-18T00:29:17.7145485Z Installing collected packages: psutil, nvidia-ml-py
2024-12-18T00:29:17.8717643Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1
2024-12-18T00:29:18.0702643Z Prepare all required actions
2024-12-18T00:29:18.0703765Z Getting action download info
2024-12-18T00:29:18.2180625Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6)
2024-12-18T00:29:18.4900864Z Download action repository 'actions/download-artifact@v4' (SHA:fa0a91b85d4f404e444e00e005971372dc801d16)
2024-12-18T00:29:18.8742098Z ##[group]Run ./.github/actions/download-build-artifacts
2024-12-18T00:29:18.8742470Z with:
2024-12-18T00:29:18.8742733Z   name: linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-12-18T00:29:18.8743277Z   s3-bucket: gha-artifacts
2024-12-18T00:29:18.8743575Z env:
2024-12-18T00:29:18.8743799Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:18.8744133Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:18.8744493Z ##[endgroup]
2024-12-18T00:29:18.8781145Z ##[group]Run seemethere/download-artifact-s3@v4
2024-12-18T00:29:18.8781490Z with:
2024-12-18T00:29:18.8781778Z   name: linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-12-18T00:29:18.8782126Z   s3-bucket: gha-artifacts
2024-12-18T00:29:18.8782396Z   region: us-east-1
2024-12-18T00:29:18.8782626Z env:
2024-12-18T00:29:18.8782849Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:18.8783184Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:18.8783592Z ##[endgroup]
2024-12-18T00:29:19.3534138Z (node:84092) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2024-12-18T00:29:19.3534888Z 
2024-12-18T00:29:19.3535196Z Please migrate your code to use AWS SDK for JavaScript (v3).
2024-12-18T00:29:19.3536084Z For more information, check the migration guide at https://a.co/7PzMCcy
2024-12-18T00:29:19.3536978Z (Use `node --trace-warnings ...` to show where the warning was created)
2024-12-18T00:29:19.6052477Z Found 1 objects with prefix pytorch/pytorch/12383255690/linux-focal-cuda12.4-py3.10-gcc9-sm86/
2024-12-18T00:29:19.6053685Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2024-12-18T00:29:27.3960762Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2024-12-18T00:29:27.3965430Z Artifact download has finished successfully
2024-12-18T00:29:27.4320386Z ##[group]Run unzip -o artifacts.zip
2024-12-18T00:29:27.4320737Z unzip -o artifacts.zip
2024-12-18T00:29:27.4329665Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:27.4330049Z env:
2024-12-18T00:29:27.4330270Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:27.4330617Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:27.4331000Z ##[endgroup]
2024-12-18T00:29:27.4408854Z Archive:  artifacts.zip
2024-12-18T00:29:27.4410325Z    creating: dist/
2024-12-18T00:29:29.6223919Z   inflating: dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl  
2024-12-18T00:29:29.6224918Z    creating: build/custom_test_artifacts/
2024-12-18T00:29:29.6225722Z    creating: build/custom_test_artifacts/custom-op-build/
2024-12-18T00:29:29.6226285Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2024-12-18T00:29:29.6226876Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/
2024-12-18T00:29:29.6232679Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml  
2024-12-18T00:29:29.6233340Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/
2024-12-18T00:29:29.6233978Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2024-12-18T00:29:29.6234665Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/
2024-12-18T00:29:29.6235341Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-12-18T00:29:29.6237442Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2024-12-18T00:29:29.6239181Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2024-12-18T00:29:29.6239880Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-12-18T00:29:29.6240564Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-12-18T00:29:29.6242946Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2024-12-18T00:29:29.6244294Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2024-12-18T00:29:29.6246703Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2024-12-18T00:29:29.6247732Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2024-12-18T00:29:29.6248966Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2024-12-18T00:29:29.6250391Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2024-12-18T00:29:29.6251120Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2024-12-18T00:29:29.6251825Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2024-12-18T00:29:29.6293780Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2024-12-18T00:29:29.6335114Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2024-12-18T00:29:29.6336166Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2024-12-18T00:29:29.6383353Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2024-12-18T00:29:29.6384370Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2024-12-18T00:29:29.6385366Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2024-12-18T00:29:29.6387142Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2024-12-18T00:29:29.6389117Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2024-12-18T00:29:29.6391058Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2024-12-18T00:29:29.6393004Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2024-12-18T00:29:29.6394966Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2024-12-18T00:29:29.6396387Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2024-12-18T00:29:29.6397293Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2024-12-18T00:29:29.6398163Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2024-12-18T00:29:29.6399022Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2024-12-18T00:29:29.6399886Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2024-12-18T00:29:29.6400730Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2024-12-18T00:29:29.6401660Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2024-12-18T00:29:29.6474070Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2024-12-18T00:29:29.6546800Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2024-12-18T00:29:29.6547765Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2024-12-18T00:29:29.6548555Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/
2024-12-18T00:29:29.6549212Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2024-12-18T00:29:29.6549842Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2024-12-18T00:29:29.6550720Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2024-12-18T00:29:29.6551545Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2024-12-18T00:29:29.6552481Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2024-12-18T00:29:29.6553383Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2024-12-18T00:29:29.6554215Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2024-12-18T00:29:29.6555078Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2024-12-18T00:29:29.6555841Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2024-12-18T00:29:29.6556603Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2024-12-18T00:29:29.6557360Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2024-12-18T00:29:29.6558099Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2024-12-18T00:29:29.6579051Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2024-12-18T00:29:29.6723170Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2024-12-18T00:29:29.6723870Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2024-12-18T00:29:29.6724624Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2024-12-18T00:29:29.6725491Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2024-12-18T00:29:29.6726338Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2024-12-18T00:29:29.6727132Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2024-12-18T00:29:29.6727931Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2024-12-18T00:29:29.6728738Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2024-12-18T00:29:29.6729527Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2024-12-18T00:29:29.6730337Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2024-12-18T00:29:29.6731132Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2024-12-18T00:29:29.6752942Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2024-12-18T00:29:29.6836539Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2024-12-18T00:29:29.6837400Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2024-12-18T00:29:29.6838180Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2024-12-18T00:29:29.6839031Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2024-12-18T00:29:29.6840092Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2024-12-18T00:29:29.6841994Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2024-12-18T00:29:29.6842633Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2024-12-18T00:29:29.6845417Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2024-12-18T00:29:29.6846011Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2024-12-18T00:29:29.6846984Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2024-12-18T00:29:29.6968258Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2024-12-18T00:29:29.7031951Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2024-12-18T00:29:29.7032955Z    creating: build/custom_test_artifacts/jit-hook-build/
2024-12-18T00:29:29.7033872Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2024-12-18T00:29:29.7034967Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/
2024-12-18T00:29:29.7039973Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml  
2024-12-18T00:29:29.7040623Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/
2024-12-18T00:29:29.7041328Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2024-12-18T00:29:29.7042010Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/
2024-12-18T00:29:29.7042665Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-12-18T00:29:29.7044917Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2024-12-18T00:29:29.7046219Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2024-12-18T00:29:29.7046916Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-12-18T00:29:29.7047595Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-12-18T00:29:29.7050273Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2024-12-18T00:29:29.7051567Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2024-12-18T00:29:29.7053934Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2024-12-18T00:29:29.7054738Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2024-12-18T00:29:29.7056300Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2024-12-18T00:29:29.7057699Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2024-12-18T00:29:29.7058430Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2024-12-18T00:29:29.7059119Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2024-12-18T00:29:29.7100892Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2024-12-18T00:29:29.7141958Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2024-12-18T00:29:29.7142965Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2024-12-18T00:29:29.7190788Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2024-12-18T00:29:29.7192567Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2024-12-18T00:29:29.7194701Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2024-12-18T00:29:29.7196480Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2024-12-18T00:29:29.7197458Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2024-12-18T00:29:29.7198415Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2024-12-18T00:29:29.7199498Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2024-12-18T00:29:29.7200461Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2024-12-18T00:29:29.7201454Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2024-12-18T00:29:29.7202344Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2024-12-18T00:29:29.7203204Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2024-12-18T00:29:29.7204069Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2024-12-18T00:29:29.7204923Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2024-12-18T00:29:29.7205757Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2024-12-18T00:29:29.7206599Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2024-12-18T00:29:29.7279054Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2024-12-18T00:29:29.7352814Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2024-12-18T00:29:29.7353651Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2024-12-18T00:29:29.7354348Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/
2024-12-18T00:29:29.7354945Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2024-12-18T00:29:29.7355567Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2024-12-18T00:29:29.7356226Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2024-12-18T00:29:29.7356967Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2024-12-18T00:29:29.7358075Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2024-12-18T00:29:29.7358878Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2024-12-18T00:29:29.7359658Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2024-12-18T00:29:29.7360430Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2024-12-18T00:29:29.7361759Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2024-12-18T00:29:29.7362668Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2024-12-18T00:29:29.7363486Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2024-12-18T00:29:29.7364483Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2024-12-18T00:29:29.7386498Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2024-12-18T00:29:29.7450665Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2024-12-18T00:29:29.7451724Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2024-12-18T00:29:29.7452523Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2024-12-18T00:29:29.7453220Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2024-12-18T00:29:29.7454025Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2024-12-18T00:29:29.7455010Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2024-12-18T00:29:29.7455790Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2024-12-18T00:29:29.7458870Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2024-12-18T00:29:29.7459592Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2024-12-18T00:29:29.7460443Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2024-12-18T00:29:29.7510640Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2024-12-18T00:29:29.7511229Z    creating: build/custom_test_artifacts/custom-backend-build/
2024-12-18T00:29:29.7511805Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2024-12-18T00:29:29.7512499Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/
2024-12-18T00:29:29.7519571Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml  
2024-12-18T00:29:29.7520282Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/
2024-12-18T00:29:29.7520965Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2024-12-18T00:29:29.7521750Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/
2024-12-18T00:29:29.7522469Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-12-18T00:29:29.7524462Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2024-12-18T00:29:29.7525847Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2024-12-18T00:29:29.7526591Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-12-18T00:29:29.7527334Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-12-18T00:29:29.7529884Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2024-12-18T00:29:29.7531334Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2024-12-18T00:29:29.7533462Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2024-12-18T00:29:29.7534332Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2024-12-18T00:29:29.7535863Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2024-12-18T00:29:29.7537291Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2024-12-18T00:29:29.7538072Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2024-12-18T00:29:29.7538825Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2024-12-18T00:29:29.7581002Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2024-12-18T00:29:29.7622681Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2024-12-18T00:29:29.7623923Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2024-12-18T00:29:29.7671691Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2024-12-18T00:29:29.7673634Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2024-12-18T00:29:29.7675561Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2024-12-18T00:29:29.7677212Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2024-12-18T00:29:29.7678261Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2024-12-18T00:29:29.7679280Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2024-12-18T00:29:29.7680306Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2024-12-18T00:29:29.7681381Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2024-12-18T00:29:29.7682385Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2024-12-18T00:29:29.7683337Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2024-12-18T00:29:29.7684259Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2024-12-18T00:29:29.7685176Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2024-12-18T00:29:29.7686105Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2024-12-18T00:29:29.7686998Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2024-12-18T00:29:29.7687909Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2024-12-18T00:29:29.7760757Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2024-12-18T00:29:29.7834053Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2024-12-18T00:29:29.7834969Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2024-12-18T00:29:29.7835725Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/
2024-12-18T00:29:29.7836422Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2024-12-18T00:29:29.7837106Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2024-12-18T00:29:29.7837805Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2024-12-18T00:29:29.7838608Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2024-12-18T00:29:29.7839517Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2024-12-18T00:29:29.7840389Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2024-12-18T00:29:29.7841278Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2024-12-18T00:29:29.7842118Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2024-12-18T00:29:29.7843120Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2024-12-18T00:29:29.7843971Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2024-12-18T00:29:29.7844822Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2024-12-18T00:29:29.7845653Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2024-12-18T00:29:29.7849575Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2024-12-18T00:29:29.7972377Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2024-12-18T00:29:29.7974023Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2024-12-18T00:29:29.7975680Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2024-12-18T00:29:29.7976676Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2024-12-18T00:29:29.7977585Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2024-12-18T00:29:29.7978430Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2024-12-18T00:29:29.7979300Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2024-12-18T00:29:29.7980181Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2024-12-18T00:29:29.7981074Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2024-12-18T00:29:29.7981956Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2024-12-18T00:29:29.7982824Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2024-12-18T00:29:29.8001725Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2024-12-18T00:29:29.8056841Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2024-12-18T00:29:29.8057777Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2024-12-18T00:29:29.8058596Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2024-12-18T00:29:29.8059346Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2024-12-18T00:29:29.8060556Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2024-12-18T00:29:29.8061524Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2024-12-18T00:29:29.8062214Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2024-12-18T00:29:29.8065143Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2024-12-18T00:29:29.8066187Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2024-12-18T00:29:29.8067306Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2024-12-18T00:29:29.8168168Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2024-12-18T00:29:29.8211167Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2024-12-18T00:29:29.8211666Z    creating: build/lib/
2024-12-18T00:29:29.8300389Z   inflating: build/lib/libprotobuf-lite.a  
2024-12-18T00:29:29.8756013Z   inflating: build/lib/libprotobuf.a  
2024-12-18T00:29:29.8765933Z   inflating: build/lib/libpthreadpool.a  
2024-12-18T00:29:29.8774495Z   inflating: build/lib/libcpuinfo.a  
2024-12-18T00:29:29.8782835Z   inflating: build/lib/libcpuinfo_internals.a  
2024-12-18T00:29:29.8783493Z   inflating: build/lib/libclog.a     
2024-12-18T00:29:29.8802031Z   inflating: build/lib/libnnpack.a   
2024-12-18T00:29:29.8804760Z   inflating: build/lib/libnnpack_reference_layers.a  
2024-12-18T00:29:29.8992006Z   inflating: build/lib/libmicrokernels-prod.a  
2024-12-18T00:29:29.9060922Z   inflating: build/lib/libgtest.a    
2024-12-18T00:29:29.9135906Z   inflating: build/lib/libbenchmark.a  
2024-12-18T00:29:29.9199496Z   inflating: build/lib/libasmjit.a   
2024-12-18T00:29:29.9207177Z   inflating: build/lib/libittnotify.a  
2024-12-18T00:29:29.9235550Z   inflating: build/lib/libtensorpipe_uv.a  
2024-12-18T00:29:29.9362661Z   inflating: build/lib/libgloo.a     
2024-12-18T00:29:29.9401712Z   inflating: build/lib/libfmt.a      
2024-12-18T00:29:29.9483405Z   inflating: build/lib/libc10.so     
2024-12-18T00:29:29.9485204Z   inflating: build/lib/libcaffe2_nvrtc.so  
2024-12-18T00:29:29.9486728Z   inflating: build/lib/libtorch_global_deps.so  
2024-12-18T00:29:29.9991243Z   inflating: build/lib/libprotoc.a   
2024-12-18T00:29:30.0011049Z   inflating: build/lib/libpytorch_qnnpack.a  
2024-12-18T00:29:30.0028599Z   inflating: build/lib/libgmock.a    
2024-12-18T00:29:30.0029358Z   inflating: build/lib/libgtest_main.a  
2024-12-18T00:29:30.0030244Z   inflating: build/lib/libbenchmark_main.a  
2024-12-18T00:29:30.0605446Z   inflating: build/lib/libtensorpipe.a  
2024-12-18T00:29:30.0994565Z   inflating: build/lib/libgloo_cuda.a  
2024-12-18T00:29:31.1305051Z   inflating: build/lib/libdnnl.a     
2024-12-18T00:29:31.1364120Z   inflating: build/lib/libc10_cuda.so  
2024-12-18T00:29:31.1364939Z   inflating: build/lib/libgmock_main.a  
2024-12-18T00:29:31.2625993Z   inflating: build/lib/libfbgemm.a   
2024-12-18T00:29:31.2883365Z   inflating: build/lib/libtensorpipe_cuda.a  
2024-12-18T00:29:31.3392333Z   inflating: build/lib/libkineto.a   
2024-12-18T00:29:31.3482197Z   inflating: build/lib/libXNNPACK.a  
2024-12-18T00:29:31.3524636Z   inflating: build/lib/libonnx_proto.a  
2024-12-18T00:29:31.4255235Z   inflating: build/lib/libonnx.a     
2024-12-18T00:29:31.5123179Z   inflating: build/lib/libmicrokernels-all.a  
2024-12-18T00:29:34.0409609Z   inflating: build/lib/libtorch_cpu.so  
2024-12-18T00:29:34.0414526Z   inflating: build/lib/libunbox_lib.a  
2024-12-18T00:29:34.0418815Z   inflating: build/lib/libshm.so     
2024-12-18T00:29:36.1366492Z   inflating: build/lib/libtorch_cuda.so  
2024-12-18T00:29:36.1368721Z   inflating: build/lib/libtorch.so   
2024-12-18T00:29:36.1371316Z   inflating: build/lib/libc10d_cuda_test.so  
2024-12-18T00:29:37.0173353Z   inflating: build/lib/libtorch_cuda_linalg.so  
2024-12-18T00:29:37.2259483Z   inflating: build/lib/libtorch_python.so  
2024-12-18T00:29:37.2329699Z   inflating: build/lib/libtorchbind_test.so  
2024-12-18T00:29:37.2349828Z   inflating: build/lib/libjitbackend_test.so  
2024-12-18T00:29:37.2375121Z   inflating: build/lib/libbackend_with_compiler.so  
2024-12-18T00:29:37.2400375Z   inflating: build/lib/libaoti_custom_ops.so  
2024-12-18T00:29:37.2435135Z   inflating: build/lib/libnnapi_backend.so  
2024-12-18T00:29:37.2435463Z    creating: build/bin/
2024-12-18T00:29:37.2489776Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2024-12-18T00:29:37.2544753Z   inflating: build/bin/c10_DeviceGuard_test  
2024-12-18T00:29:37.2600133Z   inflating: build/bin/c10_Device_test  
2024-12-18T00:29:37.2662933Z   inflating: build/bin/c10_DispatchKeySet_test  
2024-12-18T00:29:37.2719963Z   inflating: build/bin/c10_Scalar_test  
2024-12-18T00:29:37.2772285Z   inflating: build/bin/c10_StreamGuard_test  
2024-12-18T00:29:37.2826942Z   inflating: build/bin/c10_SymInt_test  
2024-12-18T00:29:37.2884966Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2024-12-18T00:29:37.2944439Z   inflating: build/bin/c10_InlineStreamGuard_test  
2024-12-18T00:29:37.3004829Z   inflating: build/bin/c10_SizesAndStrides_test  
2024-12-18T00:29:37.3079669Z   inflating: build/bin/c10_cow_test  
2024-12-18T00:29:37.3132945Z   inflating: build/bin/c10_ArrayRef_test  
2024-12-18T00:29:37.3190414Z   inflating: build/bin/c10_Bitset_test  
2024-12-18T00:29:37.3242085Z   inflating: build/bin/c10_ConstexprCrc_test  
2024-12-18T00:29:37.3295447Z   inflating: build/bin/c10_DeadlockDetection_test  
2024-12-18T00:29:37.3349351Z   inflating: build/bin/c10_Half_test  
2024-12-18T00:29:37.3410020Z   inflating: build/bin/c10_LeftRight_test  
2024-12-18T00:29:37.3468259Z   inflating: build/bin/c10_Metaprogramming_test  
2024-12-18T00:29:37.3524712Z   inflating: build/bin/c10_NetworkFlow_test  
2024-12-18T00:29:37.3578583Z   inflating: build/bin/c10_Synchronized_test  
2024-12-18T00:29:37.3637770Z   inflating: build/bin/c10_ThreadLocal_test  
2024-12-18T00:29:37.3693420Z   inflating: build/bin/c10_TypeIndex_test  
2024-12-18T00:29:37.3748307Z   inflating: build/bin/c10_TypeList_test  
2024-12-18T00:29:37.3800900Z   inflating: build/bin/c10_TypeTraits_test  
2024-12-18T00:29:37.3855973Z   inflating: build/bin/c10_accumulate_test  
2024-12-18T00:29:37.3915786Z   inflating: build/bin/c10_bfloat16_test  
2024-12-18T00:29:37.3969574Z   inflating: build/bin/c10_bit_cast_test  
2024-12-18T00:29:37.4029882Z   inflating: build/bin/c10_complex_math_test  
2024-12-18T00:29:37.4088595Z   inflating: build/bin/c10_complex_test  
2024-12-18T00:29:37.4142010Z   inflating: build/bin/c10_error_test  
2024-12-18T00:29:37.4198992Z   inflating: build/bin/c10_exception_test  
2024-12-18T00:29:37.4253160Z   inflating: build/bin/c10_flags_test  
2024-12-18T00:29:37.4307749Z   inflating: build/bin/c10_generic_math_test  
2024-12-18T00:29:37.4480226Z   inflating: build/bin/c10_intrusive_ptr_test  
2024-12-18T00:29:37.4534443Z   inflating: build/bin/c10_irange_test  
2024-12-18T00:29:37.4591658Z   inflating: build/bin/c10_lazy_test  
2024-12-18T00:29:37.4652367Z   inflating: build/bin/c10_logging_test  
2024-12-18T00:29:37.4731936Z   inflating: build/bin/c10_optional_test  
2024-12-18T00:29:37.4798882Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2024-12-18T00:29:37.4855562Z   inflating: build/bin/c10_registry_test  
2024-12-18T00:29:37.5016065Z   inflating: build/bin/c10_small_vector_test  
2024-12-18T00:29:37.5071567Z   inflating: build/bin/c10_ssize_test  
2024-12-18T00:29:37.5126459Z   inflating: build/bin/c10_string_util_test  
2024-12-18T00:29:37.5190414Z   inflating: build/bin/c10_string_view_test  
2024-12-18T00:29:37.5244581Z   inflating: build/bin/c10_tempfile_test  
2024-12-18T00:29:37.5304314Z   inflating: build/bin/c10_typeid_test  
2024-12-18T00:29:37.5353191Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2024-12-18T00:29:37.5802293Z   inflating: build/bin/protoc-3.13.0.0  
2024-12-18T00:29:37.6250464Z   inflating: build/bin/protoc        
2024-12-18T00:29:37.6307048Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test  
2024-12-18T00:29:37.6364384Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream  
2024-12-18T00:29:37.6420007Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device  
2024-12-18T00:29:37.6475660Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes  
2024-12-18T00:29:37.6531642Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads  
2024-12-18T00:29:37.6588167Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks  
2024-12-18T00:29:37.6644367Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block  
2024-12-18T00:29:37.6696838Z   inflating: build/bin/c10_cuda_CUDATest  
2024-12-18T00:29:37.7098260Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2024-12-18T00:29:37.7509606Z   inflating: build/bin/vec_test_all_types_AVX512  
2024-12-18T00:29:37.7937405Z   inflating: build/bin/vec_test_all_types_AVX2  
2024-12-18T00:29:37.7993973Z   inflating: build/bin/BackoffTest   
2024-12-18T00:29:37.8050499Z   inflating: build/bin/FileStoreTest  
2024-12-18T00:29:37.8110247Z   inflating: build/bin/TCPStoreTest  
2024-12-18T00:29:37.8167594Z   inflating: build/bin/HashStoreTest  
2024-12-18T00:29:37.8180866Z   inflating: build/bin/ProcessGroupMPITest  
2024-12-18T00:29:37.8238301Z   inflating: build/bin/test_edge_op_registration  
2024-12-18T00:29:37.8242708Z   inflating: build/bin/torch_shm_manager  
2024-12-18T00:29:37.8246426Z   inflating: build/bin/example_allreduce  
2024-12-18T00:29:37.8304554Z   inflating: build/bin/test_dist_autograd  
2024-12-18T00:29:37.8377596Z   inflating: build/bin/test_cpp_rpc  
2024-12-18T00:29:37.8380309Z   inflating: build/bin/parallel_benchmark  
2024-12-18T00:29:37.8450637Z   inflating: build/bin/test_mobile_nnc  
2024-12-18T00:29:37.8459979Z   inflating: build/bin/aot_model_compiler_test  
2024-12-18T00:29:37.8814256Z   inflating: build/bin/test_lazy     
2024-12-18T00:29:38.0004040Z   inflating: build/bin/test_api      
2024-12-18T00:29:38.0082829Z   inflating: build/bin/Dict_test     
2024-12-18T00:29:38.0138997Z   inflating: build/bin/Dimname_test  
2024-12-18T00:29:38.0208212Z   inflating: build/bin/MaybeOwned_test  
2024-12-18T00:29:38.0270022Z   inflating: build/bin/NamedTensor_test  
2024-12-18T00:29:38.0331951Z   inflating: build/bin/apply_utils_test  
2024-12-18T00:29:38.0395643Z   inflating: build/bin/atest         
2024-12-18T00:29:38.0462912Z   inflating: build/bin/basic         
2024-12-18T00:29:38.0521800Z   inflating: build/bin/broadcast_test  
2024-12-18T00:29:38.0576705Z   inflating: build/bin/cpu_allocator_test  
2024-12-18T00:29:38.0639128Z   inflating: build/bin/cpu_generator_test  
2024-12-18T00:29:38.0696566Z   inflating: build/bin/cpu_profiling_allocator_test  
2024-12-18T00:29:38.0793747Z   inflating: build/bin/cpu_rng_test  
2024-12-18T00:29:38.0847321Z   inflating: build/bin/dispatch_key_set_test  
2024-12-18T00:29:38.0901734Z   inflating: build/bin/dlconvertor_test  
2024-12-18T00:29:38.0963032Z   inflating: build/bin/extension_backend_test  
2024-12-18T00:29:38.1021881Z   inflating: build/bin/half_test     
2024-12-18T00:29:38.1122421Z   inflating: build/bin/ivalue_test   
2024-12-18T00:29:38.1175556Z   inflating: build/bin/lazy_tensor_test  
2024-12-18T00:29:38.1233486Z   inflating: build/bin/math_kernel_test  
2024-12-18T00:29:38.1291481Z   inflating: build/bin/memory_format_test  
2024-12-18T00:29:38.1348215Z   inflating: build/bin/memory_overlapping_test  
2024-12-18T00:29:38.1405337Z   inflating: build/bin/mobile_memory_cleanup  
2024-12-18T00:29:38.1465088Z   inflating: build/bin/native_test   
2024-12-18T00:29:38.1519752Z   inflating: build/bin/operator_name_test  
2024-12-18T00:29:38.1574759Z   inflating: build/bin/operators_test  
2024-12-18T00:29:38.1630760Z   inflating: build/bin/packedtensoraccessor_test  
2024-12-18T00:29:38.1701464Z   inflating: build/bin/pow_test      
2024-12-18T00:29:38.1762361Z   inflating: build/bin/quantized_test  
2024-12-18T00:29:38.1816076Z   inflating: build/bin/reduce_ops_test  
2024-12-18T00:29:38.1870724Z   inflating: build/bin/reportMemoryUsage_test  
2024-12-18T00:29:38.1931777Z   inflating: build/bin/scalar_tensor_test  
2024-12-18T00:29:38.1994864Z   inflating: build/bin/scalar_test   
2024-12-18T00:29:38.2049940Z   inflating: build/bin/StorageUtils_test  
2024-12-18T00:29:38.2106541Z   inflating: build/bin/stride_properties_test  
2024-12-18T00:29:38.2190020Z   inflating: build/bin/tensor_iterator_test  
2024-12-18T00:29:38.2247623Z   inflating: build/bin/test_parallel  
2024-12-18T00:29:38.2250734Z   inflating: build/bin/thread_init_test  
2024-12-18T00:29:38.2310003Z   inflating: build/bin/type_ptr_test  
2024-12-18T00:29:38.2373542Z   inflating: build/bin/type_test     
2024-12-18T00:29:38.2429634Z   inflating: build/bin/undefined_tensor_test  
2024-12-18T00:29:38.2430868Z   inflating: build/bin/verify_api_visibility  
2024-12-18T00:29:38.2505786Z   inflating: build/bin/legacy_vmap_test  
2024-12-18T00:29:38.2560914Z   inflating: build/bin/weakref_test  
2024-12-18T00:29:38.2616791Z   inflating: build/bin/wrapdim_test  
2024-12-18T00:29:38.2672658Z   inflating: build/bin/xla_tensor_test  
2024-12-18T00:29:38.2735768Z   inflating: build/bin/IListRef_test  
2024-12-18T00:29:38.2847752Z   inflating: build/bin/List_test     
2024-12-18T00:29:38.2917809Z   inflating: build/bin/KernelFunction_test  
2024-12-18T00:29:38.3044205Z   inflating: build/bin/kernel_function_legacy_test  
2024-12-18T00:29:38.3145196Z   inflating: build/bin/kernel_function_test  
2024-12-18T00:29:38.3278428Z   inflating: build/bin/kernel_lambda_legacy_test  
2024-12-18T00:29:38.3386532Z   inflating: build/bin/kernel_lambda_test  
2024-12-18T00:29:38.3451433Z   inflating: build/bin/kernel_stackbased_test  
2024-12-18T00:29:38.3551918Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2024-12-18T00:29:38.3606956Z   inflating: build/bin/CppSignature_test  
2024-12-18T00:29:38.3666520Z   inflating: build/bin/backend_fallback_test  
2024-12-18T00:29:38.3718667Z   inflating: build/bin/op_allowlist_test  
2024-12-18T00:29:38.4026855Z   inflating: build/bin/op_registration_test  
2024-12-18T00:29:38.4094412Z   inflating: build/bin/inline_container_test  
2024-12-18T00:29:38.4150604Z   inflating: build/bin/cuda_apply_test  
2024-12-18T00:29:38.4206347Z   inflating: build/bin/cuda_allocator_test  
2024-12-18T00:29:38.4264705Z   inflating: build/bin/cuda_caching_host_allocator_test  
2024-12-18T00:29:38.4327933Z   inflating: build/bin/cuda_atomic_ops_test  
2024-12-18T00:29:38.4401861Z   inflating: build/bin/cuda_complex_math_test  
2024-12-18T00:29:38.4464873Z   inflating: build/bin/cuda_complex_test  
2024-12-18T00:29:38.4518524Z   inflating: build/bin/cuda_device_test  
2024-12-18T00:29:38.4581264Z   inflating: build/bin/cuda_cub_test  
2024-12-18T00:29:38.4635940Z   inflating: build/bin/cuda_dlconvertor_test  
2024-12-18T00:29:38.4705136Z   inflating: build/bin/cuda_distributions_test  
2024-12-18T00:29:38.4765613Z   inflating: build/bin/cuda_generator_test  
2024-12-18T00:29:38.4819305Z   inflating: build/bin/cuda_half_test  
2024-12-18T00:29:38.4874566Z   inflating: build/bin/cuda_integer_divider_test  
2024-12-18T00:29:38.4927440Z   inflating: build/bin/cuda_optional_test  
2024-12-18T00:29:38.4982879Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2024-12-18T00:29:38.5039043Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2024-12-18T00:29:38.5092818Z   inflating: build/bin/cuda_allocatorTraceTracker_test  
2024-12-18T00:29:38.5157189Z   inflating: build/bin/cuda_stream_test  
2024-12-18T00:29:38.5210901Z   inflating: build/bin/cuda_cudnn_test  
2024-12-18T00:29:38.5267160Z   inflating: build/bin/cuda_vectorized_test  
2024-12-18T00:29:38.5281314Z   inflating: build/bin/tutorial_tensorexpr  
2024-12-18T00:29:38.5351011Z   inflating: build/bin/ProcessGroupGlooTest  
2024-12-18T00:29:38.5411908Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2024-12-18T00:29:38.5479676Z   inflating: build/bin/ProcessGroupNCCLTest  
2024-12-18T00:29:38.5545594Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2024-12-18T00:29:38.6385652Z   inflating: build/bin/test_tensorexpr  
2024-12-18T00:29:38.6971297Z   inflating: build/bin/test_jit      
2024-12-18T00:29:38.6971645Z    creating: .additional_ci_files/
2024-12-18T00:29:38.7062103Z   inflating: .additional_ci_files/test-times.json  
2024-12-18T00:29:38.7421479Z   inflating: .additional_ci_files/test-class-times.json  
2024-12-18T00:29:38.7456492Z ##[group]Run rm artifacts.zip
2024-12-18T00:29:38.7456803Z rm artifacts.zip
2024-12-18T00:29:38.7465583Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:38.7465966Z env:
2024-12-18T00:29:38.7466191Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:38.7466525Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:38.7467266Z ##[endgroup]
2024-12-18T00:29:38.8949702Z ##[group]Run df -H
2024-12-18T00:29:38.8949965Z df -H
2024-12-18T00:29:38.8958794Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:38.8959178Z env:
2024-12-18T00:29:38.8959406Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:38.8959740Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:38.8960096Z ##[endgroup]
2024-12-18T00:29:38.9197221Z Filesystem        Size  Used Avail Use% Mounted on
2024-12-18T00:29:38.9197671Z devtmpfs          4.2M     0  4.2M   0% /dev
2024-12-18T00:29:38.9198006Z tmpfs              34G     0   34G   0% /dev/shm
2024-12-18T00:29:38.9198339Z tmpfs              14G  553k   14G   1% /run
2024-12-18T00:29:38.9198662Z /dev/nvme0n1p1    161G   81G   81G  50% /
2024-12-18T00:29:38.9198990Z tmpfs              34G   21k   34G   1% /tmp
2024-12-18T00:29:38.9199322Z /dev/nvme0n1p128   11M  1.4M  9.2M  13% /boot/efi
2024-12-18T00:29:38.9199687Z tmpfs             6.7G     0  6.7G   0% /run/user/0
2024-12-18T00:29:38.9234309Z Prepare all required actions
2024-12-18T00:29:38.9234687Z Getting action download info
2024-12-18T00:29:39.0992766Z ##[group]Run ./.github/actions/download-td-artifacts
2024-12-18T00:29:39.0993121Z with:
2024-12-18T00:29:39.0993334Z env:
2024-12-18T00:29:39.0993557Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:39.0993890Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:39.0994253Z ##[endgroup]
2024-12-18T00:29:39.1117698Z ##[group]Run seemethere/download-artifact-s3@v4
2024-12-18T00:29:39.1118058Z with:
2024-12-18T00:29:39.1118272Z   name: td_results
2024-12-18T00:29:39.1118527Z   s3-bucket: gha-artifacts
2024-12-18T00:29:39.1118791Z   region: us-east-1
2024-12-18T00:29:39.1119029Z env:
2024-12-18T00:29:39.1119251Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:39.1119583Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:39.1119943Z ##[endgroup]
2024-12-18T00:29:39.5784808Z (node:84113) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2024-12-18T00:29:39.5785553Z 
2024-12-18T00:29:39.5785870Z Please migrate your code to use AWS SDK for JavaScript (v3).
2024-12-18T00:29:39.5786690Z For more information, check the migration guide at https://a.co/7PzMCcy
2024-12-18T00:29:39.5787579Z (Use `node --trace-warnings ...` to show where the warning was created)
2024-12-18T00:29:39.7028256Z Found 0 objects with prefix pytorch/pytorch/12383255690/td_results/
2024-12-18T00:29:39.7034675Z Artifact download has finished successfully
2024-12-18T00:29:39.7441655Z ##[group]Run mkdir -p .additional_ci_files
2024-12-18T00:29:39.7442024Z mkdir -p .additional_ci_files
2024-12-18T00:29:39.7442455Z mv td_results.json .additional_ci_files/td_results.json || true
2024-12-18T00:29:39.7451608Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:39.7451984Z env:
2024-12-18T00:29:39.7452203Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:39.7452547Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:39.7452914Z ##[endgroup]
2024-12-18T00:29:39.7788087Z mv: cannot stat 'td_results.json': No such file or directory
2024-12-18T00:29:39.8289488Z ##[group]Run .github/scripts/parse_ref.py
2024-12-18T00:29:39.8290005Z .github/scripts/parse_ref.py
2024-12-18T00:29:39.8300118Z shell: /usr/bin/bash -e {0}
2024-12-18T00:29:39.8300452Z env:
2024-12-18T00:29:39.8300815Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:39.8301289Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:39.8301751Z ##[endgroup]
2024-12-18T00:29:39.8719313Z Prepare all required actions
2024-12-18T00:29:39.8820038Z ##[group]Run ./.github/actions/get-workflow-job-id
2024-12-18T00:29:39.8820379Z with:
2024-12-18T00:29:39.8820775Z   github-token: ***
2024-12-18T00:29:39.8821008Z env:
2024-12-18T00:29:39.8821233Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:39.8821567Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:39.8821926Z ##[endgroup]
2024-12-18T00:29:39.8895740Z ##[group]Run set -eux
2024-12-18T00:29:39.8896006Z set -eux
2024-12-18T00:29:39.8896436Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2024-12-18T00:29:39.8905506Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:39.8905879Z env:
2024-12-18T00:29:39.8906098Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:39.8906429Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:39.8907030Z   GITHUB_TOKEN: ***
2024-12-18T00:29:39.8907279Z ##[endgroup]
2024-12-18T00:29:39.8935686Z + python3 .github/scripts/get_workflow_job_id.py 12383255690 i-029fac8e3f5b7f275
2024-12-18T00:29:41.0656735Z setting job-id=34567153568
2024-12-18T00:29:41.0657303Z setting job-name=cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:41.0833698Z Prepare all required actions
2024-12-18T00:29:41.0834078Z Getting action download info
2024-12-18T00:29:41.2335296Z ##[group]Run ./.github/actions/filter-test-configs
2024-12-18T00:29:41.2335833Z with:
2024-12-18T00:29:41.2336218Z   github-token: ***
2024-12-18T00:29:41.2337933Z   test-matrix: {"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}
2024-12-18T00:29:41.2339918Z   job-name: cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:41.2340441Z env:
2024-12-18T00:29:41.2340870Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:41.2341215Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:41.2341569Z ##[endgroup]
2024-12-18T00:29:41.2385296Z ##[group]Run nick-fields/retry@v3.0.0
2024-12-18T00:29:41.2385598Z with:
2024-12-18T00:29:41.2385807Z   shell: bash
2024-12-18T00:29:41.2386026Z   timeout_minutes: 10
2024-12-18T00:29:41.2386274Z   max_attempts: 5
2024-12-18T00:29:41.2386513Z   retry_wait_seconds: 30
2024-12-18T00:29:41.2387272Z   command: set -eux
# PyYAML 6.0 doesn't work with MacOS x86 anymore
# This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2
python3 -m pip install requests==2.27.1 pyyaml==6.0.1

2024-12-18T00:29:41.2388067Z   polling_interval_seconds: 1
2024-12-18T00:29:41.2388344Z   warning_on_retry: true
2024-12-18T00:29:41.2388628Z   continue_on_error: false
2024-12-18T00:29:41.2388912Z env:
2024-12-18T00:29:41.2389126Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:41.2389463Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:41.2389967Z   GITHUB_TOKEN: ***
2024-12-18T00:29:41.2390204Z ##[endgroup]
2024-12-18T00:29:41.8736558Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1
2024-12-18T00:29:42.3308202Z Collecting requests==2.27.1
2024-12-18T00:29:42.3657331Z   Downloading requests-2.27.1-py2.py3-none-any.whl.metadata (5.0 kB)
2024-12-18T00:29:42.4154637Z Collecting pyyaml==6.0.1
2024-12-18T00:29:42.4181186Z   Downloading PyYAML-6.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB)
2024-12-18T00:29:42.4639231Z Collecting urllib3<1.27,>=1.21.1 (from requests==2.27.1)
2024-12-18T00:29:42.4663847Z   Downloading urllib3-1.26.20-py2.py3-none-any.whl.metadata (50 kB)
2024-12-18T00:29:42.4849384Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/miniconda/lib/python3.12/site-packages (from requests==2.27.1) (2024.12.14)
2024-12-18T00:29:42.5562352Z Collecting charset-normalizer~=2.0.0 (from requests==2.27.1)
2024-12-18T00:29:42.5590085Z   Downloading charset_normalizer-2.0.12-py3-none-any.whl.metadata (11 kB)
2024-12-18T00:29:42.5658983Z Requirement already satisfied: idna<4,>=2.5 in /home/ec2-user/miniconda/lib/python3.12/site-packages (from requests==2.27.1) (3.7)
2024-12-18T00:29:42.5734690Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB)
2024-12-18T00:29:42.5838967Z Downloading PyYAML-6.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (724 kB)
2024-12-18T00:29:42.6006733Z    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 725.0/725.0 kB 41.1 MB/s eta 0:00:00
2024-12-18T00:29:42.6038151Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB)
2024-12-18T00:29:42.6113973Z Downloading urllib3-1.26.20-py2.py3-none-any.whl (144 kB)
2024-12-18T00:29:42.6768682Z Installing collected packages: urllib3, pyyaml, charset-normalizer, requests
2024-12-18T00:29:42.6770198Z   Attempting uninstall: urllib3
2024-12-18T00:29:42.6792065Z     Found existing installation: urllib3 2.2.3
2024-12-18T00:29:42.6833922Z     Uninstalling urllib3-2.2.3:
2024-12-18T00:29:42.6843544Z       Successfully uninstalled urllib3-2.2.3
2024-12-18T00:29:42.8015577Z   Attempting uninstall: charset-normalizer
2024-12-18T00:29:42.8043333Z     Found existing installation: charset-normalizer 3.3.2
2024-12-18T00:29:42.8077449Z     Uninstalling charset-normalizer-3.3.2:
2024-12-18T00:29:42.8096335Z       Successfully uninstalled charset-normalizer-3.3.2
2024-12-18T00:29:42.8370680Z   Attempting uninstall: requests
2024-12-18T00:29:42.8395556Z     Found existing installation: requests 2.32.3
2024-12-18T00:29:42.8426102Z     Uninstalling requests-2.32.3:
2024-12-18T00:29:42.8431677Z       Successfully uninstalled requests-2.32.3
2024-12-18T00:29:42.8815640Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
2024-12-18T00:29:42.8816605Z conda 24.11.1 requires requests<3,>=2.28.0, but you have requests 2.27.1 which is incompatible.
2024-12-18T00:29:42.8817306Z Successfully installed charset-normalizer-2.0.12 pyyaml-6.0.1 requests-2.27.1 urllib3-1.26.20
2024-12-18T00:29:43.3168924Z Command completed after 1 attempt(s).
2024-12-18T00:29:43.3252802Z ##[group]Run set -x
2024-12-18T00:29:43.3253071Z set -x
2024-12-18T00:29:43.3253306Z 
2024-12-18T00:29:43.3253680Z # Use relative path here as this could be checked out anywhere, not necessarily
2024-12-18T00:29:43.3254150Z # in runner workspace
2024-12-18T00:29:43.3254533Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py"
2024-12-18T00:29:43.3264742Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:43.3265122Z env:
2024-12-18T00:29:43.3265344Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:43.3265684Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:43.3266042Z ##[endgroup]
2024-12-18T00:29:43.3295047Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py
2024-12-18T00:29:43.3542314Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}"
2024-12-18T00:29:43.3542715Z echo "Workflow: ${GITHUB_WORKFLOW}"
2024-12-18T00:29:43.3543065Z echo "Job name: ${JOB_NAME}"
2024-12-18T00:29:43.3543364Z 
2024-12-18T00:29:43.3543736Z # Use relative path here as this could be checked out anywhere, not necessarily
2024-12-18T00:29:43.3544213Z # in runner workspace
2024-12-18T00:29:43.3544633Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \
2024-12-18T00:29:43.3545105Z   --workflow "${GITHUB_WORKFLOW}" \
2024-12-18T00:29:43.3545444Z   --job-name "${JOB_NAME}" \
2024-12-18T00:29:43.3547258Z   --test-matrix "{"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \
2024-12-18T00:29:43.3549125Z   --selected-test-configs "" \
2024-12-18T00:29:43.3549516Z   --pr-number "${PR_NUMBER}" \
2024-12-18T00:29:43.3549835Z   --tag "${TAG}" \
2024-12-18T00:29:43.3550125Z   --event-name "${EVENT_NAME}" \
2024-12-18T00:29:43.3550452Z   --schedule "${SCHEDULE}" \
2024-12-18T00:29:43.3550764Z   --branch "${HEAD_BRANCH}"
2024-12-18T00:29:43.3558680Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:43.3559054Z env:
2024-12-18T00:29:43.3559265Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:43.3559595Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:43.3560119Z   GITHUB_TOKEN: ***
2024-12-18T00:29:43.3560772Z   JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:43.3561387Z   PR_NUMBER: 
2024-12-18T00:29:43.3561766Z   TAG: 
2024-12-18T00:29:43.3561976Z   EVENT_NAME: push
2024-12-18T00:29:43.3562213Z   SCHEDULE: 
2024-12-18T00:29:43.3562438Z   HEAD_BRANCH: 
2024-12-18T00:29:43.3562666Z ##[endgroup]
2024-12-18T00:29:43.3600774Z Workflow: inductor
2024-12-18T00:29:43.3601358Z Job name: cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:43.6288862Z ##[group]Run echo "Filtered matrix:"
2024-12-18T00:29:43.6289215Z echo "Filtered matrix:"
2024-12-18T00:29:43.6291035Z echo "{"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}"
2024-12-18T00:29:43.6292797Z 
2024-12-18T00:29:43.6293007Z echo
2024-12-18T00:29:43.6293291Z echo "Is the current job unstable? False"
2024-12-18T00:29:43.6293627Z 
2024-12-18T00:29:43.6293840Z echo
2024-12-18T00:29:43.6294102Z echo "Is keep-going label set? False"
2024-12-18T00:29:43.6294418Z 
2024-12-18T00:29:43.6294626Z echo
2024-12-18T00:29:43.6294867Z echo "Renabled issues? "
2024-12-18T00:29:43.6303570Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:43.6303943Z env:
2024-12-18T00:29:43.6304158Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:43.6304491Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:43.6304844Z ##[endgroup]
2024-12-18T00:29:43.6332785Z Filtered matrix:
2024-12-18T00:29:43.6334576Z {include: [{config: inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}]}
2024-12-18T00:29:43.6336513Z 
2024-12-18T00:29:43.6336665Z Is the current job unstable? False
2024-12-18T00:29:43.6336870Z 
2024-12-18T00:29:43.6336988Z Is keep-going label set? False
2024-12-18T00:29:43.6337173Z 
2024-12-18T00:29:43.6337272Z Renabled issues? 
2024-12-18T00:29:43.6380946Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2024-12-18T00:29:43.6381455Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2024-12-18T00:29:43.6389646Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T00:29:43.6390015Z env:
2024-12-18T00:29:43.6390232Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:43.6390576Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:43.6390930Z   JOB_TIMEOUT: 240
2024-12-18T00:29:43.6391159Z ##[endgroup]
2024-12-18T00:29:43.6481231Z ##[group]Run set -x
2024-12-18T00:29:43.6481543Z set -x
2024-12-18T00:29:43.6481769Z 
2024-12-18T00:29:43.6482020Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2024-12-18T00:29:43.6482408Z   TEST_COMMAND=.ci/pytorch/multigpu-test.sh
2024-12-18T00:29:43.6482796Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2024-12-18T00:29:43.6483160Z   TEST_COMMAND=.ci/onnx/test.sh
2024-12-18T00:29:43.6483458Z else
2024-12-18T00:29:43.6483711Z   TEST_COMMAND=.ci/pytorch/test.sh
2024-12-18T00:29:43.6484014Z fi
2024-12-18T00:29:43.6484218Z 
2024-12-18T00:29:43.6484559Z # detached container should get cleaned up by teardown_ec2_linux
2024-12-18T00:29:43.6485076Z # TODO: Stop building test binaries as part of the build phase
2024-12-18T00:29:43.6485719Z # Used for GPU_FLAG since that doesn't play nice
2024-12-18T00:29:43.6486124Z # shellcheck disable=SC2086,SC2090
2024-12-18T00:29:43.6486459Z container_name=$(docker run \
2024-12-18T00:29:43.6486763Z   ${GPU_FLAG:-} \
2024-12-18T00:29:43.6487064Z   ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \
2024-12-18T00:29:43.6487403Z   -e BUILD_ENVIRONMENT \
2024-12-18T00:29:43.6487693Z   -e PR_NUMBER \
2024-12-18T00:29:43.6487965Z   -e GITHUB_ACTIONS \
2024-12-18T00:29:43.6488254Z   -e GITHUB_REPOSITORY \
2024-12-18T00:29:43.6488554Z   -e GITHUB_WORKFLOW \
2024-12-18T00:29:43.6488840Z   -e GITHUB_JOB \
2024-12-18T00:29:43.6489104Z   -e GITHUB_RUN_ID \
2024-12-18T00:29:43.6489381Z   -e GITHUB_RUN_NUMBER \
2024-12-18T00:29:43.6489679Z   -e GITHUB_RUN_ATTEMPT \
2024-12-18T00:29:43.6490019Z   -e JOB_ID \
2024-12-18T00:29:43.6490280Z   -e JOB_NAME \
2024-12-18T00:29:43.6490536Z   -e BASE_SHA \
2024-12-18T00:29:43.6490794Z   -e BRANCH \
2024-12-18T00:29:43.6491047Z   -e SHA1 \
2024-12-18T00:29:43.6491302Z   -e AWS_DEFAULT_REGION \
2024-12-18T00:29:43.6491601Z   -e IN_WHEEL_TEST \
2024-12-18T00:29:43.6491870Z   -e SHARD_NUMBER \
2024-12-18T00:29:43.6492144Z   -e TEST_CONFIG \
2024-12-18T00:29:43.6492425Z   -e NUM_TEST_SHARDS \
2024-12-18T00:29:43.6492715Z   -e REENABLED_ISSUES \
2024-12-18T00:29:43.6493016Z   -e CONTINUE_THROUGH_ERROR \
2024-12-18T00:29:43.6493334Z   -e VERBOSE_TEST_LOGS \
2024-12-18T00:29:43.6493626Z   -e TEST_SHOWLOCALS \
2024-12-18T00:29:43.6493910Z   -e NO_TEST_TIMEOUT \
2024-12-18T00:29:43.6494186Z   -e NO_TD \
2024-12-18T00:29:43.6494441Z   -e TD_DISTRIBUTED \
2024-12-18T00:29:43.6494721Z   -e PR_LABELS \
2024-12-18T00:29:43.6495016Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2024-12-18T00:29:43.6495344Z   -e SCCACHE_BUCKET \
2024-12-18T00:29:43.6495633Z   -e SCCACHE_REGION \
2024-12-18T00:29:43.6495921Z   -e SCCACHE_S3_KEY_PREFIX \
2024-12-18T00:29:43.6496220Z   -e XLA_CUDA \
2024-12-18T00:29:43.6496502Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2024-12-18T00:29:43.6496858Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2024-12-18T00:29:43.6497222Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2024-12-18T00:29:43.6497585Z   -e SKIP_SCCACHE_INITIALIZATION=1 \
2024-12-18T00:29:43.6497926Z   -e HUGGING_FACE_HUB_TOKEN \
2024-12-18T00:29:43.6498248Z   -e SCRIBE_GRAPHQL_ACCESS_TOKEN \
2024-12-18T00:29:43.6498569Z   -e DASHBOARD_TAG \
2024-12-18T00:29:43.6498847Z   -e IS_A100_RUNNER \
2024-12-18T00:29:43.6499135Z   -e ARTIFACTS_FILE_SUFFIX \
2024-12-18T00:29:43.6499498Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2024-12-18T00:29:43.6499929Z   --security-opt seccomp=unconfined \
2024-12-18T00:29:43.6500415Z   --cap-add=SYS_PTRACE \
2024-12-18T00:29:43.6500708Z   --ipc=host \
2024-12-18T00:29:43.6500975Z   --shm-size="${SHM_SIZE}" \
2024-12-18T00:29:43.6501266Z   --tty \
2024-12-18T00:29:43.6501507Z   --detach \
2024-12-18T00:29:43.6501766Z   --name="${container_name}" \
2024-12-18T00:29:43.6502074Z   --user jenkins \
2024-12-18T00:29:43.6502418Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2024-12-18T00:29:43.6502811Z   -w /var/lib/jenkins/workspace \
2024-12-18T00:29:43.6503124Z   "${DOCKER_IMAGE}"
2024-12-18T00:29:43.6503378Z )
2024-12-18T00:29:43.6503667Z # Propagate download.pytorch.org IP to container
2024-12-18T00:29:43.6504311Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts"
2024-12-18T00:29:43.6505092Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}"
2024-12-18T00:29:43.6505773Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"
2024-12-18T00:29:43.6513960Z shell: /usr/bin/bash -e {0}
2024-12-18T00:29:43.6514233Z env:
2024-12-18T00:29:43.6514449Z   GIT_DEFAULT_BRANCH: main
2024-12-18T00:29:43.6514783Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:29:43.6515224Z   BUILD_ENVIRONMENT: linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-12-18T00:29:43.6515601Z   PR_NUMBER: 
2024-12-18T00:29:43.6515851Z   GITHUB_REPOSITORY: pytorch/pytorch
2024-12-18T00:29:43.6516161Z   GITHUB_WORKFLOW: inductor
2024-12-18T00:29:43.6516431Z   GITHUB_JOB: test
2024-12-18T00:29:43.6516676Z   GITHUB_RUN_ID: 12383255690
2024-12-18T00:29:43.6516950Z   GITHUB_RUN_NUMBER: 109429
2024-12-18T00:29:43.6517219Z   GITHUB_RUN_ATTEMPT: 1
2024-12-18T00:29:43.6517466Z   JOB_ID: 34567153568
2024-12-18T00:29:43.6517940Z   JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:29:43.6518463Z   BRANCH: release/2.6
2024-12-18T00:29:43.6518748Z   SHA1: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:43.6519124Z   BASE_SHA: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:29:43.6519473Z   TEST_CONFIG: inductor_torchbench
2024-12-18T00:29:43.6519766Z   SHARD_NUMBER: 2
2024-12-18T00:29:43.6520001Z   NUM_TEST_SHARDS: 2
2024-12-18T00:29:43.6520249Z   REENABLED_ISSUES: 
2024-12-18T00:29:43.6520505Z   CONTINUE_THROUGH_ERROR: False
2024-12-18T00:29:43.6520786Z   VERBOSE_TEST_LOGS: False
2024-12-18T00:29:43.6521058Z   TEST_SHOWLOCALS: False
2024-12-18T00:29:43.6521387Z   NO_TEST_TIMEOUT: False
2024-12-18T00:29:43.6521640Z   NO_TD: False
2024-12-18T00:29:43.6521871Z   TD_DISTRIBUTED: False
2024-12-18T00:29:43.6522177Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2024-12-18T00:29:43.6522539Z   SCCACHE_REGION: us-east-1
2024-12-18T00:29:43.6522826Z   SCCACHE_S3_KEY_PREFIX: inductor
2024-12-18T00:29:43.6523114Z   SHM_SIZE: 2g
2024-12-18T00:29:43.6523913Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:29:43.6524764Z   XLA_CUDA: 
2024-12-18T00:29:43.6525117Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2024-12-18T00:29:43.6525567Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2024-12-18T00:29:43.6525889Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2024-12-18T00:29:43.6526193Z   DASHBOARD_TAG: 
2024-12-18T00:29:43.6526598Z   HUGGING_FACE_HUB_TOKEN: ***
2024-12-18T00:29:43.6527011Z   SCRIBE_GRAPHQL_ACCESS_TOKEN: ***
2024-12-18T00:29:43.6527302Z   IS_A100_RUNNER: 0
2024-12-18T00:29:43.6527753Z   ARTIFACTS_FILE_SUFFIX: test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_34567153568
2024-12-18T00:29:43.6528273Z ##[endgroup]
2024-12-18T00:29:43.6557660Z + [[ inductor_torchbench == \m\u\l\t\i\g\p\u ]]
2024-12-18T00:29:43.6558185Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *onnx* ]]
2024-12-18T00:29:43.6558731Z + TEST_COMMAND=.ci/pytorch/test.sh
2024-12-18T00:29:43.6567184Z +++ nproc --ignore=2
2024-12-18T00:29:43.6591481Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_REGION -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e IS_A100_RUNNER -e ARTIFACTS_FILE_SUFFIX --env-file=/tmp/github_env_12383255690 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T00:29:48.0194649Z + container_name=273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T00:29:48.0198042Z + grep download.pytorch.org /etc/hosts
2024-12-18T00:29:48.0198998Z + docker exec -i 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128 sudo bash -c '/bin/cat >> /etc/hosts'
2024-12-18T00:29:48.1329211Z + echo DOCKER_CONTAINER_ID=273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T00:29:48.1333556Z ++ echo dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl
2024-12-18T00:29:48.1336738Z + docker exec -t 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128 sh -c 'python3 -m pip install dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh'
2024-12-18T00:29:48.5382550Z Processing ./dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl (from torch==2.6.0a0+git0cdf8b1)
2024-12-18T00:29:48.8687295Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.16.1)
2024-12-18T00:29:48.8690599Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (4.12.2)
2024-12-18T00:29:48.8693702Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (2.8.8)
2024-12-18T00:29:48.8696874Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.1.4)
2024-12-18T00:29:48.8700110Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (2024.10.0)
2024-12-18T00:29:48.8705294Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.13.1)
2024-12-18T00:29:48.8721885Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.3.0)
2024-12-18T00:29:48.8735315Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.3.0)
2024-12-18T00:29:48.8752806Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.22.4)
2024-12-18T00:29:48.9153763Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.0.2)
2024-12-18T00:29:49.5245498Z Installing collected packages: torch
2024-12-18T00:29:59.9623869Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
2024-12-18T00:29:59.9624897Z timm 1.0.9.dev0 requires torchvision, which is not installed.
2024-12-18T00:29:59.9625423Z Successfully installed torch-2.6.0a0+git0cdf8b1
2024-12-18T00:30:00.0482676Z + export TERM=vt100
2024-12-18T00:30:00.0483019Z + TERM=vt100
2024-12-18T00:30:00.0486861Z ++ dirname .ci/pytorch/test.sh
2024-12-18T00:30:00.0638327Z + source .ci/pytorch/common.sh
2024-12-18T00:30:00.0642464Z +++ dirname .ci/pytorch/common.sh
2024-12-18T00:30:00.0651315Z ++ source .ci/pytorch/common_utils.sh
2024-12-18T00:30:00.0653110Z +++ declare -f -t trap_add
2024-12-18T00:30:00.0658214Z ++ set -ex
2024-12-18T00:30:00.0658591Z ++ [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *rocm* ]]
2024-12-18T00:30:00.0658987Z ++ BUILD_TEST_LIBTORCH=0
2024-12-18T00:30:00.0659483Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *rocm* ]]
2024-12-18T00:30:00.0659876Z + [[ -d /var/lib/jenkins/workspace ]]
2024-12-18T00:30:00.0663679Z ++ stat -c %u /var/lib/jenkins/workspace
2024-12-18T00:30:00.0907010Z + WORKSPACE_ORIGINAL_OWNER_ID=1000
2024-12-18T00:30:00.0907356Z + trap_add cleanup_workspace EXIT
2024-12-18T00:30:00.0907672Z + trap_add_cmd=cleanup_workspace
2024-12-18T00:30:00.0907959Z + shift
2024-12-18T00:30:00.0908186Z + for trap_add_name in "$@"
2024-12-18T00:30:00.0915653Z +++ trap -p EXIT
2024-12-18T00:30:00.0919364Z ++ eval 'extract_trap_cmd '
2024-12-18T00:30:00.0919672Z +++ extract_trap_cmd
2024-12-18T00:30:00.0919951Z +++ printf '%s\n' ''
2024-12-18T00:30:00.0920219Z ++ printf '%s\n' cleanup_workspace
2024-12-18T00:30:00.0923084Z + trap -- '
2024-12-18T00:30:00.0923335Z cleanup_workspace' EXIT
2024-12-18T00:30:00.0923660Z + sudo chown -R jenkins /var/lib/jenkins/workspace
2024-12-18T00:30:00.8635645Z + git config --global --add safe.directory /var/lib/jenkins/workspace
2024-12-18T00:30:00.8658317Z + echo 'Environment variables:'
2024-12-18T00:30:00.8658653Z Environment variables:
2024-12-18T00:30:00.8658928Z + env
2024-12-18T00:30:00.8796582Z INSTALLED_DB=yes
2024-12-18T00:30:00.8797067Z NV_LIBCUBLAS_VERSION=12.4.5.8-1
2024-12-18T00:30:00.8797478Z NVIDIA_VISIBLE_DEVICES=all
2024-12-18T00:30:00.8797834Z NV_NVML_DEV_VERSION=12.4.127-1
2024-12-18T00:30:00.8798328Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-12-18T00:30:00.8798870Z CONTINUE_THROUGH_ERROR=False
2024-12-18T00:30:00.8799249Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.21.5-1+cuda12.4
2024-12-18T00:30:00.8799692Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.21.5-1
2024-12-18T00:30:00.8800096Z BUILD_ENVIRONMENT=linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-12-18T00:30:00.8800488Z HOSTNAME=273fc4bc868e
2024-12-18T00:30:00.8801053Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_76548210-8b39-4d82-93c9-06af64c968bb
2024-12-18T00:30:00.8801757Z GITHUB_ACTION=__self
2024-12-18T00:30:00.8802034Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2024-12-18T00:30:00.8806556Z NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536
2024-12-18T00:30:00.8810853Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-4=12.4.5.8-1
2024-12-18T00:30:00.8811222Z NV_NVTX_VERSION=12.4.127-1
2024-12-18T00:30:00.8811499Z GITHUB_RUN_NUMBER=109429
2024-12-18T00:30:00.8811773Z TEST_CONFIG=inductor_torchbench
2024-12-18T00:30:00.8812072Z GITHUB_REPOSITORY_OWNER_ID=21003710
2024-12-18T00:30:00.8812389Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2024-12-18T00:30:00.8812696Z IS_A100_RUNNER=0
2024-12-18T00:30:00.8813116Z NV_CUDA_CUDART_DEV_VERSION=12.4.127-1
2024-12-18T00:30:00.8813426Z NV_LIBCUSPARSE_VERSION=12.3.1.170-1
2024-12-18T00:30:00.8813939Z SCRIBE_GRAPHQL_ACCESS_TOKEN=***
2024-12-18T00:30:00.8814240Z NV_LIBNPP_VERSION=12.2.5.30-1
2024-12-18T00:30:00.8814572Z GITHUB_TRIGGERING_ACTOR=malfet
2024-12-18T00:30:00.8814903Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2024-12-18T00:30:00.8815255Z GITHUB_REF_TYPE=branch
2024-12-18T00:30:00.8815519Z TORCH_CUDA_ARCH_LIST=Maxwell
2024-12-18T00:30:00.8815784Z NCCL_VERSION=2.21.5-1
2024-12-18T00:30:00.8816079Z BASE_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:30:00.8816407Z XLA_CUDA=
2024-12-18T00:30:00.8816754Z HUGGING_FACE_HUB_TOKEN=***
2024-12-18T00:30:00.8818963Z ***
2024-12-18T00:30:00.8819202Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2024-12-18T00:30:00.8819504Z GITHUB_REPOSITORY_ID=65600975
2024-12-18T00:30:00.8819784Z GITHUB_ACTIONS=true
2024-12-18T00:30:00.8820044Z NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:30:00.8820388Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4=12.4.127-1
2024-12-18T00:30:00.8820758Z NV_LIBNPP_PACKAGE=libnpp-12-4=12.2.5.30-1
2024-12-18T00:30:00.8821110Z SHA1=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:30:00.8821460Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2024-12-18T00:30:00.8821824Z GITHUB_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:30:00.8822415Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/release/2.6
2024-12-18T00:30:00.8822919Z UCC_HOME=/usr
2024-12-18T00:30:00.8823164Z NV_LIBCUBLAS_DEV_VERSION=12.4.5.8-1
2024-12-18T00:30:00.8823471Z VERBOSE_TEST_LOGS=False
2024-12-18T00:30:00.8823731Z NVIDIA_PRODUCT_NAME=CUDA
2024-12-18T00:30:00.8824050Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-4
2024-12-18T00:30:00.8824410Z GITHUB_REF=refs/heads/release/2.6
2024-12-18T00:30:00.8824713Z NV_CUDA_CUDART_VERSION=12.4.127-1
2024-12-18T00:30:00.8824998Z SHARD_NUMBER=2
2024-12-18T00:30:00.8825237Z GITHUB_REF_PROTECTED=true
2024-12-18T00:30:00.8825510Z HOME=/var/lib/jenkins
2024-12-18T00:30:00.8825799Z GITHUB_API_URL=https://api.github.com
2024-12-18T00:30:00.8826132Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2024-12-18T00:30:00.8826483Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb
2024-12-18T00:30:00.8826843Z SCCACHE_S3_KEY_PREFIX=inductor
2024-12-18T00:30:00.8827127Z CUDA_VERSION=12.4.1
2024-12-18T00:30:00.8827408Z NV_LIBCUBLAS_PACKAGE=libcublas-12-4=12.4.5.8-1
2024-12-18T00:30:00.8827747Z NUM_TEST_SHARDS=2
2024-12-18T00:30:00.8827986Z UCX_HOME=/usr
2024-12-18T00:30:00.8828326Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-4=12.4.1-1
2024-12-18T00:30:00.8829070Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_76548210-8b39-4d82-93c9-06af64c968bb
2024-12-18T00:30:00.8829916Z JOB_NAME=cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:30:00.8830723Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_76548210-8b39-4d82-93c9-06af64c968bb
2024-12-18T00:30:00.8831513Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2024-12-18T00:30:00.8832136Z GITHUB_EVENT_NAME=push
2024-12-18T00:30:00.8832391Z DASHBOARD_TAG=
2024-12-18T00:30:00.8832630Z GITHUB_RUN_ID=12383255690
2024-12-18T00:30:00.8832939Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-4=12.2.5.30-1
2024-12-18T00:30:00.8833310Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-4
2024-12-18T00:30:00.8833986Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_76548210-8b39-4d82-93c9-06af64c968bb
2024-12-18T00:30:00.8834639Z GITHUB_ACTOR=malfet
2024-12-18T00:30:00.8834891Z NV_LIBNPP_DEV_VERSION=12.2.5.30-1
2024-12-18T00:30:00.8835179Z PR_NUMBER=
2024-12-18T00:30:00.8835404Z GITHUB_RUN_ATTEMPT=1
2024-12-18T00:30:00.8835667Z ANACONDA_PYTHON_VERSION=3.10
2024-12-18T00:30:00.8835998Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2024-12-18T00:30:00.8836343Z TERM=vt100
2024-12-18T00:30:00.8836621Z NV_LIBCUSPARSE_DEV_VERSION=12.3.1.170-1
2024-12-18T00:30:00.8837135Z INSTALLED_VISION=yes
2024-12-18T00:30:00.8837387Z BRANCH=release/2.6
2024-12-18T00:30:00.8837635Z SCCACHE_REGION=us-east-1
2024-12-18T00:30:00.8837912Z OPENSSL_ROOT_DIR=/opt/openssl
2024-12-18T00:30:00.8838215Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2024-12-18T00:30:00.8838535Z CUDA_PATH=/usr/local/cuda
2024-12-18T00:30:00.8839054Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2024-12-18T00:30:00.8839622Z GITHUB_SERVER_URL=https://github.com
2024-12-18T00:30:00.8839983Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b
2024-12-18T00:30:00.8840339Z REENABLED_ISSUES=
2024-12-18T00:30:00.8840573Z SHLVL=1
2024-12-18T00:30:00.8840782Z MAX_JOBS=14
2024-12-18T00:30:00.8841007Z NV_CUDA_LIB_VERSION=12.4.1-1
2024-12-18T00:30:00.8841373Z NVARCH=x86_64
2024-12-18T00:30:00.8841605Z GITHUB_ACTOR_ID=2453524
2024-12-18T00:30:00.8841946Z GITHUB_WORKFLOW_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:30:00.8842350Z GITHUB_REF_NAME=release/2.6
2024-12-18T00:30:00.8842641Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-4
2024-12-18T00:30:00.8843091Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2024-12-18T00:30:00.8843516Z GITHUB_JOB=test
2024-12-18T00:30:00.8843803Z NV_LIBNCCL_PACKAGE=libnccl2=2.21.5-1+cuda12.4
2024-12-18T00:30:00.8844252Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2024-12-18T00:30:00.8844629Z NO_TEST_TIMEOUT=False
2024-12-18T00:30:00.8844887Z TD_DISTRIBUTED=False
2024-12-18T00:30:00.8845160Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.1-1
2024-12-18T00:30:00.8845495Z GITHUB_REPOSITORY=pytorch/pytorch
2024-12-18T00:30:00.8845803Z NV_NVPROF_VERSION=12.4.127-1
2024-12-18T00:30:00.8846084Z GITHUB_RETENTION_DAYS=90
2024-12-18T00:30:00.8846369Z OPENSSL_DIR=/opt/openssl
2024-12-18T00:30:00.8846643Z GITHUB_ACTION_REPOSITORY=
2024-12-18T00:30:00.8847429Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-12-18T00:30:00.8848350Z GITHUB_BASE_REF=
2024-12-18T00:30:00.8848809Z ARTIFACTS_FILE_SUFFIX=test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_34567153568
2024-12-18T00:30:00.8849348Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2024-12-18T00:30:00.8849632Z CI=true
2024-12-18T00:30:00.8849867Z NV_LIBNCCL_PACKAGE_VERSION=2.21.5-1
2024-12-18T00:30:00.8850181Z GITHUB_REPOSITORY_OWNER=pytorch
2024-12-18T00:30:00.8850460Z JOB_ID=34567153568
2024-12-18T00:30:00.8850704Z INSTALLED_PROTOBUF=yes
2024-12-18T00:30:00.8850964Z GITHUB_HEAD_REF=
2024-12-18T00:30:00.8851204Z GITHUB_ACTION_REF=
2024-12-18T00:30:00.8851504Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2024-12-18T00:30:00.8851853Z TEST_SHOWLOCALS=False
2024-12-18T00:30:00.8852112Z GITHUB_WORKFLOW=inductor
2024-12-18T00:30:00.8852392Z DEBIAN_FRONTEND=noninteractive
2024-12-18T00:30:00.8853002Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_76548210-8b39-4d82-93c9-06af64c968bb
2024-12-18T00:30:00.8853621Z NO_TD=False
2024-12-18T00:30:00.8853852Z SKIP_SCCACHE_INITIALIZATION=1
2024-12-18T00:30:00.8854130Z _=/usr/bin/env
2024-12-18T00:30:00.8854561Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2024-12-18T00:30:00.9030623Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch
2024-12-18T00:30:00.9031353Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2024-12-18T00:30:00.9032036Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib
2024-12-18T00:30:00.9032613Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test
2024-12-18T00:30:00.9033050Z + BUILD_DIR=build
2024-12-18T00:30:00.9033301Z + BUILD_RENAMED_DIR=build_renamed
2024-12-18T00:30:00.9033594Z + BUILD_BIN_DIR=build/bin
2024-12-18T00:30:00.9033850Z + SHARD_NUMBER=2
2024-12-18T00:30:00.9034095Z + NUM_TEST_SHARDS=2
2024-12-18T00:30:00.9034366Z + export VALGRIND=ON
2024-12-18T00:30:00.9034609Z + VALGRIND=ON
2024-12-18T00:30:00.9035171Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *clang9* ]]
2024-12-18T00:30:00.9035612Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *xpu* ]]
2024-12-18T00:30:00.9036055Z + [[ 0 == \1 ]]
2024-12-18T00:30:00.9036390Z + [[ False == \1 ]]
2024-12-18T00:30:00.9036750Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *bazel* ]]
2024-12-18T00:30:00.9037948Z ++ realpath build/custom_test_artifacts
2024-12-18T00:30:00.9255523Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2024-12-18T00:30:00.9256017Z + [[ -n '' ]]
2024-12-18T00:30:00.9256252Z + echo 'Environment variables'
2024-12-18T00:30:00.9256535Z Environment variables
2024-12-18T00:30:00.9256776Z + env
2024-12-18T00:30:00.9265220Z INSTALLED_DB=yes
2024-12-18T00:30:00.9265611Z NV_LIBCUBLAS_VERSION=12.4.5.8-1
2024-12-18T00:30:00.9266014Z NVIDIA_VISIBLE_DEVICES=all
2024-12-18T00:30:00.9266385Z NV_NVML_DEV_VERSION=12.4.127-1
2024-12-18T00:30:00.9267149Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-12-18T00:30:00.9267691Z CONTINUE_THROUGH_ERROR=False
2024-12-18T00:30:00.9268017Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.21.5-1+cuda12.4
2024-12-18T00:30:00.9268384Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.21.5-1
2024-12-18T00:30:00.9268787Z BUILD_ENVIRONMENT=linux-focal-cuda12.4-py3.10-gcc9-sm86
2024-12-18T00:30:00.9269312Z HOSTNAME=273fc4bc868e
2024-12-18T00:30:00.9270083Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_76548210-8b39-4d82-93c9-06af64c968bb
2024-12-18T00:30:00.9270740Z GITHUB_ACTION=__self
2024-12-18T00:30:00.9271009Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2024-12-18T00:30:00.9275005Z NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536
2024-12-18T00:30:00.9279231Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-4=12.4.5.8-1
2024-12-18T00:30:00.9279586Z NV_NVTX_VERSION=12.4.127-1
2024-12-18T00:30:00.9279848Z GITHUB_RUN_NUMBER=109429
2024-12-18T00:30:00.9280121Z TEST_CONFIG=inductor_torchbench
2024-12-18T00:30:00.9280422Z GITHUB_REPOSITORY_OWNER_ID=21003710
2024-12-18T00:30:00.9280743Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2024-12-18T00:30:00.9281360Z IS_A100_RUNNER=0
2024-12-18T00:30:00.9281626Z NV_CUDA_CUDART_DEV_VERSION=12.4.127-1
2024-12-18T00:30:00.9281936Z NV_LIBCUSPARSE_VERSION=12.3.1.170-1
2024-12-18T00:30:00.9282451Z SCRIBE_GRAPHQL_ACCESS_TOKEN=***
2024-12-18T00:30:00.9282748Z NV_LIBNPP_VERSION=12.2.5.30-1
2024-12-18T00:30:00.9283034Z GITHUB_TRIGGERING_ACTOR=malfet
2024-12-18T00:30:00.9283360Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2024-12-18T00:30:00.9283757Z GITHUB_REF_TYPE=branch
2024-12-18T00:30:00.9284018Z TORCH_CUDA_ARCH_LIST=Maxwell
2024-12-18T00:30:00.9284288Z NCCL_VERSION=2.21.5-1
2024-12-18T00:30:00.9284583Z BASE_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:30:00.9284909Z XLA_CUDA=
2024-12-18T00:30:00.9285228Z HUGGING_FACE_HUB_TOKEN=***
2024-12-18T00:30:00.9285733Z ***
2024-12-18T00:30:00.9285954Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2024-12-18T00:30:00.9286433Z GITHUB_REPOSITORY_ID=65600975
2024-12-18T00:30:00.9286709Z GITHUB_ACTIONS=true
2024-12-18T00:30:00.9286958Z NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T00:30:00.9287292Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4=12.4.127-1
2024-12-18T00:30:00.9287654Z NV_LIBNPP_PACKAGE=libnpp-12-4=12.2.5.30-1
2024-12-18T00:30:00.9288004Z SHA1=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:30:00.9288352Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2024-12-18T00:30:00.9288699Z GITHUB_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:30:00.9289241Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/release/2.6
2024-12-18T00:30:00.9289740Z UCC_HOME=/usr
2024-12-18T00:30:00.9289981Z NV_LIBCUBLAS_DEV_VERSION=12.4.5.8-1
2024-12-18T00:30:00.9290276Z VERBOSE_TEST_LOGS=False
2024-12-18T00:30:00.9290536Z NVIDIA_PRODUCT_NAME=CUDA
2024-12-18T00:30:00.9290882Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-4
2024-12-18T00:30:00.9291226Z GITHUB_REF=refs/heads/release/2.6
2024-12-18T00:30:00.9291531Z NV_CUDA_CUDART_VERSION=12.4.127-1
2024-12-18T00:30:00.9291818Z SHARD_NUMBER=2
2024-12-18T00:30:00.9292055Z GITHUB_REF_PROTECTED=true
2024-12-18T00:30:00.9292319Z HOME=/var/lib/jenkins
2024-12-18T00:30:00.9292596Z GITHUB_API_URL=https://api.github.com
2024-12-18T00:30:00.9292924Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2024-12-18T00:30:00.9293272Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb
2024-12-18T00:30:00.9293628Z SCCACHE_S3_KEY_PREFIX=inductor
2024-12-18T00:30:00.9293900Z CUDA_VERSION=12.4.1
2024-12-18T00:30:00.9294175Z NV_LIBCUBLAS_PACKAGE=libcublas-12-4=12.4.5.8-1
2024-12-18T00:30:00.9294498Z NUM_TEST_SHARDS=2
2024-12-18T00:30:00.9294725Z UCX_HOME=/usr
2024-12-18T00:30:00.9295062Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-4=12.4.1-1
2024-12-18T00:30:00.9295775Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_76548210-8b39-4d82-93c9-06af64c968bb
2024-12-18T00:30:00.9296597Z JOB_NAME=cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T00:30:00.9297405Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_76548210-8b39-4d82-93c9-06af64c968bb
2024-12-18T00:30:00.9298175Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2024-12-18T00:30:00.9298659Z GITHUB_EVENT_NAME=push
2024-12-18T00:30:00.9298910Z DASHBOARD_TAG=
2024-12-18T00:30:00.9299138Z GITHUB_RUN_ID=12383255690
2024-12-18T00:30:00.9299446Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-4=12.2.5.30-1
2024-12-18T00:30:00.9299811Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-4
2024-12-18T00:30:00.9300470Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_76548210-8b39-4d82-93c9-06af64c968bb
2024-12-18T00:30:00.9301108Z GITHUB_ACTOR=malfet
2024-12-18T00:30:00.9301352Z NV_LIBNPP_DEV_VERSION=12.2.5.30-1
2024-12-18T00:30:00.9301635Z PR_NUMBER=
2024-12-18T00:30:00.9301852Z GITHUB_RUN_ATTEMPT=1
2024-12-18T00:30:00.9302095Z VALGRIND=ON
2024-12-18T00:30:00.9302333Z ANACONDA_PYTHON_VERSION=3.10
2024-12-18T00:30:00.9302657Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2024-12-18T00:30:00.9303103Z TERM=vt100
2024-12-18T00:30:00.9303343Z NV_LIBCUSPARSE_DEV_VERSION=12.3.1.170-1
2024-12-18T00:30:00.9303657Z INSTALLED_VISION=yes
2024-12-18T00:30:00.9303903Z BRANCH=release/2.6
2024-12-18T00:30:00.9304141Z SCCACHE_REGION=us-east-1
2024-12-18T00:30:00.9304413Z OPENSSL_ROOT_DIR=/opt/openssl
2024-12-18T00:30:00.9304711Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2024-12-18T00:30:00.9315813Z CUDA_PATH=/usr/local/cuda
2024-12-18T00:30:00.9316350Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2024-12-18T00:30:00.9316928Z GITHUB_SERVER_URL=https://github.com
2024-12-18T00:30:00.9317295Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b
2024-12-18T00:30:00.9317645Z REENABLED_ISSUES=
2024-12-18T00:30:00.9317879Z SHLVL=1
2024-12-18T00:30:00.9318079Z MAX_JOBS=14
2024-12-18T00:30:00.9318304Z NV_CUDA_LIB_VERSION=12.4.1-1
2024-12-18T00:30:00.9318695Z NVARCH=x86_64
2024-12-18T00:30:00.9318930Z GITHUB_ACTOR_ID=2453524
2024-12-18T00:30:00.9319263Z GITHUB_WORKFLOW_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T00:30:00.9319653Z GITHUB_REF_NAME=release/2.6
2024-12-18T00:30:00.9319950Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-4
2024-12-18T00:30:00.9320395Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2024-12-18T00:30:00.9320821Z GITHUB_JOB=test
2024-12-18T00:30:00.9321083Z NV_LIBNCCL_PACKAGE=libnccl2=2.21.5-1+cuda12.4
2024-12-18T00:30:00.9321585Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2024-12-18T00:30:00.9321969Z NO_TEST_TIMEOUT=False
2024-12-18T00:30:00.9322228Z TD_DISTRIBUTED=False
2024-12-18T00:30:00.9322503Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.1-1
2024-12-18T00:30:00.9322828Z GITHUB_REPOSITORY=pytorch/pytorch
2024-12-18T00:30:00.9323135Z NV_NVPROF_VERSION=12.4.127-1
2024-12-18T00:30:00.9323419Z GITHUB_RETENTION_DAYS=90
2024-12-18T00:30:00.9323686Z OPENSSL_DIR=/opt/openssl
2024-12-18T00:30:00.9323963Z GITHUB_ACTION_REPOSITORY=
2024-12-18T00:30:00.9324811Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-12-18T00:30:00.9325693Z GITHUB_BASE_REF=
2024-12-18T00:30:00.9326200Z ARTIFACTS_FILE_SUFFIX=test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_34567153568
2024-12-18T00:30:00.9326797Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2024-12-18T00:30:00.9327091Z CI=true
2024-12-18T00:30:00.9327328Z NV_LIBNCCL_PACKAGE_VERSION=2.21.5-1
2024-12-18T00:30:00.9327656Z GITHUB_REPOSITORY_OWNER=pytorch
2024-12-18T00:30:00.9327956Z JOB_ID=34567153568
2024-12-18T00:30:00.9328207Z INSTALLED_PROTOBUF=yes
2024-12-18T00:30:00.9328470Z GITHUB_HEAD_REF=
2024-12-18T00:30:00.9328708Z GITHUB_ACTION_REF=
2024-12-18T00:30:00.9329021Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2024-12-18T00:30:00.9329408Z TEST_SHOWLOCALS=False
2024-12-18T00:30:00.9329669Z GITHUB_WORKFLOW=inductor
2024-12-18T00:30:00.9329951Z DEBIAN_FRONTEND=noninteractive
2024-12-18T00:30:00.9330551Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_76548210-8b39-4d82-93c9-06af64c968bb
2024-12-18T00:30:00.9331172Z NO_TD=False
2024-12-18T00:30:00.9331408Z SKIP_SCCACHE_INITIALIZATION=1
2024-12-18T00:30:00.9331688Z _=/usr/bin/env
2024-12-18T00:30:00.9331924Z + echo 'Testing pytorch'
2024-12-18T00:30:00.9332183Z Testing pytorch
2024-12-18T00:30:00.9332441Z + export LANG=C.UTF-8
2024-12-18T00:30:00.9332691Z + LANG=C.UTF-8
2024-12-18T00:30:01.0053346Z + PR_NUMBER=
2024-12-18T00:30:01.0053638Z + [[ inductor_torchbench == \d\e\f\a\u\l\t ]]
2024-12-18T00:30:01.0054073Z + [[ inductor_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]]
2024-12-18T00:30:01.0054434Z + [[ inductor_torchbench == \s\l\o\w ]]
2024-12-18T00:30:01.0054849Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *slow-gradcheck* ]]
2024-12-18T00:30:01.0055327Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *cuda* ]]
2024-12-18T00:30:01.0055752Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2024-12-18T00:30:01.0056235Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2024-12-18T00:30:01.0056835Z + [[ inductor_torchbench == *crossref* ]]
2024-12-18T00:30:01.0057223Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *rocm* ]]
2024-12-18T00:30:01.0057634Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *xpu* ]]
2024-12-18T00:30:01.0058066Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]]
2024-12-18T00:30:01.0058454Z + pip_install --user ninja==1.10.2
2024-12-18T00:30:01.0058831Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T00:30:01.0059364Z + python3 -m pip install --progress-bar off --user ninja==1.10.2
2024-12-18T00:30:02.0767433Z Collecting ninja==1.10.2
2024-12-18T00:30:02.1178567Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB)
2024-12-18T00:30:02.1774625Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2024-12-18T00:30:02.7976628Z Installing collected packages: ninja
2024-12-18T00:30:02.8055868Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2024-12-18T00:30:02.8056835Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2024-12-18T00:30:02.8515755Z Successfully installed ninja-1.10.2
2024-12-18T00:30:02.9308968Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-12-18T00:30:02.9310563Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-12-18T00:30:02.9311551Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *aarch64* ]]
2024-12-18T00:30:02.9311927Z + install_tlparse
2024-12-18T00:30:02.9312202Z + pip_install --user tlparse==0.3.25
2024-12-18T00:30:02.9312633Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T00:30:02.9313143Z + python3 -m pip install --progress-bar off --user tlparse==0.3.25
2024-12-18T00:30:03.3647895Z Collecting tlparse==0.3.25
2024-12-18T00:30:03.3830036Z   Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB)
2024-12-18T00:30:03.4386128Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB)
2024-12-18T00:30:04.0995009Z Installing collected packages: tlparse
2024-12-18T00:30:04.1346098Z Successfully installed tlparse-0.3.25
2024-12-18T00:30:04.2144148Z ++ python -m site --user-base
2024-12-18T00:30:04.2392855Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-12-18T00:30:04.2394835Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *asan* ]]
2024-12-18T00:30:04.2395315Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *-debug* ]]
2024-12-18T00:30:04.2395759Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]]
2024-12-18T00:30:04.2396369Z + echo 'We are not in debug mode: linux-focal-cuda12.4-py3.10-gcc9-sm86. Expect the assertion to pass'
2024-12-18T00:30:04.2397194Z We are not in debug mode: linux-focal-cuda12.4-py3.10-gcc9-sm86. Expect the assertion to pass
2024-12-18T00:30:04.2397773Z + cd test
2024-12-18T00:30:04.2398125Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)'
2024-12-18T00:30:06.8341831Z + [[ inductor_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2024-12-18T00:30:06.8342390Z + [[ inductor_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2024-12-18T00:30:06.8346988Z + DYNAMO_BENCHMARK_FLAGS=()
2024-12-18T00:30:06.8347430Z + [[ inductor_torchbench == *pr_time_benchmarks* ]]
2024-12-18T00:30:06.8347874Z + [[ inductor_torchbench == *dynamo_eager* ]]
2024-12-18T00:30:06.8348225Z + [[ inductor_torchbench == *aot_eager* ]]
2024-12-18T00:30:06.8348647Z + [[ inductor_torchbench == *aot_inductor* ]]
2024-12-18T00:30:06.8349329Z + [[ inductor_torchbench == *inductor* ]]
2024-12-18T00:30:06.8349669Z + [[ inductor_torchbench != *perf* ]]
2024-12-18T00:30:06.8350011Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor)
2024-12-18T00:30:06.8350335Z + [[ inductor_torchbench == *dynamic* ]]
2024-12-18T00:30:06.8350656Z + [[ inductor_torchbench == *cpu* ]]
2024-12-18T00:30:06.8350977Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda)
2024-12-18T00:30:06.8380272Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *libtorch* ]]
2024-12-18T00:30:06.8380868Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *-bazel-* ]]
2024-12-18T00:30:06.8383275Z + cd test
2024-12-18T00:30:06.8383606Z + python -c 'import torch; print(torch.__config__.show())'
2024-12-18T00:30:08.3528302Z PyTorch built with:
2024-12-18T00:30:08.3528697Z   - GCC 9.4
2024-12-18T00:30:08.3528992Z   - C++ Version: 201703
2024-12-18T00:30:08.3529629Z   - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2024-12-18T00:30:08.3530632Z   - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af)
2024-12-18T00:30:08.3531077Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2024-12-18T00:30:08.3531412Z   - LAPACK is enabled (usually provided by MKL)
2024-12-18T00:30:08.3531734Z   - NNPACK is enabled
2024-12-18T00:30:08.3531998Z   - CPU capability usage: AVX2
2024-12-18T00:30:08.3532279Z   - CUDA Runtime 12.4
2024-12-18T00:30:08.3532631Z   - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86
2024-12-18T00:30:08.3533022Z   - CuDNN 90.1
2024-12-18T00:30:08.3533247Z   - Magma 2.6.1
2024-12-18T00:30:08.3538151Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7, CUDA_VERSION=12.4, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.6.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 
2024-12-18T00:30:08.3543096Z 
2024-12-18T00:30:08.6780219Z + cd test
2024-12-18T00:30:08.6780695Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2024-12-18T00:30:10.0533995Z ATen/Parallel:
2024-12-18T00:30:10.0534827Z 	at::get_num_threads() : 8
2024-12-18T00:30:10.0535468Z 	at::get_num_interop_threads() : 16
2024-12-18T00:30:10.0535884Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2024-12-18T00:30:10.0536203Z 	omp_get_max_threads() : 8
2024-12-18T00:30:10.0536760Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2024-12-18T00:30:10.0537345Z 	mkl_get_max_threads() : 8
2024-12-18T00:30:10.0537739Z Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af)
2024-12-18T00:30:10.0538193Z std::thread::hardware_concurrency() : 16
2024-12-18T00:30:10.0538515Z Environment variables:
2024-12-18T00:30:10.0538791Z 	OMP_NUM_THREADS : [not set]
2024-12-18T00:30:10.0539075Z 	MKL_NUM_THREADS : [not set]
2024-12-18T00:30:10.0539364Z ATen parallel backend: OpenMP
2024-12-18T00:30:10.0539568Z 
2024-12-18T00:30:10.3364646Z + [[ inductor_torchbench == *numpy_2* ]]
2024-12-18T00:30:10.3365476Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *aarch64* ]]
2024-12-18T00:30:10.3365913Z + [[ inductor_torchbench == *backward* ]]
2024-12-18T00:30:10.3366237Z + [[ inductor_torchbench == *xla* ]]
2024-12-18T00:30:10.3366559Z + [[ inductor_torchbench == *executorch* ]]
2024-12-18T00:30:10.3367145Z + [[ inductor_torchbench == \j\i\t\_\l\e\g\a\c\y ]]
2024-12-18T00:30:10.3367560Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *libtorch* ]]
2024-12-18T00:30:10.3367956Z + [[ inductor_torchbench == distributed ]]
2024-12-18T00:30:10.3368319Z + [[ inductor_torchbench == *inductor_distributed* ]]
2024-12-18T00:30:10.3368702Z + [[ inductor_torchbench == *inductor-halide* ]]
2024-12-18T00:30:10.3369078Z + [[ inductor_torchbench == *inductor-triton-cpu* ]]
2024-12-18T00:30:10.3369486Z + [[ inductor_torchbench == *inductor-micro-benchmark* ]]
2024-12-18T00:30:10.3369875Z + [[ inductor_torchbench == *huggingface* ]]
2024-12-18T00:30:10.3370401Z + [[ inductor_torchbench == *timm* ]]
2024-12-18T00:30:10.3370721Z + [[ inductor_torchbench == *torchbench* ]]
2024-12-18T00:30:10.3371050Z + [[ inductor_torchbench == *cpu* ]]
2024-12-18T00:30:10.3371355Z + install_torchaudio cuda
2024-12-18T00:30:10.3371697Z + local commit
2024-12-18T00:30:10.3371934Z ++ get_pinned_commit audio
2024-12-18T00:30:10.3372217Z ++ cat .github/ci_commit_pins/audio.txt
2024-12-18T00:30:10.3404172Z + commit=332760d4b300f00a0d862e3cfe1495db3b1a14f9
2024-12-18T00:30:10.3404513Z + [[ cuda == \c\u\d\a ]]
2024-12-18T00:30:10.3404773Z + TORCH_CUDA_ARCH_LIST='8.0;8.6'
2024-12-18T00:30:10.3405368Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@332760d4b300f00a0d862e3cfe1495db3b1a14f9
2024-12-18T00:30:10.3406089Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T00:30:10.3406873Z + python3 -m pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@332760d4b300f00a0d862e3cfe1495db3b1a14f9
2024-12-18T00:30:10.6749823Z Collecting git+https://github.com/pytorch/audio.git@332760d4b300f00a0d862e3cfe1495db3b1a14f9
2024-12-18T00:30:10.6755491Z   Cloning https://github.com/pytorch/audio.git (to revision 332760d4b300f00a0d862e3cfe1495db3b1a14f9) to /tmp/pip-req-build-5ncgu324
2024-12-18T00:30:10.6786540Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-5ncgu324
2024-12-18T00:30:11.4905759Z   Running command git rev-parse -q --verify 'sha^332760d4b300f00a0d862e3cfe1495db3b1a14f9'
2024-12-18T00:30:11.4935165Z   Running command git fetch -q https://github.com/pytorch/audio.git 332760d4b300f00a0d862e3cfe1495db3b1a14f9
2024-12-18T00:30:11.9033957Z   Running command git checkout -q 332760d4b300f00a0d862e3cfe1495db3b1a14f9
2024-12-18T00:30:12.0598012Z   Resolved https://github.com/pytorch/audio.git to commit 332760d4b300f00a0d862e3cfe1495db3b1a14f9
2024-12-18T00:30:12.0598954Z   Running command git submodule update --init --recursive -q
2024-12-18T00:30:14.6176758Z   Preparing metadata (setup.py) ... [?25l- done
2024-12-18T00:30:14.6206106Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.5.0a0+332760d) (2.6.0a0+git0cdf8b1)
2024-12-18T00:30:14.6231874Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.5.0a0+332760d) (3.16.1)
2024-12-18T00:30:14.6235680Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.5.0a0+332760d) (4.12.2)
2024-12-18T00:30:14.6238290Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.5.0a0+332760d) (2.8.8)
2024-12-18T00:30:14.6241085Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.5.0a0+332760d) (3.1.4)
2024-12-18T00:30:14.6244190Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.5.0a0+332760d) (2024.10.0)
2024-12-18T00:30:14.6249360Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.5.0a0+332760d) (1.13.1)
2024-12-18T00:30:14.6263346Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchaudio==2.5.0a0+332760d) (1.3.0)
2024-12-18T00:30:14.6742803Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.5.0a0+332760d) (3.0.2)
2024-12-18T00:30:14.6805997Z Building wheels for collected packages: torchaudio
2024-12-18T00:32:28.5727168Z   Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done
2024-12-18T00:32:28.5763702Z [?25h  Created wheel for torchaudio: filename=torchaudio-2.5.0a0+332760d-cp310-cp310-linux_x86_64.whl size=2326942 sha256=7aeb6f7234e42c107e5010b86ead46ec948deaf4c92a614c9be137495f126b9a
2024-12-18T00:32:28.5766865Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/78/2e/5c/66e884f380e2436e82a6c71c80865fe12e30aca3d9350545dd
2024-12-18T00:32:28.5795370Z Successfully built torchaudio
2024-12-18T00:32:29.0987485Z Installing collected packages: torchaudio
2024-12-18T00:32:29.3177712Z Successfully installed torchaudio-2.5.0a0+332760d
2024-12-18T00:32:29.4996026Z + install_torchvision
2024-12-18T00:32:29.4996334Z + local orig_preload
2024-12-18T00:32:29.4996590Z + local commit
2024-12-18T00:32:29.5001343Z ++ get_pinned_commit vision
2024-12-18T00:32:29.5001663Z ++ cat .github/ci_commit_pins/vision.txt
2024-12-18T00:32:29.5022559Z + commit=d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:32:29.5022920Z + orig_preload=
2024-12-18T00:32:29.5023152Z + '[' -n '' ']'
2024-12-18T00:32:29.5023709Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:32:29.5024405Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T00:32:29.5025194Z + python3 -m pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:32:29.8816834Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:32:29.8822491Z   Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-1vk6y4vk
2024-12-18T00:32:29.8888200Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-1vk6y4vk
2024-12-18T00:32:31.5789559Z   Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f'
2024-12-18T00:32:31.5816773Z   Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:32:33.0054108Z   Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:32:33.3415845Z   Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:32:36.0447019Z   Preparing metadata (setup.py) ... [?25l- \ done
2024-12-18T00:32:36.0479917Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4)
2024-12-18T00:32:36.0483479Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.6.0a0+git0cdf8b1)
2024-12-18T00:32:36.0488286Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (11.0.0)
2024-12-18T00:32:36.0556244Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.16.1)
2024-12-18T00:32:36.0558894Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2)
2024-12-18T00:32:36.0562511Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8)
2024-12-18T00:32:36.0565610Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4)
2024-12-18T00:32:36.0568987Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.10.0)
2024-12-18T00:32:36.0574149Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.1)
2024-12-18T00:32:36.0588226Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0)
2024-12-18T00:32:36.1083244Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (3.0.2)
2024-12-18T00:32:36.1148797Z Building wheels for collected packages: torchvision
2024-12-18T00:33:48.4385410Z   Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done
2024-12-18T00:33:48.4419815Z [?25h  Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2028127 sha256=2725c0479eff810951f142f8bb83ee94589a68ae95bf3f27a998416d83f397f7
2024-12-18T00:33:48.4422297Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96
2024-12-18T00:33:48.4455624Z Successfully built torchvision
2024-12-18T00:33:48.9601606Z Installing collected packages: torchvision
2024-12-18T00:33:49.3752253Z Successfully installed torchvision-0.19.0a0+d23a6e1
2024-12-18T00:33:49.5219176Z + '[' -n '' ']'
2024-12-18T00:33:49.5219562Z + TORCH_CUDA_ARCH_LIST='8.0;8.6'
2024-12-18T00:33:49.5220038Z + pip_install git+https://github.com/pytorch/ao.git
2024-12-18T00:33:49.5220493Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T00:33:49.5221039Z + python3 -m pip install --progress-bar off git+https://github.com/pytorch/ao.git
2024-12-18T00:33:49.8598999Z Collecting git+https://github.com/pytorch/ao.git
2024-12-18T00:33:49.8603201Z   Cloning https://github.com/pytorch/ao.git to /tmp/pip-req-build-g0j_qlwm
2024-12-18T00:33:49.8633371Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/ao.git /tmp/pip-req-build-g0j_qlwm
2024-12-18T00:33:50.6432432Z   Resolved https://github.com/pytorch/ao.git to commit a5a53a2bd00f18401baaa00db4467f955704f71f
2024-12-18T00:33:53.1964776Z   Preparing metadata (setup.py) ... [?25l- done
2024-12-18T00:33:53.2003668Z [?25hBuilding wheels for collected packages: torchao
2024-12-18T00:35:16.5153119Z   Building wheel for torchao (setup.py) ... [?25l- \ | / - \ | / - \ done
2024-12-18T00:35:16.5192110Z [?25h  Created wheel for torchao: filename=torchao-0.8.0+gita5a53a2b-cp310-cp310-linux_x86_64.whl size=3081971 sha256=799f45e34f97113302eee0608529200e8e22da26f774a89e816e6118dd88a9a8
2024-12-18T00:35:16.5195873Z   Stored in directory: /tmp/pip-ephem-wheel-cache-3i59jr1p/wheels/c0/df/d8/b1e2f5f1ea3bf141268a83a667137647b234f04c636e1e7a5e
2024-12-18T00:35:16.5230170Z Successfully built torchao
2024-12-18T00:35:17.1061535Z Installing collected packages: torchao
2024-12-18T00:35:17.7048269Z Successfully installed torchao-0.8.0+gita5a53a2b
2024-12-18T00:35:17.8438960Z + id=1
2024-12-18T00:35:17.8439266Z + pip_install opencv-python==4.8.0.74
2024-12-18T00:35:17.8439705Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T00:35:17.8440193Z + python3 -m pip install --progress-bar off opencv-python==4.8.0.74
2024-12-18T00:35:18.3139548Z Collecting opencv-python==4.8.0.74
2024-12-18T00:35:18.3355621Z   Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB)
2024-12-18T00:35:18.3445344Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.22.4)
2024-12-18T00:35:18.3518049Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB)
2024-12-18T00:35:19.6798394Z Installing collected packages: opencv-python
2024-12-18T00:35:20.5477606Z Successfully installed opencv-python-4.8.0.74
2024-12-18T00:35:20.6481836Z + [[ inductor_torchbench == *inductor_torchbench_smoketest_perf* ]]
2024-12-18T00:35:20.6485157Z + [[ inductor_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]]
2024-12-18T00:35:20.6485638Z + [[ inductor_torchbench == *torchbench_gcp_smoketest* ]]
2024-12-18T00:35:20.6486010Z + checkout_install_torchbench
2024-12-18T00:35:20.6486288Z + local commit
2024-12-18T00:35:20.6508023Z ++ get_pinned_commit torchbench
2024-12-18T00:35:20.6508659Z ++ cat .github/ci_commit_pins/torchbench.txt
2024-12-18T00:35:20.6581526Z + commit=766a5e3a189384659fd35a68c3b17b88c761aaac
2024-12-18T00:35:20.6581962Z + git clone https://github.com/pytorch/benchmark torchbench
2024-12-18T00:35:20.6598319Z Cloning into 'torchbench'...
2024-12-18T00:35:20.8235791Z remote: Enumerating objects: 34540, done.
2024-12-18T00:35:20.8236299Z remote: Counting objects:   0% (1/1643)
2024-12-18T00:35:20.8236866Z remote: Counting objects:   1% (17/1643)
2024-12-18T00:35:20.8237355Z remote: Counting objects:   2% (33/1643)
2024-12-18T00:35:20.8237884Z remote: Counting objects:   3% (50/1643)
2024-12-18T00:35:20.8238442Z remote: Counting objects:   4% (66/1643)
2024-12-18T00:35:20.8238930Z remote: Counting objects:   5% (83/1643)
2024-12-18T00:35:20.8239390Z remote: Counting objects:   6% (99/1643)
2024-12-18T00:35:20.8239904Z remote: Counting objects:   7% (116/1643)
2024-12-18T00:35:20.8240456Z remote: Counting objects:   8% (132/1643)
2024-12-18T00:35:20.8240883Z remote: Counting objects:   9% (148/1643)
2024-12-18T00:35:20.8241483Z remote: Counting objects:  10% (165/1643)
2024-12-18T00:35:20.8241978Z remote: Counting objects:  11% (181/1643)
2024-12-18T00:35:20.8242516Z remote: Counting objects:  12% (198/1643)
2024-12-18T00:35:20.8242966Z remote: Counting objects:  13% (214/1643)
2024-12-18T00:35:20.8243443Z remote: Counting objects:  14% (231/1643)
2024-12-18T00:35:20.8243858Z remote: Counting objects:  15% (247/1643)
2024-12-18T00:35:20.8244373Z remote: Counting objects:  16% (263/1643)
2024-12-18T00:35:20.8244814Z remote: Counting objects:  17% (280/1643)
2024-12-18T00:35:20.8245292Z remote: Counting objects:  18% (296/1643)
2024-12-18T00:35:20.8245673Z remote: Counting objects:  19% (313/1643)
2024-12-18T00:35:20.8246072Z remote: Counting objects:  20% (329/1643)
2024-12-18T00:35:20.8246469Z remote: Counting objects:  21% (346/1643)
2024-12-18T00:35:20.8246862Z remote: Counting objects:  22% (362/1643)
2024-12-18T00:35:20.8247251Z remote: Counting objects:  23% (378/1643)
2024-12-18T00:35:20.8247635Z remote: Counting objects:  24% (395/1643)
2024-12-18T00:35:20.8248022Z remote: Counting objects:  25% (411/1643)
2024-12-18T00:35:20.8248409Z remote: Counting objects:  26% (428/1643)
2024-12-18T00:35:20.8248810Z remote: Counting objects:  27% (444/1643)
2024-12-18T00:35:20.8249367Z remote: Counting objects:  28% (461/1643)
2024-12-18T00:35:20.8249797Z remote: Counting objects:  29% (477/1643)
2024-12-18T00:35:20.8250188Z remote: Counting objects:  30% (493/1643)
2024-12-18T00:35:20.8250801Z remote: Counting objects:  31% (510/1643)
2024-12-18T00:35:20.8251203Z remote: Counting objects:  32% (526/1643)
2024-12-18T00:35:20.8251590Z remote: Counting objects:  33% (543/1643)
2024-12-18T00:35:20.8251973Z remote: Counting objects:  34% (559/1643)
2024-12-18T00:35:20.8252361Z remote: Counting objects:  35% (576/1643)
2024-12-18T00:35:20.8252755Z remote: Counting objects:  36% (592/1643)
2024-12-18T00:35:20.8253141Z remote: Counting objects:  37% (608/1643)
2024-12-18T00:35:20.8253528Z remote: Counting objects:  38% (625/1643)
2024-12-18T00:35:20.8253908Z remote: Counting objects:  39% (641/1643)
2024-12-18T00:35:20.8254294Z remote: Counting objects:  40% (658/1643)
2024-12-18T00:35:20.8254683Z remote: Counting objects:  41% (674/1643)
2024-12-18T00:35:20.8255069Z remote: Counting objects:  42% (691/1643)
2024-12-18T00:35:20.8255624Z remote: Counting objects:  43% (707/1643)
2024-12-18T00:35:20.8256026Z remote: Counting objects:  44% (723/1643)
2024-12-18T00:35:20.8256414Z remote: Counting objects:  45% (740/1643)
2024-12-18T00:35:20.8256798Z remote: Counting objects:  46% (756/1643)
2024-12-18T00:35:20.8257186Z remote: Counting objects:  47% (773/1643)
2024-12-18T00:35:20.8257574Z remote: Counting objects:  48% (789/1643)
2024-12-18T00:35:20.8257955Z remote: Counting objects:  49% (806/1643)
2024-12-18T00:35:20.8258346Z remote: Counting objects:  50% (822/1643)
2024-12-18T00:35:20.8258735Z remote: Counting objects:  51% (838/1643)
2024-12-18T00:35:20.8259129Z remote: Counting objects:  52% (855/1643)
2024-12-18T00:35:20.8259515Z remote: Counting objects:  53% (871/1643)
2024-12-18T00:35:20.8259896Z remote: Counting objects:  54% (888/1643)
2024-12-18T00:35:20.8260285Z remote: Counting objects:  55% (904/1643)
2024-12-18T00:35:20.8260678Z remote: Counting objects:  56% (921/1643)
2024-12-18T00:35:20.8261078Z remote: Counting objects:  57% (937/1643)
2024-12-18T00:35:20.8261470Z remote: Counting objects:  58% (953/1643)
2024-12-18T00:35:20.8261859Z remote: Counting objects:  59% (970/1643)
2024-12-18T00:35:20.8262240Z remote: Counting objects:  60% (986/1643)
2024-12-18T00:35:20.8262633Z remote: Counting objects:  61% (1003/1643)
2024-12-18T00:35:20.8263031Z remote: Counting objects:  62% (1019/1643)
2024-12-18T00:35:20.8263429Z remote: Counting objects:  63% (1036/1643)
2024-12-18T00:35:20.8263830Z remote: Counting objects:  64% (1052/1643)
2024-12-18T00:35:20.8264219Z remote: Counting objects:  65% (1068/1643)
2024-12-18T00:35:20.8264616Z remote: Counting objects:  66% (1085/1643)
2024-12-18T00:35:20.8265009Z remote: Counting objects:  67% (1101/1643)
2024-12-18T00:35:20.8265406Z remote: Counting objects:  68% (1118/1643)
2024-12-18T00:35:20.8265829Z remote: Counting objects:  69% (1134/1643)
2024-12-18T00:35:20.8266229Z remote: Counting objects:  70% (1151/1643)
2024-12-18T00:35:20.8266962Z remote: Counting objects:  71% (1167/1643)
2024-12-18T00:35:20.8267389Z remote: Counting objects:  72% (1183/1643)
2024-12-18T00:35:20.8267783Z remote: Counting objects:  73% (1200/1643)
2024-12-18T00:35:20.8268177Z remote: Counting objects:  74% (1216/1643)
2024-12-18T00:35:20.8268567Z remote: Counting objects:  75% (1233/1643)
2024-12-18T00:35:20.8268954Z remote: Counting objects:  76% (1249/1643)
2024-12-18T00:35:20.8269350Z remote: Counting objects:  77% (1266/1643)
2024-12-18T00:35:20.8269744Z remote: Counting objects:  78% (1282/1643)
2024-12-18T00:35:20.8270138Z remote: Counting objects:  79% (1298/1643)
2024-12-18T00:35:20.8270536Z remote: Counting objects:  80% (1315/1643)
2024-12-18T00:35:20.8270951Z remote: Counting objects:  81% (1331/1643)
2024-12-18T00:35:20.8271375Z remote: Counting objects:  82% (1348/1643)
2024-12-18T00:35:20.8271769Z remote: Counting objects:  83% (1364/1643)
2024-12-18T00:35:20.8272170Z remote: Counting objects:  84% (1381/1643)
2024-12-18T00:35:20.8272572Z remote: Counting objects:  85% (1397/1643)
2024-12-18T00:35:20.8273111Z remote: Counting objects:  86% (1413/1643)
2024-12-18T00:35:20.8273515Z remote: Counting objects:  87% (1430/1643)
2024-12-18T00:35:20.8273910Z remote: Counting objects:  88% (1446/1643)
2024-12-18T00:35:20.8274304Z remote: Counting objects:  89% (1463/1643)
2024-12-18T00:35:20.8274697Z remote: Counting objects:  90% (1479/1643)
2024-12-18T00:35:20.8275085Z remote: Counting objects:  91% (1496/1643)
2024-12-18T00:35:20.8275482Z remote: Counting objects:  92% (1512/1643)
2024-12-18T00:35:20.8275876Z remote: Counting objects:  93% (1528/1643)
2024-12-18T00:35:20.8276276Z remote: Counting objects:  94% (1545/1643)
2024-12-18T00:35:20.8276671Z remote: Counting objects:  95% (1561/1643)
2024-12-18T00:35:20.8277061Z remote: Counting objects:  96% (1578/1643)
2024-12-18T00:35:20.8277456Z remote: Counting objects:  97% (1594/1643)
2024-12-18T00:35:20.8277985Z remote: Counting objects:  98% (1611/1643)
2024-12-18T00:35:20.8278389Z remote: Counting objects:  99% (1627/1643)
2024-12-18T00:35:20.8278785Z remote: Counting objects: 100% (1643/1643)
2024-12-18T00:35:20.8279210Z remote: Counting objects: 100% (1643/1643), done.
2024-12-18T00:35:20.8279629Z remote: Compressing objects:   0% (1/456)
2024-12-18T00:35:20.8303361Z remote: Compressing objects:   1% (5/456)
2024-12-18T00:35:20.8338716Z remote: Compressing objects:   2% (10/456)
2024-12-18T00:35:20.8408906Z remote: Compressing objects:   3% (14/456)
2024-12-18T00:35:20.8424843Z remote: Compressing objects:   4% (19/456)
2024-12-18T00:35:20.8490615Z remote: Compressing objects:   5% (23/456)
2024-12-18T00:35:20.8527006Z remote: Compressing objects:   6% (28/456)
2024-12-18T00:35:20.8552923Z remote: Compressing objects:   7% (32/456)
2024-12-18T00:35:20.8586812Z remote: Compressing objects:   8% (37/456)
2024-12-18T00:35:20.8608670Z remote: Compressing objects:   9% (42/456)
2024-12-18T00:35:20.8637433Z remote: Compressing objects:  10% (46/456)
2024-12-18T00:35:20.8652220Z remote: Compressing objects:  11% (51/456)
2024-12-18T00:35:20.8673186Z remote: Compressing objects:  12% (55/456)
2024-12-18T00:35:20.8689936Z remote: Compressing objects:  13% (60/456)
2024-12-18T00:35:20.8713126Z remote: Compressing objects:  14% (64/456)
2024-12-18T00:35:20.8730545Z remote: Compressing objects:  15% (69/456)
2024-12-18T00:35:20.8746419Z remote: Compressing objects:  16% (73/456)
2024-12-18T00:35:20.8761262Z remote: Compressing objects:  17% (78/456)
2024-12-18T00:35:20.8769365Z remote: Compressing objects:  18% (83/456)
2024-12-18T00:35:20.8789756Z remote: Compressing objects:  19% (87/456)
2024-12-18T00:35:20.8804380Z remote: Compressing objects:  20% (92/456)
2024-12-18T00:35:20.8807354Z remote: Compressing objects:  21% (96/456)
2024-12-18T00:35:20.8815457Z remote: Compressing objects:  22% (101/456)
2024-12-18T00:35:20.8823951Z remote: Compressing objects:  23% (105/456)
2024-12-18T00:35:20.8828136Z remote: Compressing objects:  24% (110/456)
2024-12-18T00:35:20.8831707Z remote: Compressing objects:  25% (114/456)
2024-12-18T00:35:20.8836932Z remote: Compressing objects:  26% (119/456)
2024-12-18T00:35:20.8838587Z remote: Compressing objects:  27% (124/456)
2024-12-18T00:35:20.8841035Z remote: Compressing objects:  28% (128/456)
2024-12-18T00:35:20.8841573Z remote: Compressing objects:  29% (133/456)
2024-12-18T00:35:20.8842053Z remote: Compressing objects:  30% (137/456)
2024-12-18T00:35:20.8842477Z remote: Compressing objects:  31% (142/456)
2024-12-18T00:35:20.8842892Z remote: Compressing objects:  32% (146/456)
2024-12-18T00:35:20.8843385Z remote: Compressing objects:  33% (151/456)
2024-12-18T00:35:20.8843803Z remote: Compressing objects:  34% (156/456)
2024-12-18T00:35:20.8845415Z remote: Compressing objects:  35% (160/456)
2024-12-18T00:35:20.8853073Z remote: Compressing objects:  36% (165/456)
2024-12-18T00:35:20.8855796Z remote: Compressing objects:  37% (169/456)
2024-12-18T00:35:20.8856618Z remote: Compressing objects:  38% (174/456)
2024-12-18T00:35:20.8857108Z remote: Compressing objects:  39% (178/456)
2024-12-18T00:35:20.8858735Z remote: Compressing objects:  40% (183/456)
2024-12-18T00:35:20.8859161Z remote: Compressing objects:  41% (187/456)
2024-12-18T00:35:20.8862172Z remote: Compressing objects:  42% (192/456)
2024-12-18T00:35:20.8862595Z remote: Compressing objects:  43% (197/456)
2024-12-18T00:35:20.8863006Z remote: Compressing objects:  44% (201/456)
2024-12-18T00:35:20.8864575Z remote: Compressing objects:  45% (206/456)
2024-12-18T00:35:20.8865241Z remote: Compressing objects:  46% (210/456)
2024-12-18T00:35:20.8868026Z remote: Compressing objects:  47% (215/456)
2024-12-18T00:35:20.8868454Z remote: Compressing objects:  48% (219/456)
2024-12-18T00:35:20.8870636Z remote: Compressing objects:  49% (224/456)
2024-12-18T00:35:20.8872459Z remote: Compressing objects:  50% (228/456)
2024-12-18T00:35:20.8873489Z remote: Compressing objects:  51% (233/456)
2024-12-18T00:35:20.8875844Z remote: Compressing objects:  52% (238/456)
2024-12-18T00:35:20.8876940Z remote: Compressing objects:  53% (242/456)
2024-12-18T00:35:20.8879103Z remote: Compressing objects:  54% (247/456)
2024-12-18T00:35:20.8879890Z remote: Compressing objects:  55% (251/456)
2024-12-18T00:35:20.8881307Z remote: Compressing objects:  56% (256/456)
2024-12-18T00:35:20.8883897Z remote: Compressing objects:  57% (260/456)
2024-12-18T00:35:20.8886269Z remote: Compressing objects:  58% (265/456)
2024-12-18T00:35:20.8887626Z remote: Compressing objects:  59% (270/456)
2024-12-18T00:35:20.8889341Z remote: Compressing objects:  60% (274/456)
2024-12-18T00:35:20.8889916Z remote: Compressing objects:  61% (279/456)
2024-12-18T00:35:20.8892821Z remote: Compressing objects:  62% (283/456)
2024-12-18T00:35:20.8893417Z remote: Compressing objects:  63% (288/456)
2024-12-18T00:35:20.8894182Z remote: Compressing objects:  64% (292/456)
2024-12-18T00:35:20.8894833Z remote: Compressing objects:  65% (297/456)
2024-12-18T00:35:20.8896177Z remote: Compressing objects:  66% (301/456)
2024-12-18T00:35:20.8896760Z remote: Compressing objects:  67% (306/456)
2024-12-18T00:35:20.8899073Z remote: Compressing objects:  68% (311/456)
2024-12-18T00:35:20.8901226Z remote: Compressing objects:  69% (315/456)
2024-12-18T00:35:20.8901802Z remote: Compressing objects:  70% (320/456)
2024-12-18T00:35:20.8902402Z remote: Compressing objects:  71% (324/456)
2024-12-18T00:35:20.8903323Z remote: Compressing objects:  72% (329/456)
2024-12-18T00:35:20.8903889Z remote: Compressing objects:  73% (333/456)
2024-12-18T00:35:20.8904942Z remote: Compressing objects:  74% (338/456)
2024-12-18T00:35:20.8905501Z remote: Compressing objects:  75% (342/456)
2024-12-18T00:35:20.8906627Z remote: Compressing objects:  76% (347/456)
2024-12-18T00:35:20.8907219Z remote: Compressing objects:  77% (352/456)
2024-12-18T00:35:20.8907788Z remote: Compressing objects:  78% (356/456)
2024-12-18T00:35:20.8908266Z remote: Compressing objects:  79% (361/456)
2024-12-18T00:35:20.8909991Z remote: Compressing objects:  80% (365/456)
2024-12-18T00:35:20.8910580Z remote: Compressing objects:  81% (370/456)
2024-12-18T00:35:20.8911104Z remote: Compressing objects:  82% (374/456)
2024-12-18T00:35:20.8911517Z remote: Compressing objects:  83% (379/456)
2024-12-18T00:35:20.8912312Z remote: Compressing objects:  84% (384/456)
2024-12-18T00:35:20.8914167Z remote: Compressing objects:  85% (388/456)
2024-12-18T00:35:20.8916375Z remote: Compressing objects:  86% (393/456)
2024-12-18T00:35:20.8916948Z remote: Compressing objects:  87% (397/456)
2024-12-18T00:35:20.8918139Z remote: Compressing objects:  88% (402/456)
2024-12-18T00:35:20.8918719Z remote: Compressing objects:  89% (406/456)
2024-12-18T00:35:20.8919772Z remote: Compressing objects:  90% (411/456)
2024-12-18T00:35:20.8920756Z remote: Compressing objects:  91% (415/456)
2024-12-18T00:35:20.8922195Z remote: Compressing objects:  92% (420/456)
2024-12-18T00:35:20.8922773Z remote: Compressing objects:  93% (425/456)
2024-12-18T00:35:20.8923367Z remote: Compressing objects:  94% (429/456)
2024-12-18T00:35:20.8923922Z remote: Compressing objects:  95% (434/456)
2024-12-18T00:35:20.8924458Z remote: Compressing objects:  96% (438/456)
2024-12-18T00:35:20.8924865Z remote: Compressing objects:  97% (443/456)
2024-12-18T00:35:20.8925427Z remote: Compressing objects:  98% (447/456)
2024-12-18T00:35:20.8925995Z remote: Compressing objects:  99% (452/456)
2024-12-18T00:35:20.8926560Z remote: Compressing objects: 100% (456/456)
2024-12-18T00:35:20.8927007Z remote: Compressing objects: 100% (456/456), done.
2024-12-18T00:35:20.9011515Z Receiving objects:   0% (1/34540)
2024-12-18T00:35:20.9065842Z Receiving objects:   1% (346/34540)
2024-12-18T00:35:20.9113083Z Receiving objects:   2% (691/34540)
2024-12-18T00:35:20.9148466Z Receiving objects:   3% (1037/34540)
2024-12-18T00:35:20.9184140Z Receiving objects:   4% (1382/34540)
2024-12-18T00:35:20.9219917Z Receiving objects:   5% (1727/34540)
2024-12-18T00:35:20.9268317Z Receiving objects:   6% (2073/34540)
2024-12-18T00:35:20.9304736Z Receiving objects:   7% (2418/34540)
2024-12-18T00:35:20.9339453Z Receiving objects:   8% (2764/34540)
2024-12-18T00:35:20.9418089Z Receiving objects:   9% (3109/34540)
2024-12-18T00:35:20.9473887Z Receiving objects:  10% (3454/34540)
2024-12-18T00:35:20.9546874Z Receiving objects:  11% (3800/34540)
2024-12-18T00:35:20.9733343Z Receiving objects:  12% (4145/34540)
2024-12-18T00:35:20.9848912Z Receiving objects:  13% (4491/34540)
2024-12-18T00:35:20.9942507Z Receiving objects:  14% (4836/34540)
2024-12-18T00:35:21.0002882Z Receiving objects:  15% (5181/34540)
2024-12-18T00:35:21.0038962Z Receiving objects:  16% (5527/34540)
2024-12-18T00:35:21.0103433Z Receiving objects:  17% (5872/34540)
2024-12-18T00:35:21.0135515Z Receiving objects:  18% (6218/34540)
2024-12-18T00:35:21.0160537Z Receiving objects:  19% (6563/34540)
2024-12-18T00:35:21.2280053Z Receiving objects:  20% (6908/34540)
2024-12-18T00:35:21.8953870Z Receiving objects:  21% (7254/34540)
2024-12-18T00:35:22.5451224Z Receiving objects:  21% (7375/34540), 74.73 MiB | 74.72 MiB/s
2024-12-18T00:35:22.6320013Z Receiving objects:  22% (7599/34540), 112.03 MiB | 74.68 MiB/s
2024-12-18T00:35:22.7128230Z Receiving objects:  23% (7945/34540), 112.03 MiB | 74.68 MiB/s
2024-12-18T00:35:22.7942832Z Receiving objects:  24% (8290/34540), 112.03 MiB | 74.68 MiB/s
2024-12-18T00:35:22.8743916Z Receiving objects:  25% (8635/34540), 112.03 MiB | 74.68 MiB/s
2024-12-18T00:35:22.8953911Z Receiving objects:  26% (8981/34540), 112.03 MiB | 74.68 MiB/s
2024-12-18T00:35:22.9557150Z Receiving objects:  26% (9070/34540), 149.46 MiB | 74.73 MiB/s
2024-12-18T00:35:23.0361031Z Receiving objects:  27% (9326/34540), 149.46 MiB | 74.73 MiB/s
2024-12-18T00:35:23.1167991Z Receiving objects:  28% (9672/34540), 149.46 MiB | 74.73 MiB/s
2024-12-18T00:35:23.3509496Z Receiving objects:  29% (10017/34540), 149.46 MiB | 74.73 MiB/s
2024-12-18T00:35:23.4122678Z Receiving objects:  30% (10362/34540), 149.46 MiB | 74.73 MiB/s
2024-12-18T00:35:23.6187616Z Receiving objects:  31% (10708/34540), 183.98 MiB | 73.59 MiB/s
2024-12-18T00:35:23.8953144Z Receiving objects:  32% (11053/34540), 183.98 MiB | 73.59 MiB/s
2024-12-18T00:35:23.9089299Z Receiving objects:  32% (11298/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9308122Z Receiving objects:  33% (11399/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9330827Z Receiving objects:  34% (11744/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9350277Z Receiving objects:  35% (12089/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9366423Z Receiving objects:  36% (12435/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9379446Z Receiving objects:  37% (12780/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9391495Z Receiving objects:  38% (13126/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9412622Z Receiving objects:  39% (13471/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9424665Z Receiving objects:  40% (13816/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9429610Z Receiving objects:  41% (14162/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9443329Z Receiving objects:  42% (14507/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9466084Z Receiving objects:  43% (14853/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9481329Z Receiving objects:  44% (15198/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9682028Z Receiving objects:  45% (15543/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9688290Z Receiving objects:  46% (15889/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9695643Z Receiving objects:  47% (16234/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9701563Z Receiving objects:  48% (16580/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9718722Z Receiving objects:  49% (16925/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9730817Z Receiving objects:  50% (17270/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9736539Z Receiving objects:  51% (17616/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9754940Z Receiving objects:  52% (17961/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9763621Z Receiving objects:  53% (18307/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9774548Z Receiving objects:  54% (18652/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9914893Z Receiving objects:  55% (18997/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9921456Z Receiving objects:  56% (19343/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9927793Z Receiving objects:  57% (19688/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9937804Z Receiving objects:  58% (20034/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:23.9946663Z Receiving objects:  59% (20379/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.0205718Z Receiving objects:  60% (20724/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.0603460Z Receiving objects:  61% (21070/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.0616255Z Receiving objects:  62% (21415/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.0632211Z Receiving objects:  63% (21761/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.0646894Z Receiving objects:  64% (22106/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.0661106Z Receiving objects:  65% (22451/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.0671626Z Receiving objects:  66% (22797/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.0691272Z Receiving objects:  67% (23142/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.0698985Z Receiving objects:  68% (23488/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.0708051Z Receiving objects:  69% (23833/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.0922059Z Receiving objects:  70% (24178/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.0956653Z Receiving objects:  71% (24524/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.0967994Z Receiving objects:  72% (24869/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.0983589Z Receiving objects:  73% (25215/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.1024272Z Receiving objects:  74% (25560/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.1175495Z Receiving objects:  75% (25905/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.1182066Z Receiving objects:  76% (26251/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.1189072Z Receiving objects:  77% (26596/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.1196826Z Receiving objects:  78% (26942/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.1205161Z Receiving objects:  79% (27287/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.1211865Z Receiving objects:  80% (27632/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.1333669Z Receiving objects:  81% (27978/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.1352596Z Receiving objects:  82% (28323/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.1367458Z Receiving objects:  83% (28669/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.1443662Z Receiving objects:  84% (29014/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.1528202Z Receiving objects:  85% (29359/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.1610028Z Receiving objects:  86% (29705/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.1713814Z Receiving objects:  87% (30050/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.1832412Z Receiving objects:  88% (30396/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.1939534Z Receiving objects:  89% (30741/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.1960702Z Receiving objects:  90% (31086/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.1984339Z Receiving objects:  91% (31432/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.2004942Z Receiving objects:  92% (31777/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.2045963Z Receiving objects:  93% (32123/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.2096802Z Receiving objects:  94% (32468/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.2143118Z Receiving objects:  95% (32813/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.2148339Z Receiving objects:  96% (33159/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.2154246Z Receiving objects:  97% (33504/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.2197356Z Receiving objects:  98% (33850/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.2219739Z Receiving objects:  99% (34195/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.2220671Z remote: Total 34540 (delta 1440), reused 1226 (delta 1187), pack-reused 32897 (from 5)
2024-12-18T00:35:24.2239321Z Receiving objects: 100% (34540/34540), 219.29 MiB | 73.09 MiB/s
2024-12-18T00:35:24.2239924Z Receiving objects: 100% (34540/34540), 234.54 MiB | 70.47 MiB/s, done.
2024-12-18T00:35:24.2274910Z Resolving deltas:   0% (0/18569)
2024-12-18T00:35:24.2288351Z Resolving deltas:   1% (186/18569)
2024-12-18T00:35:24.2303212Z Resolving deltas:   2% (399/18569)
2024-12-18T00:35:24.2332397Z Resolving deltas:   3% (560/18569)
2024-12-18T00:35:24.2342907Z Resolving deltas:   4% (743/18569)
2024-12-18T00:35:24.2354927Z Resolving deltas:   5% (932/18569)
2024-12-18T00:35:24.2366828Z Resolving deltas:   6% (1125/18569)
2024-12-18T00:35:24.2378936Z Resolving deltas:   7% (1301/18569)
2024-12-18T00:35:24.2392170Z Resolving deltas:   8% (1487/18569)
2024-12-18T00:35:24.2405617Z Resolving deltas:   9% (1677/18569)
2024-12-18T00:35:24.2417792Z Resolving deltas:  10% (1864/18569)
2024-12-18T00:35:24.2430815Z Resolving deltas:  11% (2053/18569)
2024-12-18T00:35:24.2443507Z Resolving deltas:  12% (2229/18569)
2024-12-18T00:35:24.2461960Z Resolving deltas:  13% (2419/18569)
2024-12-18T00:35:24.2487846Z Resolving deltas:  14% (2604/18569)
2024-12-18T00:35:24.2501130Z Resolving deltas:  15% (2864/18569)
2024-12-18T00:35:24.2548549Z Resolving deltas:  16% (3020/18569)
2024-12-18T00:35:24.2579170Z Resolving deltas:  17% (3160/18569)
2024-12-18T00:35:24.2594183Z Resolving deltas:  19% (3550/18569)
2024-12-18T00:35:24.2618691Z Resolving deltas:  20% (3723/18569)
2024-12-18T00:35:24.2622779Z Resolving deltas:  21% (4049/18569)
2024-12-18T00:35:24.2658614Z Resolving deltas:  22% (4098/18569)
2024-12-18T00:35:24.2685531Z Resolving deltas:  23% (4292/18569)
2024-12-18T00:35:24.2702971Z Resolving deltas:  24% (4469/18569)
2024-12-18T00:35:24.2719968Z Resolving deltas:  25% (4667/18569)
2024-12-18T00:35:24.2740574Z Resolving deltas:  26% (4832/18569)
2024-12-18T00:35:24.2756524Z Resolving deltas:  27% (5041/18569)
2024-12-18T00:35:24.2766872Z Resolving deltas:  28% (5230/18569)
2024-12-18T00:35:24.2778751Z Resolving deltas:  29% (5401/18569)
2024-12-18T00:35:24.2796797Z Resolving deltas:  30% (5574/18569)
2024-12-18T00:35:24.2818668Z Resolving deltas:  31% (5757/18569)
2024-12-18T00:35:24.2837152Z Resolving deltas:  32% (5963/18569)
2024-12-18T00:35:24.2847933Z Resolving deltas:  33% (6129/18569)
2024-12-18T00:35:24.2870756Z Resolving deltas:  34% (6314/18569)
2024-12-18T00:35:24.2885438Z Resolving deltas:  35% (6511/18569)
2024-12-18T00:35:24.2902538Z Resolving deltas:  36% (6697/18569)
2024-12-18T00:35:24.2917498Z Resolving deltas:  37% (6886/18569)
2024-12-18T00:35:24.2929023Z Resolving deltas:  38% (7060/18569)
2024-12-18T00:35:24.2940922Z Resolving deltas:  39% (7274/18569)
2024-12-18T00:35:24.2967143Z Resolving deltas:  40% (7444/18569)
2024-12-18T00:35:24.2983850Z Resolving deltas:  41% (7614/18569)
2024-12-18T00:35:24.3004263Z Resolving deltas:  42% (7801/18569)
2024-12-18T00:35:24.3016391Z Resolving deltas:  43% (7985/18569)
2024-12-18T00:35:24.3027551Z Resolving deltas:  44% (8172/18569)
2024-12-18T00:35:24.3038418Z Resolving deltas:  45% (8357/18569)
2024-12-18T00:35:24.3061862Z Resolving deltas:  46% (8549/18569)
2024-12-18T00:35:24.3063875Z Resolving deltas:  47% (8881/18569)
2024-12-18T00:35:24.3072974Z Resolving deltas:  48% (8918/18569)
2024-12-18T00:35:24.3090387Z Resolving deltas:  49% (9100/18569)
2024-12-18T00:35:24.3098955Z Resolving deltas:  50% (9289/18569)
2024-12-18T00:35:24.3106528Z Resolving deltas:  51% (9475/18569)
2024-12-18T00:35:24.3114034Z Resolving deltas:  52% (9656/18569)
2024-12-18T00:35:24.3123378Z Resolving deltas:  53% (9854/18569)
2024-12-18T00:35:24.3132433Z Resolving deltas:  54% (10070/18569)
2024-12-18T00:35:24.3142984Z Resolving deltas:  55% (10282/18569)
2024-12-18T00:35:24.3147011Z Resolving deltas:  56% (10499/18569)
2024-12-18T00:35:24.3158510Z Resolving deltas:  57% (10591/18569)
2024-12-18T00:35:24.3178645Z Resolving deltas:  58% (10842/18569)
2024-12-18T00:35:24.3184421Z Resolving deltas:  59% (11129/18569)
2024-12-18T00:35:24.3200015Z Resolving deltas:  60% (11248/18569)
2024-12-18T00:35:24.3213711Z Resolving deltas:  61% (11332/18569)
2024-12-18T00:35:24.3224355Z Resolving deltas:  62% (11575/18569)
2024-12-18T00:35:24.3240687Z Resolving deltas:  63% (11734/18569)
2024-12-18T00:35:24.3253935Z Resolving deltas:  64% (11885/18569)
2024-12-18T00:35:24.3291577Z Resolving deltas:  65% (12071/18569)
2024-12-18T00:35:24.3303616Z Resolving deltas:  68% (12631/18569)
2024-12-18T00:35:24.3311602Z Resolving deltas:  69% (12840/18569)
2024-12-18T00:35:24.3331214Z Resolving deltas:  70% (13000/18569)
2024-12-18T00:35:24.3341845Z Resolving deltas:  71% (13198/18569)
2024-12-18T00:35:24.3354655Z Resolving deltas:  72% (13372/18569)
2024-12-18T00:35:24.3365829Z Resolving deltas:  73% (13574/18569)
2024-12-18T00:35:24.3378294Z Resolving deltas:  74% (13753/18569)
2024-12-18T00:35:24.3387481Z Resolving deltas:  75% (13931/18569)
2024-12-18T00:35:24.3404756Z Resolving deltas:  76% (14113/18569)
2024-12-18T00:35:24.3414794Z Resolving deltas:  77% (14317/18569)
2024-12-18T00:35:24.3426674Z Resolving deltas:  78% (14485/18569)
2024-12-18T00:35:24.3440504Z Resolving deltas:  79% (14676/18569)
2024-12-18T00:35:24.3447428Z Resolving deltas:  80% (14904/18569)
2024-12-18T00:35:24.3459703Z Resolving deltas:  81% (15045/18569)
2024-12-18T00:35:24.3471405Z Resolving deltas:  82% (15227/18569)
2024-12-18T00:35:24.3478942Z Resolving deltas:  83% (15415/18569)
2024-12-18T00:35:24.3499699Z Resolving deltas:  84% (15598/18569)
2024-12-18T00:35:24.3506387Z Resolving deltas:  85% (15873/18569)
2024-12-18T00:35:24.3519817Z Resolving deltas:  86% (15983/18569)
2024-12-18T00:35:24.3536550Z Resolving deltas:  87% (16156/18569)
2024-12-18T00:35:24.3567590Z Resolving deltas:  88% (16341/18569)
2024-12-18T00:35:24.3588071Z Resolving deltas:  89% (16527/18569)
2024-12-18T00:35:24.3628568Z Resolving deltas:  90% (16714/18569)
2024-12-18T00:35:24.3662961Z Resolving deltas:  91% (16904/18569)
2024-12-18T00:35:24.3682100Z Resolving deltas:  92% (17084/18569)
2024-12-18T00:35:24.3698069Z Resolving deltas:  93% (17275/18569)
2024-12-18T00:35:24.3701839Z Resolving deltas:  94% (17568/18569)
2024-12-18T00:35:24.3711169Z Resolving deltas:  95% (17642/18569)
2024-12-18T00:35:24.3721572Z Resolving deltas:  96% (17845/18569)
2024-12-18T00:35:24.3737851Z Resolving deltas:  97% (18012/18569)
2024-12-18T00:35:24.3768014Z Resolving deltas:  98% (18285/18569)
2024-12-18T00:35:24.4013839Z Resolving deltas:  99% (18399/18569)
2024-12-18T00:35:24.4015053Z Resolving deltas: 100% (18569/18569)
2024-12-18T00:35:24.4015388Z Resolving deltas: 100% (18569/18569), done.
2024-12-18T00:35:25.4809206Z + pushd torchbench
2024-12-18T00:35:25.4809514Z ~/workspace/torchbench ~/workspace
2024-12-18T00:35:25.4809892Z + git checkout 766a5e3a189384659fd35a68c3b17b88c761aaac
2024-12-18T00:35:25.5883181Z Note: switching to '766a5e3a189384659fd35a68c3b17b88c761aaac'.
2024-12-18T00:35:25.5883492Z 
2024-12-18T00:35:25.5883706Z You are in 'detached HEAD' state. You can look around, make experimental
2024-12-18T00:35:25.5884246Z changes and commit them, and you can discard any commits you make in this
2024-12-18T00:35:25.5884783Z state without impacting any branches by switching back to a branch.
2024-12-18T00:35:25.5885094Z 
2024-12-18T00:35:25.5885307Z If you want to create a new branch to retain commits you create, you may
2024-12-18T00:35:25.5886084Z do so (now or later) by using -c with the switch command. Example:
2024-12-18T00:35:25.5886367Z 
2024-12-18T00:35:25.5886494Z   git switch -c <new-branch-name>
2024-12-18T00:35:25.5886701Z 
2024-12-18T00:35:25.5886809Z Or undo this operation with:
2024-12-18T00:35:25.5886998Z 
2024-12-18T00:35:25.5887092Z   git switch -
2024-12-18T00:35:25.5887237Z 
2024-12-18T00:35:25.5887471Z Turn off this advice by setting config variable advice.detachedHead to false
2024-12-18T00:35:25.5887808Z 
2024-12-18T00:35:25.5888025Z HEAD is now at 766a5e3a Account for older numpy versions in #2514 (#2524)
2024-12-18T00:35:25.5897948Z + '[' '' ']'
2024-12-18T00:35:25.5898221Z + python install.py --continue_on_fail
2024-12-18T00:35:28.8479101Z checking packages torch, torchvision, torchaudio are installed...OK
2024-12-18T00:35:51.2522960Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK
2024-12-18T00:35:55.4774402Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK
2024-12-18T00:35:57.4773382Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK
2024-12-18T00:35:59.2512511Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK
2024-12-18T00:35:59.2750002Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK
2024-12-18T00:36:07.0847139Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK
2024-12-18T00:36:09.8769495Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK
2024-12-18T00:36:12.6551849Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK
2024-12-18T00:36:15.4812792Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK
2024-12-18T00:36:15.4816008Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found
2024-12-18T00:36:16.5545363Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK
2024-12-18T00:36:19.6460622Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK
2024-12-18T00:36:19.6695464Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK
2024-12-18T00:37:15.4371879Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK
2024-12-18T00:37:24.8611953Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK
2024-12-18T00:37:32.3951292Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK
2024-12-18T00:37:39.5575721Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK
2024-12-18T00:37:49.0823663Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK
2024-12-18T00:37:56.8520713Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK
2024-12-18T00:38:03.7817105Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK
2024-12-18T00:38:11.0629767Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK
2024-12-18T00:38:18.8982236Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK
2024-12-18T00:38:26.4029302Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK
2024-12-18T00:38:33.6040803Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK
2024-12-18T00:38:41.0243474Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK
2024-12-18T00:38:42.2816399Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK
2024-12-18T00:39:21.8933132Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK
2024-12-18T00:39:26.4548870Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK
2024-12-18T00:39:31.7879426Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK
2024-12-18T00:39:44.1810461Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK
2024-12-18T00:39:45.3879778Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK
2024-12-18T00:39:46.8914889Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK
2024-12-18T00:39:54.1562337Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK
2024-12-18T00:40:01.2242909Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK
2024-12-18T00:40:06.7466415Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK
2024-12-18T00:40:14.4405317Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK
2024-12-18T00:40:20.5610954Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK
2024-12-18T00:40:25.6654490Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK
2024-12-18T00:40:31.9879966Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK
2024-12-18T00:40:48.0424178Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK
2024-12-18T00:40:54.3046481Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK
2024-12-18T00:40:58.3031801Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK
2024-12-18T00:41:03.1457751Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK
2024-12-18T00:41:09.9833183Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK
2024-12-18T00:41:09.9836746Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found
2024-12-18T00:41:22.9535229Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK
2024-12-18T00:41:26.9662750Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK
2024-12-18T00:41:26.9665546Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found
2024-12-18T00:41:34.5995183Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK
2024-12-18T00:41:35.8361165Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK
2024-12-18T00:41:37.0973275Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK
2024-12-18T00:42:21.1521342Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK
2024-12-18T00:43:49.2039425Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK
2024-12-18T00:43:49.2314146Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK
2024-12-18T00:43:50.7873032Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK
2024-12-18T00:43:50.8116195Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK
2024-12-18T00:43:50.8351903Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK
2024-12-18T00:43:50.8587856Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK
2024-12-18T00:43:50.8823468Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK
2024-12-18T00:43:50.9065825Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK
2024-12-18T00:44:11.9187599Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK
2024-12-18T00:44:11.9190211Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found
2024-12-18T00:44:13.1980269Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK
2024-12-18T00:44:15.7521524Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK
2024-12-18T00:44:15.7767127Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK
2024-12-18T00:44:15.8006057Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK
2024-12-18T00:44:15.8253988Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK
2024-12-18T00:44:15.8496035Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK
2024-12-18T00:44:15.8737186Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK
2024-12-18T00:44:22.7715292Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK
2024-12-18T00:44:24.2434911Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK
2024-12-18T00:44:28.1302180Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK
2024-12-18T00:44:28.1545382Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK
2024-12-18T00:44:28.1792644Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK
2024-12-18T00:44:28.2033959Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK
2024-12-18T00:44:28.2273299Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK
2024-12-18T00:44:28.2512024Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK
2024-12-18T00:44:44.1816056Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK
2024-12-18T00:45:00.8628091Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK
2024-12-18T00:45:00.8874309Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK
2024-12-18T00:45:00.8878492Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found
2024-12-18T00:45:00.8880833Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found
2024-12-18T00:45:03.6141550Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK
2024-12-18T00:45:05.3530574Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK
2024-12-18T00:45:05.3779331Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK
2024-12-18T00:45:35.4929585Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK
2024-12-18T00:45:42.1431149Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK
2024-12-18T00:45:45.4311756Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK
2024-12-18T00:45:53.5763018Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK
2024-12-18T00:45:53.6013387Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK
2024-12-18T00:45:53.6257596Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK
2024-12-18T00:45:53.6501564Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK
2024-12-18T00:45:53.6740801Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK
2024-12-18T00:45:53.6983303Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK
2024-12-18T00:45:53.7227482Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK
2024-12-18T00:45:53.7474552Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK
2024-12-18T00:46:01.7337917Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK
2024-12-18T00:46:13.7398125Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK
2024-12-18T00:46:13.7653447Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK
2024-12-18T00:46:15.9379764Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK
2024-12-18T00:46:17.9600674Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK
2024-12-18T00:46:18.4753528Z + echo 'Print all dependencies after TorchBench is installed'
2024-12-18T00:46:18.4754020Z Print all dependencies after TorchBench is installed
2024-12-18T00:46:18.4754399Z + python -mpip freeze
2024-12-18T00:46:18.9998017Z absl-py==2.1.0
2024-12-18T00:46:18.9998416Z accelerate==1.2.1
2024-12-18T00:46:18.9998752Z aiohappyeyeballs==2.4.4
2024-12-18T00:46:18.9999117Z aiohttp==3.11.10
2024-12-18T00:46:18.9999484Z aiosignal==1.3.1
2024-12-18T00:46:18.9999825Z annotated-types==0.7.0
2024-12-18T00:46:19.0000200Z antlr4-python3-runtime==4.9.3
2024-12-18T00:46:19.0000602Z anyascii==0.3.2
2024-12-18T00:46:19.0000902Z astroid==3.3.6
2024-12-18T00:46:19.0001322Z astunparse==1.6.3
2024-12-18T00:46:19.0001646Z async-timeout==5.0.1
2024-12-18T00:46:19.0001971Z attrs==23.1.0
2024-12-18T00:46:19.0002267Z audioread==3.0.1
2024-12-18T00:46:19.0002520Z babel==2.16.0
2024-12-18T00:46:19.0002792Z beautifulsoup4==4.12.3
2024-12-18T00:46:19.0003850Z -e git+https://github.com/pytorch/benchmark@766a5e3a189384659fd35a68c3b17b88c761aaac#egg=bert_pytorch&subdirectory=torchbenchmark/models/BERT_pytorch
2024-12-18T00:46:19.0004824Z bibtexparser==2.0.0b8
2024-12-18T00:46:19.0005100Z black==24.10.0
2024-12-18T00:46:19.0005403Z blinker==1.9.0
2024-12-18T00:46:19.0005713Z blis==1.1.0
2024-12-18T00:46:19.0005977Z blobfile==3.0.0
2024-12-18T00:46:19.0006297Z bokeh==3.6.2
2024-12-18T00:46:19.0006548Z boto3==1.35.42
2024-12-18T00:46:19.0006872Z botocore==1.35.78
2024-12-18T00:46:19.0007165Z bs4==0.0.2
2024-12-18T00:46:19.0007442Z cachetools==5.5.0
2024-12-18T00:46:19.0008061Z cardboardlint==1.3.1
2024-12-18T00:46:19.0008349Z catalogue==2.0.10
2024-12-18T00:46:19.0008579Z certifi==2024.8.30
2024-12-18T00:46:19.0008907Z cffi==1.17.1
2024-12-18T00:46:19.0009174Z charset-normalizer==3.4.0
2024-12-18T00:46:19.0009503Z click==8.1.7
2024-12-18T00:46:19.0009717Z clldutils==3.24.0
2024-12-18T00:46:19.0010034Z cloudpathlib==0.20.0
2024-12-18T00:46:19.0010351Z cloudpickle==3.1.0
2024-12-18T00:46:19.0010665Z colorama==0.4.6
2024-12-18T00:46:19.0010981Z colorlog==6.9.0
2024-12-18T00:46:19.0011272Z confection==0.1.5
2024-12-18T00:46:19.0011554Z contourpy==1.2.1
2024-12-18T00:46:19.0011872Z coremltools==5.0b5
2024-12-18T00:46:19.0012182Z cryptography==44.0.0
2024-12-18T00:46:19.0012494Z csvw==3.5.1
2024-12-18T00:46:19.0012792Z cycler==0.12.1
2024-12-18T00:46:19.0013104Z cymem==2.0.10
2024-12-18T00:46:19.0013401Z Cython==3.0.11
2024-12-18T00:46:19.0013898Z DALL-E==0.1
2024-12-18T00:46:19.0014112Z datasets==3.2.0
2024-12-18T00:46:19.0014394Z decorator==5.1.1
2024-12-18T00:46:19.0014721Z defusedxml==0.7.1
2024-12-18T00:46:19.0015051Z Deprecated==1.2.15
2024-12-18T00:46:19.0015716Z detectron2 @ git+https://github.com/facebookresearch/detectron2.git@1a4df4d954089534daebbd7a0a45f08ece87056e
2024-12-18T00:46:19.0016439Z diffusers==0.30.3
2024-12-18T00:46:19.0016704Z dill==0.3.8
2024-12-18T00:46:19.0016921Z diskcache==5.6.3
2024-12-18T00:46:19.0017153Z distro==1.9.0
2024-12-18T00:46:19.0017368Z dlinfo==1.2.1
2024-12-18T00:46:19.0017597Z docker-pycreds==0.4.0
2024-12-18T00:46:19.0017905Z dominate==2.9.1
2024-12-18T00:46:19.0018421Z effdet @ git+https://github.com/rwightman/efficientdet-pytorch.git@d43c9e34cd62d22b4205831bb735f6dd83b8e881
2024-12-18T00:46:19.0019004Z einops==0.8.0
2024-12-18T00:46:19.0019237Z exceptiongroup==1.2.2
2024-12-18T00:46:19.0019483Z execnet==2.1.1
2024-12-18T00:46:19.0019708Z expecttest==0.2.1
2024-12-18T00:46:19.0019940Z FastNLP==0.6.0
2024-12-18T00:46:19.0020169Z fbscribelogger==0.1.7
2024-12-18T00:46:19.0020423Z ffmpeg-python==0.2.0
2024-12-18T00:46:19.0020678Z filelock==3.16.1
2024-12-18T00:46:19.0020937Z Flask==3.1.0
2024-12-18T00:46:19.0021172Z flatbuffers==2.0
2024-12-18T00:46:19.0021405Z fonttools==4.55.3
2024-12-18T00:46:19.0021637Z frozenlist==1.5.0
2024-12-18T00:46:19.0021863Z fsspec==2024.9.0
2024-12-18T00:46:19.0022078Z ftfy==6.3.1
2024-12-18T00:46:19.0022523Z functorch @ git+https://github.com/pytorch/functorch.git@b71aa0b4387b86c278132209b99538be48ef4c74
2024-12-18T00:46:19.0023053Z future==1.0.0
2024-12-18T00:46:19.0023284Z fvcore==0.1.5.post20221221
2024-12-18T00:46:19.0023542Z gdown==5.2.0
2024-12-18T00:46:19.0023749Z ghstack==0.8.0
2024-12-18T00:46:19.0023969Z gitdb==4.0.11
2024-12-18T00:46:19.0024187Z GitPython==3.1.43
2024-12-18T00:46:19.0024422Z google-auth==2.36.0
2024-12-18T00:46:19.0024689Z google-auth-oauthlib==1.0.0
2024-12-18T00:46:19.0024956Z grpcio==1.68.1
2024-12-18T00:46:19.0025175Z gym==0.26.2
2024-12-18T00:46:19.0025399Z gym-notices==0.0.8
2024-12-18T00:46:19.0025629Z h5py==3.12.1
2024-12-18T00:46:19.0025835Z higher==0.2.1
2024-12-18T00:46:19.0026070Z huggingface-hub==0.26.5
2024-12-18T00:46:19.0026322Z hydra-core==1.3.2
2024-12-18T00:46:19.0026555Z hypothesis==5.35.1
2024-12-18T00:46:19.0026785Z idna==3.10
2024-12-18T00:46:19.0026990Z imageio==2.36.1
2024-12-18T00:46:19.0027225Z importlib_metadata==8.5.0
2024-12-18T00:46:19.0027489Z inflect==7.4.0
2024-12-18T00:46:19.0027715Z iniconfig==2.0.0
2024-12-18T00:46:19.0027943Z iopath==0.1.9
2024-12-18T00:46:19.0028155Z isodate==0.7.2
2024-12-18T00:46:19.0028374Z isort==5.13.2
2024-12-18T00:46:19.0028596Z itsdangerous==2.2.0
2024-12-18T00:46:19.0028827Z Jinja2==3.1.4
2024-12-18T00:46:19.0029038Z jmespath==1.0.1
2024-12-18T00:46:19.0029258Z joblib==1.4.2
2024-12-18T00:46:19.0029477Z jsonpatch==1.33
2024-12-18T00:46:19.0029706Z jsonpointer==3.0.0
2024-12-18T00:46:19.0029940Z jsonschema==4.23.0
2024-12-18T00:46:19.0030190Z jsonschema-specifications==2024.10.1
2024-12-18T00:46:19.0030501Z junitparser==2.1.1
2024-12-18T00:46:19.0030765Z kaldi-io==0.9.8
2024-12-18T00:46:19.0031022Z kiwisolver==1.4.7
2024-12-18T00:46:19.0031356Z kornia==0.7.4
2024-12-18T00:46:19.0031572Z kornia_rs==0.1.7
2024-12-18T00:46:19.0031797Z lameenc==1.7.0
2024-12-18T00:46:19.0032018Z langcodes==3.5.0
2024-12-18T00:46:19.0032247Z langdetect==1.0.9
2024-12-18T00:46:19.0032475Z language-tags==1.2.0
2024-12-18T00:46:19.0032725Z language_data==1.3.0
2024-12-18T00:46:19.0032961Z lark==0.12.0
2024-12-18T00:46:19.0033177Z lazy_loader==0.4
2024-12-18T00:46:19.0033399Z libcst==1.5.1
2024-12-18T00:46:19.0033611Z librosa==0.9.2
2024-12-18T00:46:19.0033838Z lintrunner==0.12.5
2024-12-18T00:46:19.0034067Z llvmlite==0.38.1
2024-12-18T00:46:19.0034287Z lxml==5.3.0
2024-12-18T00:46:19.0034495Z marisa-trie==1.2.1
2024-12-18T00:46:19.0034724Z Markdown==3.7
2024-12-18T00:46:19.0034951Z markdown-it-py==3.0.0
2024-12-18T00:46:19.0035211Z MarkupSafe==3.0.2
2024-12-18T00:46:19.0035443Z matplotlib==3.8.4
2024-12-18T00:46:19.0035755Z mccabe==0.7.0
2024-12-18T00:46:19.0035971Z mdurl==0.1.2
2024-12-18T00:46:19.0036188Z ml_dtypes==0.5.0
2024-12-18T00:46:19.0036428Z MonkeyType==23.3.0
2024-12-18T00:46:19.0036669Z more-itertools==10.5.0
2024-12-18T00:46:19.0036912Z mpmath==1.3.0
2024-12-18T00:46:19.0037133Z msgpack==1.1.0
2024-12-18T00:46:19.0037356Z multidict==6.1.0
2024-12-18T00:46:19.0037591Z multiprocess==0.70.16
2024-12-18T00:46:19.0037834Z murmurhash==1.0.11
2024-12-18T00:46:19.0038063Z musdb==0.4.2
2024-12-18T00:46:19.0038280Z museval==0.4.1
2024-12-18T00:46:19.0038500Z mypy==1.13.0
2024-12-18T00:46:19.0038723Z mypy-extensions==1.0.0
2024-12-18T00:46:19.0038971Z networkx==2.8.8
2024-12-18T00:46:19.0039198Z ninja==1.10.2
2024-12-18T00:46:19.0039413Z nose==1.3.7
2024-12-18T00:46:19.0039624Z numba==0.55.2
2024-12-18T00:46:19.0039836Z numpy==1.21.2
2024-12-18T00:46:19.0040063Z nvidia-cublas-cu12==12.4.5.8
2024-12-18T00:46:19.0040355Z nvidia-cuda-cupti-cu12==12.4.127
2024-12-18T00:46:19.0040654Z nvidia-cuda-nvrtc-cu12==12.4.127
2024-12-18T00:46:19.0040959Z nvidia-cuda-runtime-cu12==12.4.127
2024-12-18T00:46:19.0041367Z nvidia-cudnn-cu12==9.1.0.70
2024-12-18T00:46:19.0041647Z nvidia-cufft-cu12==11.2.1.3
2024-12-18T00:46:19.0041924Z nvidia-curand-cu12==10.3.5.147
2024-12-18T00:46:19.0042207Z nvidia-cusolver-cu12==11.6.1.9
2024-12-18T00:46:19.0042495Z nvidia-cusparse-cu12==12.3.1.170
2024-12-18T00:46:19.0042774Z nvidia-ml-py==12.560.30
2024-12-18T00:46:19.0043031Z nvidia-nccl-cu12==2.21.5
2024-12-18T00:46:19.0043303Z nvidia-nvjitlink-cu12==12.4.127
2024-12-18T00:46:19.0043587Z nvidia-nvtx-cu12==12.4.127
2024-12-18T00:46:19.0043845Z oauthlib==3.2.2
2024-12-18T00:46:19.0044065Z omegaconf==2.3.0
2024-12-18T00:46:19.0044292Z onnx==1.17.0
2024-12-18T00:46:19.0044524Z onnxscript==0.1.0.dev20240817
2024-12-18T00:46:19.0044793Z opacus==1.5.2
2024-12-18T00:46:19.0045021Z opencv-python==4.8.0.74
2024-12-18T00:46:19.0045267Z opt-einsum==3.3.0
2024-12-18T00:46:19.0045496Z optree==0.13.0
2024-12-18T00:46:19.0045717Z packaging==24.2
2024-12-18T00:46:19.0045954Z pandas==2.0.3
2024-12-18T00:46:19.0046175Z parameterized==0.8.1
2024-12-18T00:46:19.0046418Z patch==1.16
2024-12-18T00:46:19.0046635Z pathspec==0.12.1
2024-12-18T00:46:19.0046868Z phonemizer==3.3.0
2024-12-18T00:46:19.0047099Z pillow==11.0.0
2024-12-18T00:46:19.0047323Z platformdirs==4.3.6
2024-12-18T00:46:19.0047559Z pluggy==1.5.0
2024-12-18T00:46:19.0047778Z ply==3.11
2024-12-18T00:46:19.0047991Z pooch==1.8.2
2024-12-18T00:46:19.0048217Z portalocker==3.0.0
2024-12-18T00:46:19.0048442Z preshed==3.0.9
2024-12-18T00:46:19.0048667Z prettytable==3.12.0
2024-12-18T00:46:19.0048905Z propcache==0.2.1
2024-12-18T00:46:19.0049135Z protobuf==3.20.2
2024-12-18T00:46:19.0049358Z psutil==6.1.0
2024-12-18T00:46:19.0049577Z PuLP==2.9.0
2024-12-18T00:46:19.0049793Z pwlf==2.2.1
2024-12-18T00:46:19.0050018Z py-cpuinfo==9.0.0
2024-12-18T00:46:19.0050251Z pyaml==24.12.1
2024-12-18T00:46:19.0050469Z pyarrow==18.1.0
2024-12-18T00:46:19.0050693Z pyasn1==0.6.1
2024-12-18T00:46:19.0050920Z pyasn1_modules==0.4.1
2024-12-18T00:46:19.0051180Z pyclipper==1.3.0.post6
2024-12-18T00:46:19.0051429Z pycocotools==2.0.8
2024-12-18T00:46:19.0051679Z pycparser==2.22
2024-12-18T00:46:19.0052007Z pycryptodomex==3.21.0
2024-12-18T00:46:19.0052262Z pydantic==2.10.3
2024-12-18T00:46:19.0052502Z pydantic_core==2.27.1
2024-12-18T00:46:19.0052745Z pyDOE==0.3.8
2024-12-18T00:46:19.0052963Z pydot==3.0.3
2024-12-18T00:46:19.0053181Z pygame==2.6.1
2024-12-18T00:46:19.0053403Z PyGithub==2.3.0
2024-12-18T00:46:19.0053632Z Pygments==2.15.0
2024-12-18T00:46:19.0053857Z PyJWT==2.10.1
2024-12-18T00:46:19.0054081Z pylatexenc==2.10
2024-12-18T00:46:19.0054313Z pylint==3.3.2
2024-12-18T00:46:19.0054533Z PyNaCl==1.5.0
2024-12-18T00:46:19.0054752Z pynvml==12.0.0
2024-12-18T00:46:19.0054977Z pyparsing==3.2.0
2024-12-18T00:46:19.0055211Z pypdfium2==4.30.0
2024-12-18T00:46:19.0055442Z pysbd==0.3.4
2024-12-18T00:46:19.0055661Z PySocks==1.7.1
2024-12-18T00:46:19.0055878Z pytest==8.3.4
2024-12-18T00:46:19.0056112Z pytest-benchmark==5.1.0
2024-12-18T00:46:19.0056469Z pytest-cpp==2.3.0
2024-12-18T00:46:19.0056724Z pytest-flakefinder==1.1.0
2024-12-18T00:46:19.0057012Z pytest-rerunfailures==14.0
2024-12-18T00:46:19.0057294Z pytest-subtests==0.13.1
2024-12-18T00:46:19.0057551Z pytest-xdist==3.3.1
2024-12-18T00:46:19.0057807Z python-dateutil==2.9.0.post0
2024-12-18T00:46:19.0058324Z python-doctr @ git+https://github.com/xuzhao9/doctr.git@7f8a3e61bd6480c4e7b7cde47af0aa2153309c56
2024-12-18T00:46:19.0059308Z pytorch-labs-segment-anything-fast @ git+https://github.com/pytorch-labs/segment-anything-fast.git@e6aadeb86f3ae1f58c3f98e2a91e251716e0f2aa
2024-12-18T00:46:19.0060044Z pytz==2024.2
2024-12-18T00:46:19.0060275Z PyWavelets==1.4.1
2024-12-18T00:46:19.0060510Z PyYAML==6.0.2
2024-12-18T00:46:19.0060747Z RapidFuzz==3.11.0
2024-12-18T00:46:19.0061019Z rdflib==7.1.1
2024-12-18T00:46:19.0061234Z redis==5.2.1
2024-12-18T00:46:19.0061457Z referencing==0.35.1
2024-12-18T00:46:19.0061695Z regex==2024.11.6
2024-12-18T00:46:19.0061923Z requests==2.32.3
2024-12-18T00:46:19.0062178Z requests-oauthlib==2.0.0
2024-12-18T00:46:19.0062445Z resampy==0.4.3
2024-12-18T00:46:19.0062670Z rfc3986==1.5.0
2024-12-18T00:46:19.0062900Z rich==13.9.4
2024-12-18T00:46:19.0063120Z rpds-py==0.22.3
2024-12-18T00:46:19.0063340Z rsa==4.9
2024-12-18T00:46:19.0063554Z s3transfer==0.10.4
2024-12-18T00:46:19.0063791Z safetensors==0.4.5
2024-12-18T00:46:19.0064032Z scikit-image==0.21.0
2024-12-18T00:46:19.0064281Z scikit-learn==1.6.0
2024-12-18T00:46:19.0064511Z scipy==1.10.1
2024-12-18T00:46:19.0064735Z segments==2.2.1
2024-12-18T00:46:19.0064978Z sentencepiece==0.2.0
2024-12-18T00:46:19.0065230Z sentry-sdk==2.19.2
2024-12-18T00:46:19.0065463Z setproctitle==1.3.4
2024-12-18T00:46:19.0065701Z shapely==2.0.6
2024-12-18T00:46:19.0065932Z shellingham==1.5.4
2024-12-18T00:46:19.0066172Z simplejson==3.19.3
2024-12-18T00:46:19.0066403Z six==1.17.0
2024-12-18T00:46:19.0067033Z smart-open==7.1.0
2024-12-18T00:46:19.0067281Z smmap==5.0.1
2024-12-18T00:46:19.0067518Z sortedcontainers==2.4.0
2024-12-18T00:46:19.0067783Z soundfile==0.12.1
2024-12-18T00:46:19.0068011Z soupsieve==2.6
2024-12-18T00:46:19.0068236Z soxr==0.5.0.post1
2024-12-18T00:46:19.0068460Z spacy==3.8.3
2024-12-18T00:46:19.0068692Z spacy-legacy==3.0.12
2024-12-18T00:46:19.0068945Z spacy-loggers==1.0.5
2024-12-18T00:46:19.0069183Z srsly==2.5.0
2024-12-18T00:46:19.0069401Z stempeg==0.2.3
2024-12-18T00:46:19.0069626Z submitit==1.5.2
2024-12-18T00:46:19.0069850Z sympy==1.13.1
2024-12-18T00:46:19.0070070Z tabulate==0.9.0
2024-12-18T00:46:19.0070299Z tb-nightly==2.13.0a20230426
2024-12-18T00:46:19.0070575Z tensorboard==2.13.0
2024-12-18T00:46:19.0070836Z tensorboard-data-server==0.7.2
2024-12-18T00:46:19.0071179Z tensorboardX==2.6.2.2
2024-12-18T00:46:19.0071424Z termcolor==2.5.0
2024-12-18T00:46:19.0071657Z thinc==8.3.3
2024-12-18T00:46:19.0071889Z threadpoolctl==3.5.0
2024-12-18T00:46:19.0072139Z thriftpy2==0.5.2
2024-12-18T00:46:19.0081797Z tifffile==2024.9.20
2024-12-18T00:46:19.0082361Z timm @ git+https://github.com/huggingface/pytorch-image-models.git@730b907b4d45a4713cbc425cbf224c46089fd514
2024-12-18T00:46:19.0082969Z tlparse==0.3.25
2024-12-18T00:46:19.0083217Z tokenizers==0.15.2
2024-12-18T00:46:19.0083659Z tomli==2.2.1
2024-12-18T00:46:19.0083894Z tomlkit==0.13.2
2024-12-18T00:46:19.0084709Z torch @ file:///var/lib/jenkins/workspace/dist/torch-2.6.0a0%2Bgit0cdf8b1-cp310-cp310-linux_x86_64.whl#sha256=dcadd2cd6f8025bfa9997113da51bf28d112b9db4fb82c725ed7dd046c33c456
2024-12-18T00:46:19.0085872Z torch_geometric @ git+https://github.com/pyg-team/pytorch_geometric.git@cabcd4097442ba60aa1efa11e1619dd9bb8fb527
2024-12-18T00:46:19.0086668Z torchao @ git+https://github.com/pytorch/ao.git@a5a53a2bd00f18401baaa00db4467f955704f71f
2024-12-18T00:46:19.0087390Z torchaudio @ git+https://github.com/pytorch/audio.git@332760d4b300f00a0d862e3cfe1495db3b1a14f9
2024-12-18T00:46:19.0088229Z torchmultimodal @ git+https://github.com/facebookresearch/multimodal.git@6569fcc03450c2360b50d772bf9b18ec3487fcf4
2024-12-18T00:46:19.0089071Z torchvision @ git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-12-18T00:46:19.0089744Z tornado==6.4.2
2024-12-18T00:46:19.0089979Z tqdm==4.67.1
2024-12-18T00:46:19.0090222Z transformers==4.38.1
2024-12-18T00:46:19.0090480Z treetable==0.2.5
2024-12-18T00:46:19.0091043Z -e git+https://github.com/triton-lang/triton@35c6c7c6284582b3f41c71c150e11b517acf074a#egg=triton&subdirectory=python
2024-12-18T00:46:19.0091676Z typeguard==4.4.1
2024-12-18T00:46:19.0091920Z typer==0.15.1
2024-12-18T00:46:19.0092168Z typing_extensions==4.12.2
2024-12-18T00:46:19.0092444Z tzdata==2024.2
2024-12-18T00:46:19.0092673Z Unidecode==1.3.8
2024-12-18T00:46:19.0092939Z unittest-xml-reporting==3.2.0
2024-12-18T00:46:19.0093242Z uritemplate==4.1.1
2024-12-18T00:46:19.0093488Z urllib3==2.2.3
2024-12-18T00:46:19.0093721Z visdom==0.2.4
2024-12-18T00:46:19.0093939Z wandb==0.19.1
2024-12-18T00:46:19.0094166Z wasabi==1.1.3
2024-12-18T00:46:19.0094402Z wcwidth==0.2.13
2024-12-18T00:46:19.0094637Z weasel==0.4.1
2024-12-18T00:46:19.0094872Z websocket-client==1.8.0
2024-12-18T00:46:19.0095149Z Werkzeug==3.1.3
2024-12-18T00:46:19.0095386Z wrapt==1.17.0
2024-12-18T00:46:19.0095615Z xdoctest==1.1.0
2024-12-18T00:46:19.0095852Z xxhash==3.5.0
2024-12-18T00:46:19.0096085Z xyzservices==2024.9.0
2024-12-18T00:46:19.0096337Z yacs==0.1.8
2024-12-18T00:46:19.0096552Z yarl==1.18.3
2024-12-18T00:46:19.0096777Z z3-solver==4.12.2.0
2024-12-18T00:46:19.0097016Z zipp==3.21.0
2024-12-18T00:46:19.0307371Z + popd
2024-12-18T00:46:19.0307595Z ~/workspace
2024-12-18T00:46:19.0307832Z + [[ inductor_torchbench != *cpu* ]]
2024-12-18T00:46:19.0308148Z + install_torchrec_and_fbgemm
2024-12-18T00:46:19.0313847Z + local torchrec_commit
2024-12-18T00:46:19.0318478Z ++ get_pinned_commit torchrec
2024-12-18T00:46:19.0318870Z ++ cat .github/ci_commit_pins/torchrec.txt
2024-12-18T00:46:19.0338757Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-12-18T00:46:19.0339141Z + local fbgemm_commit
2024-12-18T00:46:19.0342654Z ++ get_pinned_commit fbgemm
2024-12-18T00:46:19.0343035Z ++ cat .github/ci_commit_pins/fbgemm.txt
2024-12-18T00:46:19.0361008Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd
2024-12-18T00:46:19.0361547Z + pip_uninstall torchrec-nightly
2024-12-18T00:46:19.0361871Z + pip3 uninstall -y torchrec-nightly
2024-12-18T00:46:19.5375554Z WARNING: Skipping torchrec-nightly as it is not installed.
2024-12-18T00:46:19.5376121Z + pip_uninstall fbgemm-gpu-nightly
2024-12-18T00:46:19.5376466Z + pip3 uninstall -y fbgemm-gpu-nightly
2024-12-18T00:46:19.8180895Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed.
2024-12-18T00:46:19.8535628Z + pip_install setuptools-git-versioning scikit-build pyre-extensions
2024-12-18T00:46:19.8536416Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T00:46:19.8537227Z + python3 -m pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions
2024-12-18T00:46:20.3541641Z Collecting setuptools-git-versioning
2024-12-18T00:46:20.3709328Z   Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl.metadata (5.8 kB)
2024-12-18T00:46:20.4998695Z Collecting scikit-build
2024-12-18T00:46:20.5038077Z   Downloading scikit_build-0.18.1-py3-none-any.whl.metadata (18 kB)
2024-12-18T00:46:20.6077639Z Collecting pyre-extensions
2024-12-18T00:46:20.6108844Z   Downloading pyre_extensions-0.0.32-py3-none-any.whl.metadata (4.0 kB)
2024-12-18T00:46:20.6590763Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.2)
2024-12-18T00:46:20.6594105Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (75.1.0)
2024-12-18T00:46:20.7064062Z Collecting toml>=0.10.2 (from setuptools-git-versioning)
2024-12-18T00:46:20.7097459Z   Downloading toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB)
2024-12-18T00:46:20.7616662Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.9.0)
2024-12-18T00:46:20.7622533Z Requirement already satisfied: tomli in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (2.2.1)
2024-12-18T00:46:20.7626401Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.44.0)
2024-12-18T00:46:20.8170155Z Collecting typing-inspect (from pyre-extensions)
2024-12-18T00:46:20.8201064Z   Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB)
2024-12-18T00:46:20.8720615Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.12.2)
2024-12-18T00:46:20.8805977Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0)
2024-12-18T00:46:20.8846707Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl (10 kB)
2024-12-18T00:46:20.9378261Z Downloading scikit_build-0.18.1-py3-none-any.whl (85 kB)
2024-12-18T00:46:20.9890170Z Downloading pyre_extensions-0.0.32-py3-none-any.whl (12 kB)
2024-12-18T00:46:21.0349604Z Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)
2024-12-18T00:46:21.0876511Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB)
2024-12-18T00:46:21.9771587Z Installing collected packages: typing-inspect, toml, scikit-build, setuptools-git-versioning, pyre-extensions
2024-12-18T00:46:22.3262723Z Successfully installed pyre-extensions-0.0.32 scikit-build-0.18.1 setuptools-git-versioning-2.0.0 toml-0.10.2 typing-inspect-0.9.0
2024-12-18T00:46:22.4353280Z + [[ 0 == \1 ]]
2024-12-18T00:46:22.4353673Z + CUDA_PATH=/usr/local/cuda-12.1
2024-12-18T00:46:22.4354484Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu'
2024-12-18T00:46:22.4355364Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T00:46:22.4356340Z + python3 -m pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu'
2024-12-18T00:46:22.8037029Z Collecting fbgemm-gpu
2024-12-18T00:46:22.8039857Z   Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-t1g5yrs2/fbgemm-gpu_41e5868bc79b4bf4b1331f6f12722e56
2024-12-18T00:46:22.8070734Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-t1g5yrs2/fbgemm-gpu_41e5868bc79b4bf4b1331f6f12722e56
2024-12-18T00:46:23.9735162Z   Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd'
2024-12-18T00:46:23.9764563Z   Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd
2024-12-18T00:46:24.5994863Z   Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd
2024-12-18T00:46:25.1391657Z   Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd
2024-12-18T00:46:25.1392611Z   Running command git submodule update --init --recursive -q
2024-12-18T00:46:34.6803983Z   Preparing metadata (setup.py) ... [?25l- done
2024-12-18T00:46:34.6823315Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.21.2)
2024-12-18T00:46:34.6830416Z Building wheels for collected packages: fbgemm-gpu
2024-12-18T01:30:02.8891164Z   Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done
2024-12-18T01:30:03.0739572Z [?25h  Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=253176194 sha256=427f8fcf46dab93c67b777012bb0a92a33ffbad0dbca136de7bb432b67056c3c
2024-12-18T01:30:03.0740848Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39
2024-12-18T01:30:03.0761373Z Successfully built fbgemm-gpu
2024-12-18T01:30:03.9344299Z Installing collected packages: fbgemm-gpu
2024-12-18T01:30:07.2387842Z Successfully installed fbgemm-gpu-0.4.1rc0.post421
2024-12-18T01:30:08.1609336Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-12-18T01:30:08.1610074Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2024-12-18T01:30:08.1610891Z + python3 -m pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-12-18T01:30:08.5318917Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-12-18T01:30:08.5324086Z   Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-vl1rfahl
2024-12-18T01:30:08.5354021Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-vl1rfahl
2024-12-18T01:30:09.8380341Z   Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2'
2024-12-18T01:30:09.8408524Z   Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-12-18T01:30:10.4167674Z   Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-12-18T01:30:11.0460309Z   Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-12-18T01:30:11.3200010Z   Preparing metadata (setup.py) ... [?25l- done
2024-12-18T01:30:11.3236403Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9)
2024-12-18T01:30:11.3239377Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.0.3)
2024-12-18T01:30:11.3243620Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0)
2024-12-18T01:30:11.3794407Z Collecting torchmetrics (from torchrec==0.3.2)
2024-12-18T01:30:11.4000560Z   Downloading torchmetrics-1.6.0-py3-none-any.whl.metadata (20 kB)
2024-12-18T01:30:11.4483712Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.67.1)
2024-12-18T01:30:11.4486529Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421)
2024-12-18T01:30:11.4496897Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.21.2)
2024-12-18T01:30:11.4505936Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (3.0.0)
2024-12-18T01:30:11.4778608Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0)
2024-12-18T01:30:11.4782073Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.2)
2024-12-18T01:30:11.4785749Z Requirement already satisfied: tzdata>=2022.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.2)
2024-12-18T01:30:11.4808300Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.2)
2024-12-18T01:30:11.4812030Z Requirement already satisfied: torch>=2.0.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.6.0a0+git0cdf8b1)
2024-12-18T01:30:11.5002034Z Collecting lightning-utilities>=0.8.0 (from torchmetrics->torchrec==0.3.2)
2024-12-18T01:30:11.5036661Z   Downloading lightning_utilities-0.11.9-py3-none-any.whl.metadata (5.2 kB)
2024-12-18T01:30:11.5144376Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (75.1.0)
2024-12-18T01:30:11.5146128Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.12.2)
2024-12-18T01:30:11.5165503Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.17.0)
2024-12-18T01:30:11.5199040Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=2.0.0->torchmetrics->torchrec==0.3.2) (3.16.1)
2024-12-18T01:30:11.5202012Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=2.0.0->torchmetrics->torchrec==0.3.2) (2.8.8)
2024-12-18T01:30:11.5204747Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=2.0.0->torchmetrics->torchrec==0.3.2) (3.1.4)
2024-12-18T01:30:11.5206340Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=2.0.0->torchmetrics->torchrec==0.3.2) (2024.9.0)
2024-12-18T01:30:11.5211556Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=2.0.0->torchmetrics->torchrec==0.3.2) (1.13.1)
2024-12-18T01:30:11.5227001Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch>=2.0.0->torchmetrics->torchrec==0.3.2) (1.3.0)
2024-12-18T01:30:11.5633310Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=2.0.0->torchmetrics->torchrec==0.3.2) (3.0.2)
2024-12-18T01:30:11.5837227Z Downloading torchmetrics-1.6.0-py3-none-any.whl (926 kB)
2024-12-18T01:30:11.6102331Z Downloading lightning_utilities-0.11.9-py3-none-any.whl (28 kB)
2024-12-18T01:30:11.6228866Z Building wheels for collected packages: torchrec
2024-12-18T01:30:12.2047015Z   Building wheel for torchrec (setup.py) ... [?25l- \ | / done
2024-12-18T01:30:12.2056448Z [?25h  Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374487 sha256=c1cd62c32c806ba71c8d788ecffcdaffcae140fd989c6cb3cb5166afb218b27b
2024-12-18T01:30:12.2057644Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8
2024-12-18T01:30:12.2080970Z Successfully built torchrec
2024-12-18T01:30:13.0103300Z Installing collected packages: lightning-utilities, torchmetrics, torchrec
2024-12-18T01:30:13.6574327Z Successfully installed lightning-utilities-0.11.9 torchmetrics-1.6.0 torchrec-0.3.2
2024-12-18T01:30:13.8016328Z + [[ 0 == \1 ]]
2024-12-18T01:30:13.8020857Z ++ pwd
2024-12-18T01:30:13.8024224Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench
2024-12-18T01:30:13.8024608Z + test_dynamo_benchmark torchbench 1
2024-12-18T01:30:13.8038806Z ++ pwd
2024-12-18T01:30:13.8042910Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2024-12-18T01:30:13.8043345Z + local suite=torchbench
2024-12-18T01:30:13.8043599Z + shift
2024-12-18T01:30:13.8043815Z + local shard_id=1
2024-12-18T01:30:13.8044053Z + shift
2024-12-18T01:30:13.8044300Z + [[ inductor_torchbench == *perf_compare* ]]
2024-12-18T01:30:13.8044648Z + [[ inductor_torchbench == *perf* ]]
2024-12-18T01:30:13.8044961Z + [[ inductor_torchbench == *cpu* ]]
2024-12-18T01:30:13.8045295Z + [[ inductor_torchbench == *aot_inductor* ]]
2024-12-18T01:30:13.8045829Z + test_single_dynamo_benchmark inference torchbench 1 --inference --bfloat16
2024-12-18T01:30:13.8064764Z ++ pwd
2024-12-18T01:30:13.8068277Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2024-12-18T01:30:13.8068770Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports
2024-12-18T01:30:13.8118386Z + local name=inference
2024-12-18T01:30:13.8119235Z + shift
2024-12-18T01:30:13.8119508Z + local suite=torchbench
2024-12-18T01:30:13.8119784Z + shift
2024-12-18T01:30:13.8119988Z + local shard_id=1
2024-12-18T01:30:13.8120224Z + shift
2024-12-18T01:30:13.8120441Z + partition_flags=()
2024-12-18T01:30:13.8120708Z + local partition_flags
2024-12-18T01:30:13.8120965Z + [[ -n 2 ]]
2024-12-18T01:30:13.8121176Z + [[ -n 1 ]]
2024-12-18T01:30:13.8121574Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id")
2024-12-18T01:30:13.8122088Z + [[ inductor_torchbench == *perf_compare* ]]
2024-12-18T01:30:13.8122432Z + [[ inductor_torchbench == *perf* ]]
2024-12-18T01:30:13.8122780Z + [[ inductor_torchbench == *_avx2* ]]
2024-12-18T01:30:13.8123096Z + [[ inductor_torchbench == *_avx512* ]]
2024-12-18T01:30:13.8124176Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --inductor --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv
2024-12-18T01:30:20.8796012Z 
2024-12-18T01:30:21.9237502Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:30:21.9238122Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:30:21.9271907Z cuda eval  llama                              
2024-12-18T01:30:48.6460313Z pass
2024-12-18T01:30:48.6706843Z TIMING: _recursive_pre_grad_passes:0.03282 _recursive_joint_graph_passes:1.33529 _recursive_post_grad_passes:0.63058 async_compile.wait:2.04003 code_gen:4.61146 inductor_compile:7.56879 backend_compile:15.34419 entire_frame_compile:24.79104 total_wall_time:24.79104
2024-12-18T01:30:48.6708366Z STATS: call_* op count: 1058 | FakeTensorMode.__torch_dispatch__:26618 | FakeTensor.__torch_dispatch__:2433 | ProxyTorchDispatchMode.__torch_dispatch__:10347
2024-12-18T01:30:48.6709507Z Dynamo produced 1 graphs covering 1058 ops with 0 graph breaks (0 unique)
2024-12-18T01:30:53.5624980Z 
2024-12-18T01:30:53.9295584Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:30:53.9295991Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:30:53.9296320Z cuda eval  llama_v2_7b_16h                    
2024-12-18T01:30:53.9327426Z Traceback (most recent call last):
2024-12-18T01:30:53.9327958Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run
2024-12-18T01:30:53.9328463Z     ) = runner.load_model(
2024-12-18T01:30:53.9328958Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 319, in load_model
2024-12-18T01:30:53.9329498Z     benchmark = benchmark_cls(
2024-12-18T01:30:53.9330031Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__
2024-12-18T01:30:53.9331180Z     obj = type.__call__(cls, *args, **kwargs)
2024-12-18T01:30:53.9332093Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h/__init__.py", line 12, in __init__
2024-12-18T01:30:53.9332972Z     super().__init__(name="llama_v2_7b_16h", test=test, device=device, batch_size=batch_size, extra_args=extra_args)
2024-12-18T01:30:53.9333912Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__
2024-12-18T01:30:53.9334806Z     super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args)
2024-12-18T01:30:53.9335529Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__
2024-12-18T01:30:53.9336082Z     self._skip_by_device_name()
2024-12-18T01:30:53.9336680Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name
2024-12-18T01:30:53.9337600Z     raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.")
2024-12-18T01:30:53.9338448Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llama_v2_7b_16h/metadata.yaml`.
2024-12-18T01:30:53.9338891Z 
2024-12-18T01:30:53.9338994Z model_fail_to_load
2024-12-18T01:30:58.8238576Z 
2024-12-18T01:30:59.1854230Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:30:59.1855114Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:30:59.1855971Z cuda eval  llava                              
2024-12-18T01:30:59.1860013Z Traceback (most recent call last):
2024-12-18T01:30:59.1860703Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run
2024-12-18T01:30:59.1861316Z     ) = runner.load_model(
2024-12-18T01:30:59.1861863Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 319, in load_model
2024-12-18T01:30:59.1862390Z     benchmark = benchmark_cls(
2024-12-18T01:30:59.1862921Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__
2024-12-18T01:30:59.1863521Z     obj = type.__call__(cls, *args, **kwargs)
2024-12-18T01:30:59.1864152Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava/__init__.py", line 11, in __init__
2024-12-18T01:30:59.1864974Z     super().__init__(name="llava", test=test, device=device, batch_size=batch_size, extra_args=extra_args)
2024-12-18T01:30:59.1865891Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__
2024-12-18T01:30:59.1867004Z     super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args)
2024-12-18T01:30:59.1867730Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__
2024-12-18T01:30:59.1868293Z     self._skip_by_device_name()
2024-12-18T01:30:59.1868882Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name
2024-12-18T01:30:59.1869783Z     raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.")
2024-12-18T01:30:59.1870828Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llava/metadata.yaml`.
2024-12-18T01:30:59.1871254Z 
2024-12-18T01:30:59.1871352Z model_fail_to_load
2024-12-18T01:31:03.0864623Z 
2024-12-18T01:31:04.0259644Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:31:04.0260105Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:31:04.0266129Z cuda eval  maml                               
2024-12-18T01:31:04.0504225Z pass_due_to_skip
2024-12-18T01:31:04.0514572Z TIMING: total_wall_time:0
2024-12-18T01:31:04.0515046Z STATS: call_* op count: 0
2024-12-18T01:31:04.0515496Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-12-18T01:31:08.0584377Z 
2024-12-18T01:31:08.7731102Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:31:08.7731472Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:31:08.7737073Z cuda eval  maml_omniglot                      
2024-12-18T01:31:13.1051270Z pass
2024-12-18T01:31:13.1052395Z TIMING: _recursive_pre_grad_passes:0.00429 _recursive_joint_graph_passes:0.35728 _recursive_post_grad_passes:0.175 async_compile.wait:0.54757 code_gen:0.86882 inductor_compile:1.26754 backend_compile:2.57568 entire_frame_compile:2.97796 total_wall_time:2.97796
2024-12-18T01:31:13.1053876Z STATS: call_* op count: 28 | FakeTensorMode.__torch_dispatch__:1596 | ProxyTorchDispatchMode.__torch_dispatch__:543 | FakeTensor.__torch_dispatch__:176
2024-12-18T01:31:13.1054708Z Dynamo produced 1 graphs covering 28 ops with 0 graph breaks (0 unique)
2024-12-18T01:31:17.4456879Z 
2024-12-18T01:31:17.7751390Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/mnasnet1.0_top1_73.512-f206786ef8.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/mnasnet1.0_top1_73.512-f206786ef8.pth
2024-12-18T01:31:17.7942294Z 
2024-12-18T01:31:17.7942425Z 
2024-12-18T01:31:17.8944020Z   0% 0.00/16.9M [00:00<?, ?B/s]
2024-12-18T01:31:17.8944289Z 
2024-12-18T01:31:17.9305187Z  75% 12.8M/16.9M [00:00<00:00, 134MB/s]
2024-12-18T01:31:17.9305506Z 100% 16.9M/16.9M [00:00<00:00, 130MB/s]
2024-12-18T01:31:18.6976799Z 
2024-12-18T01:31:18.6977732Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:31:18.7039560Z cuda eval  mnasnet1_0                         
2024-12-18T01:31:34.9946135Z pass
2024-12-18T01:31:34.9957992Z TIMING: _recursive_pre_grad_passes:0.01703 _recursive_joint_graph_passes:0.40089 _recursive_post_grad_passes:0.21948 async_compile.wait:0.42295 code_gen:1.75091 inductor_compile:4.17464 backend_compile:9.02951 entire_frame_compile:13.35923 total_wall_time:13.35923
2024-12-18T01:31:34.9960410Z STATS: call_* op count: 304 | FakeTensorMode.__torch_dispatch__:14684 | ProxyTorchDispatchMode.__torch_dispatch__:6817 | FakeTensor.__torch_dispatch__:2625
2024-12-18T01:31:34.9961632Z Dynamo produced 1 graphs covering 304 ops with 0 graph breaks (0 unique)
2024-12-18T01:31:39.8381501Z 
2024-12-18T01:31:40.9306364Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:31:40.9307023Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:31:40.9366512Z cuda eval  mobilenet_v2                       
2024-12-18T01:31:56.9344357Z pass
2024-12-18T01:31:56.9355636Z TIMING: _recursive_pre_grad_passes:0.01758 _recursive_joint_graph_passes:0.20514 _recursive_post_grad_passes:0.22806 async_compile.wait:0.31592 code_gen:1.63435 inductor_compile:4.12317 backend_compile:9.16747 entire_frame_compile:13.67223 total_wall_time:13.67223
2024-12-18T01:31:56.9357850Z STATS: call_* op count: 306 | FakeTensorMode.__torch_dispatch__:15200 | FakeTensor.__torch_dispatch__:2650 | ProxyTorchDispatchMode.__torch_dispatch__:7745
2024-12-18T01:31:56.9359079Z Dynamo produced 1 graphs covering 306 ops with 0 graph breaks (0 unique)
2024-12-18T01:32:01.7207632Z 
2024-12-18T01:32:01.9868212Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:32:01.9868682Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:32:01.9869109Z cuda eval  mobilenet_v2_quantized_qat         
2024-12-18T01:32:01.9875581Z Traceback (most recent call last):
2024-12-18T01:32:01.9876275Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run
2024-12-18T01:32:01.9877836Z     ) = runner.load_model(
2024-12-18T01:32:01.9878338Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 319, in load_model
2024-12-18T01:32:01.9878875Z     benchmark = benchmark_cls(
2024-12-18T01:32:01.9879407Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__
2024-12-18T01:32:01.9879981Z     obj = type.__call__(cls, *args, **kwargs)
2024-12-18T01:32:01.9880715Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat/__init__.py", line 21, in __init__
2024-12-18T01:32:01.9881516Z     raise NotImplementedError("The eval test only supports CPU.")
2024-12-18T01:32:01.9881989Z NotImplementedError: The eval test only supports CPU.
2024-12-18T01:32:01.9882265Z 
2024-12-18T01:32:01.9882363Z model_fail_to_load
2024-12-18T01:32:06.8808511Z 
2024-12-18T01:32:08.1398102Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:32:08.1398569Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:32:08.1460958Z cuda eval  mobilenet_v3_large                 
2024-12-18T01:32:27.0808284Z pass
2024-12-18T01:32:27.0819934Z TIMING: _recursive_pre_grad_passes:0.01844 _recursive_joint_graph_passes:0.64834 _recursive_post_grad_passes:0.26152 async_compile.wait:0.38664 code_gen:2.81855 inductor_compile:5.62004 backend_compile:11.2871 entire_frame_compile:16.18672 total_wall_time:16.18672
2024-12-18T01:32:27.0822259Z STATS: call_* op count: 374 | FakeTensorMode.__torch_dispatch__:15988 | FakeTensor.__torch_dispatch__:2828 | ProxyTorchDispatchMode.__torch_dispatch__:7903
2024-12-18T01:32:27.0823484Z Dynamo produced 1 graphs covering 374 ops with 0 graph breaks (0 unique)
2024-12-18T01:32:31.9265679Z 
2024-12-18T01:32:34.1107508Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:32:34.1108195Z loading model: 0it [00:02, ?it/s]
2024-12-18T01:32:34.1223284Z cuda eval  moco                               
2024-12-18T01:32:37.0574841Z [rank0]:W1218 01:32:37.056000 37059 site-packages/torch/_logging/_internal.py:1089] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T01:32:48.4847383Z [rank0]:W1218 01:32:48.483000 37059 site-packages/torch/_dynamo/backends/distributed.py:88] [2/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance.
2024-12-18T01:32:48.5093096Z [rank0]:W1218 01:32:48.508000 37059 site-packages/torch/_dynamo/backends/distributed.py:105] [2/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes:
2024-12-18T01:32:48.5108832Z [rank0]:W1218 01:32:48.508000 37059 site-packages/torch/_dynamo/backends/distributed.py:105] [2/0_1] ┌─────────┬─────────────┬────────────────────────┐
2024-12-18T01:32:48.5110059Z [rank0]:W1218 01:32:48.508000 37059 site-packages/torch/_dynamo/backends/distributed.py:105] [2/0_1] │   Index │   Extra Ops │   Extra Param Size (b) │
2024-12-18T01:32:48.5111257Z [rank0]:W1218 01:32:48.508000 37059 site-packages/torch/_dynamo/backends/distributed.py:105] [2/0_1] ├─────────┼─────────────┼────────────────────────┤
2024-12-18T01:32:48.5112381Z [rank0]:W1218 01:32:48.508000 37059 site-packages/torch/_dynamo/backends/distributed.py:105] [2/0_1] │       0 │         470 │               44910720 │
2024-12-18T01:32:48.5113522Z [rank0]:W1218 01:32:48.508000 37059 site-packages/torch/_dynamo/backends/distributed.py:105] [2/0_1] └─────────┴─────────────┴────────────────────────┘
2024-12-18T01:33:07.9762730Z skipping cudagraphs due to mutated inputs (161 instances). Found from : 
2024-12-18T01:33:07.9764136Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 130, in forward
2024-12-18T01:33:07.9765784Z     self._momentum_update_key_encoder()  # update the key encoder
2024-12-18T01:33:07.9766995Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2024-12-18T01:33:07.9767637Z     return func(*args, **kwargs)
2024-12-18T01:33:07.9768337Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 50, in _momentum_update_key_encoder
2024-12-18T01:33:07.9769127Z     param_k.mul_(self.m).add_(param_q.mul(1. - self.m))
2024-12-18T01:33:07.9769380Z 
2024-12-18T01:33:08.1033432Z [rank0]:W1218 01:33:08.102000 37059 site-packages/torch/_inductor/utils.py:1543] [3/0_1] DeviceCopy in input program
2024-12-18T01:33:08.1171080Z skipping cudagraphs due to skipping cudagraphs due to cpu device (randperm). Found from : 
2024-12-18T01:33:08.1171980Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 82, in _batch_shuffle_ddp
2024-12-18T01:33:08.1172989Z     idx_shuffle = torch.randperm(batch_size_all).cuda()
2024-12-18T01:33:08.1173257Z 
2024-12-18T01:33:10.7895836Z [rank0]:W1218 01:33:10.788000 37059 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0] Graph break from `Tensor.item()`, consider setting:
2024-12-18T01:33:10.7897047Z [rank0]:W1218 01:33:10.788000 37059 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0]     torch._dynamo.config.capture_scalar_outputs = True
2024-12-18T01:33:10.7898026Z [rank0]:W1218 01:33:10.788000 37059 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0] or:
2024-12-18T01:33:10.7898971Z [rank0]:W1218 01:33:10.788000 37059 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2024-12-18T01:33:10.7900096Z [rank0]:W1218 01:33:10.788000 37059 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0] to include these operations in the captured graph.
2024-12-18T01:33:10.7900923Z [rank0]:W1218 01:33:10.788000 37059 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0] 
2024-12-18T01:33:10.7901799Z [rank0]:W1218 01:33:10.788000 37059 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0] Graph break: from user code at:
2024-12-18T01:33:10.7903146Z [rank0]:W1218 01:33:10.788000 37059 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0]   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 158, in torch_dynamo_resume_in_forward_at_133
2024-12-18T01:33:10.7904510Z [rank0]:W1218 01:33:10.788000 37059 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0]     self._dequeue_and_enqueue(k)
2024-12-18T01:33:10.7905714Z [rank0]:W1218 01:33:10.788000 37059 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2024-12-18T01:33:10.7906910Z [rank0]:W1218 01:33:10.788000 37059 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0]     return func(*args, **kwargs)
2024-12-18T01:33:10.7908187Z [rank0]:W1218 01:33:10.788000 37059 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0]   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 59, in _dequeue_and_enqueue
2024-12-18T01:33:10.7909449Z [rank0]:W1218 01:33:10.788000 37059 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0]     ptr = int(self.queue_ptr)
2024-12-18T01:33:10.7910196Z [rank0]:W1218 01:33:10.788000 37059 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0] 
2024-12-18T01:33:10.7910839Z [rank0]:W1218 01:33:10.788000 37059 site-packages/torch/_dynamo/variables/tensor.py:869] [5/0] 
2024-12-18T01:33:25.5330933Z pass
2024-12-18T01:33:25.5343252Z TIMING: entire_frame_compile:45.75606 _recursive_pre_grad_passes:0.14063 _recursive_joint_graph_passes:0.79143 _recursive_post_grad_passes:0.65332 async_compile.wait:2.2292 code_gen:8.33852 inductor_compile:15.57209 backend_compile:26.30558 total_wall_time:45.75606
2024-12-18T01:33:25.5345035Z STATS: call_* op count: 879 | FakeTensorMode.__torch_dispatch__:31606 | FakeTensor.__torch_dispatch__:7960 | ProxyTorchDispatchMode.__torch_dispatch__:11972
2024-12-18T01:33:25.5345872Z Dynamo produced 7 graphs covering 879 ops with 5 graph breaks (3 unique)
2024-12-18T01:33:26.5709294Z [rank0]:[W1218 01:33:26.503697855 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2024-12-18T01:33:31.7085751Z 
2024-12-18T01:33:32.0781451Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:33:32.0781846Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:33:32.0782165Z cuda eval  moondream                          
2024-12-18T01:33:32.0788939Z Traceback (most recent call last):
2024-12-18T01:33:32.0789642Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run
2024-12-18T01:33:32.0790627Z     ) = runner.load_model(
2024-12-18T01:33:32.0791394Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 319, in load_model
2024-12-18T01:33:32.0792215Z     benchmark = benchmark_cls(
2024-12-18T01:33:32.0792794Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__
2024-12-18T01:33:32.0793370Z     obj = type.__call__(cls, *args, **kwargs)
2024-12-18T01:33:32.0794006Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream/__init__.py", line 11, in __init__
2024-12-18T01:33:32.0794860Z     super().__init__(name="moondream", test=test, device=device, batch_size=batch_size, extra_args=extra_args)
2024-12-18T01:33:32.0795839Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__
2024-12-18T01:33:32.0796708Z     super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args)
2024-12-18T01:33:32.0797422Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__
2024-12-18T01:33:32.0798005Z     self._skip_by_device_name()
2024-12-18T01:33:32.0798603Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name
2024-12-18T01:33:32.0799521Z     raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.")
2024-12-18T01:33:32.0800336Z NotImplementedError: The current device NVIDIA A10G is skipped by its `moondream/metadata.yaml`.
2024-12-18T01:33:32.0800760Z 
2024-12-18T01:33:32.0800867Z model_fail_to_load
2024-12-18T01:33:36.9795896Z 
2024-12-18T01:33:39.0020948Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M
2024-12-18T01:33:39.2712670Z num decayed parameter tensors: 50, with 124,354,560 parameters
2024-12-18T01:33:39.2714467Z num non-decayed parameter tensors: 98, with 121,344 parameters
2024-12-18T01:33:39.2715595Z using fused AdamW: True
2024-12-18T01:33:39.6690953Z 
2024-12-18T01:33:39.6691227Z loading model: 0it [00:02, ?it/s]
2024-12-18T01:33:39.6727526Z cuda eval  nanogpt                            
2024-12-18T01:33:52.3068879Z pass
2024-12-18T01:33:52.3090132Z TIMING: _recursive_pre_grad_passes:0.0214 _recursive_joint_graph_passes:0.599 _recursive_post_grad_passes:0.36473 async_compile.wait:0.74529 code_gen:1.92983 inductor_compile:3.89195 backend_compile:8.29563 entire_frame_compile:10.83755 total_wall_time:10.83755
2024-12-18T01:33:52.3091872Z STATS: call_* op count: 786 | FakeTensorMode.__torch_dispatch__:18186 | FakeTensor.__torch_dispatch__:1773 | ProxyTorchDispatchMode.__torch_dispatch__:8281
2024-12-18T01:33:52.3092706Z Dynamo produced 1 graphs covering 786 ops with 0 graph breaks (0 unique)
2024-12-18T01:33:56.9936679Z 
2024-12-18T01:33:59.6148102Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:33:59.6148616Z loading model: 0it [00:02, ?it/s]
2024-12-18T01:33:59.6151812Z cuda eval  nvidia_deeprecommender             
2024-12-18T01:34:04.3353480Z pass
2024-12-18T01:34:04.3393601Z TIMING: _recursive_pre_grad_passes:0.00399 _recursive_joint_graph_passes:1.3748 _recursive_post_grad_passes:0.21669 async_compile.wait:0.18045 code_gen:0.33116 inductor_compile:0.71004 backend_compile:2.93229 entire_frame_compile:3.28383 total_wall_time:3.28383
2024-12-18T01:34:04.3395088Z STATS: call_* op count: 26 | FakeTensorMode.__torch_dispatch__:1356 | ProxyTorchDispatchMode.__torch_dispatch__:579 | FakeTensor.__torch_dispatch__:170
2024-12-18T01:34:04.3395914Z Dynamo produced 1 graphs covering 26 ops with 0 graph breaks (0 unique)
2024-12-18T01:34:08.6585086Z 
2024-12-18T01:34:09.7998340Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:34:09.7998793Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:34:09.8016667Z cuda eval  opacus_cifar10                     
2024-12-18T01:34:19.4375907Z pass
2024-12-18T01:34:19.4377627Z TIMING: _recursive_pre_grad_passes:0.00771 _recursive_joint_graph_passes:0.40274 _recursive_post_grad_passes:0.08025 async_compile.wait:0.33196 code_gen:2.02322 inductor_compile:3.06841 backend_compile:5.51073 entire_frame_compile:8.07512 total_wall_time:8.07512
2024-12-18T01:34:19.4379455Z STATS: call_* op count: 138 | FakeTensorMode.__torch_dispatch__:4942 | ProxyTorchDispatchMode.__torch_dispatch__:2467 | FakeTensor.__torch_dispatch__:866
2024-12-18T01:34:19.4380296Z Dynamo produced 1 graphs covering 138 ops with 0 graph breaks (0 unique)
2024-12-18T01:34:24.0093469Z 
2024-12-18T01:34:24.8604288Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:34:24.8604929Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:34:24.8664214Z cuda eval  phlippe_densenet                   
2024-12-18T01:34:43.2652772Z pass
2024-12-18T01:34:43.2654637Z TIMING: _recursive_pre_grad_passes:0.01902 _recursive_joint_graph_passes:0.41323 _recursive_post_grad_passes:0.23077 async_compile.wait:0.71656 code_gen:4.26898 inductor_compile:6.72417 backend_compile:11.89212 entire_frame_compile:16.71785 total_wall_time:16.71785
2024-12-18T01:34:43.2656160Z STATS: call_* op count: 372 | FakeTensorMode.__torch_dispatch__:16129 | ProxyTorchDispatchMode.__torch_dispatch__:7380 | FakeTensor.__torch_dispatch__:2741
2024-12-18T01:34:43.2657034Z Dynamo produced 1 graphs covering 372 ops with 0 graph breaks (0 unique)
2024-12-18T01:34:48.0927861Z 
2024-12-18T01:34:48.8500521Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:34:48.8501248Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:34:48.8525760Z cuda eval  phlippe_resnet                     
2024-12-18T01:34:56.3126746Z pass
2024-12-18T01:34:56.3128506Z TIMING: _recursive_pre_grad_passes:0.00885 _recursive_joint_graph_passes:0.52722 _recursive_post_grad_passes:0.09444 async_compile.wait:0.29031 code_gen:1.03747 inductor_compile:1.95173 backend_compile:4.38468 entire_frame_compile:6.01821 total_wall_time:6.01821
2024-12-18T01:34:56.3129983Z STATS: call_* op count: 142 | FakeTensorMode.__torch_dispatch__:6009 | ProxyTorchDispatchMode.__torch_dispatch__:2780 | FakeTensor.__torch_dispatch__:1018
2024-12-18T01:34:56.3130813Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique)
2024-12-18T01:35:00.8019758Z 
2024-12-18T01:35:01.2566160Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:35:01.2566547Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:35:01.2567163Z cuda eval  pyhpc_equation_of_state            
2024-12-18T01:35:07.4777781Z pass
2024-12-18T01:35:07.4778857Z TIMING: _recursive_pre_grad_passes:0.01615 _recursive_joint_graph_passes:0.28643 _recursive_post_grad_passes:0.09694 async_compile.wait:0.30384 code_gen:1.49462 inductor_compile:2.25914 backend_compile:4.32165 entire_frame_compile:4.92008 total_wall_time:4.92008
2024-12-18T01:35:07.4780360Z STATS: call_* op count: 732 | FakeTensorMode.__torch_dispatch__:8848 | ProxyTorchDispatchMode.__torch_dispatch__:3745 | FakeTensor.__torch_dispatch__:1061
2024-12-18T01:35:07.4781213Z Dynamo produced 1 graphs covering 732 ops with 0 graph breaks (0 unique)
2024-12-18T01:35:11.8621040Z 
2024-12-18T01:35:12.3082064Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:35:12.3082442Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:35:12.3085820Z cuda eval  pyhpc_isoneutral_mixing            
2024-12-18T01:35:33.8277460Z skipping cudagraphs due to mutated inputs (7 instances). Found from : 
2024-12-18T01:35:33.8278231Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2771, in run_n_iterations
2024-12-18T01:35:33.8279105Z     return self.model_iter_fn(mod, inputs, collect_outputs=True)
2024-12-18T01:35:33.8279964Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 444, in forward_pass
2024-12-18T01:35:33.8280517Z     return mod(*inputs)
2024-12-18T01:35:33.8281157Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing/__init__.py", line 96, in forward
2024-12-18T01:35:33.8281894Z     return isoneutral_pytorch.isoneutral_diffusion_pre(
2024-12-18T01:35:33.8282756Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing/isoneutral_pytorch.py", line 165, in isoneutral_diffusion_pre
2024-12-18T01:35:33.8283830Z     K_11[1:-2, 2:-2, :] = sumz / (4.0 * dzt[None, None, :])
2024-12-18T01:35:33.8284072Z 
2024-12-18T01:35:33.8588183Z pass
2024-12-18T01:35:33.8589315Z TIMING: _recursive_pre_grad_passes:0.04108 _recursive_joint_graph_passes:0.39171 _recursive_post_grad_passes:1.67721 async_compile.wait:0.24492 code_gen:3.0438 inductor_compile:9.62721 backend_compile:17.88477 entire_frame_compile:20.04181 total_wall_time:20.04181
2024-12-18T01:35:33.8590809Z STATS: call_* op count: 1488 | FakeTensorMode.__torch_dispatch__:41402 | FakeTensor.__torch_dispatch__:5051 | ProxyTorchDispatchMode.__torch_dispatch__:14005
2024-12-18T01:35:33.8591656Z Dynamo produced 1 graphs covering 1488 ops with 0 graph breaks (0 unique)
2024-12-18T01:35:38.9643452Z 
2024-12-18T01:35:39.8344850Z loading model: 0it [00:00, ?it/s]WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead
2024-12-18T01:35:39.9551989Z 
2024-12-18T01:35:39.9552865Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:35:39.9553506Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead
2024-12-18T01:35:39.9554102Z cuda eval  pyhpc_turbulent_kinetic_energy     
2024-12-18T01:35:40.0011816Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead
2024-12-18T01:36:05.5014335Z pass
2024-12-18T01:36:05.5074282Z TIMING: _recursive_pre_grad_passes:0.04926 _recursive_joint_graph_passes:0.25532 _recursive_post_grad_passes:0.59566 async_compile.wait:1.4083 code_gen:8.65724 inductor_compile:13.12577 backend_compile:21.55125 entire_frame_compile:23.97329 total_wall_time:23.97329
2024-12-18T01:36:05.5075847Z STATS: call_* op count: 1992 | FakeTensorMode.__torch_dispatch__:38441 | FakeTensor.__torch_dispatch__:2563 | ProxyTorchDispatchMode.__torch_dispatch__:14635
2024-12-18T01:36:05.5076682Z Dynamo produced 1 graphs covering 1992 ops with 0 graph breaks (0 unique)
2024-12-18T01:36:10.6664137Z 
2024-12-18T01:36:11.6079213Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:36:11.6079612Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:36:11.6091255Z cuda eval  pytorch_CycleGAN_and_pix2pix       
2024-12-18T01:36:28.2668580Z pass
2024-12-18T01:36:28.2676016Z TIMING: _recursive_pre_grad_passes:0.00892 _recursive_joint_graph_passes:0.17884 _recursive_post_grad_passes:0.10451 async_compile.wait:0.73002 code_gen:2.67777 inductor_compile:3.98032 backend_compile:6.47887 entire_frame_compile:8.09033 total_wall_time:8.09033
2024-12-18T01:36:28.2677534Z STATS: call_* op count: 182 | FakeTensorMode.__torch_dispatch__:7168 | ProxyTorchDispatchMode.__torch_dispatch__:3743 | FakeTensor.__torch_dispatch__:1038
2024-12-18T01:36:28.2678689Z Dynamo produced 1 graphs covering 182 ops with 0 graph breaks (0 unique)
2024-12-18T01:36:32.8614477Z 
2024-12-18T01:36:34.1575318Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:36:34.1575648Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:36:34.1595727Z cuda eval  pytorch_stargan                    
2024-12-18T01:36:57.7824224Z pass
2024-12-18T01:36:57.7836862Z TIMING: _recursive_pre_grad_passes:0.00801 _recursive_joint_graph_passes:0.25637 _recursive_post_grad_passes:0.09907 async_compile.wait:0.24302 code_gen:1.73711 inductor_compile:2.93719 backend_compile:6.08032 entire_frame_compile:7.66277 total_wall_time:7.66277
2024-12-18T01:36:57.7838610Z STATS: call_* op count: 112 | FakeTensorMode.__torch_dispatch__:10633 | FakeTensor.__torch_dispatch__:1190 | ProxyTorchDispatchMode.__torch_dispatch__:4591
2024-12-18T01:36:57.7839449Z Dynamo produced 1 graphs covering 112 ops with 0 graph breaks (0 unique)
2024-12-18T01:37:02.4407740Z 
2024-12-18T01:37:03.3276531Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:37:03.3276882Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:37:03.3297949Z cuda eval  pytorch_unet                       
2024-12-18T01:37:25.5356397Z pass
2024-12-18T01:37:25.5381264Z TIMING: _recursive_pre_grad_passes:0.00914 _recursive_joint_graph_passes:0.15962 _recursive_post_grad_passes:0.11817 async_compile.wait:1.50184 code_gen:4.02065 inductor_compile:5.30601 backend_compile:8.07944 entire_frame_compile:10.02253 total_wall_time:10.02253
2024-12-18T01:37:25.5384735Z STATS: call_* op count: 142 | FakeTensorMode.__torch_dispatch__:9496 | ProxyTorchDispatchMode.__torch_dispatch__:4337 | FakeTensor.__torch_dispatch__:1309
2024-12-18T01:37:25.5385761Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique)
2024-12-18T01:37:30.1312087Z 
2024-12-18T01:37:31.0404895Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/resnet152-394f9c45.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/resnet152-394f9c45.pth
2024-12-18T01:37:31.0579697Z 
2024-12-18T01:37:31.0579833Z 
2024-12-18T01:37:31.1585227Z   0% 0.00/230M [00:00<?, ?B/s]
2024-12-18T01:37:31.1585464Z 
2024-12-18T01:37:31.2802540Z  17% 38.8M/230M [00:00<00:00, 405MB/s]
2024-12-18T01:37:31.2802779Z 
2024-12-18T01:37:31.3843889Z  34% 77.5M/230M [00:00<00:00, 360MB/s]
2024-12-18T01:37:31.3844109Z 
2024-12-18T01:37:31.4844954Z  50% 115M/230M [00:00<00:00, 368MB/s] 
2024-12-18T01:37:31.4845183Z 
2024-12-18T01:37:31.5889498Z  67% 154M/230M [00:00<00:00, 383MB/s]
2024-12-18T01:37:31.5889730Z 
2024-12-18T01:37:31.6888590Z  83% 191M/230M [00:00<00:00, 378MB/s]
2024-12-18T01:37:31.6888824Z 
2024-12-18T01:37:31.6907311Z 100% 230M/230M [00:00<00:00, 390MB/s]
2024-12-18T01:37:31.6907632Z 100% 230M/230M [00:00<00:00, 382MB/s]
2024-12-18T01:37:32.7410861Z 
2024-12-18T01:37:32.7411312Z loading model: 0it [00:02, ?it/s]
2024-12-18T01:37:32.7588849Z cuda eval  resnet152                          
2024-12-18T01:38:11.2386606Z pass
2024-12-18T01:38:11.2426190Z TIMING: _recursive_pre_grad_passes:0.04821 _recursive_joint_graph_passes:0.54973 _recursive_post_grad_passes:0.66916 async_compile.wait:0.13427 code_gen:3.97532 inductor_compile:10.55074 backend_compile:23.52325 entire_frame_compile:34.7631 total_wall_time:34.7631
2024-12-18T01:38:11.2427696Z STATS: call_* op count: 1030 | FakeTensorMode.__torch_dispatch__:41110 | ProxyTorchDispatchMode.__torch_dispatch__:20286 | FakeTensor.__torch_dispatch__:8053
2024-12-18T01:38:11.2428578Z Dynamo produced 1 graphs covering 1030 ops with 0 graph breaks (0 unique)
2024-12-18T01:38:16.9269948Z 
2024-12-18T01:38:17.3219240Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/resnet18-f37072fd.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/resnet18-f37072fd.pth
2024-12-18T01:38:17.3382152Z 
2024-12-18T01:38:17.3382157Z 
2024-12-18T01:38:17.4386969Z   0% 0.00/44.7M [00:00<?, ?B/s]
2024-12-18T01:38:17.4387298Z 
2024-12-18T01:38:17.4577426Z  84% 37.4M/44.7M [00:00<00:00, 391MB/s]
2024-12-18T01:38:17.4577772Z 100% 44.7M/44.7M [00:00<00:00, 392MB/s]
2024-12-18T01:38:18.2424477Z 
2024-12-18T01:38:18.2424737Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:38:18.2449593Z cuda eval  resnet18                           
2024-12-18T01:38:26.6204629Z pass
2024-12-18T01:38:26.6221879Z TIMING: _recursive_pre_grad_passes:0.00898 _recursive_joint_graph_passes:0.39939 _recursive_post_grad_passes:0.09615 async_compile.wait:0.25237 code_gen:1.3025 inductor_compile:2.23608 backend_compile:4.74426 entire_frame_compile:6.35609 total_wall_time:6.35609
2024-12-18T01:38:26.6223419Z STATS: call_* op count: 138 | FakeTensorMode.__torch_dispatch__:5934 | ProxyTorchDispatchMode.__torch_dispatch__:2707 | FakeTensor.__torch_dispatch__:1046
2024-12-18T01:38:26.6224277Z Dynamo produced 1 graphs covering 138 ops with 0 graph breaks (0 unique)
2024-12-18T01:38:31.1599015Z 
2024-12-18T01:38:31.6921553Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/resnet50-0676ba61.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/resnet50-0676ba61.pth
2024-12-18T01:38:31.7129603Z 
2024-12-18T01:38:31.7130022Z 
2024-12-18T01:38:31.8132132Z   0% 0.00/97.8M [00:00<?, ?B/s]
2024-12-18T01:38:31.8132349Z 
2024-12-18T01:38:31.9131964Z  31% 30.5M/97.8M [00:00<00:00, 320MB/s]
2024-12-18T01:38:31.9132190Z 
2024-12-18T01:38:32.0221352Z  63% 61.9M/97.8M [00:00<00:00, 325MB/s]
2024-12-18T01:38:32.0222132Z 
2024-12-18T01:38:32.0480448Z  95% 92.9M/97.8M [00:00<00:00, 312MB/s]
2024-12-18T01:38:32.0480780Z 100% 97.8M/97.8M [00:00<00:00, 306MB/s]
2024-12-18T01:38:32.9526084Z 
2024-12-18T01:38:32.9526342Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:38:32.9590914Z cuda eval  resnet50                           
2024-12-18T01:38:49.1860187Z pass
2024-12-18T01:38:49.1885692Z TIMING: _recursive_pre_grad_passes:0.01898 _recursive_joint_graph_passes:0.20034 _recursive_post_grad_passes:0.23262 async_compile.wait:0.10391 code_gen:1.70546 inductor_compile:4.20221 backend_compile:9.12445 entire_frame_compile:13.2567 total_wall_time:13.2567
2024-12-18T01:38:49.1887206Z STATS: call_* op count: 350 | FakeTensorMode.__torch_dispatch__:14726 | ProxyTorchDispatchMode.__torch_dispatch__:6992 | FakeTensor.__torch_dispatch__:2749
2024-12-18T01:38:49.1888045Z Dynamo produced 1 graphs covering 350 ops with 0 graph breaks (0 unique)
2024-12-18T01:38:53.9768200Z 
2024-12-18T01:38:54.2438433Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:38:54.2438839Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:38:54.2439172Z cuda eval  resnet50_quantized_qat             
2024-12-18T01:38:54.2445582Z Traceback (most recent call last):
2024-12-18T01:38:54.2446099Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run
2024-12-18T01:38:54.2446613Z     ) = runner.load_model(
2024-12-18T01:38:54.2447099Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 319, in load_model
2024-12-18T01:38:54.2447635Z     benchmark = benchmark_cls(
2024-12-18T01:38:54.2448165Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__
2024-12-18T01:38:54.2448740Z     obj = type.__call__(cls, *args, **kwargs)
2024-12-18T01:38:54.2449436Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat/__init__.py", line 21, in __init__
2024-12-18T01:38:54.2450215Z     raise NotImplementedError("The eval test only supports CPU.")
2024-12-18T01:38:54.2450697Z NotImplementedError: The eval test only supports CPU.
2024-12-18T01:38:54.2450976Z 
2024-12-18T01:38:54.2451080Z model_fail_to_load
2024-12-18T01:38:58.1284735Z 
2024-12-18T01:38:58.6548204Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/resnext50_32x4d-7cdf4587.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/resnext50_32x4d-7cdf4587.pth
2024-12-18T01:38:58.6702746Z 
2024-12-18T01:38:58.6702931Z 
2024-12-18T01:38:58.7703475Z   0% 0.00/95.8M [00:00<?, ?B/s]
2024-12-18T01:38:58.7703719Z 
2024-12-18T01:38:58.8705280Z  39% 37.1M/95.8M [00:00<00:00, 389MB/s]
2024-12-18T01:38:58.8705603Z 
2024-12-18T01:38:58.9059239Z  84% 80.8M/95.8M [00:00<00:00, 429MB/s]
2024-12-18T01:38:58.9059606Z 100% 95.8M/95.8M [00:00<00:00, 426MB/s]
2024-12-18T01:38:59.8792387Z 
2024-12-18T01:38:59.8792776Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:38:59.8853543Z cuda eval  resnext50_32x4d                    
2024-12-18T01:39:16.9037751Z pass
2024-12-18T01:39:16.9067257Z TIMING: _recursive_pre_grad_passes:0.01826 _recursive_joint_graph_passes:0.19524 _recursive_post_grad_passes:0.23149 async_compile.wait:0.25933 code_gen:1.86815 inductor_compile:4.32712 backend_compile:9.09776 entire_frame_compile:13.1859 total_wall_time:13.1859
2024-12-18T01:39:16.9068822Z STATS: call_* op count: 350 | FakeTensorMode.__torch_dispatch__:14651 | ProxyTorchDispatchMode.__torch_dispatch__:6992 | FakeTensor.__torch_dispatch__:2749
2024-12-18T01:39:16.9069672Z Dynamo produced 1 graphs covering 350 ops with 0 graph breaks (0 unique)
2024-12-18T01:39:21.7365987Z 
2024-12-18T01:39:29.4596288Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:39:29.4596663Z loading model: 0it [00:07, ?it/s]
2024-12-18T01:39:29.4708925Z cuda eval  sam                                
2024-12-18T01:41:15.4441805Z pass
2024-12-18T01:41:15.4817613Z TIMING: _recursive_pre_grad_passes:0.11413 _recursive_joint_graph_passes:4.83411 _recursive_post_grad_passes:3.50406 async_compile.wait:4.12118 code_gen:13.50597 inductor_compile:25.73781 backend_compile:54.81117 entire_frame_compile:68.31189 total_wall_time:68.31189
2024-12-18T01:41:15.4819465Z STATS: call_* op count: 4569 | FakeTensorMode.__torch_dispatch__:136253 | FakeTensor.__torch_dispatch__:9504 | ProxyTorchDispatchMode.__torch_dispatch__:55802
2024-12-18T01:41:15.4820308Z Dynamo produced 1 graphs covering 4569 ops with 0 graph breaks (0 unique)
2024-12-18T01:41:22.5306802Z 
2024-12-18T01:43:12.2670944Z loading model: 0it [00:00, ?it/s]E1218 01:43:12.266000 41443 site-packages/torch/_inductor/select_algorithm.py:1477] [0/0] Exception out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. for benchmark choice TritonTemplateCaller(/tmp/tmpiq9zzxaj/4y/c4yhvjgiahofk7lqkbafpmkciqkxp3y6mhrrtwx5qo5im6gb3v6s.py, ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=4, num_warps=4)
2024-12-18T01:43:12.2674717Z E1218 01:43:12.266000 41443 site-packages/torch/_inductor/select_algorithm.py:1477] [0/0] Exception out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. for benchmark choice TritonTemplateCaller(/tmp/tmpiq9zzxaj/6q/c6qzg47yw4dv53zlqiti7r3cbyiuzemov6cdqayxvpxn2jnpok5h.py, ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=4)
2024-12-18T01:43:12.2678858Z E1218 01:43:12.266000 41443 site-packages/torch/_inductor/select_algorithm.py:1477] [0/0] Exception out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. for benchmark choice TritonTemplateCaller(/tmp/tmpiq9zzxaj/ro/croyoqlc3o6reo2a72x7ivxejgwx53dgwoou6bopmxixcnbxrmfq.py, ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=8)
2024-12-18T01:43:13.5029914Z W1218 01:43:13.502000 41443 site-packages/torch/_inductor/select_algorithm.py:1696] [0/0] out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-12-18T01:43:13.9183836Z W1218 01:43:13.917000 41443 site-packages/torch/_inductor/select_algorithm.py:1696] [0/0] out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-12-18T01:43:14.5931399Z W1218 01:43:14.592000 41443 site-packages/torch/_inductor/select_algorithm.py:1696] [0/0] out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-12-18T01:43:14.5936977Z AUTOTUNE mm(4096x1280, 1280x5120)
2024-12-18T01:43:14.5938461Z   triton_mm_90 0.7834 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:14.5940690Z   triton_mm_86 0.7905 ms 99.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:14.5942677Z   triton_mm_93 0.7905 ms 99.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:14.5944451Z   triton_mm_83 0.7936 ms 98.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=2, num_warps=4
2024-12-18T01:43:14.5946163Z   triton_mm_88 0.7956 ms 98.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:14.5947915Z   triton_mm_84 0.8131 ms 96.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=8
2024-12-18T01:43:14.5950002Z   triton_mm_91 0.8141 ms 96.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=4, num_warps=8
2024-12-18T01:43:14.5951862Z   triton_mm_87 0.8212 ms 95.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=4, num_warps=8
2024-12-18T01:43:14.5953617Z   triton_mm_94 0.8315 ms 94.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:14.5954707Z   mm 0.8407 ms 93.2% 
2024-12-18T01:43:14.5955463Z SingleProcess AUTOTUNE benchmarking takes 2.3250 seconds and 0.0025 seconds precompiling for 20 choices
2024-12-18T01:43:17.6034772Z E1218 01:43:17.602000 41443 site-packages/torch/_inductor/select_algorithm.py:1477] [0/0] Exception out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. for benchmark choice TritonTemplateCaller(/tmp/tmpiq9zzxaj/4x/c4xbvt2rrf4xi7nbak2uk2tk5qcbf2r3f4miok4fvlsgjniyl66j.py, ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=8)
2024-12-18T01:43:17.6039304Z E1218 01:43:17.602000 41443 site-packages/torch/_inductor/select_algorithm.py:1477] [0/0] Exception out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. for benchmark choice TritonTemplateCaller(/tmp/tmpiq9zzxaj/ob/cob6k43ujxp6njkc64lj5fhkhmxree6qm4gw45qtwqzm7u4wfkqw.py, ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=4)
2024-12-18T01:43:17.6044016Z E1218 01:43:17.603000 41443 site-packages/torch/_inductor/select_algorithm.py:1477] [0/0] Exception out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. for benchmark choice TritonTemplateCaller(/tmp/tmpiq9zzxaj/is/cisgodjxn7qberaybuhoxnraigyren4iwrjjchqajaekrxxvykae.py, ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=4, num_warps=4)
2024-12-18T01:43:18.8493288Z W1218 01:43:18.848000 41443 site-packages/torch/_inductor/select_algorithm.py:1696] [0/0] out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-12-18T01:43:19.2529831Z W1218 01:43:19.252000 41443 site-packages/torch/_inductor/select_algorithm.py:1696] [0/0] out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-12-18T01:43:19.9280092Z W1218 01:43:19.927000 41443 site-packages/torch/_inductor/select_algorithm.py:1696] [0/0] out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-12-18T01:43:19.9287410Z AUTOTUNE mm(4096x5120, 5120x1280)
2024-12-18T01:43:19.9288466Z   triton_mm_3476 0.7813 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:19.9289636Z   triton_mm_3473 0.7864 ms 99.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:19.9290359Z   mm 0.7905 ms 98.8% 
2024-12-18T01:43:19.9291035Z   triton_mm_3470 0.7905 ms 98.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=4, num_warps=8
2024-12-18T01:43:19.9292172Z   triton_mm_3474 0.7905 ms 98.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=4, num_warps=8
2024-12-18T01:43:19.9293471Z   triton_mm_3477 0.7916 ms 98.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:19.9294607Z   triton_mm_3469 0.7926 ms 98.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:19.9295734Z   triton_mm_3471 0.7926 ms 98.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:19.9296856Z   triton_mm_3467 0.8090 ms 96.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=8
2024-12-18T01:43:19.9297964Z   triton_mm_3466 0.8120 ms 96.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=2, num_warps=4
2024-12-18T01:43:19.9298935Z SingleProcess AUTOTUNE benchmarking takes 2.3239 seconds and 0.0021 seconds precompiling for 20 choices
2024-12-18T01:43:25.5338355Z AUTOTUNE convolution(1x3x1024x1024, 1280x3x16x16)
2024-12-18T01:43:25.5339374Z   triton_convolution2d_3 0.9605 ms 100.0% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=128, BLOCK_N=128, GROUPS=1, KERNEL_H=16, KERNEL_W=16, PADDING_H=0, PADDING_W=0, STRIDE_H=16, STRIDE_W=16, UNROLL=False, num_stages=2, num_warps=8
2024-12-18T01:43:25.5340819Z   triton_convolution2d_6 1.0506 ms 91.4% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=256, BLOCK_N=64, GROUPS=1, KERNEL_H=16, KERNEL_W=16, PADDING_H=0, PADDING_W=0, STRIDE_H=16, STRIDE_W=16, UNROLL=False, num_stages=2, num_warps=8
2024-12-18T01:43:25.5342282Z   triton_convolution2d_1 1.1100 ms 86.5% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=256, BLOCK_N=64, GROUPS=1, KERNEL_H=16, KERNEL_W=16, PADDING_H=0, PADDING_W=0, STRIDE_H=16, STRIDE_W=16, UNROLL=False, num_stages=2, num_warps=4
2024-12-18T01:43:25.5343684Z   triton_convolution2d_0 1.1971 ms 80.2% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=256, GROUPS=1, KERNEL_H=16, KERNEL_W=16, PADDING_H=0, PADDING_W=0, STRIDE_H=16, STRIDE_W=16, UNROLL=False, num_stages=2, num_warps=4
2024-12-18T01:43:25.5344522Z   convolution 1.4735 ms 65.2% 
2024-12-18T01:43:25.5345334Z   triton_convolution2d_5 1.6067 ms 59.8% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=256, GROUPS=1, KERNEL_H=16, KERNEL_W=16, PADDING_H=0, PADDING_W=0, STRIDE_H=16, STRIDE_W=16, UNROLL=False, num_stages=2, num_warps=8
2024-12-18T01:43:25.5346699Z   triton_convolution2d_4 1.6835 ms 57.1% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=64, GROUPS=1, KERNEL_H=16, KERNEL_W=16, PADDING_H=0, PADDING_W=0, STRIDE_H=16, STRIDE_W=16, UNROLL=False, num_stages=2, num_warps=4
2024-12-18T01:43:25.5348050Z   triton_convolution2d_2 3.9926 ms 24.1% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=1024, BLOCK_N=16, GROUPS=1, KERNEL_H=16, KERNEL_W=16, PADDING_H=0, PADDING_W=0, STRIDE_H=16, STRIDE_W=16, UNROLL=False, num_stages=1, num_warps=8
2024-12-18T01:43:25.5349468Z SingleProcess AUTOTUNE benchmarking takes 1.2104 seconds and 0.0008 seconds precompiling for 8 choices
2024-12-18T01:43:25.5351676Z E1218 01:43:25.533000 41443 site-packages/torch/_inductor/select_algorithm.py:1477] [0/0] Exception out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. for benchmark choice TritonTemplateCaller(/tmp/tmpiq9zzxaj/at/cat2l4rykyfmjerbtoirkicd7ppgoxy3usjp7ad2arhpmgnw6qys.py, ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=4)
2024-12-18T01:43:25.5355225Z E1218 01:43:25.533000 41443 site-packages/torch/_inductor/select_algorithm.py:1477] [0/0] Exception out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. for benchmark choice TritonTemplateCaller(/tmp/tmpiq9zzxaj/f2/cf2qc5cwwf3xjaf2t6bg3airktfgpi6okzp2ywyum6cnobooqakw.py, ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=8)
2024-12-18T01:43:25.5358976Z E1218 01:43:25.534000 41443 site-packages/torch/_inductor/select_algorithm.py:1477] [0/0] Exception out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. for benchmark choice TritonTemplateCaller(/tmp/tmpiq9zzxaj/wz/cwzxnuv2krvwvh7v644mjupiigbcvo6yzpfnkdnyut355t643zmr.py, ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=4, num_warps=4)
2024-12-18T01:43:26.7559026Z W1218 01:43:26.755000 41443 site-packages/torch/_inductor/select_algorithm.py:1696] [0/0] out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-12-18T01:43:27.1583073Z W1218 01:43:27.157000 41443 site-packages/torch/_inductor/select_algorithm.py:1696] [0/0] out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-12-18T01:43:27.8326782Z W1218 01:43:27.831000 41443 site-packages/torch/_inductor/select_algorithm.py:1696] [0/0] out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-12-18T01:43:27.8333001Z AUTOTUNE mm(4904x1280, 1280x3840)
2024-12-18T01:43:27.8333882Z   triton_mm_16 0.7178 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:27.8335023Z   triton_mm_13 0.7219 ms 99.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=2, num_warps=4
2024-12-18T01:43:27.8336128Z   triton_mm_18 0.7260 ms 98.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:27.8337281Z   triton_mm_20 0.7342 ms 97.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:27.8338372Z   triton_mm_14 0.7363 ms 97.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=8
2024-12-18T01:43:27.8339475Z   triton_mm_23 0.7404 ms 97.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:27.8340593Z   triton_mm_17 0.7444 ms 96.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=4, num_warps=8
2024-12-18T01:43:27.8342061Z   triton_mm_21 0.7598 ms 94.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=4, num_warps=8
2024-12-18T01:43:27.8343186Z   triton_mm_24 0.7752 ms 92.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:27.8344494Z   triton_mm_12 0.7885 ms 91.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=2, num_warps=4
2024-12-18T01:43:27.8345740Z SingleProcess AUTOTUNE benchmarking takes 2.2972 seconds and 0.0023 seconds precompiling for 20 choices
2024-12-18T01:43:29.7846399Z AUTOTUNE bmm(14x5600x80, 14x80x14)
2024-12-18T01:43:29.7847174Z   triton_bmm_27 0.0369 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=32, BLOCK_N=16, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=2, num_warps=2
2024-12-18T01:43:29.7849161Z   triton_bmm_33 0.0369 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=16, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:29.7850906Z   triton_bmm_36 0.0369 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=16, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=4, num_warps=4
2024-12-18T01:43:29.7852520Z   triton_bmm_30 0.0369 ms 99.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=16, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=5, num_warps=4
2024-12-18T01:43:29.7854128Z   triton_bmm_40 0.0379 ms 97.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=16, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:29.7855504Z   triton_bmm_41 0.0379 ms 97.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=16, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=5, num_warps=8
2024-12-18T01:43:29.7856706Z   triton_bmm_28 0.0399 ms 92.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=32, BLOCK_N=16, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=5, num_warps=2
2024-12-18T01:43:29.7857902Z   triton_bmm_29 0.0399 ms 92.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=16, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=5, num_warps=4
2024-12-18T01:43:29.7858997Z   triton_bmm_35 0.0420 ms 87.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=16, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=4, num_warps=4
2024-12-18T01:43:29.7860378Z   triton_bmm_37 0.0420 ms 87.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=16, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:29.7861343Z SingleProcess AUTOTUNE benchmarking takes 1.9496 seconds and 0.0013 seconds precompiling for 17 choices
2024-12-18T01:43:31.7500709Z AUTOTUNE bmm(14x5600x80, 14x80x14)
2024-12-18T01:43:31.7501888Z   triton_bmm_49 0.0410 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=16, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:31.7503379Z   triton_bmm_43 0.0420 ms 97.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=32, BLOCK_N=16, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=2, num_warps=2
2024-12-18T01:43:31.7504724Z   triton_bmm_46 0.0420 ms 97.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=16, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=5, num_warps=4
2024-12-18T01:43:31.7506019Z   triton_bmm_52 0.0420 ms 97.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=16, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=4, num_warps=4
2024-12-18T01:43:31.7507703Z   triton_bmm_56 0.0430 ms 95.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=16, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:31.7508845Z   triton_bmm_57 0.0430 ms 95.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=16, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=5, num_warps=8
2024-12-18T01:43:31.7509969Z   triton_bmm_44 0.0451 ms 90.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=32, BLOCK_N=16, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=5, num_warps=2
2024-12-18T01:43:31.7510677Z   bmm 0.0461 ms 88.9% 
2024-12-18T01:43:31.7511326Z   triton_bmm_45 0.0461 ms 88.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=16, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=5, num_warps=4
2024-12-18T01:43:31.7512429Z   triton_bmm_51 0.0471 ms 87.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=16, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=4, num_warps=4
2024-12-18T01:43:31.7513871Z SingleProcess AUTOTUNE benchmarking takes 1.9636 seconds and 0.0014 seconds precompiling for 17 choices
2024-12-18T01:43:31.7516881Z E1218 01:43:31.749000 41443 site-packages/torch/_inductor/select_algorithm.py:1477] [0/0] Exception out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. for benchmark choice TritonTemplateCaller(/tmp/tmpiq9zzxaj/ee/ceeska2pnkr24yc3n7xorq3izglsjsbrbi6cjrcz5qddg7ckucej.py, ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=8)
2024-12-18T01:43:31.7520631Z E1218 01:43:31.749000 41443 site-packages/torch/_inductor/select_algorithm.py:1477] [0/0] Exception out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. for benchmark choice TritonTemplateCaller(/tmp/tmpiq9zzxaj/re/creg2z22q4ll2kdhee7i6uasjvqtpr7425rk7vbbdw62di6t2y2m.py, ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=4, num_warps=4)
2024-12-18T01:43:31.7524214Z E1218 01:43:31.750000 41443 site-packages/torch/_inductor/select_algorithm.py:1477] [0/0] Exception out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. for benchmark choice TritonTemplateCaller(/tmp/tmpiq9zzxaj/4b/c4bx6rikzp6ym36gpekpbaqynf7qtz6ojamjsumzxpaqdaovvmam.py, ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=4)
2024-12-18T01:43:32.8819069Z W1218 01:43:32.881000 41443 site-packages/torch/_inductor/select_algorithm.py:1696] [0/0] out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-12-18T01:43:33.2582191Z W1218 01:43:33.257000 41443 site-packages/torch/_inductor/select_algorithm.py:1696] [0/0] out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-12-18T01:43:33.8862620Z W1218 01:43:33.885000 41443 site-packages/torch/_inductor/select_algorithm.py:1696] [0/0] out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-12-18T01:43:33.8871828Z AUTOTUNE mm(4900x1280, 1280x1280)
2024-12-18T01:43:33.8872841Z   triton_mm_67 0.2509 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:33.8874066Z   triton_mm_71 0.2509 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:33.8875187Z   triton_mm_64 0.2519 ms 99.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=2, num_warps=4
2024-12-18T01:43:33.8876564Z   triton_mm_74 0.2540 ms 98.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:33.8877684Z   triton_mm_65 0.2550 ms 98.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=8
2024-12-18T01:43:33.8878805Z   triton_mm_69 0.2560 ms 98.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:33.8879922Z   triton_mm_68 0.2601 ms 96.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=4, num_warps=8
2024-12-18T01:43:33.8881202Z   triton_mm_72 0.2611 ms 96.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=4, num_warps=8
2024-12-18T01:43:33.8882325Z   triton_mm_75 0.2652 ms 94.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:33.8883435Z   triton_mm_63 0.2683 ms 93.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=2, num_warps=4
2024-12-18T01:43:33.8884394Z SingleProcess AUTOTUNE benchmarking takes 2.1351 seconds and 0.0021 seconds precompiling for 20 choices
2024-12-18T01:43:33.8975422Z E1218 01:43:33.896000 41443 site-packages/torch/_inductor/select_algorithm.py:1477] [0/0] Exception out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. for benchmark choice TritonTemplateCaller(/tmp/tmpiq9zzxaj/mu/cmufhpxtnvc6lmqjrhuoaknqxaocbrhdgrg2djegu33qesgnxft2.py, ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=4, num_warps=4)
2024-12-18T01:43:33.8979549Z E1218 01:43:33.897000 41443 site-packages/torch/_inductor/select_algorithm.py:1477] [0/0] Exception out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. for benchmark choice TritonTemplateCaller(/tmp/tmpiq9zzxaj/2n/c2nhghdrp6nawotbc77ntnqeijeer7qhry7kfmgu4iyhwzgddztv.py, ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=8)
2024-12-18T01:43:33.8984842Z E1218 01:43:33.897000 41443 site-packages/torch/_inductor/select_algorithm.py:1477] [0/0] Exception out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. for benchmark choice TritonTemplateCaller(/tmp/tmpiq9zzxaj/ti/cti7f3s6oxgbrlyd3yudyb46iyn5c2qit3yr45tf3hbbtygbgxwg.py, ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=4)
2024-12-18T01:43:35.2303531Z W1218 01:43:35.229000 41443 site-packages/torch/_inductor/select_algorithm.py:1696] [0/0] out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-12-18T01:43:35.6330723Z W1218 01:43:35.632000 41443 site-packages/torch/_inductor/select_algorithm.py:1696] [0/0] out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-12-18T01:43:36.2981691Z W1218 01:43:36.297000 41443 site-packages/torch/_inductor/select_algorithm.py:1696] [0/0] out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-12-18T01:43:36.2992134Z AUTOTUNE addmm(4096x3840, 4096x1280, 1280x3840)
2024-12-18T01:43:36.2993471Z   triton_mm_776 0.5919 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:36.2994637Z   triton_mm_772 0.5980 ms 99.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:36.2995909Z   triton_mm_779 0.5990 ms 98.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:36.2997501Z   triton_mm_769 0.6042 ms 98.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=2, num_warps=4
2024-12-18T01:43:36.2998992Z   triton_mm_774 0.6062 ms 97.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:36.3000609Z   triton_mm_777 0.6185 ms 95.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=4, num_warps=8
2024-12-18T01:43:36.3001833Z   triton_mm_770 0.6195 ms 95.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=8
2024-12-18T01:43:36.3002943Z   triton_mm_773 0.6216 ms 95.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=4, num_warps=8
2024-12-18T01:43:36.3004068Z   triton_mm_780 0.6380 ms 92.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:36.3004799Z   bias_addmm 0.6431 ms 92.0% 
2024-12-18T01:43:36.3005334Z SingleProcess AUTOTUNE benchmarking takes 2.3999 seconds and 0.0022 seconds precompiling for 21 choices
2024-12-18T01:43:38.4924701Z AUTOTUNE bmm(64x1024x80, 64x80x64)
2024-12-18T01:43:38.4925772Z   triton_bmm_793 0.0440 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:38.4926621Z   bmm 0.0451 ms 97.7% 
2024-12-18T01:43:38.4927303Z   triton_bmm_783 0.0451 ms 97.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=32, BLOCK_N=32, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=2, num_warps=4
2024-12-18T01:43:38.4928439Z   triton_bmm_786 0.0451 ms 97.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=32, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=5, num_warps=4
2024-12-18T01:43:38.4929570Z   triton_bmm_789 0.0451 ms 97.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=3, num_warps=8
2024-12-18T01:43:38.4930710Z   triton_bmm_784 0.0471 ms 93.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=32, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=5, num_warps=8
2024-12-18T01:43:38.4931832Z   triton_bmm_785 0.0471 ms 93.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=32, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=5, num_warps=8
2024-12-18T01:43:38.4933067Z   triton_bmm_790 0.0471 ms 93.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=5, num_warps=4
2024-12-18T01:43:38.4934215Z   triton_bmm_794 0.0471 ms 93.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=4, num_warps=4
2024-12-18T01:43:38.4935685Z   triton_bmm_791 0.0481 ms 91.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=False, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:38.4936767Z SingleProcess AUTOTUNE benchmarking takes 2.1916 seconds and 0.0012 seconds precompiling for 19 choices
2024-12-18T01:43:38.4938992Z E1218 01:43:38.492000 41443 site-packages/torch/_inductor/select_algorithm.py:1477] [0/0] Exception out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. for benchmark choice TritonTemplateCaller(/tmp/tmpiq9zzxaj/mn/cmn3qoxcww65gglkyvqu3ymlemluc5kgr4ncbxczcikkhfperk5y.py, ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=4, num_warps=4)
2024-12-18T01:43:38.4942764Z E1218 01:43:38.492000 41443 site-packages/torch/_inductor/select_algorithm.py:1477] [0/0] Exception out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. for benchmark choice TritonTemplateCaller(/tmp/tmpiq9zzxaj/ew/cew53ud4xxhw3eesr7jc7vuq7api2gj6yl52qh2b43dzr2uo3yys.py, ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=4)
2024-12-18T01:43:38.4947319Z E1218 01:43:38.493000 41443 site-packages/torch/_inductor/select_algorithm.py:1477] [0/0] Exception out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help. for benchmark choice TritonTemplateCaller(/tmp/tmpiq9zzxaj/yu/cyuufwme4fizodeaenkrlocepb5szw4p6ppqif3fwmsdvhgvlyaq.py, ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=8)
2024-12-18T01:43:39.6102698Z W1218 01:43:39.609000 41443 site-packages/torch/_inductor/select_algorithm.py:1696] [0/0] out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-12-18T01:43:39.9829282Z W1218 01:43:39.982000 41443 site-packages/torch/_inductor/select_algorithm.py:1696] [0/0] out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-12-18T01:43:40.6025709Z W1218 01:43:40.601000 41443 site-packages/torch/_inductor/select_algorithm.py:1696] [0/0] out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.
2024-12-18T01:43:40.6037222Z AUTOTUNE mm(4096x1280, 1280x1280)
2024-12-18T01:43:40.6038105Z   triton_mm_827 0.2017 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:40.6039263Z   triton_mm_831 0.2017 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:40.6040421Z   triton_mm_824 0.2038 ms 99.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=2, num_warps=4
2024-12-18T01:43:40.6041544Z   triton_mm_834 0.2048 ms 98.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:40.6042684Z   triton_mm_829 0.2068 ms 97.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:40.6043858Z   triton_mm_828 0.2109 ms 95.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=4, num_warps=8
2024-12-18T01:43:40.6044979Z   triton_mm_832 0.2109 ms 95.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=4, num_warps=8
2024-12-18T01:43:40.6046321Z   triton_mm_825 0.2120 ms 95.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=8
2024-12-18T01:43:40.6047033Z   mm 0.2140 ms 94.3% 
2024-12-18T01:43:40.6047696Z   triton_mm_835 0.2140 ms 94.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4
2024-12-18T01:43:40.6048669Z SingleProcess AUTOTUNE benchmarking takes 2.1088 seconds and 0.0020 seconds precompiling for 20 choices
2024-12-18T01:43:41.7037144Z AUTOTUNE convolution(1x1280x64x64, 256x1280x1x1)
2024-12-18T01:43:41.7037544Z   convolution 0.0584 ms 100.0% 
2024-12-18T01:43:41.7038361Z   triton_convolution2d_3483 0.0604 ms 96.6% ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, GROUPS=1, KERNEL_H=1, KERNEL_W=1, PADDING_H=0, PADDING_W=0, STRIDE_H=1, STRIDE_W=1, UNROLL=True, num_stages=2, num_warps=4
2024-12-18T01:43:41.7040005Z   triton_convolution2d_3482 0.0676 ms 86.4% ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, GROUPS=1, KERNEL_H=1, KERNEL_W=1, PADDING_H=0, PADDING_W=0, STRIDE_H=1, STRIDE_W=1, UNROLL=True, num_stages=2, num_warps=8
2024-12-18T01:43:41.7041371Z   triton_convolution2d_3485 0.0717 ms 81.4% ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=256, BLOCK_N=64, GROUPS=1, KERNEL_H=1, KERNEL_W=1, PADDING_H=0, PADDING_W=0, STRIDE_H=1, STRIDE_W=1, UNROLL=True, num_stages=2, num_warps=8
2024-12-18T01:43:41.7042741Z   triton_convolution2d_3484 0.0758 ms 77.0% ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=256, GROUPS=1, KERNEL_H=1, KERNEL_W=1, PADDING_H=0, PADDING_W=0, STRIDE_H=1, STRIDE_W=1, UNROLL=True, num_stages=2, num_warps=8
2024-12-18T01:43:41.7043582Z   conv1x1_via_mm 0.0788 ms 74.0% 
2024-12-18T01:43:41.7044390Z   triton_convolution2d_3480 0.0942 ms 62.0% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=256, BLOCK_N=64, GROUPS=1, KERNEL_H=1, KERNEL_W=1, PADDING_H=0, PADDING_W=0, STRIDE_H=1, STRIDE_W=1, UNROLL=True, num_stages=2, num_warps=4
2024-12-18T01:43:41.7045732Z   triton_convolution2d_3479 0.1066 ms 54.8% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=256, GROUPS=1, KERNEL_H=1, KERNEL_W=1, PADDING_H=0, PADDING_W=0, STRIDE_H=1, STRIDE_W=1, UNROLL=True, num_stages=2, num_warps=4
2024-12-18T01:43:41.7047067Z   triton_convolution2d_3481 0.1526 ms 38.3% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=1024, BLOCK_N=16, GROUPS=1, KERNEL_H=1, KERNEL_W=1, PADDING_H=0, PADDING_W=0, STRIDE_H=1, STRIDE_W=1, UNROLL=True, num_stages=1, num_warps=8
2024-12-18T01:43:41.7048130Z SingleProcess AUTOTUNE benchmarking takes 1.0567 seconds and 0.0007 seconds precompiling for 9 choices
2024-12-18T01:43:42.7142064Z AUTOTUNE convolution(1x256x64x64, 256x256x3x3)
2024-12-18T01:43:42.7142597Z   convolution 0.1198 ms 100.0% 
2024-12-18T01:43:42.7143615Z   triton_convolution2d_3492 0.1997 ms 60.0% ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=256, BLOCK_N=64, GROUPS=1, KERNEL_H=3, KERNEL_W=3, PADDING_H=1, PADDING_W=1, STRIDE_H=1, STRIDE_W=1, UNROLL=False, num_stages=2, num_warps=8
2024-12-18T01:43:42.7145014Z   triton_convolution2d_3487 0.2632 ms 45.5% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=256, BLOCK_N=64, GROUPS=1, KERNEL_H=3, KERNEL_W=3, PADDING_H=1, PADDING_W=1, STRIDE_H=1, STRIDE_W=1, UNROLL=False, num_stages=2, num_warps=4
2024-12-18T01:43:42.7146369Z   triton_convolution2d_3489 0.2959 ms 40.5% ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, GROUPS=1, KERNEL_H=3, KERNEL_W=3, PADDING_H=1, PADDING_W=1, STRIDE_H=1, STRIDE_W=1, UNROLL=False, num_stages=2, num_warps=8
2024-12-18T01:43:42.7147710Z   triton_convolution2d_3490 0.3604 ms 33.2% ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, GROUPS=1, KERNEL_H=3, KERNEL_W=3, PADDING_H=1, PADDING_W=1, STRIDE_H=1, STRIDE_W=1, UNROLL=False, num_stages=2, num_warps=4
2024-12-18T01:43:42.7149046Z   triton_convolution2d_3488 0.3666 ms 32.7% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=1024, BLOCK_N=16, GROUPS=1, KERNEL_H=3, KERNEL_W=3, PADDING_H=1, PADDING_W=1, STRIDE_H=1, STRIDE_W=1, UNROLL=False, num_stages=1, num_warps=8
2024-12-18T01:43:42.7150657Z   triton_convolution2d_3491 0.4117 ms 29.1% ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=256, GROUPS=1, KERNEL_H=3, KERNEL_W=3, PADDING_H=1, PADDING_W=1, STRIDE_H=1, STRIDE_W=1, UNROLL=False, num_stages=2, num_warps=8
2024-12-18T01:43:42.7152004Z   triton_convolution2d_3486 0.7475 ms 16.0% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=256, GROUPS=1, KERNEL_H=3, KERNEL_W=3, PADDING_H=1, PADDING_W=1, STRIDE_H=1, STRIDE_W=1, UNROLL=False, num_stages=2, num_warps=4
2024-12-18T01:43:42.7153126Z SingleProcess AUTOTUNE benchmarking takes 1.0081 seconds and 0.0006 seconds precompiling for 8 choices
2024-12-18T01:44:21.8520432Z Warning: Custom flash attention kernels were written specifically for A100.
2024-12-18T01:44:21.8521200Z We will try to read previously created kernel configurations from /var/lib/jenkins/workspace/flash_4_configs.p.
2024-12-18T01:44:21.8521922Z You can disable this kernel by setting SEGMENT_ANYTHING_FAST_USE_FLASH_4=0
2024-12-18T01:44:21.8523437Z key  (torch.Size([1, 16, 4096, 128]), torch.Size([1, 16, 4096, 128]), torch.Size([1, 16, 4096, 128]), torch.Size([1, 16, 4096, 128]), torch.Size([1, 16, 4096, 128]), (8388608, 524288, 128, 1), (8388608, 524288, 128, 1), (8388608, 524288, 128, 1), (8388608, 524288, 128, 1), (8388608, 524288, 128, 1))  not found. Running autotune. This might take a while.
2024-12-18T01:44:21.8524523Z all configs len:  60
2024-12-18T01:44:26.5980967Z (64, 64, 1, 1)  : 37811.41280005613
2024-12-18T01:44:28.7055190Z (64, 64, 2, 1)  : 9421.091566673567
2024-12-18T01:44:31.9114628Z (64, 64, 2, 2)  : 7284.9230599968
2024-12-18T01:44:33.5479898Z (64, 64, 4, 1)  : 3933.8233299986314
2024-12-18T01:44:35.0907822Z (64, 64, 4, 2)  : 3311.1352600008104
2024-12-18T01:44:36.5775741Z (64, 64, 4, 3)  : 2894.7981300007086
2024-12-18T01:44:37.4845098Z (64, 64, 4, 4)  : None
2024-12-18T01:44:38.8628280Z (64, 64, 8, 1)  : 3902.7832900046633
2024-12-18T01:44:40.2234564Z (64, 64, 8, 2)  : 3770.528610002657
2024-12-18T01:44:41.4735878Z (64, 64, 8, 3)  : 3218.129830001999
2024-12-18T01:44:42.0852090Z (64, 64, 8, 4)  : None
2024-12-18T01:44:42.7109676Z (64, 64, 8, 5)  : None
2024-12-18T01:44:43.3499276Z (64, 64, 8, 6)  : None
2024-12-18T01:44:44.0054412Z (64, 64, 8, 7)  : None
2024-12-18T01:44:44.6847889Z (64, 64, 8, 8)  : None
2024-12-18T01:44:52.7633612Z (64, 128, 1, 1)  : 66285.3128000279
2024-12-18T01:44:56.0543898Z (64, 128, 2, 1)  : 15954.466899984254
2024-12-18T01:44:59.1417238Z (64, 128, 2, 2)  : 10294.347000035486
2024-12-18T01:45:01.1944558Z (64, 128, 4, 1)  : 3780.54499999962
2024-12-18T01:45:03.1173531Z (64, 128, 4, 2)  : 3374.146819996895
2024-12-18T01:45:04.3849118Z (64, 128, 4, 3)  : None
2024-12-18T01:45:05.7501372Z (64, 128, 4, 4)  : None
2024-12-18T01:45:07.2200449Z (64, 128, 8, 1)  : 3442.0349700030783
2024-12-18T01:45:08.6655616Z (64, 128, 8, 2)  : 3330.900709997877
2024-12-18T01:45:09.4481419Z (64, 128, 8, 3)  : None
2024-12-18T01:45:10.2594084Z (64, 128, 8, 4)  : None
2024-12-18T01:45:11.1117289Z (64, 128, 8, 5)  : None
2024-12-18T01:45:11.9869568Z (64, 128, 8, 6)  : None
2024-12-18T01:45:12.9340867Z (64, 128, 8, 7)  : None
2024-12-18T01:45:13.9109761Z (64, 128, 8, 8)  : None
2024-12-18T01:45:25.7719846Z (128, 64, 1, 1)  : 68735.31689998345
2024-12-18T01:45:30.2115420Z (128, 64, 2, 1)  : 35320.42380002167
2024-12-18T01:45:34.7353802Z (128, 64, 2, 2)  : 36864.92070000895
2024-12-18T01:45:37.1343094Z (128, 64, 4, 1)  : 3960.598060002667
2024-12-18T01:45:39.1698881Z (128, 64, 4, 2)  : 13587.365450030122
2024-12-18T01:45:41.2265769Z (128, 64, 4, 3)  : 12803.289850035071
2024-12-18T01:45:42.9006333Z (128, 64, 4, 4)  : None
2024-12-18T01:45:44.2796276Z (128, 64, 8, 1)  : 2756.0355399964465
2024-12-18T01:45:45.9946661Z (128, 64, 8, 2)  : 4243.754569997691
2024-12-18T01:45:46.8309638Z (128, 64, 8, 3)  : None
2024-12-18T01:45:47.7029155Z (128, 64, 8, 4)  : None
2024-12-18T01:45:48.5877823Z (128, 64, 8, 5)  : None
2024-12-18T01:45:49.4868422Z (128, 64, 8, 6)  : None
2024-12-18T01:45:50.4082317Z (128, 64, 8, 7)  : None
2024-12-18T01:45:51.3509440Z (128, 64, 8, 8)  : None
2024-12-18T01:46:11.7766118Z (128, 128, 1, 1)  : 81372.32389999554
2024-12-18T01:46:18.6642981Z (128, 128, 2, 1)  : 100070.15800010777
2024-12-18T01:46:25.7579845Z (128, 128, 2, 2)  : 88402.45033267517
2024-12-18T01:46:28.7368369Z (128, 128, 4, 1)  : 11089.976700031912
2024-12-18T01:46:32.2194704Z (128, 128, 4, 2)  : 33011.15739996021
2024-12-18T01:46:35.0675818Z (128, 128, 4, 3)  : None
2024-12-18T01:46:37.9840406Z (128, 128, 4, 4)  : None
2024-12-18T01:46:39.2524614Z (128, 128, 8, 1)  : None
2024-12-18T01:46:40.5670679Z (128, 128, 8, 2)  : None
2024-12-18T01:46:41.9281189Z (128, 128, 8, 3)  : None
2024-12-18T01:46:43.3346039Z (128, 128, 8, 4)  : None
2024-12-18T01:46:44.8020646Z (128, 128, 8, 5)  : None
2024-12-18T01:46:46.2951008Z (128, 128, 8, 6)  : None
2024-12-18T01:46:47.8569003Z (128, 128, 8, 7)  : None
2024-12-18T01:46:49.4486166Z (128, 128, 8, 8)  : None
2024-12-18T01:46:49.4487322Z Found best_config  (128, 64, 8, 1)  with time  2756.0355399964465  for key  (torch.Size([1, 16, 4096, 128]), torch.Size([1, 16, 4096, 128]), torch.Size([1, 16, 4096, 128]), torch.Size([1, 16, 4096, 128]), torch.Size([1, 16, 4096, 128]), (8388608, 524288, 128, 1), (8388608, 524288, 128, 1), (8388608, 524288, 128, 1), (8388608, 524288, 128, 1), (8388608, 524288, 128, 1))
2024-12-18T01:46:49.4488882Z Warning: Custom flash attention kernels were written specifically for A100.
2024-12-18T01:46:49.4489509Z Storing configs for NVIDIA A10G locally under /var/lib/jenkins/workspace/flash_4_configs.p
2024-12-18T01:46:49.4490102Z Saving best configs to file /var/lib/jenkins/workspace/flash_4_configs.p
2024-12-18T01:47:01.2044530Z 
2024-12-18T01:47:01.2045024Z loading model: 0it [05:38, ?it/s]
2024-12-18T01:47:01.2089191Z cuda eval  sam_fast                           
2024-12-18T01:47:35.8042389Z AUTOTUNE addmm(4096x3840, 4096x1280, 1280x3840)
2024-12-18T01:47:35.8042895Z   bias_addmm 85.9228 ms 100.0% 
2024-12-18T01:47:35.8043277Z   addmm 86.2720 ms 99.6% 
2024-12-18T01:47:35.8043795Z SingleProcess AUTOTUNE benchmarking takes 1.4084 seconds and 0.0002 seconds precompiling for 2 choices
2024-12-18T01:51:08.7052558Z E1218 01:51:08.704000 41443 site-packages/torch/_dynamo/utils.py:2154] Accuracy failed: uint8 tensor did not match
2024-12-18T01:51:08.7053378Z E1218 01:51:08.704000 41443 site-packages/torch/_dynamo/utils.py:2117] Accuracy failed for key name masks
2024-12-18T01:51:08.7314178Z pass
2024-12-18T01:51:08.7727764Z TIMING: _recursive_pre_grad_passes:0.11423 _recursive_joint_graph_passes:3.85101 _recursive_post_grad_passes:2.89818 async_compile.wait:4.54809 code_gen:15.8035 inductor_compile:27.1699 backend_compile:54.73322 entire_frame_compile:68.58847 total_wall_time:68.58847
2024-12-18T01:51:08.7730717Z STATS: call_* op count: 8837 | FakeTensorMode.__torch_dispatch__:286413 | FakeTensor.__torch_dispatch__:28142 | ProxyTorchDispatchMode.__torch_dispatch__:110476
2024-12-18T01:51:08.7732022Z Dynamo produced 2 graphs covering 6751 ops with 0 graph breaks (0 unique)
2024-12-18T01:51:21.3294416Z 
2024-12-18T01:51:21.6251923Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/shufflenetv2_x1-5666bf0f80.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/shufflenetv2_x1-5666bf0f80.pth
2024-12-18T01:51:21.6404194Z 
2024-12-18T01:51:21.6404397Z 
2024-12-18T01:51:21.6689107Z   0% 0.00/8.79M [00:00<?, ?B/s]
2024-12-18T01:51:21.6689423Z 100% 8.79M/8.79M [00:00<00:00, 326MB/s]
2024-12-18T01:51:22.4169668Z 
2024-12-18T01:51:22.4170532Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:51:22.4246220Z cuda eval  shufflenet_v2_x1_0                 
2024-12-18T01:51:40.1699813Z pass
2024-12-18T01:51:40.1702420Z TIMING: _recursive_pre_grad_passes:0.02267 _recursive_joint_graph_passes:0.64406 _recursive_post_grad_passes:0.24458 async_compile.wait:0.46365 code_gen:2.73779 inductor_compile:5.36113 backend_compile:11.11428 entire_frame_compile:15.89041 total_wall_time:15.89041
2024-12-18T01:51:40.1703907Z STATS: call_* op count: 542 | FakeTensorMode.__torch_dispatch__:17387 | ProxyTorchDispatchMode.__torch_dispatch__:8165 | FakeTensor.__torch_dispatch__:2931
2024-12-18T01:51:40.1705140Z Dynamo produced 1 graphs covering 542 ops with 0 graph breaks (0 unique)
2024-12-18T01:51:45.0269364Z 
2024-12-18T01:51:46.1841442Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:51:46.1841795Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:51:46.1843618Z cuda eval  soft_actor_critic                  
2024-12-18T01:51:49.3892358Z pass
2024-12-18T01:51:49.3895871Z TIMING: _recursive_pre_grad_passes:0.00379 _recursive_joint_graph_passes:0.6017 _recursive_post_grad_passes:0.03017 async_compile.precompile:0.18117 async_compile.wait:0.00027 code_gen:0.24987 inductor_compile:0.36163 backend_compile:1.68845 entire_frame_compile:1.92897 total_wall_time:1.92897
2024-12-18T01:51:49.3897494Z STATS: call_* op count: 30 | FakeTensorMode.__torch_dispatch__:887 | ProxyTorchDispatchMode.__torch_dispatch__:267 | FakeTensor.__torch_dispatch__:50
2024-12-18T01:51:49.3898638Z Dynamo produced 1 graphs covering 30 ops with 0 graph breaks (0 unique)
2024-12-18T01:51:53.7467455Z 
2024-12-18T01:51:55.3235533Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:51:55.3235888Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:51:55.3300115Z cuda eval  speech_transformer                 
2024-12-18T01:52:06.7760902Z skipping cudagraphs due to skipping cudagraphs due to cpu device (arg13_1). Found from : 
2024-12-18T01:52:06.7762049Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/encoder.py", line 58, in torch_dynamo_resume_in_forward_at_50
2024-12-18T01:52:06.7762996Z     enc_output, enc_slf_attn = enc_layer(
2024-12-18T01:52:06.7763820Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/encoder.py", line 84, in forward
2024-12-18T01:52:06.7764650Z     enc_output, enc_slf_attn = self.slf_attn(
2024-12-18T01:52:06.7765464Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 54, in forward
2024-12-18T01:52:06.7766352Z     output, attn = self.attention(q, k, v, mask=mask)
2024-12-18T01:52:06.7767342Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 77, in forward
2024-12-18T01:52:06.7768142Z     attn = attn / self.temperature
2024-12-18T01:52:06.7768346Z 
2024-12-18T01:52:07.5051716Z W1218 01:52:07.504000 46322 site-packages/torch/_functorch/_aot_autograd/autograd_cache.py:856] [11/0_1] Bypassing autograd cache due to: Cannot cache a graph with functional tensor
2024-12-18T01:52:24.1906579Z skipping cudagraphs due to skipping cudagraphs due to cpu device (arg11_1). Found from : 
2024-12-18T01:52:24.1909037Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/decoder.py", line 105, in torch_dynamo_resume_in_forward_at_96
2024-12-18T01:52:24.1910106Z     dec_output, dec_slf_attn, dec_enc_attn = dec_layer(
2024-12-18T01:52:24.1911034Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/decoder.py", line 243, in forward
2024-12-18T01:52:24.1911934Z     dec_output, dec_slf_attn = self.slf_attn(
2024-12-18T01:52:24.1912853Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 54, in forward
2024-12-18T01:52:24.1913773Z     output, attn = self.attention(q, k, v, mask=mask)
2024-12-18T01:52:24.1914694Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 77, in forward
2024-12-18T01:52:24.1915603Z     attn = attn / self.temperature
2024-12-18T01:52:24.1915872Z 
2024-12-18T01:52:26.3764109Z pass
2024-12-18T01:52:26.3790043Z TIMING: entire_frame_compile:27.63741 _recursive_pre_grad_passes:0.04248 _recursive_joint_graph_passes:3.05483 _recursive_post_grad_passes:1.49543 async_compile.precompile:0.08316 async_compile.wait:1.15584 code_gen:4.61334 inductor_compile:9.18199 backend_compile:20.91008 total_wall_time:27.63741
2024-12-18T01:52:26.3791651Z STATS: call_* op count: 1454 | FakeTensorMode.__torch_dispatch__:38752 | FakeTensor.__torch_dispatch__:2738 | ProxyTorchDispatchMode.__torch_dispatch__:17456
2024-12-18T01:52:26.3792483Z Dynamo produced 7 graphs covering 1454 ops with 10 graph breaks (2 unique)
2024-12-18T01:52:31.4837294Z 
2024-12-18T01:52:31.7711956Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/squeezenet1_1-b8a52dc0.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/squeezenet1_1-b8a52dc0.pth
2024-12-18T01:52:31.7913501Z 
2024-12-18T01:52:31.7913769Z 
2024-12-18T01:52:31.8058734Z   0% 0.00/4.73M [00:00<?, ?B/s]
2024-12-18T01:52:31.8059100Z 100% 4.73M/4.73M [00:00<00:00, 346MB/s]
2024-12-18T01:52:32.4943737Z 
2024-12-18T01:52:32.4944126Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:52:32.4958176Z cuda eval  squeezenet1_1                      
2024-12-18T01:52:39.1255846Z pass
2024-12-18T01:52:39.1258337Z TIMING: _recursive_pre_grad_passes:0.00757 _recursive_joint_graph_passes:0.12817 _recursive_post_grad_passes:0.18438 async_compile.wait:0.43186 code_gen:1.6647 inductor_compile:2.33719 backend_compile:3.71256 entire_frame_compile:4.81772 total_wall_time:4.81772
2024-12-18T01:52:39.1260594Z STATS: call_* op count: 132 | FakeTensorMode.__torch_dispatch__:2764 | ProxyTorchDispatchMode.__torch_dispatch__:815 | FakeTensor.__torch_dispatch__:268
2024-12-18T01:52:39.1261755Z Dynamo produced 1 graphs covering 132 ops with 0 graph breaks (0 unique)
2024-12-18T01:52:43.6341145Z 
2024-12-18T01:52:44.3108553Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:52:44.3108815Z 
2024-12-18T01:52:45.3649965Z Loading pipeline components...:   0% 0/6 [00:00<?, ?it/s]
2024-12-18T01:52:45.3650293Z 
2024-12-18T01:52:45.6934167Z Loading pipeline components...:  50% 3/6 [00:01<00:01,  2.85it/s]
2024-12-18T01:52:45.6934866Z 
2024-12-18T01:52:45.8323127Z Loading pipeline components...:  67% 4/6 [00:01<00:00,  2.91it/s]
2024-12-18T01:52:45.8323463Z 
2024-12-18T01:52:45.8323741Z Loading pipeline components...: 100% 6/6 [00:01<00:00,  4.71it/s]
2024-12-18T01:52:45.8324236Z Loading pipeline components...: 100% 6/6 [00:01<00:00,  3.94it/s]
2024-12-18T01:52:47.3144733Z 
2024-12-18T01:52:47.3144922Z loading model: 0it [00:03, ?it/s]
2024-12-18T01:52:47.3234396Z cuda eval  stable_diffusion_text_encoder      
2024-12-18T01:52:47.3775942Z WARNING:common:fp64 golden ref were not generated for stable_diffusion_text_encoder. Setting accuracy check to cosine
2024-12-18T01:53:17.1184698Z pass
2024-12-18T01:53:17.1313422Z TIMING: _recursive_pre_grad_passes:0.04321 _recursive_joint_graph_passes:2.07901 _recursive_post_grad_passes:1.9528 async_compile.wait:0.60077 code_gen:4.11858 inductor_compile:9.45661 backend_compile:20.90346 entire_frame_compile:27.79982 total_wall_time:27.79982
2024-12-18T01:53:17.1315001Z STATS: call_* op count: 1647 | FakeTensorMode.__torch_dispatch__:44565 | FakeTensor.__torch_dispatch__:4352 | ProxyTorchDispatchMode.__torch_dispatch__:21058
2024-12-18T01:53:17.1315856Z Dynamo produced 1 graphs covering 1647 ops with 0 graph breaks (0 unique)
2024-12-18T01:53:22.3477567Z 
2024-12-18T01:53:23.0131618Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:53:23.0131887Z 
2024-12-18T01:53:23.6160830Z Loading pipeline components...:   0% 0/6 [00:00<?, ?it/s]
2024-12-18T01:53:23.6161306Z 
2024-12-18T01:53:24.4319854Z Loading pipeline components...:  33% 2/6 [00:00<00:01,  3.32it/s]
2024-12-18T01:53:24.4320198Z 
2024-12-18T01:53:24.5094325Z Loading pipeline components...:  50% 3/6 [00:01<00:01,  1.94it/s]
2024-12-18T01:53:24.5094864Z Loading pipeline components...: 100% 6/6 [00:01<00:00,  4.01it/s]
2024-12-18T01:53:26.2615677Z 
2024-12-18T01:53:26.2616117Z loading model: 0it [00:03, ?it/s]
2024-12-18T01:53:26.2793932Z cuda eval  stable_diffusion_unet              
2024-12-18T01:53:26.3028005Z pass_due_to_skip
2024-12-18T01:53:26.5090095Z TIMING: total_wall_time:0
2024-12-18T01:53:26.5090484Z STATS: call_* op count: 0
2024-12-18T01:53:26.5091186Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-12-18T01:53:30.7207783Z 
2024-12-18T01:53:32.5876408Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:53:32.5876759Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:53:32.5951920Z cuda eval  timm_efficientnet                  
2024-12-18T01:53:54.2627608Z pass
2024-12-18T01:53:54.2641361Z TIMING: _recursive_pre_grad_passes:0.02225 _recursive_joint_graph_passes:0.21575 _recursive_post_grad_passes:0.51095 async_compile.wait:0.46384 code_gen:3.34172 inductor_compile:6.73125 backend_compile:12.53292 entire_frame_compile:18.37765 total_wall_time:18.37765
2024-12-18T01:53:54.2642844Z STATS: call_* op count: 478 | FakeTensorMode.__torch_dispatch__:17973 | FakeTensor.__torch_dispatch__:2977 | ProxyTorchDispatchMode.__torch_dispatch__:8400
2024-12-18T01:53:54.2643984Z Dynamo produced 1 graphs covering 478 ops with 0 graph breaks (0 unique)
2024-12-18T01:53:59.3403084Z 
2024-12-18T01:54:01.8043824Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:54:01.8044173Z loading model: 0it [00:02, ?it/s]
2024-12-18T01:54:01.8141421Z cuda eval  timm_regnet                        
2024-12-18T01:54:26.7722769Z pass
2024-12-18T01:54:26.7757948Z TIMING: _recursive_pre_grad_passes:0.02764 _recursive_joint_graph_passes:0.43608 _recursive_post_grad_passes:0.30822 async_compile.wait:0.34174 code_gen:2.64598 inductor_compile:5.85357 backend_compile:12.80181 entire_frame_compile:19.594 total_wall_time:19.594
2024-12-18T01:54:26.7759490Z STATS: call_* op count: 638 | FakeTensorMode.__torch_dispatch__:21220 | ProxyTorchDispatchMode.__torch_dispatch__:9787 | FakeTensor.__torch_dispatch__:3595
2024-12-18T01:54:26.7760335Z Dynamo produced 1 graphs covering 638 ops with 0 graph breaks (0 unique)
2024-12-18T01:54:31.8980919Z 
2024-12-18T01:54:33.7810394Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:54:33.7819670Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:54:33.7841647Z cuda eval  timm_resnest                       
2024-12-18T01:54:46.3602787Z pass
2024-12-18T01:54:46.3614846Z TIMING: _recursive_pre_grad_passes:0.01243 _recursive_joint_graph_passes:0.37268 _recursive_post_grad_passes:0.11607 async_compile.wait:0.50716 code_gen:3.38575 inductor_compile:4.77837 backend_compile:7.5894 entire_frame_compile:9.94503 total_wall_time:9.94503
2024-12-18T01:54:46.3616675Z STATS: call_* op count: 262 | FakeTensorMode.__torch_dispatch__:8592 | ProxyTorchDispatchMode.__torch_dispatch__:3730 | FakeTensor.__torch_dispatch__:1321
2024-12-18T01:54:46.3617520Z Dynamo produced 1 graphs covering 262 ops with 0 graph breaks (0 unique)
2024-12-18T01:54:51.1540929Z 
2024-12-18T01:54:53.0273877Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:54:53.0274556Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:54:53.0314403Z cuda eval  timm_vision_transformer            
2024-12-18T01:55:06.2470048Z pass
2024-12-18T01:55:06.2485676Z TIMING: _recursive_pre_grad_passes:0.01645 _recursive_joint_graph_passes:1.46709 _recursive_post_grad_passes:0.39631 async_compile.wait:0.39957 code_gen:1.74162 inductor_compile:3.56088 backend_compile:8.70954 entire_frame_compile:11.32102 total_wall_time:11.32102
2024-12-18T01:55:06.2487205Z STATS: call_* op count: 526 | FakeTensorMode.__torch_dispatch__:15558 | FakeTensor.__torch_dispatch__:1710 | ProxyTorchDispatchMode.__torch_dispatch__:7371
2024-12-18T01:55:06.2488366Z Dynamo produced 1 graphs covering 526 ops with 0 graph breaks (0 unique)
2024-12-18T01:55:11.0885706Z 
2024-12-18T01:55:23.7837779Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:55:23.7838141Z loading model: 0it [00:12, ?it/s]
2024-12-18T01:55:23.7977322Z cuda eval  timm_vision_transformer_large      
2024-12-18T01:55:23.8213639Z pass_due_to_skip
2024-12-18T01:55:23.9294469Z TIMING: total_wall_time:0
2024-12-18T01:55:23.9294769Z STATS: call_* op count: 0
2024-12-18T01:55:23.9295155Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-12-18T01:55:28.0847015Z 
2024-12-18T01:55:30.0156574Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:55:30.0157364Z loading model: 0it [00:01, ?it/s]
2024-12-18T01:55:30.0204074Z cuda eval  timm_vovnet                        
2024-12-18T01:55:46.1121669Z pass
2024-12-18T01:55:46.1145013Z TIMING: _recursive_pre_grad_passes:0.01519 _recursive_joint_graph_passes:0.17824 _recursive_post_grad_passes:0.17386 async_compile.wait:0.37475 code_gen:3.13991 inductor_compile:4.90278 backend_compile:8.68939 entire_frame_compile:12.23704 total_wall_time:12.23704
2024-12-18T01:55:46.1146498Z STATS: call_* op count: 260 | FakeTensorMode.__torch_dispatch__:11242 | ProxyTorchDispatchMode.__torch_dispatch__:5252 | FakeTensor.__torch_dispatch__:2025
2024-12-18T01:55:46.1147343Z Dynamo produced 1 graphs covering 260 ops with 0 graph breaks (0 unique)
2024-12-18T01:55:50.9454493Z 
2024-12-18T01:55:51.2790775Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:55:51.2791128Z 
2024-12-18T01:55:51.2872384Z clip_merges.bpe: 0.00B [00:00, ?B/s]
2024-12-18T01:55:51.2872788Z clip_merges.bpe: 532kB [00:00, 66.3MB/s]
2024-12-18T01:55:52.9667595Z 
2024-12-18T01:55:52.9668254Z loading model: 0it [00:02, ?it/s]
2024-12-18T01:55:52.9749922Z cuda eval  torch_multimodal_clip              
2024-12-18T01:56:18.2545583Z pass
2024-12-18T01:56:18.2575082Z TIMING: _recursive_pre_grad_passes:0.02571 _recursive_joint_graph_passes:0.71611 _recursive_post_grad_passes:0.88839 async_compile.wait:1.26892 code_gen:4.98822 inductor_compile:8.7675 backend_compile:16.23454 entire_frame_compile:22.14594 total_wall_time:22.14594
2024-12-18T01:56:18.2576676Z STATS: call_* op count: 740 | FakeTensorMode.__torch_dispatch__:30657 | FakeTensor.__torch_dispatch__:4469 | ProxyTorchDispatchMode.__torch_dispatch__:13860
2024-12-18T01:56:18.2577567Z Dynamo produced 1 graphs covering 740 ops with 0 graph breaks (0 unique)
2024-12-18T01:56:23.3568276Z 
2024-12-18T01:56:23.8551594Z loading model: 0it [00:00, ?it/s]WARNING:common:Model tts_angular does not support bfloat16, running with amp instead
2024-12-18T01:56:24.0828984Z 
2024-12-18T01:56:24.0829557Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:56:24.0830084Z WARNING:common:Model tts_angular does not support bfloat16, running with amp instead
2024-12-18T01:56:24.0830588Z cuda eval  tts_angular                        
2024-12-18T01:56:24.1519762Z WARNING:common:Model tts_angular does not support bfloat16, running with amp instead
2024-12-18T01:56:26.7845247Z pass
2024-12-18T01:56:26.7858492Z TIMING: entire_frame_compile:1.16425 _recursive_pre_grad_passes:0.00339 _recursive_joint_graph_passes:0.12495 inductor_compile:0.23702 backend_compile:1.02026 _recursive_post_grad_passes:0.01527 async_compile.precompile:0.1003 async_compile.wait:0.00026 code_gen:0.15062 total_wall_time:1.16425
2024-12-18T01:56:26.7860114Z STATS: call_* op count: 6 | FakeTensorMode.__torch_dispatch__:364 | ProxyTorchDispatchMode.__torch_dispatch__:54 | FakeTensor.__torch_dispatch__:14
2024-12-18T01:56:26.7860922Z Dynamo produced 3 graphs covering 6 ops with 2 graph breaks (1 unique)
2024-12-18T01:56:31.0968224Z 
2024-12-18T01:56:32.2167446Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/vgg16-397923af.pth
2024-12-18T01:56:32.2364761Z 
2024-12-18T01:56:32.2365097Z 
2024-12-18T01:56:32.3367203Z   0% 0.00/528M [00:00<?, ?B/s]
2024-12-18T01:56:32.3367526Z 
2024-12-18T01:56:32.4369589Z   8% 42.8M/528M [00:00<00:01, 448MB/s]
2024-12-18T01:56:32.4369923Z 
2024-12-18T01:56:32.5371521Z  17% 87.9M/528M [00:00<00:00, 462MB/s]
2024-12-18T01:56:32.5371862Z 
2024-12-18T01:56:32.6373320Z  25% 133M/528M [00:00<00:00, 466MB/s] 
2024-12-18T01:56:32.6373644Z 
2024-12-18T01:56:32.7375383Z  34% 178M/528M [00:00<00:00, 469MB/s]
2024-12-18T01:56:32.7375711Z 
2024-12-18T01:56:32.8376178Z  42% 223M/528M [00:00<00:00, 469MB/s]
2024-12-18T01:56:32.8376521Z 
2024-12-18T01:56:32.9378773Z  51% 268M/528M [00:00<00:00, 470MB/s]
2024-12-18T01:56:32.9379143Z 
2024-12-18T01:56:33.0379791Z  59% 313M/528M [00:00<00:00, 471MB/s]
2024-12-18T01:56:33.0380119Z 
2024-12-18T01:56:33.1567699Z  68% 358M/528M [00:00<00:00, 472MB/s]
2024-12-18T01:56:33.1567975Z 
2024-12-18T01:56:33.2570176Z  76% 403M/528M [00:00<00:00, 446MB/s]
2024-12-18T01:56:33.2570498Z 
2024-12-18T01:56:33.3578606Z  85% 448M/528M [00:01<00:00, 453MB/s]
2024-12-18T01:56:33.3578935Z 
2024-12-18T01:56:33.4707062Z  93% 492M/528M [00:01<00:00, 454MB/s]
2024-12-18T01:56:33.4707536Z 100% 528M/528M [00:01<00:00, 448MB/s]
2024-12-18T01:56:34.6125105Z 
2024-12-18T01:56:34.6126194Z loading model: 0it [00:03, ?it/s]
2024-12-18T01:56:34.6132637Z cuda eval  vgg16                              
2024-12-18T01:56:47.3963330Z pass
2024-12-18T01:56:47.3964379Z TIMING: _recursive_pre_grad_passes:0.00586 _recursive_joint_graph_passes:0.81301 _recursive_post_grad_passes:0.19274 async_compile.wait:0.42462 code_gen:1.22436 inductor_compile:1.73998 backend_compile:3.67594 entire_frame_compile:4.377 total_wall_time:4.377
2024-12-18T01:56:47.3966249Z STATS: call_* op count: 80 | FakeTensorMode.__torch_dispatch__:2237 | ProxyTorchDispatchMode.__torch_dispatch__:651 | FakeTensor.__torch_dispatch__:165
2024-12-18T01:56:47.3967270Z Dynamo produced 1 graphs covering 80 ops with 0 graph breaks (0 unique)
2024-12-18T01:56:51.8484311Z 
2024-12-18T01:56:52.5291746Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/maskrcnn_resnet50_fpn_coco-bf2d0c1e.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/maskrcnn_resnet50_fpn_coco-bf2d0c1e.pth
2024-12-18T01:56:52.5449835Z 
2024-12-18T01:56:52.5449976Z 
2024-12-18T01:56:52.6453984Z   0% 0.00/170M [00:00<?, ?B/s]
2024-12-18T01:56:52.6454233Z 
2024-12-18T01:56:52.7471892Z  22% 37.1M/170M [00:00<00:00, 388MB/s]
2024-12-18T01:56:52.7472133Z 
2024-12-18T01:56:52.8472880Z  44% 74.2M/170M [00:00<00:00, 385MB/s]
2024-12-18T01:56:52.8473125Z 
2024-12-18T01:56:52.9473686Z  67% 114M/170M [00:00<00:00, 398MB/s] 
2024-12-18T01:56:52.9473962Z 
2024-12-18T01:56:52.9892142Z  90% 153M/170M [00:00<00:00, 405MB/s]
2024-12-18T01:56:52.9892477Z 100% 170M/170M [00:00<00:00, 401MB/s]
2024-12-18T01:56:54.5878279Z WARNING:common:Model vision_maskrcnn does not support bfloat16, running with float16 instead
2024-12-18T01:56:55.0827312Z 
2024-12-18T01:56:55.0827493Z loading model: 0it [00:03, ?it/s]
2024-12-18T01:56:55.0828029Z WARNING:common:Model vision_maskrcnn does not support bfloat16, running with float16 instead
2024-12-18T01:56:55.0892043Z cuda eval  vision_maskrcnn                    
2024-12-18T01:56:57.2270519Z WARNING:common:Model vision_maskrcnn does not support bfloat16, running with float16 instead
2024-12-18T01:57:23.1664859Z W1218 01:57:23.165000 48951 site-packages/torch/_inductor/utils.py:1543] [3/0_1] DeviceCopy in input program
2024-12-18T01:57:23.1798791Z W1218 01:57:23.179000 48951 site-packages/torch/_inductor/utils.py:1543] [3/0_1] DeviceCopy in input program
2024-12-18T01:57:23.1932070Z W1218 01:57:23.192000 48951 site-packages/torch/_inductor/utils.py:1543] [3/0_1] DeviceCopy in input program
2024-12-18T01:57:23.2062410Z W1218 01:57:23.205000 48951 site-packages/torch/_inductor/utils.py:1543] [3/0_1] DeviceCopy in input program
2024-12-18T01:57:23.2198233Z W1218 01:57:23.219000 48951 site-packages/torch/_inductor/utils.py:1543] [3/0_1] DeviceCopy in input program
2024-12-18T01:57:24.8414587Z skipping cudagraphs due to skipping cudagraphs due to cpu device (arg11_1). Found from : 
2024-12-18T01:57:24.8415468Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/rpn.py", line 362, in forward
2024-12-18T01:57:24.8416163Z     anchors = self.anchor_generator(images, features)
2024-12-18T01:57:24.8416885Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 126, in forward
2024-12-18T01:57:24.8417572Z     self.set_cell_anchors(dtype, device)
2024-12-18T01:57:24.8418291Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 77, in set_cell_anchors
2024-12-18T01:57:24.8419543Z     self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors]
2024-12-18T01:57:24.8420436Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 77, in <listcomp>
2024-12-18T01:57:24.8421327Z     self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors]
2024-12-18T01:57:24.8421831Z 
2024-12-18T01:57:26.0939391Z W1218 01:57:26.092000 48951 site-packages/torch/_dynamo/variables/tensor.py:869] [13/0] Graph break from `Tensor.item()`, consider setting:
2024-12-18T01:57:26.0940431Z W1218 01:57:26.092000 48951 site-packages/torch/_dynamo/variables/tensor.py:869] [13/0]     torch._dynamo.config.capture_scalar_outputs = True
2024-12-18T01:57:26.0941246Z W1218 01:57:26.092000 48951 site-packages/torch/_dynamo/variables/tensor.py:869] [13/0] or:
2024-12-18T01:57:26.0942533Z W1218 01:57:26.092000 48951 site-packages/torch/_dynamo/variables/tensor.py:869] [13/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2024-12-18T01:57:26.0943478Z W1218 01:57:26.092000 48951 site-packages/torch/_dynamo/variables/tensor.py:869] [13/0] to include these operations in the captured graph.
2024-12-18T01:57:26.0944255Z W1218 01:57:26.092000 48951 site-packages/torch/_dynamo/variables/tensor.py:869] [13/0] 
2024-12-18T01:57:26.0944986Z W1218 01:57:26.092000 48951 site-packages/torch/_dynamo/variables/tensor.py:869] [13/0] Graph break: from user code at:
2024-12-18T01:57:26.0946351Z W1218 01:57:26.092000 48951 site-packages/torch/_dynamo/variables/tensor.py:869] [13/0]   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/generalized_rcnn.py", line 105, in torch_dynamo_resume_in_forward_at_104
2024-12-18T01:57:26.0947951Z W1218 01:57:26.092000 48951 site-packages/torch/_dynamo/variables/tensor.py:869] [13/0]     detections, detector_losses = self.roi_heads(features, proposals, images.image_sizes, targets)
2024-12-18T01:57:26.0949397Z W1218 01:57:26.092000 48951 site-packages/torch/_dynamo/variables/tensor.py:869] [13/0]   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 761, in forward
2024-12-18T01:57:26.0950725Z W1218 01:57:26.092000 48951 site-packages/torch/_dynamo/variables/tensor.py:869] [13/0]     box_features = self.box_roi_pool(features, proposals, image_shapes)
2024-12-18T01:57:26.0951988Z W1218 01:57:26.092000 48951 site-packages/torch/_dynamo/variables/tensor.py:869] [13/0]   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 310, in forward
2024-12-18T01:57:26.0953217Z W1218 01:57:26.092000 48951 site-packages/torch/_dynamo/variables/tensor.py:869] [13/0]     self.scales, self.map_levels = _setup_scales(
2024-12-18T01:57:26.0954423Z W1218 01:57:26.092000 48951 site-packages/torch/_dynamo/variables/tensor.py:869] [13/0]   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 122, in _setup_scales
2024-12-18T01:57:26.0955716Z W1218 01:57:26.092000 48951 site-packages/torch/_dynamo/variables/tensor.py:869] [13/0]     scales = [_infer_scale(feat, original_input_shape) for feat in features]
2024-12-18T01:57:26.0956986Z W1218 01:57:26.092000 48951 site-packages/torch/_dynamo/variables/tensor.py:869] [13/0]   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 122, in <listcomp>
2024-12-18T01:57:26.0958253Z W1218 01:57:26.092000 48951 site-packages/torch/_dynamo/variables/tensor.py:869] [13/0]     scales = [_infer_scale(feat, original_input_shape) for feat in features]
2024-12-18T01:57:26.0959524Z W1218 01:57:26.092000 48951 site-packages/torch/_dynamo/variables/tensor.py:869] [13/0]   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 104, in _infer_scale
2024-12-18T01:57:26.0960766Z W1218 01:57:26.092000 48951 site-packages/torch/_dynamo/variables/tensor.py:869] [13/0]     scale = 2 ** float(torch.tensor(approx_scale).log2().round())
2024-12-18T01:57:26.0961759Z W1218 01:57:26.092000 48951 site-packages/torch/_dynamo/variables/tensor.py:869] [13/0] 
2024-12-18T01:57:26.0962372Z W1218 01:57:26.092000 48951 site-packages/torch/_dynamo/variables/tensor.py:869] [13/0] 
2024-12-18T01:58:09.9727398Z pass
2024-12-18T01:58:09.9741109Z TIMING: entire_frame_compile:70.29883 _recursive_pre_grad_passes:0.04822 _recursive_joint_graph_passes:0.90809 _recursive_post_grad_passes:0.48662 async_compile.wait:5.49856 code_gen:13.34332 inductor_compile:21.5331 backend_compile:34.52947 total_wall_time:70.29883
2024-12-18T01:58:09.9742934Z STATS: call_* op count: 1358 | FakeTensorMode.__torch_dispatch__:60750 | FakeTensor.__torch_dispatch__:6030 | ProxyTorchDispatchMode.__torch_dispatch__:18373 | attempt fast:152 | fast is_contiguous:96 | slow both tensors nontrivially broadcast:56
2024-12-18T01:58:09.9744055Z Dynamo produced 25 graphs covering 1358 ops with 18 graph breaks (2 unique)
2024-12-18T01:58:15.7818933Z 
2024-12-18T01:58:18.0866927Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:58:18.0867352Z loading model: 0it [00:02, ?it/s]
2024-12-18T01:58:18.0950104Z cuda eval  yolov3                             
2024-12-18T01:58:41.1120708Z W1218 01:58:41.111000 49459 site-packages/torch/_inductor/utils.py:1543] [0/0] DeviceCopy in input program
2024-12-18T01:58:41.2642236Z W1218 01:58:41.263000 49459 site-packages/torch/_inductor/utils.py:1543] [0/0] DeviceCopy in input program
2024-12-18T01:58:41.4137676Z W1218 01:58:41.412000 49459 site-packages/torch/_inductor/utils.py:1543] [0/0] DeviceCopy in input program
2024-12-18T01:58:41.4870647Z W1218 01:58:41.486000 49459 site-packages/torch/_inductor/utils.py:1543] [0/0] DeviceCopy in input program
2024-12-18T01:58:41.4878949Z W1218 01:58:41.487000 49459 site-packages/torch/_inductor/utils.py:1543] [0/0] DeviceCopy in input program
2024-12-18T01:58:41.4889269Z W1218 01:58:41.488000 49459 site-packages/torch/_inductor/utils.py:1543] [0/0] DeviceCopy in input program
2024-12-18T01:58:50.3484851Z skipping cudagraphs due to skipping cudagraphs due to cpu device (arg298_1). Found from : 
2024-12-18T01:58:50.3486004Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2770, in run_n_iterations
2024-12-18T01:58:50.3486921Z     self.model_iter_fn(mod, inputs, collect_outputs=False)
2024-12-18T01:58:50.3487830Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 444, in forward_pass
2024-12-18T01:58:50.3488654Z     return mod(*inputs)
2024-12-18T01:58:50.3489364Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 238, in forward
2024-12-18T01:58:50.3490000Z     return self.forward_once(x)
2024-12-18T01:58:50.3490648Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 290, in forward_once
2024-12-18T01:58:50.3491300Z     yolo_out.append(module(x, out))
2024-12-18T01:58:50.3491920Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 188, in forward
2024-12-18T01:58:50.3492568Z     self.create_grids((nx, ny), p.device)
2024-12-18T01:58:50.3493254Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 159, in create_grids
2024-12-18T01:58:50.3493961Z     self.anchor_vec = self.anchor_vec.to(device)
2024-12-18T01:58:50.3494198Z 
2024-12-18T01:58:53.0209264Z pass
2024-12-18T01:58:53.0326154Z TIMING: _recursive_pre_grad_passes:0.02931 _recursive_joint_graph_passes:0.2738 _recursive_post_grad_passes:0.4648 async_compile.wait:2.23254 code_gen:8.5034 inductor_compile:12.46267 backend_compile:20.28269 entire_frame_compile:27.16712 total_wall_time:27.16712
2024-12-18T01:58:53.0327629Z STATS: call_* op count: 649 | FakeTensorMode.__torch_dispatch__:23956 | FakeTensor.__torch_dispatch__:3524 | ProxyTorchDispatchMode.__torch_dispatch__:12245
2024-12-18T01:58:53.0328458Z Dynamo produced 1 graphs covering 649 ops with 0 graph breaks (0 unique)
2024-12-18T01:58:55.1937353Z accuracy            pass_rate=82.61%
2024-12-18T01:58:55.1942380Z calls_captured      gmean=0.00x mean=675.022x
2024-12-18T01:58:55.1946559Z unique_graphs       gmean=0.00x mean=1.674x
2024-12-18T01:58:55.1949938Z graph_breaks        gmean=0.00x mean=0.761x
2024-12-18T01:58:55.1953698Z unique_graph_breaks gmean=0.00x mean=0.174x
2024-12-18T01:58:55.1957450Z autograd_captures   gmean=0.00x mean=0.000x
2024-12-18T01:58:55.1961233Z autograd_compiles   gmean=0.00x mean=0.000x
2024-12-18T01:58:55.1965023Z cudagraph_skips     gmean=0.00x mean=0.196x
2024-12-18T01:58:56.7820899Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_inference.csv
2024-12-18T01:58:57.1254439Z llama                               PASS
2024-12-18T01:58:57.1258467Z llama_v2_7b_16h                     XFAIL
2024-12-18T01:58:57.1262859Z llava                               XFAIL
2024-12-18T01:58:57.1267686Z maml                                XFAIL
2024-12-18T01:58:57.1272081Z maml_omniglot                       PASS
2024-12-18T01:58:57.1276646Z mnasnet1_0                          PASS
2024-12-18T01:58:57.1281021Z mobilenet_v2                        PASS
2024-12-18T01:58:57.1285487Z mobilenet_v2_quantized_qat          XFAIL
2024-12-18T01:58:57.1289995Z mobilenet_v3_large                  PASS
2024-12-18T01:58:57.1294376Z moco                                PASS
2024-12-18T01:58:57.1298667Z moondream                           XFAIL
2024-12-18T01:58:57.1303185Z nanogpt                             PASS
2024-12-18T01:58:57.1307648Z nvidia_deeprecommender              PASS
2024-12-18T01:58:57.1312054Z opacus_cifar10                      PASS
2024-12-18T01:58:57.1316564Z phlippe_densenet                    PASS
2024-12-18T01:58:57.1320902Z phlippe_resnet                      PASS
2024-12-18T01:58:57.1325328Z pyhpc_equation_of_state             PASS
2024-12-18T01:58:57.1329647Z pyhpc_isoneutral_mixing             PASS
2024-12-18T01:58:57.1334188Z pyhpc_turbulent_kinetic_energy      PASS
2024-12-18T01:58:57.1338541Z pytorch_CycleGAN_and_pix2pix        PASS
2024-12-18T01:58:57.1342960Z pytorch_stargan                     PASS
2024-12-18T01:58:57.1347308Z pytorch_unet                        PASS
2024-12-18T01:58:57.1351655Z resnet152                           PASS
2024-12-18T01:58:57.1355993Z resnet18                            PASS
2024-12-18T01:58:57.1360356Z resnet50                            PASS
2024-12-18T01:58:57.1364703Z resnet50_quantized_qat              XFAIL
2024-12-18T01:58:57.1370325Z resnext50_32x4d                     PASS
2024-12-18T01:58:57.1374546Z sam                                 PASS
2024-12-18T01:58:57.1378883Z sam_fast                            PASS
2024-12-18T01:58:57.1383364Z shufflenet_v2_x1_0                  PASS
2024-12-18T01:58:57.1387713Z soft_actor_critic                   PASS
2024-12-18T01:58:57.1392031Z speech_transformer                  PASS
2024-12-18T01:58:57.1396419Z squeezenet1_1                       PASS
2024-12-18T01:58:57.1400777Z stable_diffusion_text_encoder       PASS
2024-12-18T01:58:57.1405274Z stable_diffusion_unet               XFAIL
2024-12-18T01:58:57.1409592Z timm_efficientnet                   PASS
2024-12-18T01:58:57.1413948Z timm_regnet                         PASS
2024-12-18T01:58:57.1418269Z timm_resnest                        PASS
2024-12-18T01:58:57.1422714Z timm_vision_transformer             PASS
2024-12-18T01:58:57.1427125Z timm_vision_transformer_large       XFAIL
2024-12-18T01:58:57.1431475Z timm_vovnet                         PASS
2024-12-18T01:58:57.1435811Z torch_multimodal_clip               PASS
2024-12-18T01:58:57.1440149Z tts_angular                         PASS
2024-12-18T01:58:57.1444498Z vgg16                               PASS
2024-12-18T01:58:57.1448890Z vision_maskrcnn                     PASS
2024-12-18T01:58:57.1453245Z yolov3                              PASS
2024-12-18T01:58:57.1965788Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_inference.csv
2024-12-18T01:58:57.5343241Z llama                               PASS
2024-12-18T01:58:57.5347339Z llama_v2_7b_16h                     PASS
2024-12-18T01:58:57.5351706Z llava                               PASS
2024-12-18T01:58:57.5356025Z maml                                PASS
2024-12-18T01:58:57.5360541Z maml_omniglot                       PASS
2024-12-18T01:58:57.5364942Z mnasnet1_0                          PASS
2024-12-18T01:58:57.5370333Z mobilenet_v2                        PASS
2024-12-18T01:58:57.5374848Z mobilenet_v2_quantized_qat          PASS
2024-12-18T01:58:57.5379177Z mobilenet_v3_large                  PASS
2024-12-18T01:58:57.5383618Z moco                                PASS
2024-12-18T01:58:57.5387925Z moondream                           PASS
2024-12-18T01:58:57.5392463Z nanogpt                             PASS
2024-12-18T01:58:57.5396740Z nvidia_deeprecommender              PASS
2024-12-18T01:58:57.5401140Z opacus_cifar10                      PASS
2024-12-18T01:58:57.5405531Z phlippe_densenet                    PASS
2024-12-18T01:58:57.5409962Z phlippe_resnet                      PASS
2024-12-18T01:58:57.5414324Z pyhpc_equation_of_state             PASS
2024-12-18T01:58:57.5418712Z pyhpc_isoneutral_mixing             PASS
2024-12-18T01:58:57.5423116Z pyhpc_turbulent_kinetic_energy      PASS
2024-12-18T01:58:57.5427572Z pytorch_CycleGAN_and_pix2pix        PASS
2024-12-18T01:58:57.5431889Z pytorch_stargan                     PASS
2024-12-18T01:58:57.5436334Z pytorch_unet                        PASS
2024-12-18T01:58:57.5440615Z resnet152                           PASS
2024-12-18T01:58:57.5445043Z resnet18                            PASS
2024-12-18T01:58:57.5449419Z resnet50                            PASS
2024-12-18T01:58:57.5453891Z resnet50_quantized_qat              PASS
2024-12-18T01:58:57.5458210Z resnext50_32x4d                     PASS
2024-12-18T01:58:57.5462686Z sam                                 PASS
2024-12-18T01:58:57.5467306Z sam_fast                            PASS
2024-12-18T01:58:57.5471853Z shufflenet_v2_x1_0                  PASS
2024-12-18T01:58:57.5476062Z soft_actor_critic                   PASS
2024-12-18T01:58:57.5480486Z speech_transformer                  PASS
2024-12-18T01:58:57.5484809Z squeezenet1_1                       PASS
2024-12-18T01:58:57.5489222Z stable_diffusion_text_encoder       PASS
2024-12-18T01:58:57.5493532Z stable_diffusion_unet               PASS
2024-12-18T01:58:57.5497938Z timm_efficientnet                   PASS
2024-12-18T01:58:57.5502410Z timm_regnet                         PASS
2024-12-18T01:58:57.5506815Z timm_resnest                        PASS
2024-12-18T01:58:57.5511131Z timm_vision_transformer             PASS
2024-12-18T01:58:57.5515524Z timm_vision_transformer_large       PASS
2024-12-18T01:58:57.5519886Z timm_vovnet                         PASS
2024-12-18T01:58:57.5524286Z torch_multimodal_clip               PASS
2024-12-18T01:58:57.5528578Z tts_angular                         PASS
2024-12-18T01:58:57.5533006Z vgg16                               PASS
2024-12-18T01:58:57.5537287Z vision_maskrcnn                     PASS
2024-12-18T01:58:57.5541744Z yolov3                              PASS
2024-12-18T01:58:57.6059217Z + test_single_dynamo_benchmark training torchbench 1 --training --amp
2024-12-18T01:58:57.6062630Z ++ pwd
2024-12-18T01:58:57.6065727Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2024-12-18T01:58:57.6066348Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports
2024-12-18T01:58:57.6088132Z + local name=training
2024-12-18T01:58:57.6088447Z + shift
2024-12-18T01:58:57.6088709Z + local suite=torchbench
2024-12-18T01:58:57.6089022Z + shift
2024-12-18T01:58:57.6089238Z + local shard_id=1
2024-12-18T01:58:57.6089470Z + shift
2024-12-18T01:58:57.6089759Z + partition_flags=()
2024-12-18T01:58:57.6090021Z + local partition_flags
2024-12-18T01:58:57.6090306Z + [[ -n 2 ]]
2024-12-18T01:58:57.6090522Z + [[ -n 1 ]]
2024-12-18T01:58:57.6091166Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id")
2024-12-18T01:58:57.6091687Z + [[ inductor_torchbench == *perf_compare* ]]
2024-12-18T01:58:57.6092044Z + [[ inductor_torchbench == *perf* ]]
2024-12-18T01:58:57.6092360Z + [[ inductor_torchbench == *_avx2* ]]
2024-12-18T01:58:57.6092677Z + [[ inductor_torchbench == *_avx512* ]]
2024-12-18T01:58:57.6093712Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --inductor --device cuda --training --amp --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv
2024-12-18T01:59:03.9797993Z 
2024-12-18T01:59:04.3417184Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:59:04.3417823Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:59:04.3418446Z cuda train llava                              
2024-12-18T01:59:04.3425410Z Traceback (most recent call last):
2024-12-18T01:59:04.3426717Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run
2024-12-18T01:59:04.3427298Z     ) = runner.load_model(
2024-12-18T01:59:04.3427844Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 312, in load_model
2024-12-18T01:59:04.3428441Z     benchmark = benchmark_cls(
2024-12-18T01:59:04.3429024Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__
2024-12-18T01:59:04.3429606Z     obj = type.__call__(cls, *args, **kwargs)
2024-12-18T01:59:04.3430229Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava/__init__.py", line 11, in __init__
2024-12-18T01:59:04.3431036Z     super().__init__(name="llava", test=test, device=device, batch_size=batch_size, extra_args=extra_args)
2024-12-18T01:59:04.3431971Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__
2024-12-18T01:59:04.3432860Z     super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args)
2024-12-18T01:59:04.3433603Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__
2024-12-18T01:59:04.3434176Z     self._skip_by_device_name()
2024-12-18T01:59:04.3434773Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name
2024-12-18T01:59:04.3435673Z     raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.")
2024-12-18T01:59:04.3436471Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llava/metadata.yaml`.
2024-12-18T01:59:04.3436885Z 
2024-12-18T01:59:04.3436987Z model_fail_to_load
2024-12-18T01:59:09.2282766Z 
2024-12-18T01:59:10.0593786Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:59:10.0594280Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:59:10.0594737Z cuda train maml_omniglot                      
2024-12-18T01:59:19.1732655Z W1218 01:59:19.172000 49986 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T01:59:24.5644074Z pass
2024-12-18T01:59:24.5645437Z TIMING: entire_frame_compile:9.08725 _recursive_pre_grad_passes:0.01091 _recursive_joint_graph_passes:1.45471 _recursive_post_grad_passes:0.06286 async_compile.wait:3.799 code_gen:7.05294 inductor_compile:8.17238 backend_compile:7.88452 entire_backward_compile:3.29739 total_wall_time:12.38463
2024-12-18T01:59:24.5647012Z STATS: call_* op count: 107 | FakeTensorMode.__torch_dispatch__:4160 | ProxyTorchDispatchMode.__torch_dispatch__:1637 | FakeTensor.__torch_dispatch__:1061
2024-12-18T01:59:24.5647851Z Dynamo produced 3 graphs covering 107 ops with 7 graph breaks (5 unique)
2024-12-18T01:59:29.1902527Z 
2024-12-18T01:59:31.9606724Z loading model: 0it [00:00, ?it/s]
2024-12-18T01:59:31.9607199Z loading model: 0it [00:02, ?it/s]
2024-12-18T01:59:31.9607614Z cuda train mnasnet1_0                         
2024-12-18T02:00:07.2070350Z W1218 02:00:07.205000 50433 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:00:08.9567371Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T02:00:43.5605893Z pass
2024-12-18T02:00:43.5634740Z TIMING: entire_frame_compile:52.46201 _recursive_pre_grad_passes:0.04986 _recursive_joint_graph_passes:1.55018 _recursive_post_grad_passes:0.58054 async_compile.wait:7.24351 code_gen:21.71213 inductor_compile:33.14074 backend_compile:38.96131 entire_backward_compile:10.81375 total_wall_time:63.27576
2024-12-18T02:00:43.5637292Z STATS: call_* op count: 965 | FakeTensorMode.__torch_dispatch__:41029 | ProxyTorchDispatchMode.__torch_dispatch__:18733 | FakeTensor.__torch_dispatch__:12556
2024-12-18T02:00:43.5639532Z Dynamo produced 3 graphs covering 965 ops with 7 graph breaks (5 unique)
2024-12-18T02:00:50.2417512Z 
2024-12-18T02:00:52.5071176Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:00:52.5071539Z loading model: 0it [00:02, ?it/s]
2024-12-18T02:00:52.5071862Z cuda train mobilenet_v2                       
2024-12-18T02:01:27.5175764Z pass
2024-12-18T02:01:27.5227152Z TIMING: entire_frame_compile:18.37431 _recursive_pre_grad_passes:0.01229 _recursive_joint_graph_passes:0.76985 _recursive_post_grad_passes:0.55828 async_compile.wait:1.37056 code_gen:9.15518 inductor_compile:15.845 backend_compile:12.20036 entire_backward_compile:11.09725 total_wall_time:29.47155
2024-12-18T02:01:27.5228759Z STATS: call_* op count: 157 | FakeTensorMode.__torch_dispatch__:23076 | FakeTensor.__torch_dispatch__:6684 | ProxyTorchDispatchMode.__torch_dispatch__:10376
2024-12-18T02:01:27.5229607Z Dynamo produced 2 graphs covering 157 ops with 6 graph breaks (5 unique)
2024-12-18T02:01:32.9478362Z 
2024-12-18T02:01:35.0285052Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:01:35.0285535Z loading model: 0it [00:02, ?it/s]
2024-12-18T02:01:35.0286045Z cuda train mobilenet_v2_quantized_qat         
2024-12-18T02:01:35.0286547Z Traceback (most recent call last):
2024-12-18T02:01:35.0287081Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2732, in validate_model
2024-12-18T02:01:35.0287641Z     self.model_iter_fn(model, example_inputs)
2024-12-18T02:01:35.0288264Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 453, in forward_and_backward_pass
2024-12-18T02:01:35.0288868Z     pred = mod(*cloned_inputs)
2024-12-18T02:01:35.0289439Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 822, in call_wrapped
2024-12-18T02:01:35.0290079Z     return self._wrapped_call(self, *args, **kwargs)
2024-12-18T02:01:35.0290678Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 400, in __call__
2024-12-18T02:01:35.0291263Z     raise e
2024-12-18T02:01:35.0291735Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 387, in __call__
2024-12-18T02:01:35.0292421Z     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
2024-12-18T02:01:35.0293165Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T02:01:35.0293803Z     return self._call_impl(*args, **kwargs)
2024-12-18T02:01:35.0294400Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T02:01:35.0295017Z     return forward_call(*args, **kwargs)
2024-12-18T02:01:35.0295346Z   File "<eval_with_key>.3", line 207, in forward
2024-12-18T02:01:35.0295894Z     activation_post_process_101 = self.activation_post_process_101(classifier_1);  classifier_1 = None
2024-12-18T02:01:35.0296732Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T02:01:35.0297396Z     return self._call_impl(*args, **kwargs)
2024-12-18T02:01:35.0299059Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T02:01:35.0299682Z     return forward_call(*args, **kwargs)
2024-12-18T02:01:35.0300308Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 408, in forward
2024-12-18T02:01:35.0300988Z     return torch.fused_moving_avg_obs_fake_quant(
2024-12-18T02:01:35.0301391Z RuntimeError: expected scalar type Float but found Half
2024-12-18T02:01:35.0301750Z 
2024-12-18T02:01:35.0301970Z The above exception was the direct cause of the following exception:
2024-12-18T02:01:35.0302280Z 
2024-12-18T02:01:35.0302398Z Traceback (most recent call last):
2024-12-18T02:01:35.0302869Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run
2024-12-18T02:01:35.0303512Z     ) = runner.load_model(
2024-12-18T02:01:35.0304003Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 372, in load_model
2024-12-18T02:01:35.0304561Z     self.validate_model(model, example_inputs)
2024-12-18T02:01:35.0305107Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2734, in validate_model
2024-12-18T02:01:35.0305663Z     raise RuntimeError("Eager run failed") from e
2024-12-18T02:01:35.0306002Z RuntimeError: Eager run failed
2024-12-18T02:01:35.0306195Z 
2024-12-18T02:01:35.0306288Z eager_fail_to_run
2024-12-18T02:01:39.0887681Z 
2024-12-18T02:01:41.7602355Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:01:41.7602806Z loading model: 0it [00:02, ?it/s]
2024-12-18T02:01:41.7603223Z cuda train mobilenet_v3_large                 
2024-12-18T02:02:22.4160274Z W1218 02:02:22.415000 52312 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:02:24.3969693Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T02:03:02.2662382Z E1218 02:03:02.265000 52312 site-packages/torch/_dynamo/utils.py:2247] RMSE (res-fp64): 4.55295, (ref-fp64): 1.49158 and shape=torch.Size([112]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000, use_larger_multiplier_for_smaller_tensor: 0
2024-12-18T02:03:02.2665762Z E1218 02:03:02.265000 52312 site-packages/torch/_dynamo/utils.py:2117] Accuracy failed for key name features.11.block.3.1.weight.grad
2024-12-18T02:03:02.2924689Z pass
2024-12-18T02:03:02.2964754Z TIMING: entire_frame_compile:58.87478 _recursive_pre_grad_passes:0.05432 _recursive_joint_graph_passes:1.60662 _recursive_post_grad_passes:0.80719 async_compile.wait:6.39181 code_gen:24.47171 inductor_compile:38.22874 backend_compile:43.232 entire_backward_compile:13.14395 total_wall_time:72.01874
2024-12-18T02:03:02.2966919Z STATS: call_* op count: 1080 | FakeTensorMode.__torch_dispatch__:46351 | FakeTensor.__torch_dispatch__:15096 | ProxyTorchDispatchMode.__torch_dispatch__:20792
2024-12-18T02:03:02.2967893Z Dynamo produced 3 graphs covering 1080 ops with 7 graph breaks (5 unique)
2024-12-18T02:03:09.4314446Z 
2024-12-18T02:03:12.0094453Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:03:12.0094796Z loading model: 0it [00:02, ?it/s]
2024-12-18T02:03:12.0095114Z cuda train moco                               
2024-12-18T02:03:17.6255044Z [rank0]:W1218 02:03:17.624000 53517 site-packages/torch/_logging/_internal.py:1089] [3/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:03:26.1665930Z [rank0]:W1218 02:03:26.165000 53517 site-packages/torch/_dynamo/backends/distributed.py:88] [4/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance.
2024-12-18T02:03:26.1911588Z [rank0]:W1218 02:03:26.190000 53517 site-packages/torch/_dynamo/backends/distributed.py:105] [4/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes:
2024-12-18T02:03:26.1913461Z [rank0]:W1218 02:03:26.190000 53517 site-packages/torch/_dynamo/backends/distributed.py:105] [4/0_1] ┌─────────┬─────────────┬────────────────────────┐
2024-12-18T02:03:26.1914803Z [rank0]:W1218 02:03:26.190000 53517 site-packages/torch/_dynamo/backends/distributed.py:105] [4/0_1] │   Index │   Extra Ops │   Extra Param Size (b) │
2024-12-18T02:03:26.1915969Z [rank0]:W1218 02:03:26.190000 53517 site-packages/torch/_dynamo/backends/distributed.py:105] [4/0_1] ├─────────┼─────────────┼────────────────────────┤
2024-12-18T02:03:26.1917431Z [rank0]:W1218 02:03:26.190000 53517 site-packages/torch/_dynamo/backends/distributed.py:105] [4/0_1] │       0 │         480 │               94032128 │
2024-12-18T02:03:26.1918597Z [rank0]:W1218 02:03:26.190000 53517 site-packages/torch/_dynamo/backends/distributed.py:105] [4/0_1] └─────────┴─────────────┴────────────────────────┘
2024-12-18T02:03:50.5907127Z skipping cudagraphs due to mutated inputs (161 instances)
2024-12-18T02:03:50.7033660Z [rank0]:W1218 02:03:50.702000 53517 site-packages/torch/_inductor/utils.py:1543] [5/0_1] DeviceCopy in input program
2024-12-18T02:03:50.7173053Z skipping cudagraphs due to skipping cudagraphs due to cpu device (randperm). Found from : 
2024-12-18T02:03:50.7174430Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 82, in _batch_shuffle_ddp
2024-12-18T02:03:50.7175595Z     idx_shuffle = torch.randperm(batch_size_all).cuda()
2024-12-18T02:03:50.7176007Z 
2024-12-18T02:03:53.3695644Z [rank0]:W1218 02:03:53.368000 53517 site-packages/torch/_dynamo/variables/tensor.py:869] [7/0] Graph break from `Tensor.item()`, consider setting:
2024-12-18T02:03:53.3697682Z [rank0]:W1218 02:03:53.368000 53517 site-packages/torch/_dynamo/variables/tensor.py:869] [7/0]     torch._dynamo.config.capture_scalar_outputs = True
2024-12-18T02:03:53.3698548Z [rank0]:W1218 02:03:53.368000 53517 site-packages/torch/_dynamo/variables/tensor.py:869] [7/0] or:
2024-12-18T02:03:53.3699372Z [rank0]:W1218 02:03:53.368000 53517 site-packages/torch/_dynamo/variables/tensor.py:869] [7/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2024-12-18T02:03:53.3700341Z [rank0]:W1218 02:03:53.368000 53517 site-packages/torch/_dynamo/variables/tensor.py:869] [7/0] to include these operations in the captured graph.
2024-12-18T02:03:53.3701160Z [rank0]:W1218 02:03:53.368000 53517 site-packages/torch/_dynamo/variables/tensor.py:869] [7/0] 
2024-12-18T02:03:53.3701928Z [rank0]:W1218 02:03:53.368000 53517 site-packages/torch/_dynamo/variables/tensor.py:869] [7/0] Graph break: from user code at:
2024-12-18T02:03:53.3703368Z [rank0]:W1218 02:03:53.368000 53517 site-packages/torch/_dynamo/variables/tensor.py:869] [7/0]   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 158, in torch_dynamo_resume_in_forward_at_133
2024-12-18T02:03:53.3704736Z [rank0]:W1218 02:03:53.368000 53517 site-packages/torch/_dynamo/variables/tensor.py:869] [7/0]     self._dequeue_and_enqueue(k)
2024-12-18T02:03:53.3705938Z [rank0]:W1218 02:03:53.368000 53517 site-packages/torch/_dynamo/variables/tensor.py:869] [7/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2024-12-18T02:03:53.3707152Z [rank0]:W1218 02:03:53.368000 53517 site-packages/torch/_dynamo/variables/tensor.py:869] [7/0]     return func(*args, **kwargs)
2024-12-18T02:03:53.3708460Z [rank0]:W1218 02:03:53.368000 53517 site-packages/torch/_dynamo/variables/tensor.py:869] [7/0]   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 59, in _dequeue_and_enqueue
2024-12-18T02:03:53.3710122Z [rank0]:W1218 02:03:53.368000 53517 site-packages/torch/_dynamo/variables/tensor.py:869] [7/0]     ptr = int(self.queue_ptr)
2024-12-18T02:03:53.3710882Z [rank0]:W1218 02:03:53.368000 53517 site-packages/torch/_dynamo/variables/tensor.py:869] [7/0] 
2024-12-18T02:03:53.3711526Z [rank0]:W1218 02:03:53.368000 53517 site-packages/torch/_dynamo/variables/tensor.py:869] [7/0] 
2024-12-18T02:04:25.6495550Z pass
2024-12-18T02:04:25.6826049Z TIMING: entire_frame_compile:49.21017 _recursive_pre_grad_passes:0.14375 _recursive_joint_graph_passes:1.54467 inductor_compile:29.39135 backend_compile:32.70985 _recursive_post_grad_passes:0.80716 async_compile.wait:4.53234 code_gen:17.99844 entire_backward_compile:13.35023 total_wall_time:62.5604
2024-12-18T02:04:25.6827708Z STATS: call_* op count: 889 | FakeTensorMode.__torch_dispatch__:47096 | ProxyTorchDispatchMode.__torch_dispatch__:20314 | FakeTensor.__torch_dispatch__:10784
2024-12-18T02:04:25.6828907Z Dynamo produced 10 graphs covering 889 ops with 11 graph breaks (8 unique)
2024-12-18T02:04:27.1551131Z [rank0]:[W1218 02:04:27.087963805 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2024-12-18T02:04:32.7236007Z 
2024-12-18T02:04:34.7383406Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M
2024-12-18T02:04:35.0121490Z num decayed parameter tensors: 50, with 124,354,560 parameters
2024-12-18T02:04:35.0121976Z num non-decayed parameter tensors: 98, with 121,344 parameters
2024-12-18T02:04:35.0125388Z using fused AdamW: True
2024-12-18T02:04:35.5438448Z 
2024-12-18T02:04:35.5438939Z loading model: 0it [00:02, ?it/s]
2024-12-18T02:04:35.5442267Z cuda train nanogpt                            
2024-12-18T02:04:59.6989216Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:04:59.6990109Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 453, in torch_dynamo_resume_in_forward_and_backward_pass_at_448
2024-12-18T02:04:59.6990816Z     pred = mod(*cloned_inputs)
2024-12-18T02:04:59.6991412Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt/model.py", line 228, in forward
2024-12-18T02:04:59.6992177Z     tok_emb = self.transformer.wte(idx)  # token embeddings of shape (b, t, n_embd)
2024-12-18T02:04:59.6992524Z 
2024-12-18T02:04:59.7793108Z W1218 02:04:59.778000 54636 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:05:01.0413184Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T02:05:29.8862545Z pass
2024-12-18T02:05:29.9474577Z TIMING: entire_frame_compile:42.89876 _recursive_pre_grad_passes:0.04951 _recursive_joint_graph_passes:1.97226 _recursive_post_grad_passes:0.75519 async_compile.wait:4.15751 code_gen:16.87051 inductor_compile:26.70503 backend_compile:33.84352 entire_backward_compile:6.94129 total_wall_time:49.84005
2024-12-18T02:05:29.9476359Z STATS: call_* op count: 1156 | FakeTensorMode.__torch_dispatch__:44451 | FakeTensor.__torch_dispatch__:11519 | ProxyTorchDispatchMode.__torch_dispatch__:20481
2024-12-18T02:05:29.9477277Z Dynamo produced 3 graphs covering 1156 ops with 7 graph breaks (5 unique)
2024-12-18T02:05:36.1980813Z 
2024-12-18T02:05:38.9052077Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:05:38.9052459Z loading model: 0it [00:02, ?it/s]
2024-12-18T02:05:38.9052794Z cuda train nvidia_deeprecommender             
2024-12-18T02:05:47.3360386Z W1218 02:05:47.335000 55002 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:05:52.1932608Z pass
2024-12-18T02:05:52.2233319Z TIMING: entire_frame_compile:9.68007 _recursive_pre_grad_passes:0.00943 _recursive_joint_graph_passes:3.79872 _recursive_post_grad_passes:0.07834 async_compile.wait:2.49848 code_gen:3.49833 inductor_compile:4.54411 backend_compile:8.66999 entire_backward_compile:0.84899 total_wall_time:10.52907
2024-12-18T02:05:52.2235627Z STATS: call_* op count: 96 | FakeTensorMode.__torch_dispatch__:3816 | ProxyTorchDispatchMode.__torch_dispatch__:1592 | FakeTensor.__torch_dispatch__:1078
2024-12-18T02:05:52.2236452Z Dynamo produced 3 graphs covering 96 ops with 7 graph breaks (5 unique)
2024-12-18T02:05:56.7184628Z 
2024-12-18T02:05:57.9117916Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:05:57.9118270Z loading model: 0it [00:01, ?it/s]
2024-12-18T02:05:57.9118598Z cuda train opacus_cifar10                     
2024-12-18T02:05:57.9134291Z Traceback (most recent call last):
2024-12-18T02:05:57.9134999Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2732, in validate_model
2024-12-18T02:05:57.9135612Z     self.model_iter_fn(model, example_inputs)
2024-12-18T02:05:57.9136891Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 455, in forward_and_backward_pass
2024-12-18T02:05:57.9137540Z     self.grad_scaler.scale(loss).backward()
2024-12-18T02:05:57.9138111Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 626, in backward
2024-12-18T02:05:57.9138665Z     torch.autograd.backward(
2024-12-18T02:05:57.9139226Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward
2024-12-18T02:05:57.9139799Z     _engine_run_backward(
2024-12-18T02:05:57.9140380Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 823, in _engine_run_backward
2024-12-18T02:05:57.9141216Z     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2024-12-18T02:05:57.9142127Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 98, in __call__
2024-12-18T02:05:57.9142901Z     return self.hook(module, *args, **kwargs)
2024-12-18T02:05:57.9143617Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 328, in capture_backprops_hook
2024-12-18T02:05:57.9144380Z     activations, backprops = self.rearrange_grad_samples(
2024-12-18T02:05:57.9145137Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 384, in rearrange_grad_samples
2024-12-18T02:05:57.9145815Z     raise ValueError(
2024-12-18T02:05:57.9146365Z ValueError: No activations detected for <class 'torch.nn.modules.linear.Linear'>, run forward after add_hooks(model)
2024-12-18T02:05:57.9146860Z 
2024-12-18T02:05:57.9147078Z The above exception was the direct cause of the following exception:
2024-12-18T02:05:57.9147399Z 
2024-12-18T02:05:57.9147514Z Traceback (most recent call last):
2024-12-18T02:05:57.9147998Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run
2024-12-18T02:05:57.9148485Z     ) = runner.load_model(
2024-12-18T02:05:57.9148976Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 372, in load_model
2024-12-18T02:05:57.9149539Z     self.validate_model(model, example_inputs)
2024-12-18T02:05:57.9150097Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2734, in validate_model
2024-12-18T02:05:57.9150663Z     raise RuntimeError("Eager run failed") from e
2024-12-18T02:05:57.9151017Z RuntimeError: Eager run failed
2024-12-18T02:05:57.9151206Z 
2024-12-18T02:05:57.9151311Z eager_fail_to_run
2024-12-18T02:06:02.0027703Z 
2024-12-18T02:06:03.0983914Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:06:03.0984274Z loading model: 0it [00:01, ?it/s]
2024-12-18T02:06:03.0984611Z cuda train phlippe_densenet                   
2024-12-18T02:06:35.6380537Z pass
2024-12-18T02:06:35.6386475Z TIMING: entire_frame_compile:20.85598 _recursive_pre_grad_passes:0.01331 _recursive_joint_graph_passes:1.43549 _recursive_post_grad_passes:0.48031 async_compile.wait:0.80763 code_gen:9.19999 inductor_compile:14.59455 backend_compile:14.05257 entire_backward_compile:8.92916 total_wall_time:29.78514
2024-12-18T02:06:35.6388280Z STATS: call_* op count: 190 | FakeTensorMode.__torch_dispatch__:23806 | ProxyTorchDispatchMode.__torch_dispatch__:11247 | FakeTensor.__torch_dispatch__:5670
2024-12-18T02:06:35.6389242Z Dynamo produced 2 graphs covering 190 ops with 6 graph breaks (5 unique)
2024-12-18T02:06:40.8851456Z 
2024-12-18T02:06:41.8143902Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:06:41.8144289Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:06:41.8144630Z cuda train phlippe_resnet                     
2024-12-18T02:06:54.7340785Z pass
2024-12-18T02:06:54.7342185Z TIMING: entire_frame_compile:7.93465 _recursive_pre_grad_passes:0.00702 _recursive_joint_graph_passes:1.32286 _recursive_post_grad_passes:0.18753 async_compile.wait:0.92532 code_gen:2.90866 inductor_compile:4.91762 backend_compile:5.8042 entire_backward_compile:2.9047 total_wall_time:10.83934
2024-12-18T02:06:54.7344409Z STATS: call_* op count: 75 | FakeTensorMode.__torch_dispatch__:8717 | ProxyTorchDispatchMode.__torch_dispatch__:4228 | FakeTensor.__torch_dispatch__:2125
2024-12-18T02:06:54.7345377Z Dynamo produced 2 graphs covering 75 ops with 6 graph breaks (5 unique)
2024-12-18T02:06:59.4574619Z 
2024-12-18T02:07:00.9313161Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:07:00.9313621Z loading model: 0it [00:01, ?it/s]
2024-12-18T02:07:00.9314048Z cuda train pytorch_CycleGAN_and_pix2pix       
2024-12-18T02:07:34.5086406Z pass
2024-12-18T02:07:34.5178137Z TIMING: entire_frame_compile:11.03627 _recursive_pre_grad_passes:0.00683 _recursive_joint_graph_passes:0.41729 _recursive_post_grad_passes:0.35981 async_compile.wait:2.5789 code_gen:8.71631 inductor_compile:13.19985 backend_compile:9.05162 entire_backward_compile:8.87458 total_wall_time:19.91085
2024-12-18T02:07:34.5180270Z STATS: call_* op count: 95 | FakeTensorMode.__torch_dispatch__:13217 | ProxyTorchDispatchMode.__torch_dispatch__:7137 | FakeTensor.__torch_dispatch__:3594
2024-12-18T02:07:34.5181113Z Dynamo produced 2 graphs covering 95 ops with 6 graph breaks (5 unique)
2024-12-18T02:07:39.5650353Z 
2024-12-18T02:07:41.2550775Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:07:41.2551129Z loading model: 0it [00:01, ?it/s]
2024-12-18T02:07:41.2551454Z cuda train pytorch_stargan                    
2024-12-18T02:08:20.6084491Z pass
2024-12-18T02:08:20.6162618Z TIMING: entire_frame_compile:8.52887 _recursive_pre_grad_passes:0.00653 _recursive_joint_graph_passes:0.35556 _recursive_post_grad_passes:0.17892 async_compile.wait:0.92402 code_gen:3.32207 inductor_compile:6.06782 backend_compile:6.49611 entire_backward_compile:2.93509 total_wall_time:11.46396
2024-12-18T02:08:20.6164206Z STATS: call_* op count: 60 | FakeTensorMode.__torch_dispatch__:11805 | FakeTensor.__torch_dispatch__:2199 | ProxyTorchDispatchMode.__torch_dispatch__:5359
2024-12-18T02:08:20.6165064Z Dynamo produced 2 graphs covering 60 ops with 6 graph breaks (5 unique)
2024-12-18T02:08:25.3638203Z 
2024-12-18T02:08:26.4973139Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:08:26.4973507Z loading model: 0it [00:01, ?it/s]
2024-12-18T02:08:26.4973837Z cuda train pytorch_unet                       
2024-12-18T02:09:18.1130240Z W1218 02:09:18.112000 57017 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:09:36.4549859Z pass_due_to_skip
2024-12-18T02:09:36.4648406Z TIMING: entire_frame_compile:27.97513 _recursive_pre_grad_passes:0.02637 _recursive_joint_graph_passes:0.28712 _recursive_post_grad_passes:0.33085 async_compile.wait:5.205 code_gen:18.50567 inductor_compile:24.99595 backend_compile:22.13401 entire_backward_compile:10.09147 total_wall_time:38.0666
2024-12-18T02:09:36.4651113Z STATS: call_* op count: 464 | FakeTensorMode.__torch_dispatch__:21319 | ProxyTorchDispatchMode.__torch_dispatch__:9642 | FakeTensor.__torch_dispatch__:6155
2024-12-18T02:09:36.4652002Z Dynamo produced 3 graphs covering 464 ops with 7 graph breaks (5 unique)
2024-12-18T02:09:41.9941368Z 
2024-12-18T02:09:44.6854172Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:09:44.6854827Z loading model: 0it [00:02, ?it/s]
2024-12-18T02:09:44.6855446Z cuda train resnet152                          
2024-12-18T02:11:03.0956925Z W1218 02:11:03.094000 58046 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:11:08.4845202Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][195].grad", "L['self'].param_groups[0]['params'][204].grad", "L['self'].param_groups[0]['params'][213].grad", "L['self'].param_groups[0]['params'][222].grad", "L['self'].param_groups[0]['params'][231].grad", "L['self'].param_groups[0]['params'][240].grad", "L['self'].param_groups[0]['params'][249].grad", "L['self'].param_groups[0]['params'][258].grad", "L['self'].param_groups[0]['params'][267].grad", "L['self'].param_groups[0]['params'][276].grad", "L['self'].param_groups[0]['params'][285].grad", "L['self'].param_groups[0]['params'][294].grad", "L['self'].param_groups[0]['params'][303].grad", "L['self'].param_groups[0]['params'][312].grad", "L['self'].param_groups[0]['params'][321].grad", "L['self'].param_groups[0]['params'][330].grad", "L['self'].param_groups[0]['params'][339].grad", "L['self'].param_groups[0]['params'][348].grad", "L['self'].param_groups[0]['params'][357].grad", "L['self'].param_groups[0]['params'][366].grad", "L['self'].param_groups[0]['params'][375].grad", "L['self'].param_groups[0]['params'][384].grad", "L['self'].param_groups[0]['params'][393].grad", "L['self'].param_groups[0]['params'][402].grad", "L['self'].param_groups[0]['params'][411].grad", "L['self'].param_groups[0]['params'][420].grad", "L['self'].param_groups[0]['params'][429].grad", "L['self'].param_groups[0]['params'][438].grad", "L['self'].param_groups[0]['params'][450].grad", "L['self'].param_groups[0]['params'][459].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T02:12:52.4727709Z pass
2024-12-18T02:12:52.5113529Z TIMING: entire_frame_compile:146.99336 _recursive_pre_grad_passes:0.14673 _recursive_joint_graph_passes:2.23041 _recursive_post_grad_passes:1.75572 async_compile.wait:8.13592 code_gen:49.90659 inductor_compile:81.90665 backend_compile:105.32632 entire_backward_compile:22.75562 total_wall_time:169.74898
2024-12-18T02:12:52.5115212Z STATS: call_* op count: 2873 | FakeTensorMode.__torch_dispatch__:118049 | ProxyTorchDispatchMode.__torch_dispatch__:56933 | FakeTensor.__torch_dispatch__:36957
2024-12-18T02:12:52.5116087Z Dynamo produced 3 graphs covering 2873 ops with 7 graph breaks (5 unique)
2024-12-18T02:13:03.5368703Z 
2024-12-18T02:13:05.0611487Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:13:05.0611967Z loading model: 0it [00:01, ?it/s]
2024-12-18T02:13:05.0612313Z cuda train resnet18                           
2024-12-18T02:13:22.8528109Z pass
2024-12-18T02:13:22.8601510Z TIMING: entire_frame_compile:8.61842 _recursive_pre_grad_passes:0.00731 _recursive_joint_graph_passes:1.31255 _recursive_post_grad_passes:0.38216 async_compile.wait:1.08578 code_gen:5.81466 inductor_compile:8.45925 backend_compile:6.5182 entire_backward_compile:5.67827 total_wall_time:14.29669
2024-12-18T02:13:22.8603185Z STATS: call_* op count: 73 | FakeTensorMode.__torch_dispatch__:8948 | ProxyTorchDispatchMode.__torch_dispatch__:4224 | FakeTensor.__torch_dispatch__:2150
2024-12-18T02:13:22.8604027Z Dynamo produced 2 graphs covering 73 ops with 6 graph breaks (5 unique)
2024-12-18T02:13:27.7345185Z 
2024-12-18T02:13:29.8233441Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:13:29.8233796Z loading model: 0it [00:02, ?it/s]
2024-12-18T02:13:29.8234112Z cuda train resnet50                           
2024-12-18T02:14:03.9835410Z pass
2024-12-18T02:14:04.0003705Z TIMING: entire_frame_compile:17.89625 _recursive_pre_grad_passes:0.01282 _recursive_joint_graph_passes:0.77549 _recursive_post_grad_passes:0.46044 async_compile.wait:0.43184 code_gen:8.72095 inductor_compile:14.12974 backend_compile:12.13417 entire_backward_compile:9.43663 total_wall_time:27.33288
2024-12-18T02:14:04.0005501Z STATS: call_* op count: 179 | FakeTensorMode.__torch_dispatch__:22379 | ProxyTorchDispatchMode.__torch_dispatch__:11011 | FakeTensor.__torch_dispatch__:5585
2024-12-18T02:14:04.0006369Z Dynamo produced 2 graphs covering 179 ops with 6 graph breaks (5 unique)
2024-12-18T02:14:09.3548601Z 
2024-12-18T02:14:11.6287094Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:14:11.6287648Z loading model: 0it [00:02, ?it/s]
2024-12-18T02:14:11.6288112Z cuda train resnet50_quantized_qat             
2024-12-18T02:14:11.6288664Z Traceback (most recent call last):
2024-12-18T02:14:11.6289213Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2732, in validate_model
2024-12-18T02:14:11.6290372Z     self.model_iter_fn(model, example_inputs)
2024-12-18T02:14:11.6291014Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 453, in forward_and_backward_pass
2024-12-18T02:14:11.6291621Z     pred = mod(*cloned_inputs)
2024-12-18T02:14:11.6292193Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 822, in call_wrapped
2024-12-18T02:14:11.6292824Z     return self._wrapped_call(self, *args, **kwargs)
2024-12-18T02:14:11.6293434Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 400, in __call__
2024-12-18T02:14:11.6293980Z     raise e
2024-12-18T02:14:11.6294460Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 387, in __call__
2024-12-18T02:14:11.6295143Z     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
2024-12-18T02:14:11.6295890Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T02:14:11.6296539Z     return self._call_impl(*args, **kwargs)
2024-12-18T02:14:11.6297141Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T02:14:11.6297732Z     return forward_call(*args, **kwargs)
2024-12-18T02:14:11.6298068Z   File "<eval_with_key>.3", line 167, in forward
2024-12-18T02:14:11.6298528Z     activation_post_process_73 = self.activation_post_process_73(fc);  fc = None
2024-12-18T02:14:11.6299296Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
2024-12-18T02:14:11.6299953Z     return self._call_impl(*args, **kwargs)
2024-12-18T02:14:11.6300561Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
2024-12-18T02:14:11.6301178Z     return forward_call(*args, **kwargs)
2024-12-18T02:14:11.6301909Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 408, in forward
2024-12-18T02:14:11.6302582Z     return torch.fused_moving_avg_obs_fake_quant(
2024-12-18T02:14:11.6302984Z RuntimeError: expected scalar type Float but found Half
2024-12-18T02:14:11.6303254Z 
2024-12-18T02:14:11.6303470Z The above exception was the direct cause of the following exception:
2024-12-18T02:14:11.6303781Z 
2024-12-18T02:14:11.6303901Z Traceback (most recent call last):
2024-12-18T02:14:11.6304364Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run
2024-12-18T02:14:11.6304854Z     ) = runner.load_model(
2024-12-18T02:14:11.6305337Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 372, in load_model
2024-12-18T02:14:11.6305893Z     self.validate_model(model, example_inputs)
2024-12-18T02:14:11.6306441Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2734, in validate_model
2024-12-18T02:14:11.6307006Z     raise RuntimeError("Eager run failed") from e
2024-12-18T02:14:11.6307348Z RuntimeError: Eager run failed
2024-12-18T02:14:11.6307536Z 
2024-12-18T02:14:11.6307805Z eager_fail_to_run
2024-12-18T02:14:15.7327160Z 
2024-12-18T02:14:17.9552628Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:14:17.9552978Z loading model: 0it [00:02, ?it/s]
2024-12-18T02:14:17.9553289Z cuda train resnext50_32x4d                    
2024-12-18T02:14:52.3260605Z W1218 02:14:52.325000 59206 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:15:28.7538715Z pass
2024-12-18T02:15:28.7659943Z TIMING: entire_frame_compile:51.4672 _recursive_pre_grad_passes:0.05132 _recursive_joint_graph_passes:0.91663 _recursive_post_grad_passes:0.60179 async_compile.wait:5.20066 code_gen:20.3932 inductor_compile:31.13734 backend_compile:38.00583 entire_backward_compile:9.24663 total_wall_time:60.71383
2024-12-18T02:15:28.7663328Z STATS: call_* op count: 1003 | FakeTensorMode.__torch_dispatch__:41052 | ProxyTorchDispatchMode.__torch_dispatch__:19567 | FakeTensor.__torch_dispatch__:12775
2024-12-18T02:15:28.7665250Z Dynamo produced 3 graphs covering 1003 ops with 7 graph breaks (5 unique)
2024-12-18T02:15:35.3776349Z 
2024-12-18T02:15:43.2385582Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:15:43.2385945Z loading model: 0it [00:07, ?it/s]
2024-12-18T02:15:43.2386272Z cuda train sam                                
2024-12-18T02:15:43.2395935Z Traceback (most recent call last):
2024-12-18T02:15:43.2396497Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2732, in validate_model
2024-12-18T02:15:43.2397061Z     self.model_iter_fn(model, example_inputs)
2024-12-18T02:15:43.2397681Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 455, in forward_and_backward_pass
2024-12-18T02:15:43.2398316Z     self.grad_scaler.scale(loss).backward()
2024-12-18T02:15:43.2398873Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 626, in backward
2024-12-18T02:15:43.2399441Z     torch.autograd.backward(
2024-12-18T02:15:43.2400010Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward
2024-12-18T02:15:43.2400581Z     _engine_run_backward(
2024-12-18T02:15:43.2401157Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 823, in _engine_run_backward
2024-12-18T02:15:43.2401986Z     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2024-12-18T02:15:43.2402667Z RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
2024-12-18T02:15:43.2403027Z 
2024-12-18T02:15:43.2403238Z The above exception was the direct cause of the following exception:
2024-12-18T02:15:43.2403555Z 
2024-12-18T02:15:43.2403668Z Traceback (most recent call last):
2024-12-18T02:15:43.2404135Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4845, in run
2024-12-18T02:15:43.2404632Z     ) = runner.load_model(
2024-12-18T02:15:43.2405123Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 372, in load_model
2024-12-18T02:15:43.2405675Z     self.validate_model(model, example_inputs)
2024-12-18T02:15:43.2406209Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2734, in validate_model
2024-12-18T02:15:43.2406768Z     raise RuntimeError("Eager run failed") from e
2024-12-18T02:15:43.2407114Z RuntimeError: Eager run failed
2024-12-18T02:15:43.2407300Z 
2024-12-18T02:15:43.2407400Z eager_fail_to_run
2024-12-18T02:15:47.3030415Z 
2024-12-18T02:15:48.8750636Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:15:48.8750999Z loading model: 0it [00:01, ?it/s]
2024-12-18T02:15:48.8751331Z cuda train shufflenet_v2_x1_0                 
2024-12-18T02:16:29.4599427Z pass
2024-12-18T02:16:29.4610553Z TIMING: entire_frame_compile:21.36562 _recursive_pre_grad_passes:0.0152 _recursive_joint_graph_passes:1.48611 _recursive_post_grad_passes:0.84585 async_compile.wait:1.37219 code_gen:14.54099 inductor_compile:21.56361 backend_compile:15.07654 entire_backward_compile:15.16391 total_wall_time:36.52953
2024-12-18T02:16:29.4612201Z STATS: call_* op count: 275 | FakeTensorMode.__torch_dispatch__:25416 | ProxyTorchDispatchMode.__torch_dispatch__:11975 | FakeTensor.__torch_dispatch__:6647
2024-12-18T02:16:29.4613047Z Dynamo produced 2 graphs covering 275 ops with 6 graph breaks (5 unique)
2024-12-18T02:16:35.0770530Z 
2024-12-18T02:16:36.1777859Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:16:36.1778221Z loading model: 0it [00:01, ?it/s]
2024-12-18T02:16:36.1778546Z cuda train soft_actor_critic                  
2024-12-18T02:16:41.5398513Z W1218 02:16:41.538000 60434 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:16:44.1421314Z pass
2024-12-18T02:16:44.1430129Z TIMING: entire_frame_compile:4.97768 _recursive_pre_grad_passes:0.00838 _recursive_joint_graph_passes:1.69029 _recursive_post_grad_passes:0.05664 async_compile.wait:1.28301 code_gen:1.88241 inductor_compile:2.69631 backend_compile:4.32116 entire_backward_compile:0.70892 total_wall_time:5.6866
2024-12-18T02:16:44.1432130Z STATS: call_* op count: 70 | FakeTensorMode.__torch_dispatch__:2345 | ProxyTorchDispatchMode.__torch_dispatch__:941 | FakeTensor.__torch_dispatch__:514
2024-12-18T02:16:44.1432951Z Dynamo produced 4 graphs covering 70 ops with 6 graph breaks (5 unique)
2024-12-18T02:16:48.6579168Z 
2024-12-18T02:16:50.1158722Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:16:50.1159083Z loading model: 0it [00:01, ?it/s]
2024-12-18T02:16:50.1159422Z cuda train speech_transformer                 
2024-12-18T02:17:08.0831264Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_14). Found from : 
2024-12-18T02:17:08.0832446Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/encoder.py", line 58, in torch_dynamo_resume_in_forward_at_50
2024-12-18T02:17:08.0833417Z     enc_output, enc_slf_attn = enc_layer(
2024-12-18T02:17:08.0834253Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/encoder.py", line 84, in forward
2024-12-18T02:17:08.0835071Z     enc_output, enc_slf_attn = self.slf_attn(
2024-12-18T02:17:08.0835903Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 54, in forward
2024-12-18T02:17:08.0836753Z     output, attn = self.attention(q, k, v, mask=mask)
2024-12-18T02:17:08.0837597Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 77, in forward
2024-12-18T02:17:08.0838416Z     attn = attn / self.temperature
2024-12-18T02:17:08.0838619Z 
2024-12-18T02:17:08.9069987Z W1218 02:17:08.906000 60661 site-packages/torch/_functorch/_aot_autograd/autograd_cache.py:856] [13/0_1] Bypassing autograd cache due to: Cannot cache a graph with functional tensor
2024-12-18T02:17:35.4553391Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_12). Found from : 
2024-12-18T02:17:35.4554554Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/decoder.py", line 105, in torch_dynamo_resume_in_forward_at_96
2024-12-18T02:17:35.4555512Z     dec_output, dec_slf_attn, dec_enc_attn = dec_layer(
2024-12-18T02:17:35.4556360Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/decoder.py", line 243, in forward
2024-12-18T02:17:35.4557181Z     dec_output, dec_slf_attn = self.slf_attn(
2024-12-18T02:17:35.4558003Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 54, in forward
2024-12-18T02:17:35.4558846Z     output, attn = self.attention(q, k, v, mask=mask)
2024-12-18T02:17:35.4559947Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 77, in forward
2024-12-18T02:17:35.4560771Z     attn = attn / self.temperature
2024-12-18T02:17:35.4560969Z 
2024-12-18T02:17:52.4772351Z pass
2024-12-18T02:17:52.5132890Z TIMING: entire_frame_compile:42.8111 _recursive_pre_grad_passes:0.0438 _recursive_joint_graph_passes:6.83622 inductor_compile:24.09684 backend_compile:35.40774 _recursive_post_grad_passes:1.59316 async_compile.precompile:0.07883 async_compile.wait:3.17747 code_gen:14.83351 entire_backward_compile:13.38068 total_wall_time:56.19178
2024-12-18T02:17:52.5134635Z STATS: call_* op count: 1460 | FakeTensorMode.__torch_dispatch__:80149 | ProxyTorchDispatchMode.__torch_dispatch__:48802 | FakeTensor.__torch_dispatch__:6917
2024-12-18T02:17:52.5135495Z Dynamo produced 10 graphs covering 1460 ops with 16 graph breaks (7 unique)
2024-12-18T02:17:58.6828886Z 
2024-12-18T02:18:00.0968633Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:18:00.0968994Z loading model: 0it [00:01, ?it/s]
2024-12-18T02:18:00.0969348Z cuda train squeezenet1_1                      
2024-12-18T02:18:18.5085660Z pass
2024-12-18T02:18:18.5107567Z TIMING: entire_frame_compile:6.26612 _recursive_pre_grad_passes:0.00623 _recursive_joint_graph_passes:0.32675 _recursive_post_grad_passes:0.10492 async_compile.wait:1.08031 code_gen:10.13718 inductor_compile:12.18372 backend_compile:4.76914 entire_backward_compile:9.05437 total_wall_time:15.3205
2024-12-18T02:18:18.5109164Z STATS: call_* op count: 70 | FakeTensorMode.__torch_dispatch__:6174 | ProxyTorchDispatchMode.__torch_dispatch__:2530 | FakeTensor.__torch_dispatch__:948
2024-12-18T02:18:18.5109966Z Dynamo produced 2 graphs covering 70 ops with 6 graph breaks (5 unique)
2024-12-18T02:18:23.3864424Z 
2024-12-18T02:18:24.0516005Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:18:24.0516262Z 
2024-12-18T02:18:24.2275283Z Loading pipeline components...:   0% 0/6 [00:00<?, ?it/s]
2024-12-18T02:18:24.2275603Z 
2024-12-18T02:18:25.4862022Z Loading pipeline components...:  50% 3/6 [00:00<00:00, 17.07it/s]
2024-12-18T02:18:25.4862397Z 
2024-12-18T02:18:25.4862685Z Loading pipeline components...:  83% 5/6 [00:01<00:00,  2.97it/s]
2024-12-18T02:18:25.4863190Z Loading pipeline components...: 100% 6/6 [00:01<00:00,  4.18it/s]
2024-12-18T02:18:27.1030231Z 
2024-12-18T02:18:27.1030839Z loading model: 0it [00:03, ?it/s]
2024-12-18T02:18:27.1031218Z cuda train stable_diffusion_text_encoder      
2024-12-18T02:18:27.1388893Z WARNING:common:fp64 golden ref were not generated for stable_diffusion_text_encoder. Setting accuracy check to cosine
2024-12-18T02:19:20.9430260Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:19:20.9431534Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 453, in torch_dynamo_resume_in_forward_and_backward_pass_at_448
2024-12-18T02:19:20.9432662Z     pred = mod(*cloned_inputs)
2024-12-18T02:19:20.9433679Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 806, in forward
2024-12-18T02:19:20.9434677Z     return self.text_model(
2024-12-18T02:19:20.9435564Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 699, in forward
2024-12-18T02:19:20.9436554Z     hidden_states = self.embeddings(input_ids=input_ids, position_ids=position_ids)
2024-12-18T02:19:20.9437360Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 224, in forward
2024-12-18T02:19:20.9438047Z     inputs_embeds = self.token_embedding(input_ids)
2024-12-18T02:19:20.9438294Z 
2024-12-18T02:19:22.1990444Z pass
2024-12-18T02:19:22.2490703Z TIMING: entire_frame_compile:39.7466 _recursive_pre_grad_passes:0.02525 _recursive_joint_graph_passes:6.31225 _recursive_post_grad_passes:2.17237 async_compile.wait:0.65913 code_gen:12.11572 inductor_compile:22.85579 backend_compile:31.10777 entire_backward_compile:11.63717 total_wall_time:51.38377
2024-12-18T02:19:22.2492865Z STATS: call_* op count: 828 | FakeTensorMode.__torch_dispatch__:68759 | FakeTensor.__torch_dispatch__:11092 | ProxyTorchDispatchMode.__torch_dispatch__:32781
2024-12-18T02:19:22.2493701Z Dynamo produced 2 graphs covering 828 ops with 5 graph breaks (5 unique)
2024-12-18T02:19:28.4824004Z 
2024-12-18T02:19:29.1498560Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:19:29.1498945Z 
2024-12-18T02:19:29.2539319Z Loading pipeline components...:   0% 0/6 [00:00<?, ?it/s]
2024-12-18T02:19:29.2539777Z 
2024-12-18T02:19:30.2947301Z Loading pipeline components...:  17% 1/6 [00:00<00:00,  9.63it/s]
2024-12-18T02:19:30.2947749Z 
2024-12-18T02:19:30.9242409Z Loading pipeline components...:  67% 4/6 [00:01<00:00,  3.32it/s]
2024-12-18T02:19:30.9242767Z 
2024-12-18T02:19:30.9243062Z Loading pipeline components...: 100% 6/6 [00:01<00:00,  3.26it/s]
2024-12-18T02:19:30.9243563Z Loading pipeline components...: 100% 6/6 [00:01<00:00,  3.38it/s]
2024-12-18T02:19:33.1404033Z 
2024-12-18T02:19:33.1404659Z loading model: 0it [00:04, ?it/s]
2024-12-18T02:19:33.1405107Z cuda train stable_diffusion_unet              
2024-12-18T02:19:33.1643381Z pass_due_to_skip
2024-12-18T02:19:33.5238808Z TIMING: total_wall_time:0
2024-12-18T02:19:33.5239195Z STATS: call_* op count: 0
2024-12-18T02:19:33.5239639Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-12-18T02:19:37.8708687Z 
2024-12-18T02:19:41.4913597Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:19:41.4913947Z loading model: 0it [00:03, ?it/s]
2024-12-18T02:19:41.4914264Z cuda train timm_efficientnet                  
2024-12-18T02:20:27.8868785Z W1218 02:20:27.885000 61995 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:20:30.3634734Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T02:21:17.1196133Z pass
2024-12-18T02:21:17.1247823Z TIMING: entire_frame_compile:70.64993 _recursive_pre_grad_passes:0.06505 _recursive_joint_graph_passes:1.02894 _recursive_post_grad_passes:0.97791 async_compile.wait:6.11028 code_gen:27.84235 inductor_compile:43.91273 backend_compile:50.62716 entire_backward_compile:14.43674 total_wall_time:85.08667
2024-12-18T02:21:17.1250785Z STATS: call_* op count: 1327 | FakeTensorMode.__torch_dispatch__:56743 | FakeTensor.__torch_dispatch__:18444 | ProxyTorchDispatchMode.__torch_dispatch__:25720
2024-12-18T02:21:17.1251640Z Dynamo produced 3 graphs covering 1327 ops with 7 graph breaks (5 unique)
2024-12-18T02:21:24.9714161Z 
2024-12-18T02:21:28.1072867Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:21:28.1073218Z loading model: 0it [00:03, ?it/s]
2024-12-18T02:21:28.1073538Z cuda train timm_regnet                        
2024-12-18T02:22:20.5674497Z pass
2024-12-18T02:22:20.6063833Z TIMING: entire_frame_compile:27.81393 _recursive_pre_grad_passes:0.01932 _recursive_joint_graph_passes:1.81297 _recursive_post_grad_passes:0.8104 async_compile.wait:1.2585 code_gen:10.73781 inductor_compile:18.71404 backend_compile:18.32581 entire_backward_compile:11.72863 total_wall_time:39.54256
2024-12-18T02:22:20.6066060Z STATS: call_* op count: 323 | FakeTensorMode.__torch_dispatch__:34593 | ProxyTorchDispatchMode.__torch_dispatch__:17103 | FakeTensor.__torch_dispatch__:9036
2024-12-18T02:22:20.6067713Z Dynamo produced 2 graphs covering 323 ops with 6 graph breaks (5 unique)
2024-12-18T02:22:26.6008323Z 
2024-12-18T02:22:28.9574890Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:22:28.9575322Z loading model: 0it [00:02, ?it/s]
2024-12-18T02:22:28.9575648Z cuda train timm_resnest                       
2024-12-18T02:22:59.7908879Z W1218 02:22:59.789000 63827 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:23:20.1912537Z pass
2024-12-18T02:23:20.1976245Z TIMING: entire_frame_compile:31.46033 _recursive_pre_grad_passes:0.03022 _recursive_joint_graph_passes:0.79642 _recursive_post_grad_passes:0.36135 async_compile.wait:4.68403 code_gen:21.68695 inductor_compile:28.64784 backend_compile:24.50618 entire_backward_compile:12.62688 total_wall_time:44.08721
2024-12-18T02:23:20.1977872Z STATS: call_* op count: 569 | FakeTensorMode.__torch_dispatch__:23457 | ProxyTorchDispatchMode.__torch_dispatch__:10391 | FakeTensor.__torch_dispatch__:6920
2024-12-18T02:23:20.1978718Z Dynamo produced 3 graphs covering 569 ops with 7 graph breaks (5 unique)
2024-12-18T02:23:26.1612664Z 
2024-12-18T02:23:28.0468933Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:23:28.0469604Z loading model: 0it [00:01, ?it/s]
2024-12-18T02:23:28.0470248Z cuda train timm_vision_transformer            
2024-12-18T02:23:55.9650485Z pass
2024-12-18T02:23:55.9835909Z TIMING: entire_frame_compile:17.18406 _recursive_pre_grad_passes:0.01129 _recursive_joint_graph_passes:4.42108 _recursive_post_grad_passes:0.59407 async_compile.wait:0.69006 code_gen:7.10818 inductor_compile:11.82627 backend_compile:13.76648 entire_backward_compile:7.21538 total_wall_time:24.39944
2024-12-18T02:23:55.9838989Z STATS: call_* op count: 267 | FakeTensorMode.__torch_dispatch__:23522 | FakeTensor.__torch_dispatch__:4976 | ProxyTorchDispatchMode.__torch_dispatch__:10943
2024-12-18T02:23:55.9840162Z Dynamo produced 2 graphs covering 267 ops with 6 graph breaks (5 unique)
2024-12-18T02:24:01.4054370Z 
2024-12-18T02:24:14.3962969Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:24:14.3963309Z loading model: 0it [00:12, ?it/s]
2024-12-18T02:24:14.3963644Z cuda train timm_vision_transformer_large      
2024-12-18T02:24:14.4196157Z pass_due_to_skip
2024-12-18T02:24:14.6874360Z TIMING: total_wall_time:0
2024-12-18T02:24:14.6874702Z STATS: call_* op count: 0
2024-12-18T02:24:14.6875090Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-12-18T02:24:18.8542863Z 
2024-12-18T02:24:21.2477492Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:24:21.2477884Z loading model: 0it [00:02, ?it/s]
2024-12-18T02:24:21.2478197Z cuda train timm_vovnet                        
2024-12-18T02:24:58.6666350Z pass
2024-12-18T02:24:58.6848971Z TIMING: entire_frame_compile:15.83837 _recursive_pre_grad_passes:0.01063 _recursive_joint_graph_passes:0.73397 _recursive_post_grad_passes:0.37718 async_compile.wait:1.31102 code_gen:14.11753 inductor_compile:18.6349 backend_compile:10.98318 entire_backward_compile:13.20936 total_wall_time:29.04773
2024-12-18T02:24:58.6850574Z STATS: call_* op count: 134 | FakeTensorMode.__torch_dispatch__:17226 | ProxyTorchDispatchMode.__torch_dispatch__:8526 | FakeTensor.__torch_dispatch__:4484
2024-12-18T02:24:58.6851561Z Dynamo produced 2 graphs covering 134 ops with 6 graph breaks (5 unique)
2024-12-18T02:25:04.1162190Z 
2024-12-18T02:25:06.3001308Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:25:06.3004040Z loading model: 0it [00:02, ?it/s]
2024-12-18T02:25:06.3004606Z cuda train torch_multimodal_clip              
2024-12-18T02:25:59.5467670Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:25:59.5468516Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 453, in torch_dynamo_resume_in_forward_and_backward_pass_at_448
2024-12-18T02:25:59.5469223Z     pred = mod(*cloned_inputs)
2024-12-18T02:25:59.5469830Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torchmultimodal/models/clip/model.py", line 72, in forward
2024-12-18T02:25:59.5470485Z     embeddings_b = self.encoder_b(features_b)
2024-12-18T02:25:59.5471142Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torchmultimodal/models/clip/text_encoder.py", line 119, in forward
2024-12-18T02:25:59.5471831Z     embeddings = self.token_embedding(text)
2024-12-18T02:25:59.5472058Z 
2024-12-18T02:25:59.6732115Z W1218 02:25:59.672000 65440 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:26:02.1982782Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][163].grad", "L['self'].param_groups[0]['params'][164].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][166].grad", "L['self'].param_groups[0]['params'][167].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][169].grad", "L['self'].param_groups[0]['params'][170].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][172].grad", "L['self'].param_groups[0]['params'][173].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][175].grad", "L['self'].param_groups[0]['params'][176].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][178].grad", "L['self'].param_groups[0]['params'][179].grad", "L['self'].param_groups[0]['params'][180].grad", "L['self'].param_groups[0]['params'][181].grad", "L['self'].param_groups[0]['params'][182].grad", "L['self'].param_groups[0]['params'][183].grad", "L['self'].param_groups[0]['params'][184].grad", "L['self'].param_groups[0]['params'][185].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][187].grad", "L['self'].param_groups[0]['params'][188].grad", "L['self'].param_groups[0]['params'][189].grad", "L['self'].param_groups[0]['params'][190].grad", "L['self'].param_groups[0]['params'][191].grad", "L['self'].param_groups[0]['params'][192].grad", "L['self'].param_groups[0]['params'][193].grad", "L['self'].param_groups[0]['params'][194].grad", "L['self'].param_groups[0]['params'][195].grad", "L['self'].param_groups[0]['params'][196].grad", "L['self'].param_groups[0]['params'][197].grad", "L['self'].param_groups[0]['params'][198].grad", "L['self'].param_groups[0]['params'][199].grad", "L['self'].param_groups[0]['params'][200].grad", "L['self'].param_groups[0]['params'][201].grad", "L['self'].param_groups[0]['params'][202].grad", "L['self'].param_groups[0]['params'][203].grad", "L['self'].param_groups[0]['params'][204].grad", "L['self'].param_groups[0]['params'][205].grad", "L['self'].param_groups[0]['params'][206].grad", "L['self'].param_groups[0]['params'][207].grad", "L['self'].param_groups[0]['params'][208].grad", "L['self'].param_groups[0]['params'][209].grad", "L['self'].param_groups[0]['params'][210].grad", "L['self'].param_groups[0]['params'][211].grad", "L['self'].param_groups[0]['params'][212].grad", "L['self'].param_groups[0]['params'][213].grad", "L['self'].param_groups[0]['params'][214].grad", "L['self'].param_groups[0]['params'][215].grad", "L['self'].param_groups[0]['params'][216].grad", "L['self'].param_groups[0]['params'][217].grad", "L['self'].param_groups[0]['params'][218].grad", "L['self'].param_groups[0]['params'][219].grad", "L['self'].param_groups[0]['params'][220].grad", "L['self'].param_groups[0]['params'][221].grad", "L['self'].param_groups[0]['params'][222].grad", "L['self'].param_groups[0]['params'][223].grad", "L['self'].param_groups[0]['params'][224].grad", "L['self'].param_groups[0]['params'][225].grad", "L['self'].param_groups[0]['params'][226].grad", "L['self'].param_groups[0]['params'][227].grad", "L['self'].param_groups[0]['params'][228].grad", "L['self'].param_groups[0]['params'][229].grad", "L['self'].param_groups[0]['params'][230].grad", "L['self'].param_groups[0]['params'][231].grad", "L['self'].param_groups[0]['params'][232].grad", "L['self'].param_groups[0]['params'][233].grad", "L['self'].param_groups[0]['params'][234].grad", "L['self'].param_groups[0]['params'][235].grad", "L['self'].param_groups[0]['params'][236].grad", "L['self'].param_groups[0]['params'][237].grad", "L['self'].param_groups[0]['params'][238].grad", "L['self'].param_groups[0]['params'][239].grad", "L['self'].param_groups[0]['params'][240].grad", "L['self'].param_groups[0]['params'][241].grad", "L['self'].param_groups[0]['params'][242].grad", "L['self'].param_groups[0]['params'][243].grad", "L['self'].param_groups[0]['params'][244].grad", "L['self'].param_groups[0]['params'][245].grad", "L['self'].param_groups[0]['params'][246].grad", "L['self'].param_groups[0]['params'][247].grad", "L['self'].param_groups[0]['params'][248].grad", "L['self'].param_groups[0]['params'][249].grad", "L['self'].param_groups[0]['params'][250].grad", "L['self'].param_groups[0]['params'][251].grad", "L['self'].param_groups[0]['params'][252].grad", "L['self'].param_groups[0]['params'][253].grad", "L['self'].param_groups[0]['params'][254].grad", "L['self'].param_groups[0]['params'][255].grad", "L['self'].param_groups[0]['params'][256].grad", "L['self'].param_groups[0]['params'][257].grad", "L['self'].param_groups[0]['params'][258].grad", "L['self'].param_groups[0]['params'][259].grad", "L['self'].param_groups[0]['params'][260].grad", "L['self'].param_groups[0]['params'][261].grad", "L['self'].param_groups[0]['params'][262].grad", "L['self'].param_groups[0]['params'][263].grad", "L['self'].param_groups[0]['params'][264].grad", "L['self'].param_groups[0]['params'][265].grad", "L['self'].param_groups[0]['params'][266].grad", "L['self'].param_groups[0]['params'][267].grad", "L['self'].param_groups[0]['params'][268].grad", "L['self'].param_groups[0]['params'][269].grad", "L['self'].param_groups[0]['params'][270].grad", "L['self'].param_groups[0]['params'][271].grad", "L['self'].param_groups[0]['params'][272].grad", "L['self'].param_groups[0]['params'][273].grad", "L['self'].param_groups[0]['params'][274].grad", "L['self'].param_groups[0]['params'][275].grad", "L['self'].param_groups[0]['params'][276].grad", "L['self'].param_groups[0]['params'][277].grad", "L['self'].param_groups[0]['params'][278].grad", "L['self'].param_groups[0]['params'][279].grad", "L['self'].param_groups[0]['params'][280].grad", "L['self'].param_groups[0]['params'][281].grad", "L['self'].param_groups[0]['params'][282].grad", "L['self'].param_groups[0]['params'][283].grad", "L['self'].param_groups[0]['params'][284].grad", "L['self'].param_groups[0]['params'][285].grad", "L['self'].param_groups[0]['params'][286].grad", "L['self'].param_groups[0]['params'][287].grad", "L['self'].param_groups[0]['params'][288].grad", "L['self'].param_groups[0]['params'][289].grad", "L['self'].param_groups[0]['params'][290].grad", "L['self'].param_groups[0]['params'][291].grad", "L['self'].param_groups[0]['params'][292].grad", "L['self'].param_groups[0]['params'][293].grad", "L['self'].param_groups[0]['params'][294].grad", "L['self'].param_groups[0]['params'][295].grad", "L['self'].param_groups[0]['params'][296].grad", "L['self'].param_groups[0]['params'][297].grad", "L['self'].param_groups[0]['params'][298].grad", "L['self'].param_groups[0]['params'][299].grad", "L['self'].param_groups[0]['params'][300].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T02:27:00.5087984Z pass
2024-12-18T02:27:00.5944121Z TIMING: entire_frame_compile:89.82946 _recursive_pre_grad_passes:0.09061 _recursive_joint_graph_passes:2.87061 _recursive_post_grad_passes:1.77788 async_compile.wait:6.19191 code_gen:35.18054 inductor_compile:55.63896 backend_compile:69.56529 entire_backward_compile:14.51467 total_wall_time:104.34413
2024-12-18T02:27:00.5945834Z STATS: call_* op count: 1922 | FakeTensorMode.__torch_dispatch__:93195 | FakeTensor.__torch_dispatch__:22768 | ProxyTorchDispatchMode.__torch_dispatch__:42456
2024-12-18T02:27:00.5946694Z Dynamo produced 3 graphs covering 1922 ops with 7 graph breaks (5 unique)
2024-12-18T02:27:09.4724839Z 
2024-12-18T02:27:10.3499781Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:27:10.3500221Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:27:10.3500626Z cuda train tts_angular                        
2024-12-18T02:27:13.8157167Z W1218 02:27:13.814000 65977 site-packages/torch/_logging/_internal.py:1089] [10/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:27:14.0167365Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T02:27:19.1385244Z pass
2024-12-18T02:27:19.1453076Z TIMING: entire_frame_compile:6.16319 _recursive_pre_grad_passes:0.01016 _recursive_joint_graph_passes:0.14545 inductor_compile:4.10208 backend_compile:5.28455 _recursive_post_grad_passes:0.18666 async_compile.precompile:0.20978 async_compile.wait:2.30522 code_gen:3.27329 entire_backward_compile:0.39892 total_wall_time:6.56211
2024-12-18T02:27:19.1456224Z STATS: call_* op count: 102 | FakeTensorMode.__torch_dispatch__:2467 | ProxyTorchDispatchMode.__torch_dispatch__:990 | FakeTensor.__torch_dispatch__:817
2024-12-18T02:27:19.1457471Z Dynamo produced 5 graphs covering 102 ops with 9 graph breaks (6 unique)
2024-12-18T02:27:23.7137072Z 
2024-12-18T02:27:26.6312595Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:27:26.6312960Z loading model: 0it [00:02, ?it/s]
2024-12-18T02:27:26.6313270Z cuda train vgg16                              
2024-12-18T02:28:01.4869171Z pass
2024-12-18T02:28:01.5116072Z TIMING: entire_frame_compile:6.53981 _recursive_pre_grad_passes:0.00517 _recursive_joint_graph_passes:2.71698 _recursive_post_grad_passes:0.08327 async_compile.wait:1.00834 code_gen:5.95745 inductor_compile:7.17456 backend_compile:5.57682 entire_backward_compile:5.27216 total_wall_time:11.81197
2024-12-18T02:28:01.5117685Z STATS: call_* op count: 44 | FakeTensorMode.__torch_dispatch__:4382 | ProxyTorchDispatchMode.__torch_dispatch__:1686 | FakeTensor.__torch_dispatch__:590
2024-12-18T02:28:01.5118535Z Dynamo produced 2 graphs covering 44 ops with 6 graph breaks (5 unique)
2024-12-18T02:28:06.2342857Z 
2024-12-18T02:28:09.7800144Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:28:09.7800557Z loading model: 0it [00:03, ?it/s]
2024-12-18T02:28:09.7800902Z cuda train vision_maskrcnn                    
2024-12-18T02:28:43.7321680Z W1218 02:28:43.731000 66555 site-packages/torch/_inductor/utils.py:1543] [5/0_1] DeviceCopy in input program
2024-12-18T02:28:43.7331419Z W1218 02:28:43.732000 66555 site-packages/torch/_inductor/utils.py:1543] [5/0_1] DeviceCopy in input program
2024-12-18T02:28:43.7341367Z W1218 02:28:43.733000 66555 site-packages/torch/_inductor/utils.py:1543] [5/0_1] DeviceCopy in input program
2024-12-18T02:28:43.7351281Z W1218 02:28:43.734000 66555 site-packages/torch/_inductor/utils.py:1543] [5/0_1] DeviceCopy in input program
2024-12-18T02:28:43.7362037Z W1218 02:28:43.735000 66555 site-packages/torch/_inductor/utils.py:1543] [5/0_1] DeviceCopy in input program
2024-12-18T02:28:45.3475554Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_12). Found from : 
2024-12-18T02:28:45.3476459Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/rpn.py", line 362, in forward
2024-12-18T02:28:45.3477169Z     anchors = self.anchor_generator(images, features)
2024-12-18T02:28:45.3477887Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 126, in forward
2024-12-18T02:28:45.3478670Z     self.set_cell_anchors(dtype, device)
2024-12-18T02:28:45.3479466Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 77, in set_cell_anchors
2024-12-18T02:28:45.3480521Z     self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors]
2024-12-18T02:28:45.3481497Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 77, in <listcomp>
2024-12-18T02:28:45.3482817Z     self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors]
2024-12-18T02:28:45.3483267Z 
2024-12-18T02:28:47.1726647Z W1218 02:28:47.171000 66555 site-packages/torch/_dynamo/variables/tensor.py:869] [18/0] Graph break from `Tensor.item()`, consider setting:
2024-12-18T02:28:47.1728679Z W1218 02:28:47.171000 66555 site-packages/torch/_dynamo/variables/tensor.py:869] [18/0]     torch._dynamo.config.capture_scalar_outputs = True
2024-12-18T02:28:47.1729795Z W1218 02:28:47.171000 66555 site-packages/torch/_dynamo/variables/tensor.py:869] [18/0] or:
2024-12-18T02:28:47.1730718Z W1218 02:28:47.171000 66555 site-packages/torch/_dynamo/variables/tensor.py:869] [18/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2024-12-18T02:28:47.1731668Z W1218 02:28:47.171000 66555 site-packages/torch/_dynamo/variables/tensor.py:869] [18/0] to include these operations in the captured graph.
2024-12-18T02:28:47.1732792Z W1218 02:28:47.171000 66555 site-packages/torch/_dynamo/variables/tensor.py:869] [18/0] 
2024-12-18T02:28:47.1733536Z W1218 02:28:47.171000 66555 site-packages/torch/_dynamo/variables/tensor.py:869] [18/0] Graph break: from user code at:
2024-12-18T02:28:47.1734904Z W1218 02:28:47.171000 66555 site-packages/torch/_dynamo/variables/tensor.py:869] [18/0]   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/generalized_rcnn.py", line 105, in torch_dynamo_resume_in_forward_at_104
2024-12-18T02:28:47.1736593Z W1218 02:28:47.171000 66555 site-packages/torch/_dynamo/variables/tensor.py:869] [18/0]     detections, detector_losses = self.roi_heads(features, proposals, images.image_sizes, targets)
2024-12-18T02:28:47.1738031Z W1218 02:28:47.171000 66555 site-packages/torch/_dynamo/variables/tensor.py:869] [18/0]   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 761, in forward
2024-12-18T02:28:47.1739395Z W1218 02:28:47.171000 66555 site-packages/torch/_dynamo/variables/tensor.py:869] [18/0]     box_features = self.box_roi_pool(features, proposals, image_shapes)
2024-12-18T02:28:47.1740678Z W1218 02:28:47.171000 66555 site-packages/torch/_dynamo/variables/tensor.py:869] [18/0]   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 310, in forward
2024-12-18T02:28:47.1741984Z W1218 02:28:47.171000 66555 site-packages/torch/_dynamo/variables/tensor.py:869] [18/0]     self.scales, self.map_levels = _setup_scales(
2024-12-18T02:28:47.1743201Z W1218 02:28:47.171000 66555 site-packages/torch/_dynamo/variables/tensor.py:869] [18/0]   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 122, in _setup_scales
2024-12-18T02:28:47.1744484Z W1218 02:28:47.171000 66555 site-packages/torch/_dynamo/variables/tensor.py:869] [18/0]     scales = [_infer_scale(feat, original_input_shape) for feat in features]
2024-12-18T02:28:47.1745768Z W1218 02:28:47.171000 66555 site-packages/torch/_dynamo/variables/tensor.py:869] [18/0]   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 122, in <listcomp>
2024-12-18T02:28:47.1747043Z W1218 02:28:47.171000 66555 site-packages/torch/_dynamo/variables/tensor.py:869] [18/0]     scales = [_infer_scale(feat, original_input_shape) for feat in features]
2024-12-18T02:28:47.1748323Z W1218 02:28:47.171000 66555 site-packages/torch/_dynamo/variables/tensor.py:869] [18/0]   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 104, in _infer_scale
2024-12-18T02:28:47.1749614Z W1218 02:28:47.171000 66555 site-packages/torch/_dynamo/variables/tensor.py:869] [18/0]     scale = 2 ** float(torch.tensor(approx_scale).log2().round())
2024-12-18T02:28:47.1750425Z W1218 02:28:47.171000 66555 site-packages/torch/_dynamo/variables/tensor.py:869] [18/0] 
2024-12-18T02:28:47.1751028Z W1218 02:28:47.171000 66555 site-packages/torch/_dynamo/variables/tensor.py:869] [18/0] 
2024-12-18T02:29:04.2546101Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:29:04.2547158Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-12-18T02:29:04.2547999Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-12-18T02:29:04.2548439Z 
2024-12-18T02:29:10.6547496Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:29:10.6548601Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-12-18T02:29:10.6549835Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-12-18T02:29:10.6550458Z 
2024-12-18T02:29:17.0004269Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:29:17.0005109Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-12-18T02:29:17.0006415Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-12-18T02:29:17.0006952Z 
2024-12-18T02:29:54.1633233Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:29:54.1634332Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-12-18T02:29:54.1635546Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-12-18T02:29:54.1636195Z 
2024-12-18T02:30:03.0845649Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:30:03.0846416Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-12-18T02:30:03.0847248Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-12-18T02:30:03.0847725Z 
2024-12-18T02:30:16.5343023Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:30:16.5343824Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-12-18T02:30:16.5344675Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-12-18T02:30:16.5345112Z 
2024-12-18T02:30:22.1701368Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:30:22.1702233Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-12-18T02:30:22.1703056Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-12-18T02:30:22.1703474Z 
2024-12-18T02:30:28.7000527Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:30:28.7001500Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 415, in torch_dynamo_resume_in_paste_mask_in_image_at_407
2024-12-18T02:30:28.7002481Z     mask = F.interpolate(mask, size=(h, w), mode="bilinear", align_corners=False)
2024-12-18T02:30:28.7002821Z 
2024-12-18T02:30:29.7003104Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:30:29.7004542Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 415, in torch_dynamo_resume_in_paste_mask_in_image_at_407
2024-12-18T02:30:29.7005561Z     mask = F.interpolate(mask, size=(h, w), mode="bilinear", align_corners=False)
2024-12-18T02:30:29.7005977Z 
2024-12-18T02:30:29.8635355Z W1218 02:30:29.862000 66555 site-packages/torch/_functorch/_aot_autograd/autograd_cache.py:856] [40/0_1] Bypassing autograd cache due to: Cannot cache a graph with functional tensor
2024-12-18T02:30:30.8070716Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:30:30.8071670Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 821, in torch_dynamo_resume_in_forward_at_804
2024-12-18T02:30:30.8073091Z     masks_probs = maskrcnn_inference(mask_logits, labels)
2024-12-18T02:30:30.8073872Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 76, in maskrcnn_inference
2024-12-18T02:30:30.8074621Z     mask_prob = mask_prob[index, labels][:, None]
2024-12-18T02:30:30.8074868Z 
2024-12-18T02:30:32.6456839Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:30:32.6457765Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-12-18T02:30:32.6458770Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-12-18T02:30:32.6459214Z 
2024-12-18T02:30:36.1044044Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:30:36.1044907Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/generalized_rcnn.py", line 101, in forward
2024-12-18T02:30:36.1046931Z     features = self.backbone(images.tensors)
2024-12-18T02:30:36.1047644Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/backbone_utils.py", line 58, in forward
2024-12-18T02:30:36.1048321Z     x = self.fpn(x)
2024-12-18T02:30:36.1048912Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/feature_pyramid_network.py", line 194, in forward
2024-12-18T02:30:36.1049687Z     inner_top_down = F.interpolate(last_inner, size=feat_shape, mode="nearest")
2024-12-18T02:30:36.1050035Z 
2024-12-18T02:30:41.4462024Z W1218 02:30:41.445000 66555 site-packages/torch/_dynamo/convert_frame.py:906] [30/8] torch._dynamo hit config.cache_size_limit (8)
2024-12-18T02:30:41.4463195Z W1218 02:30:41.445000 66555 site-packages/torch/_dynamo/convert_frame.py:906] [30/8]    function: 'roi_align' (/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py:186)
2024-12-18T02:30:41.4464521Z W1218 02:30:41.445000 66555 site-packages/torch/_dynamo/convert_frame.py:906] [30/8]    last reason: 30/0: tensor 'L['boxes']' size mismatch at index 0. expected 648, actual 623
2024-12-18T02:30:41.4465608Z W1218 02:30:41.445000 66555 site-packages/torch/_dynamo/convert_frame.py:906] [30/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
2024-12-18T02:30:41.4466983Z W1218 02:30:41.445000 66555 site-packages/torch/_dynamo/convert_frame.py:906] [30/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html.
2024-12-18T02:30:42.2730046Z pass
2024-12-18T02:30:42.3332014Z TIMING: entire_frame_compile:134.75607 _recursive_pre_grad_passes:0.05643 _recursive_joint_graph_passes:2.97527 inductor_compile:63.31916 backend_compile:95.58687 _recursive_post_grad_passes:1.3059 async_compile.wait:15.95124 code_gen:41.2319 entire_backward_compile:6.78534 total_wall_time:141.54141
2024-12-18T02:30:42.3335221Z STATS: call_* op count: 1461 | FakeTensorMode.__torch_dispatch__:94895 | FakeTensor.__torch_dispatch__:10132 | ProxyTorchDispatchMode.__torch_dispatch__:31462 | attempt fast:1501 | slow no contiguity match:174 | fast is_contiguous:1050 | slow both tensors nontrivially broadcast:277
2024-12-18T02:30:42.3337193Z Dynamo produced 42 graphs covering 1461 ops with 35 graph breaks (8 unique)
2024-12-18T02:30:51.0215280Z 
2024-12-18T02:30:54.1038318Z loading model: 0it [00:00, ?it/s]
2024-12-18T02:30:54.1038801Z loading model: 0it [00:03, ?it/s]
2024-12-18T02:30:54.1039221Z cuda train yolov3                             
2024-12-18T02:31:37.3163058Z W1218 02:31:37.315000 68258 site-packages/torch/_inductor/utils.py:1543] [3/0_1] DeviceCopy in input program
2024-12-18T02:31:37.3166840Z W1218 02:31:37.316000 68258 site-packages/torch/_inductor/utils.py:1543] [3/0_1] DeviceCopy in input program
2024-12-18T02:31:37.5305318Z W1218 02:31:37.529000 68258 site-packages/torch/_inductor/utils.py:1543] [3/0_1] DeviceCopy in input program
2024-12-18T02:31:37.5309288Z W1218 02:31:37.530000 68258 site-packages/torch/_inductor/utils.py:1543] [3/0_1] DeviceCopy in input program
2024-12-18T02:31:37.7628388Z W1218 02:31:37.761000 68258 site-packages/torch/_inductor/utils.py:1543] [3/0_1] DeviceCopy in input program
2024-12-18T02:31:37.7632336Z W1218 02:31:37.762000 68258 site-packages/torch/_inductor/utils.py:1543] [3/0_1] DeviceCopy in input program
2024-12-18T02:31:52.9525673Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_358). Found from : 
2024-12-18T02:31:52.9526631Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 453, in torch_dynamo_resume_in_forward_and_backward_pass_at_448
2024-12-18T02:31:52.9527353Z     pred = mod(*cloned_inputs)
2024-12-18T02:31:52.9527959Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 238, in forward
2024-12-18T02:31:52.9528623Z     return self.forward_once(x)
2024-12-18T02:31:52.9529266Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 290, in forward_once
2024-12-18T02:31:52.9530375Z     yolo_out.append(module(x, out))
2024-12-18T02:31:52.9530982Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 188, in forward
2024-12-18T02:31:52.9531628Z     self.create_grids((nx, ny), p.device)
2024-12-18T02:31:52.9532281Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 159, in create_grids
2024-12-18T02:31:52.9532969Z     self.anchor_vec = self.anchor_vec.to(device)
2024-12-18T02:31:52.9533204Z 
2024-12-18T02:32:23.8291269Z W1218 02:32:23.828000 68258 site-packages/torch/_logging/_internal.py:1089] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-12-18T02:32:26.3289800Z ('Grad tensors ["L['self'].param_groups[0]['params'][0].grad", "L['self'].param_groups[0]['params'][1].grad", "L['self'].param_groups[0]['params'][2].grad", "L['self'].param_groups[0]['params'][3].grad", "L['self'].param_groups[0]['params'][4].grad", "L['self'].param_groups[0]['params'][5].grad", "L['self'].param_groups[0]['params'][6].grad", "L['self'].param_groups[0]['params'][7].grad", "L['self'].param_groups[0]['params'][8].grad", "L['self'].param_groups[0]['params'][9].grad", "L['self'].param_groups[0]['params'][10].grad", "L['self'].param_groups[0]['params'][11].grad", "L['self'].param_groups[0]['params'][12].grad", "L['self'].param_groups[0]['params'][13].grad", "L['self'].param_groups[0]['params'][14].grad", "L['self'].param_groups[0]['params'][15].grad", "L['self'].param_groups[0]['params'][16].grad", "L['self'].param_groups[0]['params'][17].grad", "L['self'].param_groups[0]['params'][18].grad", "L['self'].param_groups[0]['params'][19].grad", "L['self'].param_groups[0]['params'][20].grad", "L['self'].param_groups[0]['params'][21].grad", "L['self'].param_groups[0]['params'][22].grad", "L['self'].param_groups[0]['params'][23].grad", "L['self'].param_groups[0]['params'][24].grad", "L['self'].param_groups[0]['params'][25].grad", "L['self'].param_groups[0]['params'][26].grad", "L['self'].param_groups[0]['params'][27].grad", "L['self'].param_groups[0]['params'][28].grad", "L['self'].param_groups[0]['params'][29].grad", "L['self'].param_groups[0]['params'][30].grad", "L['self'].param_groups[0]['params'][31].grad", "L['self'].param_groups[0]['params'][32].grad", "L['self'].param_groups[0]['params'][33].grad", "L['self'].param_groups[0]['params'][34].grad", "L['self'].param_groups[0]['params'][35].grad", "L['self'].param_groups[0]['params'][36].grad", "L['self'].param_groups[0]['params'][37].grad", "L['self'].param_groups[0]['params'][38].grad", "L['self'].param_groups[0]['params'][39].grad", "L['self'].param_groups[0]['params'][40].grad", "L['self'].param_groups[0]['params'][41].grad", "L['self'].param_groups[0]['params'][42].grad", "L['self'].param_groups[0]['params'][43].grad", "L['self'].param_groups[0]['params'][44].grad", "L['self'].param_groups[0]['params'][45].grad", "L['self'].param_groups[0]['params'][46].grad", "L['self'].param_groups[0]['params'][47].grad", "L['self'].param_groups[0]['params'][48].grad", "L['self'].param_groups[0]['params'][49].grad", "L['self'].param_groups[0]['params'][50].grad", "L['self'].param_groups[0]['params'][51].grad", "L['self'].param_groups[0]['params'][52].grad", "L['self'].param_groups[0]['params'][53].grad", "L['self'].param_groups[0]['params'][54].grad", "L['self'].param_groups[0]['params'][55].grad", "L['self'].param_groups[0]['params'][56].grad", "L['self'].param_groups[0]['params'][57].grad", "L['self'].param_groups[0]['params'][58].grad", "L['self'].param_groups[0]['params'][59].grad", "L['self'].param_groups[0]['params'][60].grad", "L['self'].param_groups[0]['params'][61].grad", "L['self'].param_groups[0]['params'][62].grad", "L['self'].param_groups[0]['params'][63].grad", "L['self'].param_groups[0]['params'][64].grad", "L['self'].param_groups[0]['params'][65].grad", "L['self'].param_groups[0]['params'][66].grad", "L['self'].param_groups[0]['params'][67].grad", "L['self'].param_groups[0]['params'][68].grad", "L['self'].param_groups[0]['params'][69].grad", "L['self'].param_groups[0]['params'][70].grad", "L['self'].param_groups[0]['params'][71].grad", "L['self'].param_groups[0]['params'][72].grad", "L['self'].param_groups[0]['params'][73].grad", "L['self'].param_groups[0]['params'][74].grad", "L['self'].param_groups[0]['params'][75].grad", "L['self'].param_groups[0]['params'][76].grad", "L['self'].param_groups[0]['params'][77].grad", "L['self'].param_groups[0]['params'][78].grad", "L['self'].param_groups[0]['params'][79].grad", "L['self'].param_groups[0]['params'][80].grad", "L['self'].param_groups[0]['params'][81].grad", "L['self'].param_groups[0]['params'][82].grad", "L['self'].param_groups[0]['params'][83].grad", "L['self'].param_groups[0]['params'][84].grad", "L['self'].param_groups[0]['params'][85].grad", "L['self'].param_groups[0]['params'][86].grad", "L['self'].param_groups[0]['params'][87].grad", "L['self'].param_groups[0]['params'][88].grad", "L['self'].param_groups[0]['params'][89].grad", "L['self'].param_groups[0]['params'][90].grad", "L['self'].param_groups[0]['params'][91].grad", "L['self'].param_groups[0]['params'][92].grad", "L['self'].param_groups[0]['params'][93].grad", "L['self'].param_groups[0]['params'][94].grad", "L['self'].param_groups[0]['params'][95].grad", "L['self'].param_groups[0]['params'][96].grad", "L['self'].param_groups[0]['params'][97].grad", "L['self'].param_groups[0]['params'][98].grad", "L['self'].param_groups[0]['params'][99].grad", "L['self'].param_groups[0]['params'][100].grad", "L['self'].param_groups[0]['params'][101].grad", "L['self'].param_groups[0]['params'][102].grad", "L['self'].param_groups[0]['params'][103].grad", "L['self'].param_groups[0]['params'][104].grad", "L['self'].param_groups[0]['params'][105].grad", "L['self'].param_groups[0]['params'][106].grad", "L['self'].param_groups[0]['params'][107].grad", "L['self'].param_groups[0]['params'][108].grad", "L['self'].param_groups[0]['params'][109].grad", "L['self'].param_groups[0]['params'][110].grad", "L['self'].param_groups[0]['params'][111].grad", "L['self'].param_groups[0]['params'][112].grad", "L['self'].param_groups[0]['params'][113].grad", "L['self'].param_groups[0]['params'][114].grad", "L['self'].param_groups[0]['params'][115].grad", "L['self'].param_groups[0]['params'][116].grad", "L['self'].param_groups[0]['params'][117].grad", "L['self'].param_groups[0]['params'][118].grad", "L['self'].param_groups[0]['params'][119].grad", "L['self'].param_groups[0]['params'][120].grad", "L['self'].param_groups[0]['params'][121].grad", "L['self'].param_groups[0]['params'][122].grad", "L['self'].param_groups[0]['params'][123].grad", "L['self'].param_groups[0]['params'][124].grad", "L['self'].param_groups[0]['params'][125].grad", "L['self'].param_groups[0]['params'][126].grad", "L['self'].param_groups[0]['params'][127].grad", "L['self'].param_groups[0]['params'][128].grad", "L['self'].param_groups[0]['params'][129].grad", "L['self'].param_groups[0]['params'][130].grad", "L['self'].param_groups[0]['params'][131].grad", "L['self'].param_groups[0]['params'][132].grad", "L['self'].param_groups[0]['params'][133].grad", "L['self'].param_groups[0]['params'][134].grad", "L['self'].param_groups[0]['params'][135].grad", "L['self'].param_groups[0]['params'][136].grad", "L['self'].param_groups[0]['params'][137].grad", "L['self'].param_groups[0]['params'][138].grad", "L['self'].param_groups[0]['params'][139].grad", "L['self'].param_groups[0]['params'][140].grad", "L['self'].param_groups[0]['params'][141].grad", "L['self'].param_groups[0]['params'][142].grad", "L['self'].param_groups[0]['params'][143].grad", "L['self'].param_groups[0]['params'][144].grad", "L['self'].param_groups[0]['params'][145].grad", "L['self'].param_groups[0]['params'][146].grad", "L['self'].param_groups[0]['params'][147].grad", "L['self'].param_groups[0]['params'][148].grad", "L['self'].param_groups[0]['params'][149].grad", "L['self'].param_groups[0]['params'][150].grad", "L['self'].param_groups[0]['params'][151].grad", "L['self'].param_groups[0]['params'][152].grad", "L['self'].param_groups[0]['params'][153].grad", "L['self'].param_groups[0]['params'][154].grad", "L['self'].param_groups[0]['params'][155].grad", "L['self'].param_groups[0]['params'][156].grad", "L['self'].param_groups[0]['params'][157].grad", "L['self'].param_groups[0]['params'][158].grad", "L['self'].param_groups[0]['params'][159].grad", "L['self'].param_groups[0]['params'][160].grad", "L['self'].param_groups[0]['params'][161].grad", "L['self'].param_groups[0]['params'][162].grad", "L['self'].param_groups[0]['params'][163].grad", "L['self'].param_groups[0]['params'][164].grad", "L['self'].param_groups[0]['params'][165].grad", "L['self'].param_groups[0]['params'][166].grad", "L['self'].param_groups[0]['params'][167].grad", "L['self'].param_groups[0]['params'][168].grad", "L['self'].param_groups[0]['params'][169].grad", "L['self'].param_groups[0]['params'][170].grad", "L['self'].param_groups[0]['params'][171].grad", "L['self'].param_groups[0]['params'][172].grad", "L['self'].param_groups[0]['params'][173].grad", "L['self'].param_groups[0]['params'][174].grad", "L['self'].param_groups[0]['params'][175].grad", "L['self'].param_groups[0]['params'][176].grad", "L['self'].param_groups[0]['params'][177].grad", "L['self'].param_groups[0]['params'][178].grad", "L['self'].param_groups[0]['params'][179].grad", "L['self'].param_groups[0]['params'][180].grad", "L['self'].param_groups[0]['params'][181].grad", "L['self'].param_groups[0]['params'][182].grad", "L['self'].param_groups[0]['params'][183].grad", "L['self'].param_groups[0]['params'][184].grad", "L['self'].param_groups[0]['params'][185].grad", "L['self'].param_groups[0]['params'][186].grad", "L['self'].param_groups[0]['params'][187].grad", "L['self'].param_groups[0]['params'][188].grad", "L['self'].param_groups[0]['params'][189].grad", "L['self'].param_groups[0]['params'][190].grad", "L['self'].param_groups[0]['params'][191].grad", "L['self'].param_groups[0]['params'][192].grad", "L['self'].param_groups[0]['params'][193].grad", "L['self'].param_groups[0]['params'][194].grad", "L['self'].param_groups[0]['params'][195].grad", "L['self'].param_groups[0]['params'][196].grad", "L['self'].param_groups[0]['params'][197].grad", "L['self'].param_groups[0]['params'][198].grad", "L['self'].param_groups[0]['params'][199].grad", "L['self'].param_groups[0]['params'][200].grad", "L['self'].param_groups[0]['params'][201].grad", "L['self'].param_groups[0]['params'][202].grad", "L['self'].param_groups[0]['params'][203].grad", "L['self'].param_groups[0]['params'][204].grad", "L['self'].param_groups[0]['params'][205].grad", "L['self'].param_groups[0]['params'][206].grad", "L['self'].param_groups[0]['params'][207].grad", "L['self'].param_groups[0]['params'][208].grad", "L['self'].param_groups[0]['params'][209].grad", "L['self'].param_groups[0]['params'][210].grad", "L['self'].param_groups[0]['params'][211].grad", "L['self'].param_groups[0]['params'][212].grad", "L['self'].param_groups[0]['params'][213].grad", "L['self'].param_groups[0]['params'][214].grad", "L['self'].param_groups[0]['params'][215].grad", "L['self'].param_groups[0]['params'][216].grad", "L['self'].param_groups[0]['params'][217].grad", "L['self'].param_groups[0]['params'][218].grad", "L['self'].param_groups[0]['params'][219].grad", "L['self'].param_groups[0]['params'][220].grad", "L['self'].param_groups[0]['params'][221].grad", "L['self'].param_groups[0]['params'][222].grad", "L['self'].param_groups[0]['params'][223].grad", "L['self'].param_groups[0]['params'][224].grad"] will be copied during cudagraphs execution.If using cudagraphs and the grad tensor addresses will be the same across runs, use torch._dynamo.decorators.mark_static_address to elide this copy.',)
2024-12-18T02:34:12.3763562Z skipping cudagraphs due to deterministic index put. Found from : 
2024-12-18T02:34:12.3764416Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 453, in torch_dynamo_resume_in_forward_and_backward_pass_at_448
2024-12-18T02:34:12.3765581Z     pred = mod(*cloned_inputs)
2024-12-18T02:34:12.3766211Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 238, in forward
2024-12-18T02:34:12.3767228Z     return self.forward_once(x)
2024-12-18T02:34:12.3767863Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 292, in forward_once
2024-12-18T02:34:12.3768539Z     x = module(x)
2024-12-18T02:34:12.3768712Z 
2024-12-18T02:34:12.7863265Z pass
2024-12-18T02:34:12.8258911Z TIMING: entire_frame_compile:126.14498 _recursive_pre_grad_passes:0.08877 _recursive_joint_graph_passes:1.13832 _recursive_post_grad_passes:3.34874 async_compile.wait:10.89658 code_gen:78.94502 inductor_compile:111.45714 backend_compile:97.31431 entire_backward_compile:51.23055 total_wall_time:177.37553
2024-12-18T02:34:12.8260568Z STATS: call_* op count: 1829 | FakeTensorMode.__torch_dispatch__:97346 | FakeTensor.__torch_dispatch__:32902 | ProxyTorchDispatchMode.__torch_dispatch__:46238
2024-12-18T02:34:12.8261447Z Dynamo produced 4 graphs covering 1829 ops with 8 graph breaks (5 unique)
2024-12-18T02:34:21.3687033Z accuracy            pass_rate=78.95%
2024-12-18T02:34:21.3691082Z calls_captured      gmean=0.00x mean=530.079x
2024-12-18T02:34:21.3694738Z unique_graphs       gmean=0.00x mean=3.579x
2024-12-18T02:34:21.3698513Z graph_breaks        gmean=0.00x mean=6.447x
2024-12-18T02:34:21.3702478Z unique_graph_breaks gmean=0.00x mean=4.316x
2024-12-18T02:34:21.3706340Z autograd_captures   gmean=0.00x mean=0.000x
2024-12-18T02:34:21.3709943Z autograd_compiles   gmean=0.00x mean=0.000x
2024-12-18T02:34:21.3713870Z cudagraph_skips     gmean=0.00x mean=0.632x
2024-12-18T02:34:23.6833040Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_training.csv
2024-12-18T02:34:24.0267551Z llava                               XFAIL
2024-12-18T02:34:24.0273566Z maml_omniglot                       PASS
2024-12-18T02:34:24.0280105Z mnasnet1_0                          PASS
2024-12-18T02:34:24.0287284Z mobilenet_v2                        PASS
2024-12-18T02:34:24.0293988Z mobilenet_v2_quantized_qat          XFAIL
2024-12-18T02:34:24.0301028Z mobilenet_v3_large                  PASS
2024-12-18T02:34:24.0307941Z moco                                PASS
2024-12-18T02:34:24.0313459Z nanogpt                             PASS
2024-12-18T02:34:24.0319278Z nvidia_deeprecommender              PASS
2024-12-18T02:34:24.0324833Z opacus_cifar10                      XFAIL
2024-12-18T02:34:24.0329958Z phlippe_densenet                    PASS
2024-12-18T02:34:24.0335525Z phlippe_resnet                      PASS
2024-12-18T02:34:24.0341311Z pytorch_CycleGAN_and_pix2pix        PASS
2024-12-18T02:34:24.0347087Z pytorch_stargan                     PASS
2024-12-18T02:34:24.0352020Z pytorch_unet                        XFAIL
2024-12-18T02:34:24.0357911Z resnet152                           PASS
2024-12-18T02:34:24.0362063Z resnet18                            PASS
2024-12-18T02:34:24.0366439Z resnet50                            PASS
2024-12-18T02:34:24.0372694Z resnet50_quantized_qat              XFAIL
2024-12-18T02:34:24.0376655Z resnext50_32x4d                     PASS
2024-12-18T02:34:24.0381070Z sam                                 XFAIL
2024-12-18T02:34:24.0385392Z shufflenet_v2_x1_0                  PASS
2024-12-18T02:34:24.0389924Z soft_actor_critic                   PASS
2024-12-18T02:34:24.0394062Z speech_transformer                  PASS
2024-12-18T02:34:24.0398296Z squeezenet1_1                       PASS
2024-12-18T02:34:24.0402640Z stable_diffusion_text_encoder       PASS
2024-12-18T02:34:24.0406785Z stable_diffusion_unet               XFAIL
2024-12-18T02:34:24.0411223Z timm_efficientnet                   PASS
2024-12-18T02:34:24.0415421Z timm_regnet                         PASS
2024-12-18T02:34:24.0419950Z timm_resnest                        PASS
2024-12-18T02:34:24.0424175Z timm_vision_transformer             PASS
2024-12-18T02:34:24.0429176Z timm_vision_transformer_large       XFAIL
2024-12-18T02:34:24.0433746Z timm_vovnet                         PASS
2024-12-18T02:34:24.0437962Z torch_multimodal_clip               PASS
2024-12-18T02:34:24.0442682Z tts_angular                         PASS
2024-12-18T02:34:24.0447259Z vgg16                               PASS
2024-12-18T02:34:24.0451688Z vision_maskrcnn                     PASS
2024-12-18T02:34:24.0456021Z yolov3                              PASS
2024-12-18T02:34:24.0977615Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_training.csv
2024-12-18T02:34:24.4369503Z llava                               PASS
2024-12-18T02:34:24.4373468Z maml_omniglot                       PASS
2024-12-18T02:34:24.4378526Z mnasnet1_0                          PASS
2024-12-18T02:34:24.4383594Z mobilenet_v2                        PASS
2024-12-18T02:34:24.4388580Z mobilenet_v2_quantized_qat          PASS
2024-12-18T02:34:24.4393327Z mobilenet_v3_large                  PASS
2024-12-18T02:34:24.4398061Z moco                                PASS
2024-12-18T02:34:24.4403037Z nanogpt                             PASS
2024-12-18T02:34:24.4407907Z nvidia_deeprecommender              PASS
2024-12-18T02:34:24.4412936Z opacus_cifar10                      PASS
2024-12-18T02:34:24.4417897Z phlippe_densenet                    PASS
2024-12-18T02:34:24.4422591Z phlippe_resnet                      PASS
2024-12-18T02:34:24.4427732Z pytorch_CycleGAN_and_pix2pix        PASS
2024-12-18T02:34:24.4432553Z pytorch_stargan                     PASS
2024-12-18T02:34:24.4437438Z pytorch_unet                        PASS
2024-12-18T02:34:24.4442228Z resnet152                           PASS
2024-12-18T02:34:24.4447183Z resnet18                            PASS
2024-12-18T02:34:24.4451951Z resnet50                            PASS
2024-12-18T02:34:24.4456902Z resnet50_quantized_qat              PASS
2024-12-18T02:34:24.4461624Z resnext50_32x4d                     PASS
2024-12-18T02:34:24.4467075Z sam                                 PASS
2024-12-18T02:34:24.4472100Z shufflenet_v2_x1_0                  PASS
2024-12-18T02:34:24.4476972Z soft_actor_critic                   PASS
2024-12-18T02:34:24.4481800Z speech_transformer                  PASS
2024-12-18T02:34:24.4486642Z squeezenet1_1                       PASS
2024-12-18T02:34:24.4491418Z stable_diffusion_text_encoder       PASS
2024-12-18T02:34:24.4496278Z stable_diffusion_unet               PASS
2024-12-18T02:34:24.4500981Z timm_efficientnet                   PASS
2024-12-18T02:34:24.4506205Z timm_regnet                         PASS
2024-12-18T02:34:24.4510989Z timm_resnest                        PASS
2024-12-18T02:34:24.4515852Z timm_vision_transformer             PASS
2024-12-18T02:34:24.4520709Z timm_vision_transformer_large       PASS
2024-12-18T02:34:24.4525693Z timm_vovnet                         PASS
2024-12-18T02:34:24.4530408Z torch_multimodal_clip               PASS
2024-12-18T02:34:24.4535389Z tts_angular                         PASS
2024-12-18T02:34:24.4540347Z vgg16                               PASS
2024-12-18T02:34:24.4545313Z vision_maskrcnn                     PASS
2024-12-18T02:34:24.4550118Z yolov3                              PASS
2024-12-18T02:34:24.5109009Z + cleanup_workspace
2024-12-18T02:34:24.5109518Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.'
2024-12-18T02:34:24.5110290Z sudo may print the following warning message that can be ignored. The chown command will still run.
2024-12-18T02:34:24.5110902Z + echo '    sudo: setrlimit(RLIMIT_STACK): Operation not permitted'
2024-12-18T02:34:24.5111369Z     sudo: setrlimit(RLIMIT_STACK): Operation not permitted
2024-12-18T02:34:24.5111895Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42'
2024-12-18T02:34:24.5112489Z For more details refer to https://github.com/sudo-project/sudo/issues/42
2024-12-18T02:34:24.5113248Z + sudo chown -R 1000 /var/lib/jenkins/workspace
2024-12-18T02:34:25.3354765Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@release/2.6
2024-12-18T02:34:25.3355268Z with:
2024-12-18T02:34:25.3355522Z   benchmark-results-dir: test/test-reports
2024-12-18T02:34:25.3355853Z   dry-run: false
2024-12-18T02:34:25.3356088Z   schema-version: v3
2024-12-18T02:34:25.3356511Z   github-token: ***
2024-12-18T02:34:25.3356751Z env:
2024-12-18T02:34:25.3356974Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:34:25.3357311Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:34:25.3357841Z   DOCKER_CONTAINER_ID: 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T02:34:25.3358323Z ##[endgroup]
2024-12-18T02:34:25.3389501Z ##[group]Run set -eux
2024-12-18T02:34:25.3389769Z set -eux
2024-12-18T02:34:25.3390036Z python3 -mpip install boto3==1.35.33
2024-12-18T02:34:25.3404189Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:34:25.3404581Z env:
2024-12-18T02:34:25.3404838Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:34:25.3405180Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:34:25.3405715Z   DOCKER_CONTAINER_ID: 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T02:34:25.3406196Z ##[endgroup]
2024-12-18T02:34:25.3441683Z + python3 -mpip install boto3==1.35.33
2024-12-18T02:34:25.6705709Z Defaulting to user installation because normal site-packages is not writeable
2024-12-18T02:34:26.8535153Z Collecting boto3==1.35.33
2024-12-18T02:34:26.8883298Z   Downloading boto3-1.35.33-py3-none-any.whl (139 kB)
2024-12-18T02:34:26.9141251Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.0)
2024-12-18T02:34:28.1316340Z Collecting botocore<1.36.0,>=1.35.33
2024-12-18T02:34:28.1358343Z   Downloading botocore-1.35.83-py3-none-any.whl (13.3 MB)
2024-12-18T02:34:28.3894477Z Collecting s3transfer<0.11.0,>=0.10.0
2024-12-18T02:34:28.4063836Z   Downloading s3transfer-0.10.4-py3-none-any.whl (83 kB)
2024-12-18T02:34:28.4179295Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.25.10)
2024-12-18T02:34:28.4183249Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.8.1)
2024-12-18T02:34:28.5460597Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.15.0)
2024-12-18T02:34:28.6259931Z Installing collected packages: botocore, s3transfer, boto3
2024-12-18T02:34:29.2458398Z Successfully installed boto3-1.35.33 botocore-1.35.83 s3transfer-0.10.4
2024-12-18T02:34:29.3608649Z ##[group]Run set -eux
2024-12-18T02:34:29.3608933Z set -eux
2024-12-18T02:34:29.3609169Z 
2024-12-18T02:34:29.3609432Z if [[ -z "${GITHUB_TOKEN}" ]]; then
2024-12-18T02:34:29.3609799Z   echo "Missing github-token input"
2024-12-18T02:34:29.3610127Z   exit 1
2024-12-18T02:34:29.3610360Z fi
2024-12-18T02:34:29.3620574Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:34:29.3620953Z env:
2024-12-18T02:34:29.3621170Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:34:29.3621509Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:34:29.3622165Z   DOCKER_CONTAINER_ID: 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T02:34:29.3623069Z   GITHUB_TOKEN: ***
2024-12-18T02:34:29.3623319Z ##[endgroup]
2024-12-18T02:34:29.3652215Z + [[ -z *** ]]
2024-12-18T02:34:29.3704456Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main
2024-12-18T02:34:29.3704896Z with:
2024-12-18T02:34:29.3705234Z   github-token: ***
2024-12-18T02:34:29.3705480Z env:
2024-12-18T02:34:29.3705699Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:34:29.3706223Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:34:29.3706937Z   DOCKER_CONTAINER_ID: 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T02:34:29.3707425Z ##[endgroup]
2024-12-18T02:34:29.3731046Z ##[group]Run set -eux
2024-12-18T02:34:29.3731309Z set -eux
2024-12-18T02:34:29.3731541Z 
2024-12-18T02:34:29.3732007Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2024-12-18T02:34:29.3740224Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:34:29.3740611Z env:
2024-12-18T02:34:29.3740835Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:34:29.3741184Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:34:29.3741721Z   DOCKER_CONTAINER_ID: 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T02:34:29.3742587Z   GITHUB_TOKEN: ***
2024-12-18T02:34:29.3742829Z ##[endgroup]
2024-12-18T02:34:29.3772065Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 12383255690 i-029fac8e3f5b7f275
2024-12-18T02:34:30.6403110Z setting job-id=34567153568
2024-12-18T02:34:30.6404211Z setting job-name=cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T02:34:30.6513480Z ##[group]Run set -eux
2024-12-18T02:34:30.6513807Z set -eux
2024-12-18T02:34:30.6514043Z 
2024-12-18T02:34:30.6514424Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \
2024-12-18T02:34:30.6514931Z   --schema-version "${SCHEMA_VERSION}" \
2024-12-18T02:34:30.6515281Z   --repo "${REPO}" \
2024-12-18T02:34:30.6515596Z   --head-branch "${HEAD_BRANCH}" \
2024-12-18T02:34:30.6515921Z   --head-sha "${HEAD_SHA}" \
2024-12-18T02:34:30.6516264Z   --workflow-id "${WORKFLOW_RUN_ID}" \
2024-12-18T02:34:30.6516643Z   --run-attempt "${RUN_ATTEMPT}" \
2024-12-18T02:34:30.6516979Z   --job-id "${JOB_ID}" \
2024-12-18T02:34:30.6517282Z   --job-name "${JOB_NAME}"
2024-12-18T02:34:30.6526143Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:34:30.6526513Z env:
2024-12-18T02:34:30.6526736Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:34:30.6527069Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:34:30.6527600Z   DOCKER_CONTAINER_ID: 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T02:34:30.6528082Z   SCHEMA_VERSION: v3
2024-12-18T02:34:30.6528329Z   REPO: pytorch/pytorch
2024-12-18T02:34:30.6528602Z   HEAD_BRANCH: refs/heads/release/2.6
2024-12-18T02:34:30.6528950Z   HEAD_SHA: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7
2024-12-18T02:34:30.6529299Z   WORKFLOW_RUN_ID: 12383255690
2024-12-18T02:34:30.6529576Z   RUN_ATTEMPT: 1
2024-12-18T02:34:30.6529801Z   JOB_ID: 34567153568
2024-12-18T02:34:30.6530275Z   JOB_NAME: cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2024-12-18T02:34:30.6530807Z ##[endgroup]
2024-12-18T02:34:30.6561199Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/release/2.6/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/heads/release/2.6 --head-sha 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 --workflow-id 12383255690 --run-attempt 1 --job-id 34567153568 --job-name 'cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)'
2024-12-18T02:34:30.6906622Z ##[group]Run set -eux
2024-12-18T02:34:30.6906888Z set -eux
2024-12-18T02:34:30.6907117Z 
2024-12-18T02:34:30.6907373Z # TODO (huydhn): Implement this part
2024-12-18T02:34:30.6907737Z echo "runners=[]" >> "${GITHUB_OUTPUT}"
2024-12-18T02:34:30.6915765Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:34:30.6916343Z env:
2024-12-18T02:34:30.6916562Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:34:30.6917060Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:34:30.6917607Z   DOCKER_CONTAINER_ID: 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T02:34:30.6918096Z ##[endgroup]
2024-12-18T02:34:30.6946792Z + echo 'runners=[]'
2024-12-18T02:34:30.6980369Z ##[group]Run set -eux
2024-12-18T02:34:30.6980642Z set -eux
2024-12-18T02:34:30.6980867Z 
2024-12-18T02:34:30.6981121Z # TODO (huydhn): Implement this part
2024-12-18T02:34:30.6981513Z echo "dependencies={}" >> "${GITHUB_OUTPUT}"
2024-12-18T02:34:30.6990389Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:34:30.6990769Z env:
2024-12-18T02:34:30.6990990Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:34:30.6991329Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:34:30.6991868Z   DOCKER_CONTAINER_ID: 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T02:34:30.6992368Z ##[endgroup]
2024-12-18T02:34:30.7022289Z + echo 'dependencies={}'
2024-12-18T02:34:30.7054954Z ##[group]Run set -eux
2024-12-18T02:34:30.7055238Z set -eux
2024-12-18T02:34:30.7055478Z 
2024-12-18T02:34:30.7055752Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then
2024-12-18T02:34:30.7056185Z   echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping"
2024-12-18T02:34:30.7056654Z   # We don't want the job to fail if the directory doesn't exist
2024-12-18T02:34:30.7057046Z   exit 0
2024-12-18T02:34:30.7057274Z fi
2024-12-18T02:34:30.7057491Z 
2024-12-18T02:34:30.7057728Z if [[ "${DRY_RUN}" == "true" ]]; then
2024-12-18T02:34:30.7058195Z   python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \
2024-12-18T02:34:30.7058743Z     --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \
2024-12-18T02:34:30.7059176Z     --metadata "${BENCHMARK_METADATA}" \
2024-12-18T02:34:30.7059555Z     --runners "${RUNNER_INFO}" \
2024-12-18T02:34:30.7059909Z     --dependencies "${DEPENDENCIES}" \
2024-12-18T02:34:30.7060247Z     --dry-run
2024-12-18T02:34:30.7060500Z else
2024-12-18T02:34:30.7060872Z   python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \
2024-12-18T02:34:30.7061406Z     --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \
2024-12-18T02:34:30.7061914Z     --metadata "${BENCHMARK_METADATA}" \
2024-12-18T02:34:30.7062263Z     --runners "${RUNNER_INFO}" \
2024-12-18T02:34:30.7062605Z     --dependencies "${DEPENDENCIES}"
2024-12-18T02:34:30.7062918Z fi
2024-12-18T02:34:30.7071393Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:34:30.7071777Z env:
2024-12-18T02:34:30.7072005Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:34:30.7072355Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:34:30.7072909Z   DOCKER_CONTAINER_ID: 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T02:34:30.7073433Z   BENCHMARK_RESULTS_DIR: test/test-reports
2024-12-18T02:34:30.7073762Z   DRY_RUN: false
2024-12-18T02:34:30.7075094Z   BENCHMARK_METADATA: {"timestamp": 1734489270, "schema_version": "v3", "name": "cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/release/2.6", "head_sha": "0cdf8b1d09254cfda66191d1bd01e3041c3c76f7", "workflow_id": 12383255690, "run_attempt": 1, "job_id": 34567153568}
2024-12-18T02:34:30.7076477Z   RUNNER_INFO: []
2024-12-18T02:34:30.7076730Z   DEPENDENCIES: {}
2024-12-18T02:34:30.7076985Z ##[endgroup]
2024-12-18T02:34:30.7109329Z + [[ ! -d test/test-reports ]]
2024-12-18T02:34:30.7109638Z + [[ false == \t\r\u\e ]]
2024-12-18T02:34:30.7112122Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/release/2.6/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1734489270, "schema_version": "v3", "name": "cuda12.4-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/release/2.6", "head_sha": "0cdf8b1d09254cfda66191d1bd01e3041c3c76f7", "workflow_id": 12383255690, "run_attempt": 1, "job_id": 34567153568}' --runners '[]' --dependencies '{}'
2024-12-18T02:34:30.8909365Z INFO:root:Upload test/test-reports/inference_torchbench.json to s3://ossci-benchmarks/v3/pytorch/pytorch/12383255690/34567153568/inference_torchbench.json
2024-12-18T02:34:30.9328205Z INFO:botocore.credentials:Found credentials from IAM Role: gh-ci-github-action-runners-runner-role
2024-12-18T02:34:31.1452908Z INFO:root:Upload test/test-reports/inference_torchbench_graph_breaks.json to s3://ossci-benchmarks/v3/pytorch/pytorch/12383255690/34567153568/inference_torchbench_graph_breaks.json
2024-12-18T02:34:31.2528626Z INFO:root:Upload test/test-reports/training_torchbench.json to s3://ossci-benchmarks/v3/pytorch/pytorch/12383255690/34567153568/training_torchbench.json
2024-12-18T02:34:31.4094555Z INFO:root:Upload test/test-reports/training_torchbench_graph_breaks.json to s3://ossci-benchmarks/v3/pytorch/pytorch/12383255690/34567153568/training_torchbench_graph_breaks.json
2024-12-18T02:34:31.5972829Z ##[group]Run cat test/**/*_toprint.log || true
2024-12-18T02:34:31.5973211Z cat test/**/*_toprint.log || true
2024-12-18T02:34:31.5982125Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:34:31.5982505Z env:
2024-12-18T02:34:31.5982728Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:34:31.5983062Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:34:31.5983660Z   DOCKER_CONTAINER_ID: 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T02:34:31.5984139Z ##[endgroup]
2024-12-18T02:34:31.6068722Z cat: 'test/**/*_toprint.log': No such file or directory
2024-12-18T02:34:31.6106164Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2024-12-18T02:34:31.6106510Z kill "$MONITOR_SCRIPT_PID"
2024-12-18T02:34:31.6114385Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:34:31.6114761Z env:
2024-12-18T02:34:31.6114981Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:34:31.6115317Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:34:31.6115853Z   DOCKER_CONTAINER_ID: 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T02:34:31.6116338Z   MONITOR_SCRIPT_PID: 84077
2024-12-18T02:34:31.6116616Z ##[endgroup]
2024-12-18T02:34:31.6286048Z Prepare all required actions
2024-12-18T02:34:31.6286439Z Getting action download info
2024-12-18T02:34:31.7543332Z Download action repository 'actions/upload-artifact@v4' (SHA:6f51ac03b9356f520e9adb1b1b7802705f340c2b)
2024-12-18T02:34:32.1501174Z ##[group]Run ./.github/actions/upload-test-artifacts
2024-12-18T02:34:32.1501529Z with:
2024-12-18T02:34:32.1502046Z   file-suffix: test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_34567153568
2024-12-18T02:34:32.1502547Z   s3-bucket: gha-artifacts
2024-12-18T02:34:32.1502814Z env:
2024-12-18T02:34:32.1503030Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:34:32.1503385Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:34:32.1503947Z   DOCKER_CONTAINER_ID: 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T02:34:32.1504420Z ##[endgroup]
2024-12-18T02:34:32.1549151Z ##[group]Run # Remove any previous test jsons if they exist
2024-12-18T02:34:32.1549622Z # Remove any previous test jsons if they exist
2024-12-18T02:34:32.1549989Z rm -f test-jsons-*.zip
2024-12-18T02:34:32.1550474Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json'
2024-12-18T02:34:32.1559253Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:34:32.1559633Z env:
2024-12-18T02:34:32.1559858Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:34:32.1560379Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:34:32.1560915Z   DOCKER_CONTAINER_ID: 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T02:34:32.1561579Z   FILE_SUFFIX: test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_34567153568
2024-12-18T02:34:32.1562062Z ##[endgroup]
2024-12-18T02:34:32.1749589Z   adding: test/test-reports/inference_torchbench.json (deflated 99%)
2024-12-18T02:34:32.1772317Z   adding: test/test-reports/inference_torchbench_graph_breaks.json (deflated 98%)
2024-12-18T02:34:32.1817312Z   adding: test/test-reports/training_torchbench.json (deflated 99%)
2024-12-18T02:34:32.1953179Z   adding: test/test-reports/training_torchbench_graph_breaks.json (deflated 99%)
2024-12-18T02:34:32.2002709Z ##[group]Run # Remove any previous test reports if they exist
2024-12-18T02:34:32.2003295Z # Remove any previous test reports if they exist
2024-12-18T02:34:32.2003818Z rm -f test-reports-*.zip
2024-12-18T02:34:32.2004434Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv'
2024-12-18T02:34:32.2012945Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:34:32.2013412Z env:
2024-12-18T02:34:32.2013726Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:34:32.2014211Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:34:32.2014864Z   DOCKER_CONTAINER_ID: 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T02:34:32.2015607Z   FILE_SUFFIX: test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_34567153568
2024-12-18T02:34:32.2016211Z ##[endgroup]
2024-12-18T02:34:32.2068920Z   adding: test/test-reports/inference_torchbench.csv (deflated 69%)
2024-12-18T02:34:32.2071743Z   adding: test/test-reports/inference_torchbench_graph_breaks.csv (deflated 92%)
2024-12-18T02:34:32.2072755Z   adding: test/test-reports/inference_torchbench_graph_break_deduped.csv (deflated 76%)
2024-12-18T02:34:32.2073502Z   adding: test/test-reports/training_torchbench.csv (deflated 66%)
2024-12-18T02:34:32.2081588Z   adding: test/test-reports/training_torchbench_graph_breaks.csv (deflated 96%)
2024-12-18T02:34:32.2083000Z   adding: test/test-reports/training_torchbench_graph_break_deduped.csv (deflated 75%)
2024-12-18T02:34:32.2129262Z ##[group]Run # Remove any previous usage logs if they exist
2024-12-18T02:34:32.2129822Z # Remove any previous usage logs if they exist
2024-12-18T02:34:32.2130316Z rm -f logs-*.zip
2024-12-18T02:34:32.2130842Z # this workflow is also run in bazel build test, but we dont generate usage reports for it
2024-12-18T02:34:32.2131675Z # so check to see if the file exists first
2024-12-18T02:34:32.2132164Z if [ -f 'usage_log.txt' ]; then
2024-12-18T02:34:32.2132659Z     zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt'
2024-12-18T02:34:32.2133087Z fi
2024-12-18T02:34:32.2133595Z if find "test/test-reports" -name "*.log" 2>/dev/null | grep -q .; then
2024-12-18T02:34:32.2134271Z     zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log'
2024-12-18T02:34:32.2134749Z fi
2024-12-18T02:34:32.2143192Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:34:32.2143640Z env:
2024-12-18T02:34:32.2144059Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:34:32.2144482Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:34:32.2145061Z   DOCKER_CONTAINER_ID: 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T02:34:32.2145905Z   FILE_SUFFIX: test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_34567153568
2024-12-18T02:34:32.2146477Z ##[endgroup]
2024-12-18T02:34:32.2446840Z   adding: usage_log.txt (deflated 98%)
2024-12-18T02:34:32.2522853Z ##[group]Run # Remove any previous debugging artifacts if they exist
2024-12-18T02:34:32.2523358Z # Remove any previous debugging artifacts if they exist
2024-12-18T02:34:32.2523758Z rm -f debug-*.zip
2024-12-18T02:34:32.2524228Z if [ -d 'test/debug' ]; then
2024-12-18T02:34:32.2524583Z   zip -r "debug-${FILE_SUFFIX}.zip" test/debug
2024-12-18T02:34:32.2524919Z fi
2024-12-18T02:34:32.2533126Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:34:32.2533498Z env:
2024-12-18T02:34:32.2533731Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:34:32.2534100Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:34:32.2534630Z   DOCKER_CONTAINER_ID: 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T02:34:32.2535294Z   FILE_SUFFIX: test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_34567153568
2024-12-18T02:34:32.2535765Z ##[endgroup]
2024-12-18T02:34:32.2633660Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-12-18T02:34:32.2634002Z with:
2024-12-18T02:34:32.2634236Z   s3-bucket: gha-artifacts
2024-12-18T02:34:32.2634557Z   s3-prefix: pytorch/pytorch/12383255690/1/artifact

2024-12-18T02:34:32.2634912Z   retention-days: 14
2024-12-18T02:34:32.2635191Z   if-no-files-found: warn
2024-12-18T02:34:32.2635475Z   path: test-jsons-*.zip
2024-12-18T02:34:32.2635742Z   name: artifact
2024-12-18T02:34:32.2635979Z   region: us-east-1
2024-12-18T02:34:32.2636220Z env:
2024-12-18T02:34:32.2636448Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:34:32.2636788Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:34:32.2637330Z   DOCKER_CONTAINER_ID: 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T02:34:32.2637808Z ##[endgroup]
2024-12-18T02:34:32.6878034Z NOTE: s3-prefix specified, ignoring name parameter
2024-12-18T02:34:32.6878516Z With the provided path, there will be 1 file uploaded
2024-12-18T02:34:32.6878952Z Uploading to s3 prefix: pytorch/pytorch/12383255690/1/artifact
2024-12-18T02:34:32.6937783Z Starting upload of test-jsons-test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_34567153568.zip
2024-12-18T02:34:32.8549620Z Finished upload of test-jsons-test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_34567153568.zip
2024-12-18T02:34:32.8847427Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-12-18T02:34:32.8847767Z with:
2024-12-18T02:34:32.8847987Z   s3-bucket: gha-artifacts
2024-12-18T02:34:32.8848311Z   s3-prefix: pytorch/pytorch/12383255690/1/artifact

2024-12-18T02:34:32.8848666Z   retention-days: 14
2024-12-18T02:34:32.8848930Z   if-no-files-found: error
2024-12-18T02:34:32.8849220Z   path: test-reports-*.zip
2024-12-18T02:34:32.8849481Z   name: artifact
2024-12-18T02:34:32.8849718Z   region: us-east-1
2024-12-18T02:34:32.8849957Z env:
2024-12-18T02:34:32.8850178Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:34:32.8850728Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:34:32.8851268Z   DOCKER_CONTAINER_ID: 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T02:34:32.8851746Z ##[endgroup]
2024-12-18T02:34:33.2154333Z NOTE: s3-prefix specified, ignoring name parameter
2024-12-18T02:34:33.2154788Z With the provided path, there will be 1 file uploaded
2024-12-18T02:34:33.2155258Z Uploading to s3 prefix: pytorch/pytorch/12383255690/1/artifact
2024-12-18T02:34:33.2214212Z Starting upload of test-reports-test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_34567153568.zip
2024-12-18T02:34:33.3699746Z Finished upload of test-reports-test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_34567153568.zip
2024-12-18T02:34:33.3999951Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-12-18T02:34:33.4000294Z with:
2024-12-18T02:34:33.4000519Z   s3-bucket: gha-artifacts
2024-12-18T02:34:33.4000859Z   s3-prefix: pytorch/pytorch/12383255690/1/artifact

2024-12-18T02:34:33.4001230Z   retention-days: 14
2024-12-18T02:34:33.4001498Z   if-no-files-found: ignore
2024-12-18T02:34:33.4001782Z   path: logs-*.zip
2024-12-18T02:34:33.4002020Z   name: artifact
2024-12-18T02:34:33.4002265Z   region: us-east-1
2024-12-18T02:34:33.4002504Z env:
2024-12-18T02:34:33.4002724Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:34:33.4003054Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:34:33.4003781Z   DOCKER_CONTAINER_ID: 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T02:34:33.4004266Z ##[endgroup]
2024-12-18T02:34:33.7336104Z NOTE: s3-prefix specified, ignoring name parameter
2024-12-18T02:34:33.7336563Z With the provided path, there will be 1 file uploaded
2024-12-18T02:34:33.7337005Z Uploading to s3 prefix: pytorch/pytorch/12383255690/1/artifact
2024-12-18T02:34:33.7394946Z Starting upload of logs-test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_34567153568.zip
2024-12-18T02:34:33.9127350Z Finished upload of logs-test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_34567153568.zip
2024-12-18T02:34:33.9437968Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-12-18T02:34:33.9438311Z with:
2024-12-18T02:34:33.9438535Z   s3-bucket: gha-artifacts
2024-12-18T02:34:33.9438858Z   s3-prefix: pytorch/pytorch/12383255690/1/artifact

2024-12-18T02:34:33.9439208Z   retention-days: 14
2024-12-18T02:34:33.9439476Z   if-no-files-found: ignore
2024-12-18T02:34:33.9439781Z   path: debug-*.zip
2024-12-18T02:34:33.9440020Z   name: artifact
2024-12-18T02:34:33.9440262Z   region: us-east-1
2024-12-18T02:34:33.9440506Z env:
2024-12-18T02:34:33.9440729Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:34:33.9441069Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:34:33.9441599Z   DOCKER_CONTAINER_ID: 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T02:34:33.9442076Z ##[endgroup]
2024-12-18T02:34:34.2694818Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded.
2024-12-18T02:34:34.2999270Z ##[group]Run # shellcheck disable=SC2156
2024-12-18T02:34:34.2999653Z # shellcheck disable=SC2156
2024-12-18T02:34:34.3000222Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2024-12-18T02:34:34.3009310Z shell: /usr/bin/bash -e {0}
2024-12-18T02:34:34.3009583Z env:
2024-12-18T02:34:34.3009801Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:34:34.3010142Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:34:34.3010687Z   DOCKER_CONTAINER_ID: 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T02:34:34.3011172Z ##[endgroup]
2024-12-18T02:34:34.5812601Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@release/2.6
2024-12-18T02:34:34.5813059Z with:
2024-12-18T02:34:34.5813273Z env:
2024-12-18T02:34:34.5813487Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:34:34.5813829Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:34:34.5814372Z   DOCKER_CONTAINER_ID: 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T02:34:34.5814855Z ##[endgroup]
2024-12-18T02:34:34.5841791Z ##[group]Run set -eou pipefail
2024-12-18T02:34:34.5842134Z set -eou pipefail
2024-12-18T02:34:34.5842642Z 
2024-12-18T02:34:34.5843416Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2024-12-18T02:34:34.5844024Z for _ in $(seq 1440); do
2024-12-18T02:34:34.5844498Z     # Break if no ssh session exists anymore
2024-12-18T02:34:34.5845069Z     if [ "$(who)" = "" ]; then
2024-12-18T02:34:34.5845505Z       break
2024-12-18T02:34:34.5845890Z     fi
2024-12-18T02:34:34.5846313Z     echo "."
2024-12-18T02:34:34.5846658Z     sleep 5
2024-12-18T02:34:34.5847026Z done
2024-12-18T02:34:34.5855967Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:34:34.5856494Z env:
2024-12-18T02:34:34.5856852Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:34:34.5857361Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:34:34.5870019Z   DOCKER_CONTAINER_ID: 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T02:34:34.5870512Z ##[endgroup]
2024-12-18T02:34:34.5897278Z Holding runner for 2 hours until all ssh sessions have logged out
2024-12-18T02:34:34.5988740Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2024-12-18T02:34:34.5989473Z # ignore expansion of "docker ps -q" since it could be empty
2024-12-18T02:34:34.5989913Z # shellcheck disable=SC2046
2024-12-18T02:34:34.5990260Z docker stop $(docker ps -q) || true
2024-12-18T02:34:34.5990600Z # Prune all of the docker images
2024-12-18T02:34:34.5990924Z docker system prune -af
2024-12-18T02:34:34.5999510Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:34:34.5999884Z env:
2024-12-18T02:34:34.6000108Z   GIT_DEFAULT_BRANCH: main
2024-12-18T02:34:34.6000443Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-12-18T02:34:34.6000974Z   DOCKER_CONTAINER_ID: 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T02:34:34.6001465Z ##[endgroup]
2024-12-18T02:34:35.6343221Z 273fc4bc868e
2024-12-18T02:34:39.7461411Z Deleted Containers:
2024-12-18T02:34:39.7461937Z 273fc4bc868e6990bfa9a891565e6d529b6e5b2ec0576a0016699136dbc5f128
2024-12-18T02:34:39.7462311Z 
2024-12-18T02:34:48.2476666Z Deleted Images:
2024-12-18T02:34:48.2478098Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1
2024-12-18T02:34:48.2480027Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks@sha256:ff80d4c93d18e8ab7a2501f0e62b7c40cf97aa83fc9702e88b9c2d02cdcbecdb
2024-12-18T02:34:48.2481183Z deleted: sha256:67e93a8badaf799ceb664d7c453077c6c6785d56dc10c720be9358edeb91759b
2024-12-18T02:34:48.2481811Z deleted: sha256:7958cbbed07d0e7e404de5150d4669da383984a17de74e455563b7ecc409b2e8
2024-12-18T02:34:48.2482425Z deleted: sha256:dc6a8b10e3a8452ea47655ddad3b52816f12bcbe35298b3aa96aae8a0da9bc12
2024-12-18T02:34:48.2483074Z deleted: sha256:c92bf6e7cfbdecd0dfe86d452eac5dbe5aa5b2a6aa7ef2eaeffb21af2870cda7
2024-12-18T02:34:48.2483711Z deleted: sha256:f9597cb89d67c0c899e266adc5e5f04817474bad200712b3c256ed33c346b3e8
2024-12-18T02:34:48.2484346Z deleted: sha256:eb062308fcca86ab5a341555bf03af093d21f1d44dbe3dce6f9b7781b650b2f1
2024-12-18T02:34:48.2485330Z deleted: sha256:96f28e0a6369023dd4e75125d2454531ff686a14467f3e06d3b5c6da89893472
2024-12-18T02:34:48.2485949Z deleted: sha256:3e6e64b255956d1bf66c97393ae4325f7fc67414cd710e284c17190cb262c92d
2024-12-18T02:34:48.2486552Z deleted: sha256:a3c0a58b4f23e46b45dd1b385363731800042b3f04ca66f8867b99f3c9311f11
2024-12-18T02:34:48.2487163Z deleted: sha256:435e519973ffb9c85330ae8db1d1fb1fa75d673e2f27640c2ca7277c6352092b
2024-12-18T02:34:48.2487773Z deleted: sha256:69800681a8b6382d520ac4b8b8828bd5540a297961ddeb004e415bc46b33f80d
2024-12-18T02:34:48.2488384Z deleted: sha256:1e837ad662894f46afc56a73aa5ccb04064e4a974c9e3cad86ce9e5546b1d63d
2024-12-18T02:34:48.2489001Z deleted: sha256:688eccf2945022bd75f9f2782e2397ea54f1032c2435aa8f7dffa23cbcf2ae99
2024-12-18T02:34:48.2489615Z deleted: sha256:4855651bf4c3c8acad23029fda7d17f43bf32a69e8d575bb43f4491f0d653c1f
2024-12-18T02:34:48.2490234Z deleted: sha256:ca2d39907b7e10e3c7ae753b8bd230fac77c4e4b15980befd71173da8a36da93
2024-12-18T02:34:48.2490860Z deleted: sha256:d4f04a8342f272455eb816a413dc2dfcc75b9ebb9ef7f6fab32436621412ba6b
2024-12-18T02:34:48.2491487Z deleted: sha256:63ae01c22dc840b763ddc08bf4e145c14a04e0ff8a833b357ed45cb0962a8116
2024-12-18T02:34:48.2492106Z deleted: sha256:cf91e4695bf05d3607b7fa68e3ba3d6cc8762e6fac8773483f8116812f3d52c0
2024-12-18T02:34:48.2492711Z deleted: sha256:71baa7b72633376bb45367512dcb676c81b1e09964085a3151381530df7cdc28
2024-12-18T02:34:48.2493307Z deleted: sha256:68855433b69809d035da2a30cb22864ed1cd2795434571a0c77532de19344497
2024-12-18T02:34:48.2493893Z deleted: sha256:929280209b1f4e29860f386a519df32161cb608338d5338f24735201939dc91f
2024-12-18T02:34:48.2494499Z deleted: sha256:25133b63282efce5897b0e8fbf64d8b2a2625e15792dad95bdada277b78c777d
2024-12-18T02:34:48.2495124Z deleted: sha256:11fddfe713ea9ddd5acd8285319a606609b5c428111f15a019c8286e7164b9b4
2024-12-18T02:34:48.2495931Z deleted: sha256:b1464f3888e3b0a1694aa960a726584d01ae6b62df53fe8743d197d2d5051036
2024-12-18T02:34:48.2496535Z deleted: sha256:9627c2925d7a3936b439a6f519083e650d160bc2522d2a9244d6a1dfda02bc83
2024-12-18T02:34:48.2497141Z deleted: sha256:33880827ded8236cc91531c2f928d6de5c7cfb95a241da2653b7e8a8d1ea4881
2024-12-18T02:34:48.2497746Z deleted: sha256:6851b39610a99eb7ec6939a15e35657e144eb3e628a3856afa0a0adff49b8c2f
2024-12-18T02:34:48.2498353Z deleted: sha256:1a76c7f0af1e05689638a3792e8b9d4418648384aed1f56d3ccde5b0a9681d29
2024-12-18T02:34:48.2498965Z deleted: sha256:072f501c97aae0be55d1457426bc5f8238b0bb86eac9e92889b01924352d79a5
2024-12-18T02:34:48.2499579Z deleted: sha256:8277868848759cc86b200280dee8701d7e8328a811bf75330ed24aeaf490e69e
2024-12-18T02:34:48.2500187Z deleted: sha256:1da994d575d08f659512ea580648bfddd62020acc1aef2f7d4c6008d36a6f83f
2024-12-18T02:34:48.2500815Z deleted: sha256:ddf6d49d71559688ae65aeecccbbc58dd555887d4b3306d02a87d049114dd2f9
2024-12-18T02:34:48.2501444Z deleted: sha256:648d99cd03665c7b88aae7e3291f27daa65f496f7ab2c37f0ee62c72a4459151
2024-12-18T02:34:48.2502147Z deleted: sha256:6e57cd88598cddad4070d39467d8e1d4c387388f89d2970a55391653f9243e1a
2024-12-18T02:34:48.2502767Z deleted: sha256:9b0c5db353ac5d4e0995c2a70ae3c28390676afe60dbd11e4ade07c792eac4ce
2024-12-18T02:34:48.2503391Z deleted: sha256:8fa5311fe41b6caf697456e9678220a152bbf6a7645f4b7ff89ea9317ae04444
2024-12-18T02:34:48.2504028Z deleted: sha256:dc438d6465050b81bff4a8c5925e70c2167712dbab174c38f79b6a5d7af407af
2024-12-18T02:34:48.2504638Z deleted: sha256:29b8f0a42a1f3b944f2c54348348d173d322522b578e3d485428d67996f6adde
2024-12-18T02:34:48.2505237Z deleted: sha256:26231024334c7a704f9acba86430547a013e959014bfd7a31e26c6ed78521479
2024-12-18T02:34:48.2505859Z deleted: sha256:4166e9dd7564ae2f46ea2c3dcb8a8ef6bef9e06f4faf1742f4c6f21861967193
2024-12-18T02:34:48.2506476Z deleted: sha256:80a3113f256f146f0a2528ca69aa9244a1f9ba1ecdf54c3ef8ba0fb21507cbc4
2024-12-18T02:34:48.2507105Z deleted: sha256:e90bbba5d7c08e01c5bedb0a855cc5cdd371688e23994155674704963591296f
2024-12-18T02:34:48.2507788Z deleted: sha256:d3b3df431f230849965cc370c2f1853a81d78dd6eb3f7ac3bc3b4ca64aa9818a
2024-12-18T02:34:48.2508409Z deleted: sha256:7a33f9ca143b08c9f5d8c53974b2d6fe840739b5d2cd30694ba1c86db9393908
2024-12-18T02:34:48.2509173Z deleted: sha256:bad1d7dae968956b4a81369dedfbcf539fd5b61dd3051e2eccb5457174fe788c
2024-12-18T02:34:48.2509807Z deleted: sha256:08a420245ca6eed4aa5662d0fec04e7892165cdb3fd2f5b99428adbb4d6199c0
2024-12-18T02:34:48.2510416Z deleted: sha256:c3fda6b9a4d08a345824508dc5625f52e4d23163843d0caf337c60feda68a830
2024-12-18T02:34:48.2511041Z deleted: sha256:c74a8a3ca5e31670eed41e4bbb4a89f25c7c05dacabdf93cbd03984ead0330e2
2024-12-18T02:34:48.2511738Z deleted: sha256:4171d4c54045bd3035546906a7dd3b63c9ca748e0a34eeba27fe918f52cdcf85
2024-12-18T02:34:48.2512517Z deleted: sha256:f6a1d112657a4aeaca96ac2792a10ea09463dc3d633cb8572314814d5abf1ddb
2024-12-18T02:34:48.2513163Z deleted: sha256:af5d54d84b9c302254498eed8b0bbce1cdd705351bf5cc210d3f11c0deb269b3
2024-12-18T02:34:48.2513791Z deleted: sha256:45ae227353c6db8230b86fc1c4b8c50def31bd64895b879d91ea6e600669045d
2024-12-18T02:34:48.2514390Z deleted: sha256:6032ea27ad90071e3b75290e03fd0763a74155f513f356d21e56663a878b0ed9
2024-12-18T02:34:48.2514999Z deleted: sha256:d5aef8ace96787c48070530c09b68e4bcb23bc92a9fb0978220204f64bfbd97f
2024-12-18T02:34:48.2515616Z deleted: sha256:e3116153021ece2a3d29eccdce1f7e0f092316e889c1d534257789f6b5743d51
2024-12-18T02:34:48.2516235Z deleted: sha256:fb0f11460e5fa2e4895230ae84c97fe0552e9509dfbae60abf94e1b8a6563fb9
2024-12-18T02:34:48.2516854Z deleted: sha256:e262415e0abb361b284df9e05790ae27cc3a1ea7b8d4618e7eb3eca17cf1d18e
2024-12-18T02:34:48.2517468Z deleted: sha256:7f810eab57a6121f1a2aae664f7a5940d12f789661445ce3500ecd3b3688b46b
2024-12-18T02:34:48.2518073Z deleted: sha256:6ed71f55ff78520154c43468df2f05a92eaa28ee2aa2e28b8ea860b9ffb258ef
2024-12-18T02:34:48.2518687Z deleted: sha256:92a10d8d42a56a8516d0e9e14fec53263302fe6e6438ced263a3845680456dd7
2024-12-18T02:34:48.2519301Z deleted: sha256:ec67f912ff53e9b7164f48250da3bb4c1ffc94c3eb2c1c56c534a94fe55e3845
2024-12-18T02:34:48.2520026Z deleted: sha256:c85033e4f60ace56eaad80a8151b1bdd1619865b10a9ccee5638878b0a027c23
2024-12-18T02:34:48.2520641Z deleted: sha256:0e8680489f0a37851beb2cb5e42f0df129247a68b755b08a6367ea42180b3e7a
2024-12-18T02:34:48.2521247Z deleted: sha256:3a767fb38f6b1124e5fe9a230fdd383ccb188c38fe1d6ec252260676efe255c8
2024-12-18T02:34:48.2521857Z deleted: sha256:2cce59d2a43773cb7c6b83d239dab3a53659767f96bc0d47a70ee0d51cfa096f
2024-12-18T02:34:48.2522472Z deleted: sha256:51e50e75789ab5b4e40a66bb2f284e2da947e37c0e757c74d49fdeea5da9cbec
2024-12-18T02:34:48.2523084Z deleted: sha256:29e55e4feccba0660950ed92941d309527fe5f41bea609662d0e703b9c76b3a1
2024-12-18T02:34:48.2523708Z deleted: sha256:d15bee8fc455fcb45d16b0abc67f18c99ce1cf2c9e998ef5a100bf8373f2f8ab
2024-12-18T02:34:48.2524328Z deleted: sha256:363936db6a8869d44b6f0bee97a786b168093848e9c08c8042830fc7fccd956b
2024-12-18T02:34:48.2524933Z deleted: sha256:0f54d1f89ceaf71ded488007635a618fac1528bf3603d9415a593fe4926f8b2a
2024-12-18T02:34:48.2525554Z deleted: sha256:2f5796e9879cab7dac02bdbb178aec5301ac00029b38be8613333ac639c4dc3a
2024-12-18T02:34:48.2526178Z deleted: sha256:e68525d57f9da8f6c74b456d92869c4e60e451b88cb403addbc4ea917d8ab3ac
2024-12-18T02:34:48.2526843Z deleted: sha256:3386e25983944ce3b2ebdde10d840832d581a7dfafd1df66568822a18979e268
2024-12-18T02:34:48.2527455Z deleted: sha256:d3a036e6bd36f014b3a1540d5b848655ecdf72a20dc81c0da5f80430700cfa9a
2024-12-18T02:34:48.2528072Z deleted: sha256:96d0556708ff4ec0d5d727d5947d6cb29a9d6c16e1a1ee76a92dc85aa3ea8f25
2024-12-18T02:34:48.2528693Z deleted: sha256:f3c2fb3abccb3a33dbf204d3f0d8857537348e6c1d3cc87a431afe866435d41b
2024-12-18T02:34:48.2529307Z deleted: sha256:45cd6c9a41f12991ee60a89e775abb604b0892ef3936526c18184bbbf4de57da
2024-12-18T02:34:48.2529930Z deleted: sha256:b3cacd0508bbfd2fd668f30c539033e7c6b606bf669258c8e8a3cbf80bf050c6
2024-12-18T02:34:48.2530547Z deleted: sha256:de72ce930361a96c151a8a4725f8360f33df6c812b6ed6d3dd5b6e6710a5d2c0
2024-12-18T02:34:48.2531171Z deleted: sha256:2e5d223d2bfa0f47e628b631a540d20c284dabb380ad9ee4caec65bf8d0ecde2
2024-12-18T02:34:48.2531784Z deleted: sha256:14a85e604a0401093323d356b3bb7f75a8473abd712b4bf1a2ef01fa432b344c
2024-12-18T02:34:48.2532488Z deleted: sha256:15a95a1e8f47d7c18dcbf9a6e2fc530cddcb161c0093c663181f3e73dc9f599e
2024-12-18T02:34:48.2533111Z deleted: sha256:106e8431b412f51ccd75ea46a2d5cb4343b23273cbcf50188377cb93aa9a6d82
2024-12-18T02:34:48.2533476Z 
2024-12-18T02:34:48.2533598Z Total reclaimed space: 53.04GB
2024-12-18T02:34:48.2601872Z Post job cleanup.
2024-12-18T02:34:48.2649933Z Post job cleanup.
2024-12-18T02:34:48.3413273Z [command]/usr/bin/git version
2024-12-18T02:34:48.3448792Z git version 2.40.1
2024-12-18T02:34:48.3482752Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/ac4ec9f2-d7c2-4db1-9c30-e46b305c56c4/.gitconfig'
2024-12-18T02:34:48.3492375Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/ac4ec9f2-d7c2-4db1-9c30-e46b305c56c4' before making global git config changes
2024-12-18T02:34:48.3493319Z Adding repository directory to the temporary git global config as a safe directory
2024-12-18T02:34:48.3496534Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-12-18T02:34:48.3536312Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2024-12-18T02:34:48.3567059Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2024-12-18T02:34:48.3939281Z Entering 'android/libs/fbjni'
2024-12-18T02:34:48.4010319Z Entering 'third_party/FP16'
2024-12-18T02:34:48.4078577Z Entering 'third_party/FXdiv'
2024-12-18T02:34:48.4146413Z Entering 'third_party/NNPACK'
2024-12-18T02:34:48.4216410Z Entering 'third_party/NVTX'
2024-12-18T02:34:48.4290035Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T02:34:48.4359664Z Entering 'third_party/XNNPACK'
2024-12-18T02:34:48.4444615Z Entering 'third_party/benchmark'
2024-12-18T02:34:48.4512821Z Entering 'third_party/composable_kernel'
2024-12-18T02:34:48.4591024Z Entering 'third_party/cpp-httplib'
2024-12-18T02:34:48.4658131Z Entering 'third_party/cpuinfo'
2024-12-18T02:34:48.4727196Z Entering 'third_party/cudnn_frontend'
2024-12-18T02:34:48.4795708Z Entering 'third_party/cutlass'
2024-12-18T02:34:48.4872036Z Entering 'third_party/eigen'
2024-12-18T02:34:48.4942147Z Entering 'third_party/fbgemm'
2024-12-18T02:34:48.5011118Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T02:34:48.5078953Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T02:34:48.5145951Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T02:34:48.5226652Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T02:34:48.5293495Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T02:34:48.5364070Z Entering 'third_party/flatbuffers'
2024-12-18T02:34:48.5435748Z Entering 'third_party/fmt'
2024-12-18T02:34:48.5503133Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T02:34:48.5572322Z Entering 'third_party/gloo'
2024-12-18T02:34:48.5642457Z Entering 'third_party/googletest'
2024-12-18T02:34:48.5712580Z Entering 'third_party/ideep'
2024-12-18T02:34:48.5777812Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T02:34:48.5854378Z Entering 'third_party/ittapi'
2024-12-18T02:34:48.5922722Z Entering 'third_party/kineto'
2024-12-18T02:34:48.5989611Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T02:34:48.6054278Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T02:34:48.6123747Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T02:34:48.6191739Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T02:34:48.6259310Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T02:34:48.6324386Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T02:34:48.6395194Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T02:34:48.6462008Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T02:34:48.6530464Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T02:34:48.6599385Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T02:34:48.6670021Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T02:34:48.6736613Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T02:34:48.6811642Z Entering 'third_party/mimalloc'
2024-12-18T02:34:48.6882952Z Entering 'third_party/nccl/nccl'
2024-12-18T02:34:48.6953201Z Entering 'third_party/nlohmann'
2024-12-18T02:34:48.7025954Z Entering 'third_party/onnx'
2024-12-18T02:34:48.7108727Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T02:34:48.7186461Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T02:34:48.7255303Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T02:34:48.7330374Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T02:34:48.7397313Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T02:34:48.7468308Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T02:34:48.7538918Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T02:34:48.7608983Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T02:34:48.7675616Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T02:34:48.7741480Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T02:34:48.7814132Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T02:34:48.7885504Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T02:34:48.7972864Z Entering 'third_party/pocketfft'
2024-12-18T02:34:48.8041298Z Entering 'third_party/protobuf'
2024-12-18T02:34:48.8113998Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T02:34:48.8180569Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T02:34:48.8254898Z Entering 'third_party/psimd'
2024-12-18T02:34:48.8330349Z Entering 'third_party/pthreadpool'
2024-12-18T02:34:48.8403589Z Entering 'third_party/pybind11'
2024-12-18T02:34:48.8473843Z Entering 'third_party/python-peachpy'
2024-12-18T02:34:48.8543505Z Entering 'third_party/sleef'
2024-12-18T02:34:48.8614345Z Entering 'third_party/tensorpipe'
2024-12-18T02:34:48.8683229Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T02:34:48.8749630Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T02:34:48.8815520Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T02:34:48.8882039Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T02:34:48.8946319Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T02:34:48.9043161Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2024-12-18T02:34:48.9064519Z http.https://github.com/.extraheader
2024-12-18T02:34:48.9074497Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2024-12-18T02:34:48.9104861Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2024-12-18T02:34:48.9461887Z Entering 'android/libs/fbjni'
2024-12-18T02:34:48.9505600Z http.https://github.com/.extraheader
2024-12-18T02:34:48.9550436Z Entering 'third_party/FP16'
2024-12-18T02:34:48.9596414Z http.https://github.com/.extraheader
2024-12-18T02:34:48.9640496Z Entering 'third_party/FXdiv'
2024-12-18T02:34:48.9684510Z http.https://github.com/.extraheader
2024-12-18T02:34:48.9727472Z Entering 'third_party/NNPACK'
2024-12-18T02:34:48.9776345Z http.https://github.com/.extraheader
2024-12-18T02:34:48.9819907Z Entering 'third_party/NVTX'
2024-12-18T02:34:48.9862910Z http.https://github.com/.extraheader
2024-12-18T02:34:48.9906471Z Entering 'third_party/VulkanMemoryAllocator'
2024-12-18T02:34:48.9949871Z http.https://github.com/.extraheader
2024-12-18T02:34:48.9995184Z Entering 'third_party/XNNPACK'
2024-12-18T02:34:49.0040752Z http.https://github.com/.extraheader
2024-12-18T02:34:49.0100434Z Entering 'third_party/benchmark'
2024-12-18T02:34:49.0144082Z http.https://github.com/.extraheader
2024-12-18T02:34:49.0192254Z Entering 'third_party/composable_kernel'
2024-12-18T02:34:49.0235821Z http.https://github.com/.extraheader
2024-12-18T02:34:49.0285641Z Entering 'third_party/cpp-httplib'
2024-12-18T02:34:49.0329365Z http.https://github.com/.extraheader
2024-12-18T02:34:49.0373161Z Entering 'third_party/cpuinfo'
2024-12-18T02:34:49.0417236Z http.https://github.com/.extraheader
2024-12-18T02:34:49.0461516Z Entering 'third_party/cudnn_frontend'
2024-12-18T02:34:49.0505135Z http.https://github.com/.extraheader
2024-12-18T02:34:49.0552976Z Entering 'third_party/cutlass'
2024-12-18T02:34:49.0596802Z http.https://github.com/.extraheader
2024-12-18T02:34:49.0652752Z Entering 'third_party/eigen'
2024-12-18T02:34:49.0697789Z http.https://github.com/.extraheader
2024-12-18T02:34:49.0744576Z Entering 'third_party/fbgemm'
2024-12-18T02:34:49.0788933Z http.https://github.com/.extraheader
2024-12-18T02:34:49.0834329Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-12-18T02:34:49.0878181Z http.https://github.com/.extraheader
2024-12-18T02:34:49.0921512Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-12-18T02:34:49.0964237Z http.https://github.com/.extraheader
2024-12-18T02:34:49.1009363Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-12-18T02:34:49.1052612Z http.https://github.com/.extraheader
2024-12-18T02:34:49.1103907Z Entering 'third_party/fbgemm/third_party/googletest'
2024-12-18T02:34:49.1146045Z http.https://github.com/.extraheader
2024-12-18T02:34:49.1190053Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-12-18T02:34:49.1232901Z http.https://github.com/.extraheader
2024-12-18T02:34:49.1280334Z Entering 'third_party/flatbuffers'
2024-12-18T02:34:49.1323954Z http.https://github.com/.extraheader
2024-12-18T02:34:49.1370634Z Entering 'third_party/fmt'
2024-12-18T02:34:49.1413853Z http.https://github.com/.extraheader
2024-12-18T02:34:49.1456950Z Entering 'third_party/gemmlowp/gemmlowp'
2024-12-18T02:34:49.1506100Z http.https://github.com/.extraheader
2024-12-18T02:34:49.1551378Z Entering 'third_party/gloo'
2024-12-18T02:34:49.1595322Z http.https://github.com/.extraheader
2024-12-18T02:34:49.1640317Z Entering 'third_party/googletest'
2024-12-18T02:34:49.1684103Z http.https://github.com/.extraheader
2024-12-18T02:34:49.1727471Z Entering 'third_party/ideep'
2024-12-18T02:34:49.1772178Z http.https://github.com/.extraheader
2024-12-18T02:34:49.1814681Z Entering 'third_party/ideep/mkl-dnn'
2024-12-18T02:34:49.1857588Z http.https://github.com/.extraheader
2024-12-18T02:34:49.1909998Z Entering 'third_party/ittapi'
2024-12-18T02:34:49.1954377Z http.https://github.com/.extraheader
2024-12-18T02:34:49.1999669Z Entering 'third_party/kineto'
2024-12-18T02:34:49.2043328Z http.https://github.com/.extraheader
2024-12-18T02:34:49.2086789Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-12-18T02:34:49.2129920Z http.https://github.com/.extraheader
2024-12-18T02:34:49.2174158Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-12-18T02:34:49.2219205Z http.https://github.com/.extraheader
2024-12-18T02:34:49.2269113Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-12-18T02:34:49.2312211Z http.https://github.com/.extraheader
2024-12-18T02:34:49.2356167Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-12-18T02:34:49.2400734Z http.https://github.com/.extraheader
2024-12-18T02:34:49.2444761Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-12-18T02:34:49.2488268Z http.https://github.com/.extraheader
2024-12-18T02:34:49.2531792Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-12-18T02:34:49.2575116Z http.https://github.com/.extraheader
2024-12-18T02:34:49.2622892Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-12-18T02:34:49.2670913Z http.https://github.com/.extraheader
2024-12-18T02:34:49.2714692Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-12-18T02:34:49.2758286Z http.https://github.com/.extraheader
2024-12-18T02:34:49.2804435Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-12-18T02:34:49.2847860Z http.https://github.com/.extraheader
2024-12-18T02:34:49.2897550Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-12-18T02:34:49.2949557Z http.https://github.com/.extraheader
2024-12-18T02:34:49.2996890Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-12-18T02:34:49.3040030Z http.https://github.com/.extraheader
2024-12-18T02:34:49.3084496Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-12-18T02:34:49.3126609Z http.https://github.com/.extraheader
2024-12-18T02:34:49.3173591Z Entering 'third_party/mimalloc'
2024-12-18T02:34:49.3219974Z http.https://github.com/.extraheader
2024-12-18T02:34:49.3264004Z Entering 'third_party/nccl/nccl'
2024-12-18T02:34:49.3308452Z http.https://github.com/.extraheader
2024-12-18T02:34:49.3353369Z Entering 'third_party/nlohmann'
2024-12-18T02:34:49.3398348Z http.https://github.com/.extraheader
2024-12-18T02:34:49.3448229Z Entering 'third_party/onnx'
2024-12-18T02:34:49.3496265Z http.https://github.com/.extraheader
2024-12-18T02:34:49.3552748Z Entering 'third_party/onnx/third_party/pybind11'
2024-12-18T02:34:49.3596733Z http.https://github.com/.extraheader
2024-12-18T02:34:49.3645160Z Entering 'third_party/opentelemetry-cpp'
2024-12-18T02:34:49.3689331Z http.https://github.com/.extraheader
2024-12-18T02:34:49.3733746Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-12-18T02:34:49.3776663Z http.https://github.com/.extraheader
2024-12-18T02:34:49.3819345Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-12-18T02:34:49.3866072Z http.https://github.com/.extraheader
2024-12-18T02:34:49.3909525Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-12-18T02:34:49.3951696Z http.https://github.com/.extraheader
2024-12-18T02:34:49.3995134Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-12-18T02:34:49.4037118Z http.https://github.com/.extraheader
2024-12-18T02:34:49.4081955Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-12-18T02:34:49.4123616Z http.https://github.com/.extraheader
2024-12-18T02:34:49.4166082Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-12-18T02:34:49.4216712Z http.https://github.com/.extraheader
2024-12-18T02:34:49.4260405Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-12-18T02:34:49.4307615Z http.https://github.com/.extraheader
2024-12-18T02:34:49.4349181Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-12-18T02:34:49.4393858Z http.https://github.com/.extraheader
2024-12-18T02:34:49.4439718Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-12-18T02:34:49.4486403Z http.https://github.com/.extraheader
2024-12-18T02:34:49.4532764Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-12-18T02:34:49.4575550Z http.https://github.com/.extraheader
2024-12-18T02:34:49.4642175Z Entering 'third_party/pocketfft'
2024-12-18T02:34:49.4685683Z http.https://github.com/.extraheader
2024-12-18T02:34:49.4728571Z Entering 'third_party/protobuf'
2024-12-18T02:34:49.4773490Z http.https://github.com/.extraheader
2024-12-18T02:34:49.4819742Z Entering 'third_party/protobuf/third_party/benchmark'
2024-12-18T02:34:49.4863075Z http.https://github.com/.extraheader
2024-12-18T02:34:49.4906386Z Entering 'third_party/protobuf/third_party/googletest'
2024-12-18T02:34:49.4948996Z http.https://github.com/.extraheader
2024-12-18T02:34:49.4996148Z Entering 'third_party/psimd'
2024-12-18T02:34:49.5040448Z http.https://github.com/.extraheader
2024-12-18T02:34:49.5086466Z Entering 'third_party/pthreadpool'
2024-12-18T02:34:49.5130089Z http.https://github.com/.extraheader
2024-12-18T02:34:49.5174816Z Entering 'third_party/pybind11'
2024-12-18T02:34:49.5219237Z http.https://github.com/.extraheader
2024-12-18T02:34:49.5263848Z Entering 'third_party/python-peachpy'
2024-12-18T02:34:49.5307418Z http.https://github.com/.extraheader
2024-12-18T02:34:49.5349968Z Entering 'third_party/sleef'
2024-12-18T02:34:49.5394385Z http.https://github.com/.extraheader
2024-12-18T02:34:49.5437857Z Entering 'third_party/tensorpipe'
2024-12-18T02:34:49.5482820Z http.https://github.com/.extraheader
2024-12-18T02:34:49.5526029Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-12-18T02:34:49.5570034Z http.https://github.com/.extraheader
2024-12-18T02:34:49.5618286Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-12-18T02:34:49.5659925Z http.https://github.com/.extraheader
2024-12-18T02:34:49.5703193Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-12-18T02:34:49.5745659Z http.https://github.com/.extraheader
2024-12-18T02:34:49.5790426Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-12-18T02:34:49.5832584Z http.https://github.com/.extraheader
2024-12-18T02:34:49.5874202Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-12-18T02:34:49.5917187Z http.https://github.com/.extraheader
2024-12-18T02:34:49.6081084Z A job completed hook has been configured by the self-hosted runner administrator
2024-12-18T02:34:49.6108378Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh'
2024-12-18T02:34:49.6116197Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-12-18T02:34:49.6116591Z ##[endgroup]
2024-12-18T02:34:56.6153085Z Cleaning up orphan processes