2023-09-06T07:45:13.8432372Z Requested labels: linux.gcp.a100.large
2023-09-06T07:45:13.8432524Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/main
2023-09-06T07:45:13.8432688Z Reusable workflow chain:
2023-09-06T07:45:13.8432823Z pytorch/pytorch/.github/workflows/inductor-perf-test-nightly.yml@refs/heads/main (3fe8417643c8d6c2b3d95552cd90321d141b5d54)
2023-09-06T07:45:13.8432968Z -> pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/main (3fe8417643c8d6c2b3d95552cd90321d141b5d54)
2023-09-06T07:45:13.8433117Z Waiting for a runner to pick up this job...
2023-09-06T11:53:07.5826457Z Job is about to start running on the runner: gh-ci-gcp-a100-17 (organization)
2023-09-06T11:53:12.2288634Z Current runner version: '2.308.0'
2023-09-06T11:53:12.2295974Z Runner name: 'gh-ci-gcp-a100-17'
2023-09-06T11:53:12.2296630Z Runner group name: 'Default'
2023-09-06T11:53:12.2297418Z Machine name: 'gh-ci-gcp-a100-17'
2023-09-06T11:53:12.2300065Z ##[group]GITHUB_TOKEN Permissions
2023-09-06T11:53:12.2300939Z Actions: write
2023-09-06T11:53:12.2301276Z Checks: write
2023-09-06T11:53:12.2301723Z Contents: write
2023-09-06T11:53:12.2302117Z Deployments: write
2023-09-06T11:53:12.2302516Z Discussions: write
2023-09-06T11:53:12.2302939Z Issues: write
2023-09-06T11:53:12.2303336Z Metadata: read
2023-09-06T11:53:12.2303749Z Packages: write
2023-09-06T11:53:12.2304140Z Pages: write
2023-09-06T11:53:12.2304480Z PullRequests: write
2023-09-06T11:53:12.2304935Z RepositoryProjects: write
2023-09-06T11:53:12.2305478Z SecurityEvents: write
2023-09-06T11:53:12.2305881Z Statuses: write
2023-09-06T11:53:12.2306212Z ##[endgroup]
2023-09-06T11:53:12.2310408Z Secret source: Actions
2023-09-06T11:53:12.2311333Z Prepare workflow directory
2023-09-06T11:53:12.5141437Z Prepare all required actions
2023-09-06T11:53:12.5366185Z Getting action download info
2023-09-06T11:53:12.7517488Z Download action repository 'pytorch/test-infra@main' (SHA:2153cf14870e07c67ed933a1ee61a3ca2ee1ed19)
2023-09-06T11:53:13.4864394Z Download action repository 'pytorch/pytorch@main' (SHA:60bd30ee0b35e4a5d96e2e65d320e99bd07c34d3)
2023-09-06T11:53:21.8036060Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2023-09-06T11:53:22.4638823Z Getting action download info
2023-09-06T11:53:22.6231731Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c)
2023-09-06T11:53:23.1320408Z Getting action download info
2023-09-06T11:53:23.2720357Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482)
2023-09-06T11:53:23.7481500Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/main (3fe8417643c8d6c2b3d95552cd90321d141b5d54)
2023-09-06T11:53:23.7483709Z ##[group] Inputs
2023-09-06T11:53:23.7484106Z   build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm80
2023-09-06T11:53:23.7486035Z   test-matrix: {"include": [{"config": "inductor_huggingface_perf", "shard": 1, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_huggingface_perf", "shard": 2, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_huggingface_perf", "shard": 3, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 1, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 2, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 3, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 4, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 5, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 1, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 2, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 3, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 4, "num_shards": 4, "runner": "linux.gcp.a100.large"}]}
2023-09-06T11:53:23.7488278Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:9dd361d1c04129f8eaa9d6b43335917800dd6d24
2023-09-06T11:53:23.7488813Z   sync-tag: 
2023-09-06T11:53:23.7489686Z   timeout-minutes: 720
2023-09-06T11:53:23.7490003Z   use-gha: anything-non-empty-to-use-gha
2023-09-06T11:53:23.7490723Z   dashboard-tag: training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true
2023-09-06T11:53:23.7491709Z ##[endgroup]
2023-09-06T11:53:23.7492556Z Complete job name: cuda12.1-py3.10-gcc9-sm80 / test (inductor_torchbench_perf, 1, 4, linux.gcp.a100.large)
2023-09-06T11:53:23.8157988Z A job started hook has been configured by the self-hosted runner administrator
2023-09-06T11:53:23.8337580Z ##[group]Run '/home/weiwangmeta/pre-job.sh'
2023-09-06T11:53:23.8360619Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-09-06T11:53:23.8360952Z ##[endgroup]
2023-09-06T11:53:23.8975264Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main
2023-09-06T11:53:23.8975644Z with:
2023-09-06T11:53:23.8975890Z   submodules: recursive
2023-09-06T11:53:23.8976131Z   fetch-depth: 0
2023-09-06T11:53:23.8976353Z env:
2023-09-06T11:53:23.8976587Z   GIT_DEFAULT_BRANCH: main
2023-09-06T11:53:23.8977087Z ##[endgroup]
2023-09-06T11:53:23.9199175Z ##[group]Run retry () {
2023-09-06T11:53:23.9199514Z retry () {
2023-09-06T11:53:23.9199889Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2023-09-06T11:53:23.9200208Z }
2023-09-06T11:53:23.9200444Z echo "${GITHUB_WORKSPACE}"
2023-09-06T11:53:23.9200731Z if [ -z "${NO_SUDO}" ]; then
2023-09-06T11:53:23.9201042Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2023-09-06T11:53:23.9201319Z else
2023-09-06T11:53:23.9201591Z   retry rm -rf "${GITHUB_WORKSPACE}"
2023-09-06T11:53:23.9201836Z fi
2023-09-06T11:53:23.9202137Z mkdir "${GITHUB_WORKSPACE}"
2023-09-06T11:53:23.9220313Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-09-06T11:53:23.9220639Z env:
2023-09-06T11:53:23.9220863Z   GIT_DEFAULT_BRANCH: main
2023-09-06T11:53:23.9221107Z   NO_SUDO: 
2023-09-06T11:53:23.9221332Z ##[endgroup]
2023-09-06T11:53:23.9303608Z /home/weiwangmeta/actions-runner/_work/pytorch/pytorch
2023-09-06T11:53:26.5556401Z ##[group]Run malfet/checkout@silent-checkout
2023-09-06T11:53:26.5556792Z with:
2023-09-06T11:53:26.5557068Z   ref: 3fe8417643c8d6c2b3d95552cd90321d141b5d54
2023-09-06T11:53:26.5557365Z   fetch-depth: 0
2023-09-06T11:53:26.5557593Z   submodules: recursive
2023-09-06T11:53:26.5557844Z   quiet-checkout: true
2023-09-06T11:53:26.5558112Z   repository: pytorch/pytorch
2023-09-06T11:53:26.5558532Z   token: ***
2023-09-06T11:53:26.5558743Z   ssh-strict: true
2023-09-06T11:53:26.5558997Z   persist-credentials: true
2023-09-06T11:53:26.5559251Z   clean: true
2023-09-06T11:53:26.5559526Z   sparse-checkout-cone-mode: true
2023-09-06T11:53:26.5559776Z   lfs: false
2023-09-06T11:53:26.5560011Z   set-safe-directory: true
2023-09-06T11:53:26.5560247Z env:
2023-09-06T11:53:26.5560467Z   GIT_DEFAULT_BRANCH: main
2023-09-06T11:53:26.5560695Z ##[endgroup]
2023-09-06T11:53:26.6953222Z Syncing repository: pytorch/pytorch
2023-09-06T11:53:26.6956583Z ##[group]Getting Git version info
2023-09-06T11:53:26.6957688Z Working directory is '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch'
2023-09-06T11:53:26.6958738Z [command]/usr/bin/git version
2023-09-06T11:53:26.6959159Z git version 2.25.1
2023-09-06T11:53:26.6961387Z ##[endgroup]
2023-09-06T11:53:26.6981640Z Temporarily overriding HOME='/home/weiwangmeta/actions-runner/_work/_temp/ac2727b9-510c-43a2-b3e7-c2e3c405cc83' before making global git config changes
2023-09-06T11:53:26.6982629Z Adding repository directory to the temporary git global config as a safe directory
2023-09-06T11:53:26.6983841Z [command]/usr/bin/git config --global --add safe.directory /home/weiwangmeta/actions-runner/_work/pytorch/pytorch
2023-09-06T11:53:26.7032450Z Deleting the contents of '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch'
2023-09-06T11:53:26.7038949Z ##[group]Initializing the repository
2023-09-06T11:53:26.7043387Z [command]/usr/bin/git init /home/weiwangmeta/actions-runner/_work/pytorch/pytorch
2023-09-06T11:53:26.7087802Z Initialized empty Git repository in /home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/
2023-09-06T11:53:26.7098821Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2023-09-06T11:53:26.7143816Z ##[endgroup]
2023-09-06T11:53:26.7145135Z ##[group]Disabling automatic garbage collection
2023-09-06T11:53:26.7149421Z [command]/usr/bin/git config --local gc.auto 0
2023-09-06T11:53:26.7187869Z ##[endgroup]
2023-09-06T11:53:26.7188655Z ##[group]Setting up auth
2023-09-06T11:53:26.7196695Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2023-09-06T11:53:26.7233810Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2023-09-06T11:53:26.7481678Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2023-09-06T11:53:26.7517489Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2023-09-06T11:53:26.7766087Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2023-09-06T11:53:26.7810685Z ##[endgroup]
2023-09-06T11:53:26.7811481Z ##[group]Fetching the repository
2023-09-06T11:53:26.7820666Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2023-09-06T11:53:35.1113095Z remote: Enumerating objects: 1158676        
2023-09-06T11:53:35.1113785Z remote: Enumerating objects: 1161041, done.        
2023-09-06T11:53:35.1115941Z remote: Counting objects:   0% (1/2365)        
2023-09-06T11:53:35.1116327Z remote: Counting objects:   1% (24/2365)        
2023-09-06T11:53:35.1116860Z remote: Counting objects:   2% (48/2365)        
2023-09-06T11:53:35.1117245Z remote: Counting objects:   3% (71/2365)        
2023-09-06T11:53:35.1117784Z remote: Counting objects:   4% (95/2365)        
2023-09-06T11:53:35.1118119Z remote: Counting objects:   5% (119/2365)        
2023-09-06T11:53:35.1118459Z remote: Counting objects:   6% (142/2365)        
2023-09-06T11:53:35.1119059Z remote: Counting objects:   7% (166/2365)        
2023-09-06T11:53:35.1119380Z remote: Counting objects:   8% (190/2365)        
2023-09-06T11:53:35.1119691Z remote: Counting objects:   9% (213/2365)        
2023-09-06T11:53:35.1120257Z remote: Counting objects:  10% (237/2365)        
2023-09-06T11:53:35.1120596Z remote: Counting objects:  11% (261/2365)        
2023-09-06T11:53:35.1120896Z remote: Counting objects:  12% (284/2365)        
2023-09-06T11:53:35.1121725Z remote: Counting objects:  13% (308/2365)        
2023-09-06T11:53:35.1122227Z remote: Counting objects:  14% (332/2365)        
2023-09-06T11:53:35.1123157Z remote: Counting objects:  15% (355/2365)        
2023-09-06T11:53:35.1123550Z remote: Counting objects:  16% (379/2365)        
2023-09-06T11:53:35.1123863Z remote: Counting objects:  17% (403/2365)        
2023-09-06T11:53:35.1124171Z remote: Counting objects:  18% (426/2365)        
2023-09-06T11:53:35.1124586Z remote: Counting objects:  19% (450/2365)        
2023-09-06T11:53:35.1124888Z remote: Counting objects:  20% (473/2365)        
2023-09-06T11:53:35.1125190Z remote: Counting objects:  21% (497/2365)        
2023-09-06T11:53:35.1126638Z remote: Counting objects:  22% (521/2365)        
2023-09-06T11:53:35.1127253Z remote: Counting objects:  23% (544/2365)        
2023-09-06T11:53:35.1127744Z remote: Counting objects:  24% (568/2365)        
2023-09-06T11:53:35.1128088Z remote: Counting objects:  25% (592/2365)        
2023-09-06T11:53:35.1128838Z remote: Counting objects:  26% (615/2365)        
2023-09-06T11:53:35.1129401Z remote: Counting objects:  27% (639/2365)        
2023-09-06T11:53:35.1129719Z remote: Counting objects:  28% (663/2365)        
2023-09-06T11:53:35.1130025Z remote: Counting objects:  29% (686/2365)        
2023-09-06T11:53:35.1130343Z remote: Counting objects:  30% (710/2365)        
2023-09-06T11:53:35.1130747Z remote: Counting objects:  31% (734/2365)        
2023-09-06T11:53:35.1131927Z remote: Counting objects:  32% (757/2365)        
2023-09-06T11:53:35.1132494Z remote: Counting objects:  33% (781/2365)        
2023-09-06T11:53:35.1133065Z remote: Counting objects:  34% (805/2365)        
2023-09-06T11:53:35.1133462Z remote: Counting objects:  35% (828/2365)        
2023-09-06T11:53:35.1134021Z remote: Counting objects:  36% (852/2365)        
2023-09-06T11:53:35.1134834Z remote: Counting objects:  37% (876/2365)        
2023-09-06T11:53:35.1135390Z remote: Counting objects:  38% (899/2365)        
2023-09-06T11:53:35.1135886Z remote: Counting objects:  39% (923/2365)        
2023-09-06T11:53:35.1136459Z remote: Counting objects:  40% (946/2365)        
2023-09-06T11:53:35.1136958Z remote: Counting objects:  41% (970/2365)        
2023-09-06T11:53:35.1137525Z remote: Counting objects:  42% (994/2365)        
2023-09-06T11:53:35.1138116Z remote: Counting objects:  43% (1017/2365)        
2023-09-06T11:53:35.1138669Z remote: Counting objects:  44% (1041/2365)        
2023-09-06T11:53:35.1139314Z remote: Counting objects:  45% (1065/2365)        
2023-09-06T11:53:35.1139963Z remote: Counting objects:  46% (1088/2365)        
2023-09-06T11:53:35.1140545Z remote: Counting objects:  47% (1112/2365)        
2023-09-06T11:53:35.1140891Z remote: Counting objects:  48% (1136/2365)        
2023-09-06T11:53:35.1141409Z remote: Counting objects:  49% (1159/2365)        
2023-09-06T11:53:35.1142060Z remote: Counting objects:  50% (1183/2365)        
2023-09-06T11:53:35.1142662Z remote: Counting objects:  51% (1207/2365)        
2023-09-06T11:53:35.1143344Z remote: Counting objects:  52% (1230/2365)        
2023-09-06T11:53:35.1143960Z remote: Counting objects:  53% (1254/2365)        
2023-09-06T11:53:35.1144516Z remote: Counting objects:  54% (1278/2365)        
2023-09-06T11:53:35.1145134Z remote: Counting objects:  55% (1301/2365)        
2023-09-06T11:53:35.1145725Z remote: Counting objects:  56% (1325/2365)        
2023-09-06T11:53:35.1146310Z remote: Counting objects:  57% (1349/2365)        
2023-09-06T11:53:35.1146827Z remote: Counting objects:  58% (1372/2365)        
2023-09-06T11:53:35.1147146Z remote: Counting objects:  59% (1396/2365)        
2023-09-06T11:53:35.1147536Z remote: Counting objects:  60% (1419/2365)        
2023-09-06T11:53:35.1148117Z remote: Counting objects:  61% (1443/2365)        
2023-09-06T11:53:35.1148712Z remote: Counting objects:  62% (1467/2365)        
2023-09-06T11:53:35.1150144Z remote: Counting objects:  63% (1490/2365)        
2023-09-06T11:53:35.1150759Z remote: Counting objects:  64% (1514/2365)        
2023-09-06T11:53:35.1151294Z remote: Counting objects:  65% (1538/2365)        
2023-09-06T11:53:35.1151590Z remote: Counting objects:  66% (1561/2365)        
2023-09-06T11:53:35.1151939Z remote: Counting objects:  67% (1585/2365)        
2023-09-06T11:53:35.1152538Z remote: Counting objects:  68% (1609/2365)        
2023-09-06T11:53:35.1153196Z remote: Counting objects:  69% (1632/2365)        
2023-09-06T11:53:35.1153777Z remote: Counting objects:  70% (1656/2365)        
2023-09-06T11:53:35.1154386Z remote: Counting objects:  71% (1680/2365)        
2023-09-06T11:53:35.1154973Z remote: Counting objects:  72% (1703/2365)        
2023-09-06T11:53:35.1155615Z remote: Counting objects:  73% (1727/2365)        
2023-09-06T11:53:35.1156204Z remote: Counting objects:  74% (1751/2365)        
2023-09-06T11:53:35.1156810Z remote: Counting objects:  75% (1774/2365)        
2023-09-06T11:53:35.1157429Z remote: Counting objects:  76% (1798/2365)        
2023-09-06T11:53:35.1158042Z remote: Counting objects:  77% (1822/2365)        
2023-09-06T11:53:35.1158557Z remote: Counting objects:  78% (1845/2365)        
2023-09-06T11:53:35.1159028Z remote: Counting objects:  79% (1869/2365)        
2023-09-06T11:53:35.1159660Z remote: Counting objects:  80% (1892/2365)        
2023-09-06T11:53:35.1160242Z remote: Counting objects:  81% (1916/2365)        
2023-09-06T11:53:35.1160904Z remote: Counting objects:  82% (1940/2365)        
2023-09-06T11:53:35.1161519Z remote: Counting objects:  83% (1963/2365)        
2023-09-06T11:53:35.1162197Z remote: Counting objects:  84% (1987/2365)        
2023-09-06T11:53:35.1162507Z remote: Counting objects:  85% (2011/2365)        
2023-09-06T11:53:35.1162825Z remote: Counting objects:  86% (2034/2365)        
2023-09-06T11:53:35.1163170Z remote: Counting objects:  87% (2058/2365)        
2023-09-06T11:53:35.1163614Z remote: Counting objects:  88% (2082/2365)        
2023-09-06T11:53:35.1163932Z remote: Counting objects:  89% (2105/2365)        
2023-09-06T11:53:35.1164241Z remote: Counting objects:  90% (2129/2365)        
2023-09-06T11:53:35.1164533Z remote: Counting objects:  91% (2153/2365)        
2023-09-06T11:53:35.1164840Z remote: Counting objects:  92% (2176/2365)        
2023-09-06T11:53:35.1165149Z remote: Counting objects:  93% (2200/2365)        
2023-09-06T11:53:35.1165494Z remote: Counting objects:  94% (2224/2365)        
2023-09-06T11:53:35.1165802Z remote: Counting objects:  95% (2247/2365)        
2023-09-06T11:53:35.1166095Z remote: Counting objects:  96% (2271/2365)        
2023-09-06T11:53:35.1166412Z remote: Counting objects:  97% (2295/2365)        
2023-09-06T11:53:35.1166720Z remote: Counting objects:  98% (2318/2365)        
2023-09-06T11:53:35.1167032Z remote: Counting objects:  99% (2342/2365)        
2023-09-06T11:53:35.1167329Z remote: Counting objects: 100% (2365/2365)        
2023-09-06T11:53:35.1167705Z remote: Counting objects: 100% (2365/2365), done.        
2023-09-06T11:53:35.2076802Z remote: Compressing objects:   0% (1/1248)        
2023-09-06T11:53:35.2728912Z remote: Compressing objects:   1% (13/1248)        
2023-09-06T11:53:35.3906221Z remote: Compressing objects:   2% (25/1248)        
2023-09-06T11:53:35.5029988Z remote: Compressing objects:   3% (38/1248)        
2023-09-06T11:53:35.6077392Z remote: Compressing objects:   4% (50/1248)        
2023-09-06T11:53:35.7038481Z remote: Compressing objects:   5% (63/1248)        
2023-09-06T11:53:35.7672062Z remote: Compressing objects:   6% (75/1248)        
2023-09-06T11:53:35.8122301Z remote: Compressing objects:   7% (88/1248)        
2023-09-06T11:53:35.8468729Z remote: Compressing objects:   8% (100/1248)        
2023-09-06T11:53:35.8884506Z remote: Compressing objects:   9% (113/1248)        
2023-09-06T11:53:35.9135596Z remote: Compressing objects:  10% (125/1248)        
2023-09-06T11:53:35.9419623Z remote: Compressing objects:  11% (138/1248)        
2023-09-06T11:53:35.9558379Z remote: Compressing objects:  12% (150/1248)        
2023-09-06T11:53:35.9726323Z remote: Compressing objects:  13% (163/1248)        
2023-09-06T11:53:35.9854954Z remote: Compressing objects:  14% (175/1248)        
2023-09-06T11:53:35.9980020Z remote: Compressing objects:  15% (188/1248)        
2023-09-06T11:53:36.0080539Z remote: Compressing objects:  16% (200/1248)        
2023-09-06T11:53:36.0142369Z remote: Compressing objects:  17% (213/1248)        
2023-09-06T11:53:36.0203113Z remote: Compressing objects:  18% (225/1248)        
2023-09-06T11:53:36.0242022Z remote: Compressing objects:  19% (238/1248)        
2023-09-06T11:53:36.0261215Z remote: Compressing objects:  20% (250/1248)        
2023-09-06T11:53:36.0277854Z remote: Compressing objects:  21% (263/1248)        
2023-09-06T11:53:36.0311134Z remote: Compressing objects:  22% (275/1248)        
2023-09-06T11:53:36.0321908Z remote: Compressing objects:  23% (288/1248)        
2023-09-06T11:53:36.0339936Z remote: Compressing objects:  24% (300/1248)        
2023-09-06T11:53:36.0354980Z remote: Compressing objects:  25% (312/1248)        
2023-09-06T11:53:36.0393703Z remote: Compressing objects:  26% (325/1248)        
2023-09-06T11:53:36.0416644Z remote: Compressing objects:  27% (337/1248)        
2023-09-06T11:53:36.0432389Z remote: Compressing objects:  28% (350/1248)        
2023-09-06T11:53:36.0451354Z remote: Compressing objects:  29% (362/1248)        
2023-09-06T11:53:36.0464834Z remote: Compressing objects:  30% (375/1248)        
2023-09-06T11:53:36.0483958Z remote: Compressing objects:  31% (387/1248)        
2023-09-06T11:53:36.0508039Z remote: Compressing objects:  32% (400/1248)        
2023-09-06T11:53:36.0527720Z remote: Compressing objects:  33% (412/1248)        
2023-09-06T11:53:36.0538746Z remote: Compressing objects:  34% (425/1248)        
2023-09-06T11:53:36.0553172Z remote: Compressing objects:  35% (437/1248)        
2023-09-06T11:53:36.0571024Z remote: Compressing objects:  36% (450/1248)        
2023-09-06T11:53:36.0600794Z remote: Compressing objects:  37% (462/1248)        
2023-09-06T11:53:36.0616192Z remote: Compressing objects:  38% (475/1248)        
2023-09-06T11:53:36.0630147Z remote: Compressing objects:  39% (487/1248)        
2023-09-06T11:53:36.0639530Z remote: Compressing objects:  40% (500/1248)        
2023-09-06T11:53:36.0648518Z remote: Compressing objects:  41% (512/1248)        
2023-09-06T11:53:36.0660458Z remote: Compressing objects:  42% (525/1248)        
2023-09-06T11:53:36.0673679Z remote: Compressing objects:  43% (537/1248)        
2023-09-06T11:53:36.0683353Z remote: Compressing objects:  44% (550/1248)        
2023-09-06T11:53:36.0695598Z remote: Compressing objects:  45% (562/1248)        
2023-09-06T11:53:36.0711105Z remote: Compressing objects:  46% (575/1248)        
2023-09-06T11:53:36.0728709Z remote: Compressing objects:  47% (587/1248)        
2023-09-06T11:53:36.0737531Z remote: Compressing objects:  48% (600/1248)        
2023-09-06T11:53:36.0746817Z remote: Compressing objects:  49% (612/1248)        
2023-09-06T11:53:36.0755336Z remote: Compressing objects:  50% (624/1248)        
2023-09-06T11:53:36.0763492Z remote: Compressing objects:  51% (637/1248)        
2023-09-06T11:53:36.0770656Z remote: Compressing objects:  52% (649/1248)        
2023-09-06T11:53:36.0775029Z remote: Compressing objects:  53% (662/1248)        
2023-09-06T11:53:36.0781967Z remote: Compressing objects:  54% (674/1248)        
2023-09-06T11:53:36.0790626Z remote: Compressing objects:  55% (687/1248)        
2023-09-06T11:53:36.0796305Z remote: Compressing objects:  56% (699/1248)        
2023-09-06T11:53:36.0801242Z remote: Compressing objects:  57% (712/1248)        
2023-09-06T11:53:36.0805869Z remote: Compressing objects:  58% (724/1248)        
2023-09-06T11:53:36.0809275Z remote: Compressing objects:  59% (737/1248)        
2023-09-06T11:53:36.0812911Z remote: Compressing objects:  60% (749/1248)        
2023-09-06T11:53:36.0814663Z remote: Compressing objects:  61% (762/1248)        
2023-09-06T11:53:36.0817444Z remote: Compressing objects:  62% (774/1248)        
2023-09-06T11:53:36.0818868Z remote: Compressing objects:  63% (787/1248)        
2023-09-06T11:53:36.0820912Z remote: Compressing objects:  64% (799/1248)        
2023-09-06T11:53:36.0821983Z remote: Compressing objects:  65% (812/1248)        
2023-09-06T11:53:36.0822888Z remote: Compressing objects:  66% (824/1248)        
2023-09-06T11:53:36.0824442Z remote: Compressing objects:  67% (837/1248)        
2023-09-06T11:53:36.0826081Z remote: Compressing objects:  68% (849/1248)        
2023-09-06T11:53:36.0826683Z remote: Compressing objects:  69% (862/1248)        
2023-09-06T11:53:36.0827349Z remote: Compressing objects:  70% (874/1248)        
2023-09-06T11:53:36.0842014Z remote: Compressing objects:  71% (887/1248)        
2023-09-06T11:53:36.0852981Z remote: Compressing objects:  72% (899/1248)        
2023-09-06T11:53:36.0861778Z remote: Compressing objects:  73% (912/1248)        
2023-09-06T11:53:36.0866562Z remote: Compressing objects:  74% (924/1248)        
2023-09-06T11:53:36.0870951Z remote: Compressing objects:  75% (936/1248)        
2023-09-06T11:53:36.0875078Z remote: Compressing objects:  76% (949/1248)        
2023-09-06T11:53:36.0879466Z remote: Compressing objects:  77% (961/1248)        
2023-09-06T11:53:36.0884596Z remote: Compressing objects:  78% (974/1248)        
2023-09-06T11:53:36.0888442Z remote: Compressing objects:  79% (986/1248)        
2023-09-06T11:53:36.0891961Z remote: Compressing objects:  80% (999/1248)        
2023-09-06T11:53:36.0896590Z remote: Compressing objects:  81% (1011/1248)        
2023-09-06T11:53:36.0899577Z remote: Compressing objects:  82% (1024/1248)        
2023-09-06T11:53:36.0901730Z remote: Compressing objects:  83% (1036/1248)        
2023-09-06T11:53:36.0906100Z remote: Compressing objects:  84% (1049/1248)        
2023-09-06T11:53:36.0909870Z remote: Compressing objects:  85% (1061/1248)        
2023-09-06T11:53:36.0914906Z remote: Compressing objects:  86% (1074/1248)        
2023-09-06T11:53:36.0919145Z remote: Compressing objects:  87% (1086/1248)        
2023-09-06T11:53:36.0924582Z remote: Compressing objects:  88% (1099/1248)        
2023-09-06T11:53:36.0927849Z remote: Compressing objects:  89% (1111/1248)        
2023-09-06T11:53:36.0930431Z remote: Compressing objects:  90% (1124/1248)        
2023-09-06T11:53:36.0934378Z remote: Compressing objects:  91% (1136/1248)        
2023-09-06T11:53:36.0935360Z remote: Compressing objects:  92% (1149/1248)        
2023-09-06T11:53:36.0935767Z remote: Compressing objects:  93% (1161/1248)        
2023-09-06T11:53:36.0942458Z remote: Compressing objects:  94% (1174/1248)        
2023-09-06T11:53:36.0942876Z remote: Compressing objects:  95% (1186/1248)        
2023-09-06T11:53:36.0943216Z remote: Compressing objects:  96% (1199/1248)        
2023-09-06T11:53:36.0952898Z remote: Compressing objects:  97% (1211/1248)        
2023-09-06T11:53:36.0953309Z remote: Compressing objects:  98% (1224/1248)        
2023-09-06T11:53:36.0953661Z remote: Compressing objects:  99% (1236/1248)        
2023-09-06T11:53:36.0953992Z remote: Compressing objects: 100% (1248/1248)        
2023-09-06T11:53:36.0954339Z remote: Compressing objects: 100% (1248/1248), done.        
2023-09-06T11:53:58.4450175Z remote: Total 1161041 (delta 1624), reused 1715 (delta 1113), pack-reused 1158676        
2023-09-06T11:54:37.5139642Z [command]/usr/bin/git rev-parse --verify --quiet 3fe8417643c8d6c2b3d95552cd90321d141b5d54^{object}
2023-09-06T11:54:37.5180764Z 3fe8417643c8d6c2b3d95552cd90321d141b5d54
2023-09-06T11:54:37.5188772Z ##[endgroup]
2023-09-06T11:54:37.5190267Z ##[group]Determining the checkout info
2023-09-06T11:54:37.5191684Z ##[endgroup]
2023-09-06T11:54:37.5192583Z ##[group]Checking out the ref
2023-09-06T11:54:37.5196133Z [command]/usr/bin/git checkout --quiet --force 3fe8417643c8d6c2b3d95552cd90321d141b5d54
2023-09-06T11:54:39.3269994Z ##[endgroup]
2023-09-06T11:54:39.3270600Z ##[group]Setting up auth for fetching submodules
2023-09-06T11:54:39.3275348Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2023-09-06T11:54:39.3328248Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2023-09-06T11:54:39.3363914Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2023-09-06T11:54:39.3399404Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2023-09-06T11:54:39.3431385Z ##[endgroup]
2023-09-06T11:54:39.3431951Z ##[group]Fetching submodules
2023-09-06T11:54:39.3437361Z [command]/usr/bin/git submodule sync --recursive
2023-09-06T11:54:39.3711161Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2023-09-06T11:54:39.3971358Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2023-09-06T11:54:39.3980268Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2023-09-06T11:54:39.3990074Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2023-09-06T11:54:39.3997076Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2023-09-06T11:54:39.4003387Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK'
2023-09-06T11:54:39.4010572Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2023-09-06T11:54:39.4016951Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2023-09-06T11:54:39.4024981Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2023-09-06T11:54:39.4031949Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2023-09-06T11:54:39.4037883Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub'
2023-09-06T11:54:39.4045666Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2023-09-06T11:54:39.4051639Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2023-09-06T11:54:39.4059046Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2023-09-06T11:54:39.4065103Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2023-09-06T11:54:39.4071681Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2023-09-06T11:54:39.4077795Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2023-09-06T11:54:39.4084359Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2023-09-06T11:54:39.4090208Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2023-09-06T11:54:39.4096337Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2023-09-06T11:54:39.4103947Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2023-09-06T11:54:39.4110854Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2023-09-06T11:54:39.4117460Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake'
2023-09-06T11:54:39.4124549Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2023-09-06T11:54:39.4131177Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2023-09-06T11:54:39.4181767Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc'
2023-09-06T11:54:39.4183033Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2023-09-06T11:54:39.4183783Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse'
2023-09-06T11:54:39.4184514Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2023-09-06T11:54:39.4185183Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2023-09-06T11:54:39.4185916Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt'
2023-09-06T11:54:39.4186644Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2023-09-06T11:54:39.4187388Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2023-09-06T11:54:39.4193615Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2023-09-06T11:54:39.4200832Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2023-09-06T11:54:39.4208259Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2023-09-06T11:54:39.4215662Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2023-09-06T11:54:39.4223635Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2023-09-06T11:54:39.4233197Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb'
2023-09-06T11:54:39.4243602Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2023-09-06T11:54:39.4248016Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd'
2023-09-06T11:54:39.4322212Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2023-09-06T11:54:39.9497343Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2023-09-06T11:54:40.3981448Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2023-09-06T11:54:40.7762665Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2023-09-06T11:54:41.2998312Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'...
2023-09-06T11:54:41.8244591Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2023-09-06T11:54:45.1754556Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2023-09-06T11:54:58.5861265Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2023-09-06T11:54:59.3382271Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2023-09-06T11:55:00.1860040Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/cub'...
2023-09-06T11:55:02.0653491Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2023-09-06T11:55:03.5366634Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2023-09-06T11:55:05.9260900Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2023-09-06T11:55:11.3481688Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2023-09-06T11:55:12.5502399Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2023-09-06T11:55:14.7396573Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2023-09-06T11:55:16.5030428Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2023-09-06T11:55:16.8985849Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2023-09-06T11:55:17.6091757Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2023-09-06T11:55:18.1892619Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2023-09-06T11:55:19.9768865Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2023-09-06T11:55:20.6709279Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'...
2023-09-06T11:55:21.0638072Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2023-09-06T11:55:21.5880999Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2023-09-06T11:55:23.3991003Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'...
2023-09-06T11:55:24.3382670Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2023-09-06T11:55:25.0728339Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'...
2023-09-06T11:55:25.6519924Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2023-09-06T11:55:33.0351950Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2023-09-06T11:55:38.0315744Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'...
2023-09-06T11:55:38.8520930Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2023-09-06T11:55:39.3962442Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2023-09-06T11:55:47.9259990Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2023-09-06T11:55:48.3385602Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2023-09-06T11:55:48.8115206Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2023-09-06T11:55:50.0600812Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2023-09-06T11:55:50.6316662Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2023-09-06T11:55:51.4671400Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tbb'...
2023-09-06T11:55:54.3223939Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2023-09-06T11:55:55.0355744Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/zstd'...
2023-09-06T11:55:58.3126570Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2023-09-06T11:55:58.3511113Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2023-09-06T11:55:58.3856191Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2023-09-06T11:55:58.4367134Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2023-09-06T11:55:58.4863586Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c'
2023-09-06T11:55:58.5528769Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2023-09-06T11:55:59.3500263Z Submodule path 'third_party/XNNPACK': checked out '51a987591a6fc9f0fc0707077f53d763ac132cbf'
2023-09-06T11:55:59.4016008Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2023-09-06T11:55:59.5413682Z Submodule path 'third_party/cpuinfo': checked out '6481e8bef08f606ddd627e4d3be89f64d62e1b8a'
2023-09-06T11:55:59.6054114Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4'
2023-09-06T11:55:59.9467193Z Submodule path 'third_party/cudnn_frontend': checked out '12f35fa2be5994c1106367cac2fba21457b064f4'
2023-09-06T11:56:00.4906568Z Submodule path 'third_party/cutlass': checked out '6f47420213f757831fae65c686aa471749fa8d60'
2023-09-06T11:56:00.7961682Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2023-09-06T11:56:00.8947551Z Submodule path 'third_party/fbgemm': checked out 'd0ee798b1f198cc51b6ddae20cf6063f6380ba3f'
2023-09-06T11:56:00.8998507Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2023-09-06T11:56:00.9004655Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2023-09-06T11:56:00.9010817Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass'
2023-09-06T11:56:00.9016307Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2023-09-06T11:56:00.9022770Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2023-09-06T11:56:00.9069731Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2023-09-06T11:56:01.9074376Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2023-09-06T11:56:02.7827303Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'...
2023-09-06T11:56:05.1765709Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2023-09-06T11:56:06.9911253Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2023-09-06T11:56:07.6806125Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2023-09-06T11:56:07.8181663Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2023-09-06T11:56:08.2842494Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6'
2023-09-06T11:56:08.3780973Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2023-09-06T11:56:08.4111906Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1'
2023-09-06T11:56:08.5592492Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b'
2023-09-06T11:56:08.6253945Z Submodule path 'third_party/fmt': checked out 'f5e54359df4c26b6230fc61d38aa294581393084'
2023-09-06T11:56:08.6612615Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2023-09-06T11:56:08.7330662Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2023-09-06T11:56:08.7841093Z Submodule path 'third_party/gloo': checked out 'cf1e1abc95d0b961222ee82b6935f76250fbcf16'
2023-09-06T11:56:08.8619019Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2023-09-06T11:56:08.8988152Z Submodule path 'third_party/ideep': checked out '6f4d653802bd43bc4eda515460df9f90353dbebe'
2023-09-06T11:56:08.9037588Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2023-09-06T11:56:08.9080113Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2023-09-06T11:56:20.9382228Z Submodule path 'third_party/ideep/mkl-dnn': checked out '64f6bcbcbab628e96f33a62c3e975f8535a7bde4'
2023-09-06T11:56:20.9763140Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724'
2023-09-06T11:56:21.0179983Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2023-09-06T11:56:21.1512878Z Submodule path 'third_party/kineto': checked out '49e854d805d916b2031e337763928d2f8d2e1fbf'
2023-09-06T11:56:21.1566386Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog'
2023-09-06T11:56:21.1572991Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2023-09-06T11:56:21.1579184Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2023-09-06T11:56:21.1626093Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'...
2023-09-06T11:56:21.9590873Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2023-09-06T11:56:23.7344016Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2023-09-06T11:56:25.6701635Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e'
2023-09-06T11:56:25.6756659Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2023-09-06T11:56:25.6763039Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2023-09-06T11:56:25.6768449Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2023-09-06T11:56:25.6774047Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2023-09-06T11:56:25.6780132Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2023-09-06T11:56:25.6785894Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2023-09-06T11:56:25.6793302Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2023-09-06T11:56:25.6800099Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2023-09-06T11:56:25.6850063Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'...
2023-09-06T11:56:26.9459362Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'...
2023-09-06T11:56:27.5950130Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'...
2023-09-06T11:56:29.3972847Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'...
2023-09-06T11:56:30.0385818Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'...
2023-09-06T11:56:30.8246121Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'...
2023-09-06T11:56:32.6273765Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'...
2023-09-06T11:56:40.0529200Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'...
2023-09-06T11:56:40.8059422Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9'
2023-09-06T11:56:40.8484360Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400'
2023-09-06T11:56:40.9122469Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2023-09-06T11:56:40.9483251Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067'
2023-09-06T11:56:40.9533916Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2023-09-06T11:56:40.9579444Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'...
2023-09-06T11:56:41.6117468Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4'
2023-09-06T11:56:41.6528619Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446'
2023-09-06T11:56:41.7213078Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850'
2023-09-06T11:56:41.8602561Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5'
2023-09-06T11:56:41.8991249Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150'
2023-09-06T11:56:41.9568217Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50'
2023-09-06T11:56:42.0437451Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2023-09-06T11:56:42.1088948Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78'
2023-09-06T11:56:42.1585016Z Submodule path 'third_party/nccl/nccl': checked out '559b70f86c190a0d8f67f0d7a0f2c9810dd1e8c7'
2023-09-06T11:56:42.1982707Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a'
2023-09-06T11:56:42.3404977Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2023-09-06T11:56:42.7030694Z Submodule path 'third_party/onnx': checked out 'c11b6a715daeb48b056e4c9bed3dfc0dcb243d12'
2023-09-06T11:56:42.7102827Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2023-09-06T11:56:42.7108814Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2023-09-06T11:56:42.7175727Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2023-09-06T11:56:43.4745812Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2023-09-06T11:56:44.7684808Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2023-09-06T11:56:44.8291230Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '0bd8896a4010f2d91b2340570c24fa08606ec406'
2023-09-06T11:56:44.8744650Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f'
2023-09-06T11:56:44.8793205Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx'
2023-09-06T11:56:44.8838592Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'...
2023-09-06T11:56:49.9383814Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8'
2023-09-06T11:56:49.9441552Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-09-06T11:56:49.9447859Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-09-06T11:56:49.9501283Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'...
2023-09-06T11:56:50.7235329Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'...
2023-09-06T11:56:52.0634048Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2023-09-06T11:56:52.1631324Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c'
2023-09-06T11:56:52.1682514Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-09-06T11:56:52.1728050Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'...
2023-09-06T11:56:52.6406701Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2023-09-06T11:56:52.6754192Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a'
2023-09-06T11:56:52.9962559Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2023-09-06T11:56:53.0022965Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2023-09-06T11:56:53.0030486Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2023-09-06T11:56:53.0082047Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2023-09-06T11:56:53.7404798Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2023-09-06T11:56:55.4325728Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2023-09-06T11:56:55.5345726Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2023-09-06T11:56:55.5690594Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2023-09-06T11:56:55.6083721Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413'
2023-09-06T11:56:55.6693490Z Submodule path 'third_party/pybind11': checked out '8a099e44b3d5f85b20f05828d919d2332a8de841'
2023-09-06T11:56:55.7257803Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2023-09-06T11:56:55.8003586Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff'
2023-09-06T11:56:55.9468390Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9'
2023-09-06T11:56:55.9999533Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2023-09-06T11:56:56.0050304Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2023-09-06T11:56:56.0057168Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2023-09-06T11:56:56.0064842Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2023-09-06T11:56:56.0072827Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2023-09-06T11:56:56.0119272Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2023-09-06T11:56:57.8648682Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2023-09-06T11:56:58.4082592Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2023-09-06T11:57:00.1758948Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2023-09-06T11:57:01.5626680Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2023-09-06T11:57:01.5987430Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2023-09-06T11:57:01.6914730Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2023-09-06T11:57:01.7431659Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2023-09-06T11:57:01.7484734Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-09-06T11:57:01.7529891Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2023-09-06T11:57:02.2355742Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2023-09-06T11:57:02.4193422Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8'
2023-09-06T11:57:02.4292895Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2023-09-06T11:57:02.4570539Z Entering 'android/libs/fbjni'
2023-09-06T11:57:02.4610399Z Entering 'third_party/FP16'
2023-09-06T11:57:02.4650621Z Entering 'third_party/FXdiv'
2023-09-06T11:57:02.4691044Z Entering 'third_party/NNPACK'
2023-09-06T11:57:02.4731143Z Entering 'third_party/QNNPACK'
2023-09-06T11:57:02.4772047Z Entering 'third_party/VulkanMemoryAllocator'
2023-09-06T11:57:02.4812010Z Entering 'third_party/XNNPACK'
2023-09-06T11:57:02.4867126Z Entering 'third_party/benchmark'
2023-09-06T11:57:02.4906723Z Entering 'third_party/cpuinfo'
2023-09-06T11:57:02.4946695Z Entering 'third_party/cub'
2023-09-06T11:57:02.4985470Z Entering 'third_party/cudnn_frontend'
2023-09-06T11:57:02.5031649Z Entering 'third_party/cutlass'
2023-09-06T11:57:02.5080876Z Entering 'third_party/eigen'
2023-09-06T11:57:02.5123755Z Entering 'third_party/fbgemm'
2023-09-06T11:57:02.5163856Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-09-06T11:57:02.5203194Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-09-06T11:57:02.5242203Z Entering 'third_party/fbgemm/third_party/cutlass'
2023-09-06T11:57:02.5289657Z Entering 'third_party/fbgemm/third_party/googletest'
2023-09-06T11:57:02.5329876Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-09-06T11:57:02.5368991Z Entering 'third_party/flatbuffers'
2023-09-06T11:57:02.5411848Z Entering 'third_party/fmt'
2023-09-06T11:57:02.5451256Z Entering 'third_party/foxi'
2023-09-06T11:57:02.5491254Z Entering 'third_party/gemmlowp/gemmlowp'
2023-09-06T11:57:02.5531578Z Entering 'third_party/gloo'
2023-09-06T11:57:02.5572006Z Entering 'third_party/googletest'
2023-09-06T11:57:02.5611618Z Entering 'third_party/ideep'
2023-09-06T11:57:02.5649733Z Entering 'third_party/ideep/mkl-dnn'
2023-09-06T11:57:02.5697663Z Entering 'third_party/ios-cmake'
2023-09-06T11:57:02.5736833Z Entering 'third_party/ittapi'
2023-09-06T11:57:02.5775806Z Entering 'third_party/kineto'
2023-09-06T11:57:02.5815503Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2023-09-06T11:57:02.5855689Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2023-09-06T11:57:02.5897506Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2023-09-06T11:57:02.5937155Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2023-09-06T11:57:02.5977070Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2023-09-06T11:57:02.6015026Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2023-09-06T11:57:02.6056540Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2023-09-06T11:57:02.6097335Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2023-09-06T11:57:02.6137406Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2023-09-06T11:57:02.6178522Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2023-09-06T11:57:02.6220118Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-09-06T11:57:02.6259742Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-09-06T11:57:02.6301368Z Entering 'third_party/mimalloc'
2023-09-06T11:57:02.6341284Z Entering 'third_party/nccl/nccl'
2023-09-06T11:57:02.6380855Z Entering 'third_party/neon2sse'
2023-09-06T11:57:02.6420363Z Entering 'third_party/nlohmann'
2023-09-06T11:57:02.6461997Z Entering 'third_party/onnx'
2023-09-06T11:57:02.6519061Z Entering 'third_party/onnx/third_party/benchmark'
2023-09-06T11:57:02.6559337Z Entering 'third_party/onnx/third_party/pybind11'
2023-09-06T11:57:02.6601019Z Entering 'third_party/onnx-tensorrt'
2023-09-06T11:57:02.6640419Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-09-06T11:57:02.6685028Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-09-06T11:57:02.6725195Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-09-06T11:57:02.6764122Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-09-06T11:57:02.6808491Z Entering 'third_party/pocketfft'
2023-09-06T11:57:02.6848562Z Entering 'third_party/protobuf'
2023-09-06T11:57:02.6891395Z Entering 'third_party/protobuf/third_party/benchmark'
2023-09-06T11:57:02.6932160Z Entering 'third_party/protobuf/third_party/googletest'
2023-09-06T11:57:02.6974147Z Entering 'third_party/psimd'
2023-09-06T11:57:02.7012861Z Entering 'third_party/pthreadpool'
2023-09-06T11:57:02.7053560Z Entering 'third_party/pybind11'
2023-09-06T11:57:02.7093763Z Entering 'third_party/python-peachpy'
2023-09-06T11:57:02.7133984Z Entering 'third_party/sleef'
2023-09-06T11:57:02.7174230Z Entering 'third_party/tbb'
2023-09-06T11:57:02.7216547Z Entering 'third_party/tensorpipe'
2023-09-06T11:57:02.7256462Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-09-06T11:57:02.7296461Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-09-06T11:57:02.7334460Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-09-06T11:57:02.7374296Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-09-06T11:57:02.7412859Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-09-06T11:57:02.7456057Z Entering 'third_party/zstd'
2023-09-06T11:57:02.7516850Z ##[endgroup]
2023-09-06T11:57:02.7517414Z ##[group]Persisting credentials for submodules
2023-09-06T11:57:02.7534928Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :"
2023-09-06T11:57:02.7807505Z Entering 'android/libs/fbjni'
2023-09-06T11:57:02.7852611Z Entering 'third_party/FP16'
2023-09-06T11:57:02.7898088Z Entering 'third_party/FXdiv'
2023-09-06T11:57:02.7942307Z Entering 'third_party/NNPACK'
2023-09-06T11:57:02.7987369Z Entering 'third_party/QNNPACK'
2023-09-06T11:57:02.8033904Z Entering 'third_party/VulkanMemoryAllocator'
2023-09-06T11:57:02.8081048Z Entering 'third_party/XNNPACK'
2023-09-06T11:57:02.8141586Z Entering 'third_party/benchmark'
2023-09-06T11:57:02.8186256Z Entering 'third_party/cpuinfo'
2023-09-06T11:57:02.8232655Z Entering 'third_party/cub'
2023-09-06T11:57:02.8278347Z Entering 'third_party/cudnn_frontend'
2023-09-06T11:57:02.8330323Z Entering 'third_party/cutlass'
2023-09-06T11:57:02.8383736Z Entering 'third_party/eigen'
2023-09-06T11:57:02.8431511Z Entering 'third_party/fbgemm'
2023-09-06T11:57:02.8476610Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-09-06T11:57:02.8520163Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-09-06T11:57:02.8563296Z Entering 'third_party/fbgemm/third_party/cutlass'
2023-09-06T11:57:02.8613572Z Entering 'third_party/fbgemm/third_party/googletest'
2023-09-06T11:57:02.8658146Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-09-06T11:57:02.8702887Z Entering 'third_party/flatbuffers'
2023-09-06T11:57:02.8751476Z Entering 'third_party/fmt'
2023-09-06T11:57:02.8796852Z Entering 'third_party/foxi'
2023-09-06T11:57:02.8841806Z Entering 'third_party/gemmlowp/gemmlowp'
2023-09-06T11:57:02.8886118Z Entering 'third_party/gloo'
2023-09-06T11:57:02.8931522Z Entering 'third_party/googletest'
2023-09-06T11:57:02.8976226Z Entering 'third_party/ideep'
2023-09-06T11:57:02.9020235Z Entering 'third_party/ideep/mkl-dnn'
2023-09-06T11:57:02.9070614Z Entering 'third_party/ios-cmake'
2023-09-06T11:57:02.9115235Z Entering 'third_party/ittapi'
2023-09-06T11:57:02.9159741Z Entering 'third_party/kineto'
2023-09-06T11:57:02.9205616Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2023-09-06T11:57:02.9250333Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2023-09-06T11:57:02.9298148Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2023-09-06T11:57:02.9342983Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2023-09-06T11:57:02.9388268Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2023-09-06T11:57:02.9432414Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2023-09-06T11:57:02.9479037Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2023-09-06T11:57:02.9523990Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2023-09-06T11:57:02.9568315Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2023-09-06T11:57:02.9614269Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2023-09-06T11:57:02.9660360Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-09-06T11:57:02.9704775Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-09-06T11:57:02.9750249Z Entering 'third_party/mimalloc'
2023-09-06T11:57:02.9795539Z Entering 'third_party/nccl/nccl'
2023-09-06T11:57:02.9841465Z Entering 'third_party/neon2sse'
2023-09-06T11:57:02.9886167Z Entering 'third_party/nlohmann'
2023-09-06T11:57:02.9932910Z Entering 'third_party/onnx'
2023-09-06T11:57:02.9996266Z Entering 'third_party/onnx/third_party/benchmark'
2023-09-06T11:57:03.0041004Z Entering 'third_party/onnx/third_party/pybind11'
2023-09-06T11:57:03.0088042Z Entering 'third_party/onnx-tensorrt'
2023-09-06T11:57:03.0132315Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-09-06T11:57:03.0181840Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-09-06T11:57:03.0227230Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-09-06T11:57:03.0270495Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-09-06T11:57:03.0320240Z Entering 'third_party/pocketfft'
2023-09-06T11:57:03.0364721Z Entering 'third_party/protobuf'
2023-09-06T11:57:03.0412667Z Entering 'third_party/protobuf/third_party/benchmark'
2023-09-06T11:57:03.0457329Z Entering 'third_party/protobuf/third_party/googletest'
2023-09-06T11:57:03.0503638Z Entering 'third_party/psimd'
2023-09-06T11:57:03.0547558Z Entering 'third_party/pthreadpool'
2023-09-06T11:57:03.0593123Z Entering 'third_party/pybind11'
2023-09-06T11:57:03.0638603Z Entering 'third_party/python-peachpy'
2023-09-06T11:57:03.0683621Z Entering 'third_party/sleef'
2023-09-06T11:57:03.0729460Z Entering 'third_party/tbb'
2023-09-06T11:57:03.0777098Z Entering 'third_party/tensorpipe'
2023-09-06T11:57:03.0821663Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-09-06T11:57:03.0865787Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-09-06T11:57:03.0909435Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-09-06T11:57:03.0953994Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-09-06T11:57:03.0997146Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-09-06T11:57:03.1044913Z Entering 'third_party/zstd'
2023-09-06T11:57:03.1102752Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url"
2023-09-06T11:57:03.1373552Z Entering 'android/libs/fbjni'
2023-09-06T11:57:03.1414769Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2023-09-06T11:57:03.1434222Z Entering 'third_party/FP16'
2023-09-06T11:57:03.1475993Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2023-09-06T11:57:03.1494888Z Entering 'third_party/FXdiv'
2023-09-06T11:57:03.1535661Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2023-09-06T11:57:03.1555669Z Entering 'third_party/NNPACK'
2023-09-06T11:57:03.1598146Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2023-09-06T11:57:03.1617093Z Entering 'third_party/QNNPACK'
2023-09-06T11:57:03.1658380Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config	remote.origin.url
2023-09-06T11:57:03.1678178Z Entering 'third_party/VulkanMemoryAllocator'
2023-09-06T11:57:03.1719276Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2023-09-06T11:57:03.1739128Z Entering 'third_party/XNNPACK'
2023-09-06T11:57:03.1779487Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2023-09-06T11:57:03.1815357Z Entering 'third_party/benchmark'
2023-09-06T11:57:03.1856628Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2023-09-06T11:57:03.1875529Z Entering 'third_party/cpuinfo'
2023-09-06T11:57:03.1917526Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2023-09-06T11:57:03.1937400Z Entering 'third_party/cub'
2023-09-06T11:57:03.1977769Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config	remote.origin.url
2023-09-06T11:57:03.1997760Z Entering 'third_party/cudnn_frontend'
2023-09-06T11:57:03.2039385Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2023-09-06T11:57:03.2065201Z Entering 'third_party/cutlass'
2023-09-06T11:57:03.2106588Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2023-09-06T11:57:03.2135579Z Entering 'third_party/eigen'
2023-09-06T11:57:03.2176585Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2023-09-06T11:57:03.2198878Z Entering 'third_party/fbgemm'
2023-09-06T11:57:03.2240311Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2023-09-06T11:57:03.2259156Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-09-06T11:57:03.2299897Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2023-09-06T11:57:03.2318977Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-09-06T11:57:03.2358941Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2023-09-06T11:57:03.2378565Z Entering 'third_party/fbgemm/third_party/cutlass'
2023-09-06T11:57:03.2419307Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config	remote.origin.url
2023-09-06T11:57:03.2446092Z Entering 'third_party/fbgemm/third_party/googletest'
2023-09-06T11:57:03.2488579Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2023-09-06T11:57:03.2507316Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-09-06T11:57:03.2548499Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2023-09-06T11:57:03.2568734Z Entering 'third_party/flatbuffers'
2023-09-06T11:57:03.2611018Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2023-09-06T11:57:03.2634760Z Entering 'third_party/fmt'
2023-09-06T11:57:03.2676388Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2023-09-06T11:57:03.2695877Z Entering 'third_party/foxi'
2023-09-06T11:57:03.2736653Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2023-09-06T11:57:03.2755886Z Entering 'third_party/gemmlowp/gemmlowp'
2023-09-06T11:57:03.2796750Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2023-09-06T11:57:03.2816470Z Entering 'third_party/gloo'
2023-09-06T11:57:03.2858122Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2023-09-06T11:57:03.2879592Z Entering 'third_party/googletest'
2023-09-06T11:57:03.2920260Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2023-09-06T11:57:03.2940219Z Entering 'third_party/ideep'
2023-09-06T11:57:03.2982075Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2023-09-06T11:57:03.3001505Z Entering 'third_party/ideep/mkl-dnn'
2023-09-06T11:57:03.3042247Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2023-09-06T11:57:03.3068721Z Entering 'third_party/ios-cmake'
2023-09-06T11:57:03.3109069Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config	remote.origin.url
2023-09-06T11:57:03.3127449Z Entering 'third_party/ittapi'
2023-09-06T11:57:03.3169370Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2023-09-06T11:57:03.3187442Z Entering 'third_party/kineto'
2023-09-06T11:57:03.3230013Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2023-09-06T11:57:03.3248638Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2023-09-06T11:57:03.3290443Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config	remote.origin.url
2023-09-06T11:57:03.3309412Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2023-09-06T11:57:03.3350209Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config	remote.origin.url
2023-09-06T11:57:03.3370289Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2023-09-06T11:57:03.3411706Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config	remote.origin.url
2023-09-06T11:57:03.3430469Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2023-09-06T11:57:03.3471719Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config	remote.origin.url
2023-09-06T11:57:03.3490553Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2023-09-06T11:57:03.3531558Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config	remote.origin.url
2023-09-06T11:57:03.3548826Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2023-09-06T11:57:03.3590403Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config	remote.origin.url
2023-09-06T11:57:03.3611819Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2023-09-06T11:57:03.3652050Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config	remote.origin.url
2023-09-06T11:57:03.3670912Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2023-09-06T11:57:03.3711619Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config	remote.origin.url
2023-09-06T11:57:03.3730182Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2023-09-06T11:57:03.3771626Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config	remote.origin.url
2023-09-06T11:57:03.3791981Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2023-09-06T11:57:03.3833502Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config	remote.origin.url
2023-09-06T11:57:03.3853346Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-09-06T11:57:03.3894402Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2023-09-06T11:57:03.3914153Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-09-06T11:57:03.3955772Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2023-09-06T11:57:03.3976363Z Entering 'third_party/mimalloc'
2023-09-06T11:57:03.4017646Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config	remote.origin.url
2023-09-06T11:57:03.4037789Z Entering 'third_party/nccl/nccl'
2023-09-06T11:57:03.4080164Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2023-09-06T11:57:03.4098973Z Entering 'third_party/neon2sse'
2023-09-06T11:57:03.4139671Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config	remote.origin.url
2023-09-06T11:57:03.4159661Z Entering 'third_party/nlohmann'
2023-09-06T11:57:03.4200958Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2023-09-06T11:57:03.4222067Z Entering 'third_party/onnx'
2023-09-06T11:57:03.4262970Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2023-09-06T11:57:03.4299658Z Entering 'third_party/onnx/third_party/benchmark'
2023-09-06T11:57:03.4341507Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2023-09-06T11:57:03.4360540Z Entering 'third_party/onnx/third_party/pybind11'
2023-09-06T11:57:03.4402661Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2023-09-06T11:57:03.4424080Z Entering 'third_party/onnx-tensorrt'
2023-09-06T11:57:03.4465309Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config	remote.origin.url
2023-09-06T11:57:03.4483630Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-09-06T11:57:03.4525143Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config	remote.origin.url
2023-09-06T11:57:03.4548411Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-09-06T11:57:03.4589554Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2023-09-06T11:57:03.4608548Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-09-06T11:57:03.4649893Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2023-09-06T11:57:03.4667933Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-09-06T11:57:03.4709013Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2023-09-06T11:57:03.4732638Z Entering 'third_party/pocketfft'
2023-09-06T11:57:03.4774014Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2023-09-06T11:57:03.4793635Z Entering 'third_party/protobuf'
2023-09-06T11:57:03.4834830Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2023-09-06T11:57:03.4857846Z Entering 'third_party/protobuf/third_party/benchmark'
2023-09-06T11:57:03.4899281Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2023-09-06T11:57:03.4918998Z Entering 'third_party/protobuf/third_party/googletest'
2023-09-06T11:57:03.4959753Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2023-09-06T11:57:03.4980301Z Entering 'third_party/psimd'
2023-09-06T11:57:03.5021228Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2023-09-06T11:57:03.5040647Z Entering 'third_party/pthreadpool'
2023-09-06T11:57:03.5081667Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2023-09-06T11:57:03.5100905Z Entering 'third_party/pybind11'
2023-09-06T11:57:03.5141942Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2023-09-06T11:57:03.5161780Z Entering 'third_party/python-peachpy'
2023-09-06T11:57:03.5202525Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2023-09-06T11:57:03.5222351Z Entering 'third_party/sleef'
2023-09-06T11:57:03.5263435Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2023-09-06T11:57:03.5282873Z Entering 'third_party/tbb'
2023-09-06T11:57:03.5323839Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config	remote.origin.url
2023-09-06T11:57:03.5345040Z Entering 'third_party/tensorpipe'
2023-09-06T11:57:03.5386843Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2023-09-06T11:57:03.5405800Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-09-06T11:57:03.5446219Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2023-09-06T11:57:03.5465072Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-09-06T11:57:03.5506937Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2023-09-06T11:57:03.5524945Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-09-06T11:57:03.5565745Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2023-09-06T11:57:03.5585014Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-09-06T11:57:03.5626188Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2023-09-06T11:57:03.5644072Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-09-06T11:57:03.5685092Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2023-09-06T11:57:03.5706505Z Entering 'third_party/zstd'
2023-09-06T11:57:03.5747379Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config	remote.origin.url
2023-09-06T11:57:03.6094107Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2023-09-06T11:57:03.6375397Z Entering 'android/libs/fbjni'
2023-09-06T11:57:03.6415125Z Entering 'third_party/FP16'
2023-09-06T11:57:03.6454194Z Entering 'third_party/FXdiv'
2023-09-06T11:57:03.6494202Z Entering 'third_party/NNPACK'
2023-09-06T11:57:03.6533763Z Entering 'third_party/QNNPACK'
2023-09-06T11:57:03.6573379Z Entering 'third_party/VulkanMemoryAllocator'
2023-09-06T11:57:03.6613126Z Entering 'third_party/XNNPACK'
2023-09-06T11:57:03.6668371Z Entering 'third_party/benchmark'
2023-09-06T11:57:03.6707848Z Entering 'third_party/cpuinfo'
2023-09-06T11:57:03.6747467Z Entering 'third_party/cub'
2023-09-06T11:57:03.6786697Z Entering 'third_party/cudnn_frontend'
2023-09-06T11:57:03.6833485Z Entering 'third_party/cutlass'
2023-09-06T11:57:03.6883544Z Entering 'third_party/eigen'
2023-09-06T11:57:03.6926201Z Entering 'third_party/fbgemm'
2023-09-06T11:57:03.6965948Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-09-06T11:57:03.7005288Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-09-06T11:57:03.7043308Z Entering 'third_party/fbgemm/third_party/cutlass'
2023-09-06T11:57:03.7091092Z Entering 'third_party/fbgemm/third_party/googletest'
2023-09-06T11:57:03.7130178Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-09-06T11:57:03.7171706Z Entering 'third_party/flatbuffers'
2023-09-06T11:57:03.7214569Z Entering 'third_party/fmt'
2023-09-06T11:57:03.7254899Z Entering 'third_party/foxi'
2023-09-06T11:57:03.7294041Z Entering 'third_party/gemmlowp/gemmlowp'
2023-09-06T11:57:03.7333505Z Entering 'third_party/gloo'
2023-09-06T11:57:03.7374048Z Entering 'third_party/googletest'
2023-09-06T11:57:03.7415195Z Entering 'third_party/ideep'
2023-09-06T11:57:03.7453440Z Entering 'third_party/ideep/mkl-dnn'
2023-09-06T11:57:03.7499603Z Entering 'third_party/ios-cmake'
2023-09-06T11:57:03.7538880Z Entering 'third_party/ittapi'
2023-09-06T11:57:03.7578508Z Entering 'third_party/kineto'
2023-09-06T11:57:03.7617918Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2023-09-06T11:57:03.7656086Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2023-09-06T11:57:03.7698519Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2023-09-06T11:57:03.7737123Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2023-09-06T11:57:03.7776269Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2023-09-06T11:57:03.7813176Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2023-09-06T11:57:03.7854408Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2023-09-06T11:57:03.7894289Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2023-09-06T11:57:03.7933167Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2023-09-06T11:57:03.7976616Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2023-09-06T11:57:03.8016464Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-09-06T11:57:03.8055565Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-09-06T11:57:03.8098739Z Entering 'third_party/mimalloc'
2023-09-06T11:57:03.8139017Z Entering 'third_party/nccl/nccl'
2023-09-06T11:57:03.8178680Z Entering 'third_party/neon2sse'
2023-09-06T11:57:03.8218465Z Entering 'third_party/nlohmann'
2023-09-06T11:57:03.8258711Z Entering 'third_party/onnx'
2023-09-06T11:57:03.8316724Z Entering 'third_party/onnx/third_party/benchmark'
2023-09-06T11:57:03.8356794Z Entering 'third_party/onnx/third_party/pybind11'
2023-09-06T11:57:03.8400121Z Entering 'third_party/onnx-tensorrt'
2023-09-06T11:57:03.8438795Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-09-06T11:57:03.8483618Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-09-06T11:57:03.8522835Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-09-06T11:57:03.8563079Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-09-06T11:57:03.8608363Z Entering 'third_party/pocketfft'
2023-09-06T11:57:03.8648001Z Entering 'third_party/protobuf'
2023-09-06T11:57:03.8693049Z Entering 'third_party/protobuf/third_party/benchmark'
2023-09-06T11:57:03.8731548Z Entering 'third_party/protobuf/third_party/googletest'
2023-09-06T11:57:03.8772666Z Entering 'third_party/psimd'
2023-09-06T11:57:03.8812274Z Entering 'third_party/pthreadpool'
2023-09-06T11:57:03.8851480Z Entering 'third_party/pybind11'
2023-09-06T11:57:03.8892638Z Entering 'third_party/python-peachpy'
2023-09-06T11:57:03.8931675Z Entering 'third_party/sleef'
2023-09-06T11:57:03.8970971Z Entering 'third_party/tbb'
2023-09-06T11:57:03.9014356Z Entering 'third_party/tensorpipe'
2023-09-06T11:57:03.9054167Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-09-06T11:57:03.9093837Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-09-06T11:57:03.9132385Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-09-06T11:57:03.9171730Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-09-06T11:57:03.9210314Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-09-06T11:57:03.9251730Z Entering 'third_party/zstd'
2023-09-06T11:57:03.9304571Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2023-09-06T11:57:03.9576239Z Entering 'android/libs/fbjni'
2023-09-06T11:57:03.9615652Z Entering 'third_party/FP16'
2023-09-06T11:57:03.9655408Z Entering 'third_party/FXdiv'
2023-09-06T11:57:03.9694672Z Entering 'third_party/NNPACK'
2023-09-06T11:57:03.9734332Z Entering 'third_party/QNNPACK'
2023-09-06T11:57:03.9774631Z Entering 'third_party/VulkanMemoryAllocator'
2023-09-06T11:57:03.9814571Z Entering 'third_party/XNNPACK'
2023-09-06T11:57:03.9869922Z Entering 'third_party/benchmark'
2023-09-06T11:57:03.9910767Z Entering 'third_party/cpuinfo'
2023-09-06T11:57:03.9951437Z Entering 'third_party/cub'
2023-09-06T11:57:03.9991594Z Entering 'third_party/cudnn_frontend'
2023-09-06T11:57:04.0037861Z Entering 'third_party/cutlass'
2023-09-06T11:57:04.0085217Z Entering 'third_party/eigen'
2023-09-06T11:57:04.0127156Z Entering 'third_party/fbgemm'
2023-09-06T11:57:04.0167873Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-09-06T11:57:04.0207071Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-09-06T11:57:04.0246932Z Entering 'third_party/fbgemm/third_party/cutlass'
2023-09-06T11:57:04.0294980Z Entering 'third_party/fbgemm/third_party/googletest'
2023-09-06T11:57:04.0333475Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-09-06T11:57:04.0373680Z Entering 'third_party/flatbuffers'
2023-09-06T11:57:04.0417197Z Entering 'third_party/fmt'
2023-09-06T11:57:04.0456284Z Entering 'third_party/foxi'
2023-09-06T11:57:04.0495743Z Entering 'third_party/gemmlowp/gemmlowp'
2023-09-06T11:57:04.0535263Z Entering 'third_party/gloo'
2023-09-06T11:57:04.0574116Z Entering 'third_party/googletest'
2023-09-06T11:57:04.0613596Z Entering 'third_party/ideep'
2023-09-06T11:57:04.0652120Z Entering 'third_party/ideep/mkl-dnn'
2023-09-06T11:57:04.0698802Z Entering 'third_party/ios-cmake'
2023-09-06T11:57:04.0737742Z Entering 'third_party/ittapi'
2023-09-06T11:57:04.0777150Z Entering 'third_party/kineto'
2023-09-06T11:57:04.0816870Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2023-09-06T11:57:04.0856178Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2023-09-06T11:57:04.0896380Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2023-09-06T11:57:04.0935189Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2023-09-06T11:57:04.0975510Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2023-09-06T11:57:04.1012908Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2023-09-06T11:57:04.1054156Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2023-09-06T11:57:04.1093932Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2023-09-06T11:57:04.1133149Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2023-09-06T11:57:04.1172771Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2023-09-06T11:57:04.1213838Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-09-06T11:57:04.1253225Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-09-06T11:57:04.1293870Z Entering 'third_party/mimalloc'
2023-09-06T11:57:04.1333485Z Entering 'third_party/nccl/nccl'
2023-09-06T11:57:04.1373080Z Entering 'third_party/neon2sse'
2023-09-06T11:57:04.1412097Z Entering 'third_party/nlohmann'
2023-09-06T11:57:04.1452350Z Entering 'third_party/onnx'
2023-09-06T11:57:04.1508980Z Entering 'third_party/onnx/third_party/benchmark'
2023-09-06T11:57:04.1548449Z Entering 'third_party/onnx/third_party/pybind11'
2023-09-06T11:57:04.1590224Z Entering 'third_party/onnx-tensorrt'
2023-09-06T11:57:04.1628213Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-09-06T11:57:04.1673551Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-09-06T11:57:04.1713503Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-09-06T11:57:04.1751370Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-09-06T11:57:04.1795613Z Entering 'third_party/pocketfft'
2023-09-06T11:57:04.1834817Z Entering 'third_party/protobuf'
2023-09-06T11:57:04.1878675Z Entering 'third_party/protobuf/third_party/benchmark'
2023-09-06T11:57:04.1917181Z Entering 'third_party/protobuf/third_party/googletest'
2023-09-06T11:57:04.1957711Z Entering 'third_party/psimd'
2023-09-06T11:57:04.1998223Z Entering 'third_party/pthreadpool'
2023-09-06T11:57:04.2038191Z Entering 'third_party/pybind11'
2023-09-06T11:57:04.2078461Z Entering 'third_party/python-peachpy'
2023-09-06T11:57:04.2118311Z Entering 'third_party/sleef'
2023-09-06T11:57:04.2158858Z Entering 'third_party/tbb'
2023-09-06T11:57:04.2201222Z Entering 'third_party/tensorpipe'
2023-09-06T11:57:04.2240872Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-09-06T11:57:04.2280089Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-09-06T11:57:04.2318312Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-09-06T11:57:04.2356616Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-09-06T11:57:04.2394517Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-09-06T11:57:04.2436336Z Entering 'third_party/zstd'
2023-09-06T11:57:04.2489460Z ##[endgroup]
2023-09-06T11:57:04.2538791Z [command]/usr/bin/git log -1 --format='%H'
2023-09-06T11:57:04.2573275Z '3fe8417643c8d6c2b3d95552cd90321d141b5d54'
2023-09-06T11:57:04.2802581Z Prepare all required actions
2023-09-06T11:57:04.2803298Z Getting action download info
2023-09-06T11:57:04.4815280Z ##[group]Run ./.github/actions/setup-linux
2023-09-06T11:57:04.4815768Z env:
2023-09-06T11:57:04.4816004Z   GIT_DEFAULT_BRANCH: main
2023-09-06T11:57:04.4816250Z ##[endgroup]
2023-09-06T11:57:04.4885736Z ##[group]Run set -euo pipefail
2023-09-06T11:57:04.4886363Z set -euo pipefail
2023-09-06T11:57:04.4886813Z function get_ec2_metadata() {
2023-09-06T11:57:04.4887164Z   # Pulled from instance metadata endpoint for EC2
2023-09-06T11:57:04.4887660Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2023-09-06T11:57:04.4888135Z   category=$1
2023-09-06T11:57:04.4888461Z   # If it is GCP runner (runner name contains gcp), do not run this
2023-09-06T11:57:04.4888844Z   runner_name_str=gh-ci-gcp-a100-17
2023-09-06T11:57:04.4889235Z   if [[ $runner_name_str != *"gcp"* ]]; then
2023-09-06T11:57:04.4889603Z     curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2023-09-06T11:57:04.4889901Z   else
2023-09-06T11:57:04.4890244Z     echo "Runner is from Google Cloud Platform, No info on ec2 metadata"
2023-09-06T11:57:04.4890570Z   fi
2023-09-06T11:57:04.4890778Z }
2023-09-06T11:57:04.4891030Z echo "ami-id: $(get_ec2_metadata ami-id)"
2023-09-06T11:57:04.4891381Z echo "instance-id: $(get_ec2_metadata instance-id)"
2023-09-06T11:57:04.4891756Z echo "instance-type: $(get_ec2_metadata instance-type)"
2023-09-06T11:57:04.4892092Z echo "system info $(uname -a)"
2023-09-06T11:57:04.4911425Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-09-06T11:57:04.4911711Z env:
2023-09-06T11:57:04.4911946Z   GIT_DEFAULT_BRANCH: main
2023-09-06T11:57:04.4912193Z ##[endgroup]
2023-09-06T11:57:04.4958928Z ami-id: Runner is from Google Cloud Platform, No info on ec2 metadata
2023-09-06T11:57:04.4964988Z instance-id: Runner is from Google Cloud Platform, No info on ec2 metadata
2023-09-06T11:57:04.4970442Z instance-type: Runner is from Google Cloud Platform, No info on ec2 metadata
2023-09-06T11:57:04.4981864Z system info Linux gh-ci-gcp-a100-17 5.15.0-1037-gcp #45~20.04.1-Ubuntu SMP Thu Jun 22 08:31:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
2023-09-06T11:57:04.5000784Z ##[group]Run if systemctl is-active --quiet docker; then
2023-09-06T11:57:04.5001165Z if systemctl is-active --quiet docker; then
2023-09-06T11:57:04.5001508Z     echo "Docker daemon is running...";
2023-09-06T11:57:04.5001787Z else
2023-09-06T11:57:04.5002104Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2023-09-06T11:57:04.5002423Z fi
2023-09-06T11:57:04.5021137Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-09-06T11:57:04.5021434Z env:
2023-09-06T11:57:04.5021670Z   GIT_DEFAULT_BRANCH: main
2023-09-06T11:57:04.5021919Z ##[endgroup]
2023-09-06T11:57:04.5091693Z Docker daemon is running...
2023-09-06T11:57:04.5134062Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2023-09-06T11:57:04.5134611Z with:
2023-09-06T11:57:04.5135064Z   shell: bash
2023-09-06T11:57:04.5135608Z   timeout_minutes: 5
2023-09-06T11:57:04.5136253Z   max_attempts: 3
2023-09-06T11:57:04.5136629Z   retry_wait_seconds: 30
2023-09-06T11:57:04.5137348Z   command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \
    --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"

2023-09-06T11:57:04.5138171Z   polling_interval_seconds: 1
2023-09-06T11:57:04.5138524Z   warning_on_retry: true
2023-09-06T11:57:04.5138805Z   continue_on_error: false
2023-09-06T11:57:04.5139113Z env:
2023-09-06T11:57:04.5139409Z   GIT_DEFAULT_BRANCH: main
2023-09-06T11:57:04.5140013Z   AWS_RETRY_MODE: standard
2023-09-06T11:57:04.5140296Z   AWS_MAX_ATTEMPTS: 5
2023-09-06T11:57:04.5140629Z   AWS_DEFAULT_REGION: us-east-1
2023-09-06T11:57:04.5140956Z ##[endgroup]
2023-09-06T11:57:06.7344964Z WARNING! Your password will be stored unencrypted in /home/ubuntu/.docker/config.json.
2023-09-06T11:57:06.7346318Z Configure a credential helper to remove this warning. See
2023-09-06T11:57:06.7347563Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2023-09-06T11:57:06.7347991Z 
2023-09-06T11:57:06.7348242Z Login Succeeded
2023-09-06T11:57:07.5855096Z Command completed after 1 attempt(s).
2023-09-06T11:57:07.5916720Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2023-09-06T11:57:07.5917149Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2023-09-06T11:57:07.5917529Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2023-09-06T11:57:07.5935657Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-09-06T11:57:07.5935979Z env:
2023-09-06T11:57:07.5936201Z   GIT_DEFAULT_BRANCH: main
2023-09-06T11:57:07.5936452Z ##[endgroup]
2023-09-06T11:57:07.6025161Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2023-09-06T11:57:07.6025632Z # ignore expansion of "docker ps -q" since it could be empty
2023-09-06T11:57:07.6025971Z # shellcheck disable=SC2046
2023-09-06T11:57:07.6026283Z docker stop $(docker ps -q) || true
2023-09-06T11:57:07.6026603Z # Prune all of the docker images
2023-09-06T11:57:07.6026901Z docker system prune -af
2023-09-06T11:57:07.6044326Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-09-06T11:57:07.6044612Z env:
2023-09-06T11:57:07.6044845Z   GIT_DEFAULT_BRANCH: main
2023-09-06T11:57:07.6045099Z ##[endgroup]
2023-09-06T11:57:07.6471523Z "docker stop" requires at least 1 argument.
2023-09-06T11:57:07.6472165Z See 'docker stop --help'.
2023-09-06T11:57:07.6472339Z 
2023-09-06T11:57:07.6472536Z Usage:  docker stop [OPTIONS] CONTAINER [CONTAINER...]
2023-09-06T11:57:07.6472748Z 
2023-09-06T11:57:07.6472878Z Stop one or more running containers
2023-09-06T11:57:07.6698235Z Total reclaimed space: 0B
2023-09-06T11:57:07.6737332Z ##[group]Run set +e
2023-09-06T11:57:07.6737707Z set +e
2023-09-06T11:57:07.6737943Z set -x
2023-09-06T11:57:07.6738150Z 
2023-09-06T11:57:07.6738420Z PT_DOMAIN=download.pytorch.org
2023-09-06T11:57:07.6738877Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400,
2023-09-06T11:57:07.6739413Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last
2023-09-06T11:57:07.6739822Z # one is returned at random
2023-09-06T11:57:07.6740146Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1)
2023-09-06T11:57:07.6740434Z 
2023-09-06T11:57:07.6740686Z if [ -z "${RESOLVED_IP}" ]; then
2023-09-06T11:57:07.6741069Z   echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..."
2023-09-06T11:57:07.6741465Z   RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1)
2023-09-06T11:57:07.6741766Z 
2023-09-06T11:57:07.6742031Z   if [ -z "${RESOLVED_IP}" ]; then
2023-09-06T11:57:07.6742369Z     echo "Couldn't resolve ${PT_DOMAIN}, exiting..."
2023-09-06T11:57:07.6742655Z     exit 1
2023-09-06T11:57:07.6742887Z   fi
2023-09-06T11:57:07.6743101Z fi
2023-09-06T11:57:07.6743311Z 
2023-09-06T11:57:07.6743570Z if grep -r "${PT_DOMAIN}" /etc/hosts; then
2023-09-06T11:57:07.6743900Z   # Clean up any old records first
2023-09-06T11:57:07.6744227Z   sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts
2023-09-06T11:57:07.6744497Z fi
2023-09-06T11:57:07.6744693Z 
2023-09-06T11:57:07.6744993Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts
2023-09-06T11:57:07.6745480Z cat /etc/hosts
2023-09-06T11:57:07.6764044Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-09-06T11:57:07.6764345Z env:
2023-09-06T11:57:07.6764570Z   GIT_DEFAULT_BRANCH: main
2023-09-06T11:57:07.6765026Z ##[endgroup]
2023-09-06T11:57:07.6801614Z + PT_DOMAIN=download.pytorch.org
2023-09-06T11:57:07.6808236Z ++ dig -4 +short download.pytorch.org
2023-09-06T11:57:07.6808929Z ++ tail -n1
2023-09-06T11:57:07.6894995Z + RESOLVED_IP=54.230.18.128
2023-09-06T11:57:07.6895622Z + '[' -z 54.230.18.128 ']'
2023-09-06T11:57:07.6896002Z + grep -r download.pytorch.org /etc/hosts
2023-09-06T11:57:07.6910004Z 54.230.18.128 download.pytorch.org
2023-09-06T11:57:07.6911756Z + sudo sed -i /download.pytorch.org/d /etc/hosts
2023-09-06T11:57:07.7009559Z + echo '54.230.18.128 download.pytorch.org'
2023-09-06T11:57:07.7010044Z + sudo tee -a /etc/hosts
2023-09-06T11:57:07.7102508Z 54.230.18.128 download.pytorch.org
2023-09-06T11:57:07.7110080Z + cat /etc/hosts
2023-09-06T11:57:07.7118314Z 127.0.0.1 localhost
2023-09-06T11:57:07.7118687Z 
2023-09-06T11:57:07.7119151Z # The following lines are desirable for IPv6 capable hosts
2023-09-06T11:57:07.7133666Z ::1 ip6-localhost ip6-loopback
2023-09-06T11:57:07.7134049Z fe00::0 ip6-localnet
2023-09-06T11:57:07.7134359Z ff00::0 ip6-mcastprefix
2023-09-06T11:57:07.7134674Z ff02::1 ip6-allnodes
2023-09-06T11:57:07.7134964Z ff02::2 ip6-allrouters
2023-09-06T11:57:07.7135258Z ff02::3 ip6-allhosts
2023-09-06T11:57:07.7135543Z 169.254.169.254 metadata.google.internal metadata
2023-09-06T11:57:07.7135864Z 54.230.18.128 download.pytorch.org
2023-09-06T11:57:07.7215013Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main
2023-09-06T11:57:07.7215360Z with:
2023-09-06T11:57:07.7215922Z   docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:9dd361d1c04129f8eaa9d6b43335917800dd6d24
2023-09-06T11:57:07.7216496Z   docker-build-dir: .ci/docker
2023-09-06T11:57:07.7216772Z   working-directory: .
2023-09-06T11:57:07.7217104Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2023-09-06T11:57:07.7217401Z env:
2023-09-06T11:57:07.7217612Z   GIT_DEFAULT_BRANCH: main
2023-09-06T11:57:07.7217873Z ##[endgroup]
2023-09-06T11:57:07.7235405Z ##[group]Run set -ex
2023-09-06T11:57:07.7235692Z set -ex
2023-09-06T11:57:07.7235914Z 
2023-09-06T11:57:07.7236265Z # If the docker build directory or the build script doesn't exist, the action will
2023-09-06T11:57:07.7236806Z # gracefully return the docker image name as it is.  Pulling docker image in Linux
2023-09-06T11:57:07.7237231Z # job could then download the pre-built image as usual
2023-09-06T11:57:07.7237645Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then
2023-09-06T11:57:07.7238023Z   echo "skip=true" >> "${GITHUB_OUTPUT}"
2023-09-06T11:57:07.7238395Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2023-09-06T11:57:07.7238689Z 
2023-09-06T11:57:07.7239023Z   echo "There is no Docker build script in ${REPO_NAME} repo, skipping..."
2023-09-06T11:57:07.7239363Z   exit 0
2023-09-06T11:57:07.7239596Z else
2023-09-06T11:57:07.7239856Z   echo "skip=false" >> "${GITHUB_OUTPUT}"
2023-09-06T11:57:07.7240128Z fi
2023-09-06T11:57:07.7240337Z 
2023-09-06T11:57:07.7240645Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then
2023-09-06T11:57:07.7241096Z   # The docker image name already includes the ECR prefix and tag, so we can just
2023-09-06T11:57:07.7241508Z   # use it as it is, but first let's extract the tag
2023-09-06T11:57:07.7241902Z   DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}')
2023-09-06T11:57:07.7242297Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2023-09-06T11:57:07.7242681Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2023-09-06T11:57:07.7242976Z else
2023-09-06T11:57:07.7243276Z   DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}")
2023-09-06T11:57:07.7243698Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2023-09-06T11:57:07.7244312Z   echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2023-09-06T11:57:07.7244673Z fi
2023-09-06T11:57:07.7274258Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-09-06T11:57:07.7274568Z env:
2023-09-06T11:57:07.7274807Z   GIT_DEFAULT_BRANCH: main
2023-09-06T11:57:07.7275064Z   REPO_NAME: pytorch
2023-09-06T11:57:07.7275631Z   DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:9dd361d1c04129f8eaa9d6b43335917800dd6d24
2023-09-06T11:57:07.7276205Z   DOCKER_BUILD_DIR: .ci/docker
2023-09-06T11:57:07.7276548Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2023-09-06T11:57:07.7276857Z ##[endgroup]
2023-09-06T11:57:07.7315094Z + [[ ! -d .ci/docker ]]
2023-09-06T11:57:07.7315460Z + [[ ! -f .ci/docker/build.sh ]]
2023-09-06T11:57:07.7315745Z + echo skip=false
2023-09-06T11:57:07.7316774Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:9dd361d1c04129f8eaa9d6b43335917800dd6d24 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]]
2023-09-06T11:57:07.7323837Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:9dd361d1c04129f8eaa9d6b43335917800dd6d24
2023-09-06T11:57:07.7324806Z ++ awk -F '[:,]' '{print $2}'
2023-09-06T11:57:07.7345233Z + DOCKER_TAG=9dd361d1c04129f8eaa9d6b43335917800dd6d24
2023-09-06T11:57:07.7346143Z + echo docker-tag=9dd361d1c04129f8eaa9d6b43335917800dd6d24
2023-09-06T11:57:07.7347382Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:9dd361d1c04129f8eaa9d6b43335917800dd6d24
2023-09-06T11:57:07.7396656Z ##[group]Run set +e
2023-09-06T11:57:07.7396955Z set +e
2023-09-06T11:57:07.7397183Z set -x
2023-09-06T11:57:07.7397400Z 
2023-09-06T11:57:07.7397722Z # Check if image already exists, if it does then skip building it
2023-09-06T11:57:07.7398112Z if docker manifest inspect "${DOCKER_IMAGE}"; then
2023-09-06T11:57:07.7398420Z   exit 0
2023-09-06T11:57:07.7398640Z fi
2023-09-06T11:57:07.7398848Z 
2023-09-06T11:57:07.7399168Z # NB: This part requires a full checkout. Otherwise, the merge base will
2023-09-06T11:57:07.7399614Z # be empty.  The default action would be to continue rebuild the image
2023-09-06T11:57:07.7400024Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then
2023-09-06T11:57:07.7400412Z   # if we're on the base branch then use the parent commit
2023-09-06T11:57:07.7400768Z   MERGE_BASE=$(git rev-parse HEAD~)
2023-09-06T11:57:07.7401020Z else
2023-09-06T11:57:07.7401346Z   # otherwise we're on a PR, so use the most recent base commit
2023-09-06T11:57:07.7401737Z   MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION")
2023-09-06T11:57:07.7402023Z fi
2023-09-06T11:57:07.7402217Z 
2023-09-06T11:57:07.7402464Z if [[ -z "${MERGE_BASE}" ]]; then
2023-09-06T11:57:07.7402787Z   echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2023-09-06T11:57:07.7403057Z 
2023-09-06T11:57:07.7403418Z   echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..."
2023-09-06T11:57:07.7403846Z   exit 0
2023-09-06T11:57:07.7404067Z fi
2023-09-06T11:57:07.7404277Z 
2023-09-06T11:57:07.7404561Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then
2023-09-06T11:57:07.7405044Z   echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit"
2023-09-06T11:57:07.7405433Z   exit 1
2023-09-06T11:57:07.7405844Z fi
2023-09-06T11:57:07.7406054Z 
2023-09-06T11:57:07.7406360Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}")
2023-09-06T11:57:07.7406843Z # If no image exists but the hash is the same as the previous hash then we should error out here
2023-09-06T11:57:07.7407294Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then
2023-09-06T11:57:07.7407780Z   echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch"
2023-09-06T11:57:07.7408310Z   echo "         Will re-build docker image to store in local cache, TTS may be longer"
2023-09-06T11:57:07.7408634Z fi
2023-09-06T11:57:07.7408842Z 
2023-09-06T11:57:07.7409114Z echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2023-09-06T11:57:07.7426001Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-09-06T11:57:07.7426284Z env:
2023-09-06T11:57:07.7426523Z   GIT_DEFAULT_BRANCH: main
2023-09-06T11:57:07.7426792Z   DOCKER_BUILD_DIR: .ci/docker
2023-09-06T11:57:07.7427107Z   BASE_REVISION: 3fe8417643c8d6c2b3d95552cd90321d141b5d54
2023-09-06T11:57:07.7427892Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:9dd361d1c04129f8eaa9d6b43335917800dd6d24
2023-09-06T11:57:07.7428497Z   DOCKER_TAG: 9dd361d1c04129f8eaa9d6b43335917800dd6d24
2023-09-06T11:57:07.7428780Z ##[endgroup]
2023-09-06T11:57:07.7464288Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:9dd361d1c04129f8eaa9d6b43335917800dd6d24
2023-09-06T11:57:08.4671812Z {
2023-09-06T11:57:08.4672241Z 	"schemaVersion": 2,
2023-09-06T11:57:08.4672953Z 	"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
2023-09-06T11:57:08.4673592Z 	"config": {
2023-09-06T11:57:08.4674128Z 		"mediaType": "application/vnd.docker.container.image.v1+json",
2023-09-06T11:57:08.4674604Z 		"size": 45655,
2023-09-06T11:57:08.4674960Z 		"digest": "sha256:d5a715adf5384149f42b562408a0548773cf7fc344221f5b65a1c1a685e1c2a8"
2023-09-06T11:57:08.4675273Z 	},
2023-09-06T11:57:08.4675496Z 	"layers": [
2023-09-06T11:57:08.4675709Z 		{
2023-09-06T11:57:08.4676067Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4676403Z 			"size": 28578563,
2023-09-06T11:57:08.4676772Z 			"digest": "sha256:99803d4b97f3db529ae9ca4174b0951afac6b309e7deaa8ec3214c584e02b3a8"
2023-09-06T11:57:08.4677211Z 		},
2023-09-06T11:57:08.4677529Z 		{
2023-09-06T11:57:08.4678014Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4678563Z 			"size": 7920735,
2023-09-06T11:57:08.4679354Z 			"digest": "sha256:0b5414b58ed1bab309ed977b88a36dc8f22512a60ef70761c3da177ad5a32023"
2023-09-06T11:57:08.4680031Z 		},
2023-09-06T11:57:08.4680320Z 		{
2023-09-06T11:57:08.4680658Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4681025Z 			"size": 55728009,
2023-09-06T11:57:08.4681393Z 			"digest": "sha256:434ea07ac2904f60aa3aeee28f971d188ba20631f84b4bf8064719c09f5b84b5"
2023-09-06T11:57:08.4681718Z 		},
2023-09-06T11:57:08.4681916Z 		{
2023-09-06T11:57:08.4682431Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4682898Z 			"size": 185,
2023-09-06T11:57:08.4683474Z 			"digest": "sha256:7bbb6f0efdde0ffab9bbc66e5052261b9b38ccc121db5173b125a2077acad92a"
2023-09-06T11:57:08.4684047Z 		},
2023-09-06T11:57:08.4684270Z 		{
2023-09-06T11:57:08.4684607Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4684933Z 			"size": 6884,
2023-09-06T11:57:08.4685328Z 			"digest": "sha256:93036f51a832fc3daa8da5a3fcc95fc30367dd3daef6d73192d6c0e21a40a468"
2023-09-06T11:57:08.4685902Z 		},
2023-09-06T11:57:08.4686316Z 		{
2023-09-06T11:57:08.4686939Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4688221Z 			"size": 1329001351,
2023-09-06T11:57:08.4688668Z 			"digest": "sha256:d7fb9be2f79b53474246c4372e5401579c288d26dd447d950ddd0234c19956a9"
2023-09-06T11:57:08.4688984Z 		},
2023-09-06T11:57:08.4689193Z 		{
2023-09-06T11:57:08.4689526Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4689885Z 			"size": 62478,
2023-09-06T11:57:08.4690251Z 			"digest": "sha256:b7deb82cec78a179b20de0cd3344c1e9ad951051eb25f9c90ef966633b56959b"
2023-09-06T11:57:08.4690603Z 		},
2023-09-06T11:57:08.4690798Z 		{
2023-09-06T11:57:08.4691148Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4691493Z 			"size": 1685,
2023-09-06T11:57:08.4691842Z 			"digest": "sha256:ffe25f6234c7144513b6397bc22671ddb4a56feab444a854bbbae954cc36f23a"
2023-09-06T11:57:08.4692171Z 		},
2023-09-06T11:57:08.4692366Z 		{
2023-09-06T11:57:08.4692807Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4693548Z 			"size": 1523,
2023-09-06T11:57:08.4694556Z 			"digest": "sha256:9e5d1aba80d9e5af83aa303810fb0f403caaa7a588c688ce175d8b8c62afd991"
2023-09-06T11:57:08.4695263Z 		},
2023-09-06T11:57:08.4695614Z 		{
2023-09-06T11:57:08.4696050Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4696400Z 			"size": 2530498212,
2023-09-06T11:57:08.4697001Z 			"digest": "sha256:04430174d25302e5ca6708e5eacd748e2ab0953297d3fdbc833e4f1e6bd646b4"
2023-09-06T11:57:08.4697330Z 		},
2023-09-06T11:57:08.4697511Z 		{
2023-09-06T11:57:08.4697844Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4698188Z 			"size": 86037,
2023-09-06T11:57:08.4698554Z 			"digest": "sha256:845c662a61cb45227f2458cc4009153619547484896345f02966f43c2364f22d"
2023-09-06T11:57:08.4698855Z 		},
2023-09-06T11:57:08.4699203Z + exit 0
2023-09-06T11:57:08.4699507Z 		{
2023-09-06T11:57:08.4700026Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4700524Z 			"size": 1801,
2023-09-06T11:57:08.4700903Z 			"digest": "sha256:8d60f017b61c0c815862cbb0c9ea30b2339b93ce73ac82a1adfc4fe36ade415b"
2023-09-06T11:57:08.4701222Z 		},
2023-09-06T11:57:08.4701417Z 		{
2023-09-06T11:57:08.4701752Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4702083Z 			"size": 285556557,
2023-09-06T11:57:08.4702467Z 			"digest": "sha256:2c2cb8fdcb6b4bda01e1646b940149c0a12946ce0c11f7ba11dc2c094606259c"
2023-09-06T11:57:08.4702818Z 		},
2023-09-06T11:57:08.4703013Z 		{
2023-09-06T11:57:08.4703330Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4703675Z 			"size": 544,
2023-09-06T11:57:08.4704168Z 			"digest": "sha256:b4849290c5549d46acc73e89947994fd65789c9510f1603b3e7bada192d06d04"
2023-09-06T11:57:08.4704693Z 		},
2023-09-06T11:57:08.4704992Z 		{
2023-09-06T11:57:08.4705518Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4706025Z 			"size": 1278,
2023-09-06T11:57:08.4706615Z 			"digest": "sha256:9c1eadd32ad20cdbe96b735f3b8c46c90b234ed1d4951f7dac6221cd2434526e"
2023-09-06T11:57:08.4707218Z 		},
2023-09-06T11:57:08.4707527Z 		{
2023-09-06T11:57:08.4708310Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4708709Z 			"size": 483,
2023-09-06T11:57:08.4709276Z 			"digest": "sha256:daf57644add8caf8aab13964eb4a80d50b51046bb5e3522339c652e2eb8472eb"
2023-09-06T11:57:08.4709821Z 		},
2023-09-06T11:57:08.4710021Z 		{
2023-09-06T11:57:08.4710361Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4710697Z 			"size": 91702064,
2023-09-06T11:57:08.4711055Z 			"digest": "sha256:7c8d8836abad7488de22c76d160d79a4319c93b477c6f4283999148ba6741221"
2023-09-06T11:57:08.4711386Z 		},
2023-09-06T11:57:08.4711583Z 		{
2023-09-06T11:57:08.4711902Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4712243Z 			"size": 2754,
2023-09-06T11:57:08.4712600Z 			"digest": "sha256:580e0fea9066d77e63dc4183414f89cb4326165a6fcea764c145fa6e7c223b62"
2023-09-06T11:57:08.4713171Z 		},
2023-09-06T11:57:08.4713355Z 		{
2023-09-06T11:57:08.4713691Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4714035Z 			"size": 2020,
2023-09-06T11:57:08.4714415Z 			"digest": "sha256:95a486a5bceb5bfc11eb732f2866cd45a8df742edbbb51b48e5672e790013df1"
2023-09-06T11:57:08.4714746Z 		},
2023-09-06T11:57:08.4714944Z 		{
2023-09-06T11:57:08.4715283Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4715630Z 			"size": 699,
2023-09-06T11:57:08.4715963Z 			"digest": "sha256:03870a46e125521f9df3ee2930c8b4984dd497a7e5f074d4e33f1e65e4b1b3ec"
2023-09-06T11:57:08.4716283Z 		},
2023-09-06T11:57:08.4716480Z 		{
2023-09-06T11:57:08.4716796Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4717141Z 			"size": 2364499408,
2023-09-06T11:57:08.4717511Z 			"digest": "sha256:2f4342e3f8f3caf07a77df64df3bfa63f00c6e9061395423b057cbf8cafcf2ad"
2023-09-06T11:57:08.4717853Z 		},
2023-09-06T11:57:08.4718033Z 		{
2023-09-06T11:57:08.4718426Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4718766Z 			"size": 380,
2023-09-06T11:57:08.4719122Z 			"digest": "sha256:aea5db51af841c7a632ff79f3023c943669404f76c54986d8b01b381715ff044"
2023-09-06T11:57:08.4719562Z 		},
2023-09-06T11:57:08.4719767Z 		{
2023-09-06T11:57:08.4720103Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4720444Z 			"size": 6330,
2023-09-06T11:57:08.4720798Z 			"digest": "sha256:fe13553c51588e18d622df69cb3cfcf2bab88e9767ddbcb17b963c9e1ade99d4"
2023-09-06T11:57:08.4721164Z 		},
2023-09-06T11:57:08.4721363Z 		{
2023-09-06T11:57:08.4721692Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4722013Z 			"size": 803,
2023-09-06T11:57:08.4722376Z 			"digest": "sha256:c16cf9a5795aee759413d04ebb567fd42656d58c153b28ce1d3b2768efaddb11"
2023-09-06T11:57:08.4722706Z 		},
2023-09-06T11:57:08.4723011Z 		{
2023-09-06T11:57:08.4723341Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4723680Z 			"size": 106,
2023-09-06T11:57:08.4724038Z 			"digest": "sha256:efb74f88ccdd1a3811e30bf53f84f148beb499f71443c64129bac9e26d3c1425"
2023-09-06T11:57:08.4724379Z 		},
2023-09-06T11:57:08.4724581Z 		{
2023-09-06T11:57:08.4724912Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4725234Z 			"size": 888,
2023-09-06T11:57:08.4725591Z 			"digest": "sha256:e5fa01783c35e4092951b1375b41c6f1acc3d0666f0af3f10aebbe9f3ffa1050"
2023-09-06T11:57:08.4725921Z 		},
2023-09-06T11:57:08.4726113Z 		{
2023-09-06T11:57:08.4726431Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4726772Z 			"size": 121477409,
2023-09-06T11:57:08.4727129Z 			"digest": "sha256:d073ad6760c426d74881a0138cd10f5b7c0f78f203ad05888c9053ec981728a1"
2023-09-06T11:57:08.4727448Z 		},
2023-09-06T11:57:08.4727627Z 		{
2023-09-06T11:57:08.4727969Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4728344Z 			"size": 109,
2023-09-06T11:57:08.4728700Z 			"digest": "sha256:3bc7d371ee77ce3b96061c9d93ff835987dfff30877594e91340b22a514fa63e"
2023-09-06T11:57:08.4729014Z 		},
2023-09-06T11:57:08.4729207Z 		{
2023-09-06T11:57:08.4729545Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4729885Z 			"size": 558,
2023-09-06T11:57:08.4730230Z 			"digest": "sha256:ba2991171fbc7cfdb6d53cf43b59b59782602cf64933a506f1e899c47c959243"
2023-09-06T11:57:08.4730564Z 		},
2023-09-06T11:57:08.4730759Z 		{
2023-09-06T11:57:08.4731091Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4731417Z 			"size": 1176221,
2023-09-06T11:57:08.4731792Z 			"digest": "sha256:9df7805d993b3b4f9f62144fcdf04f2bede10fbc5d23adf9fde77c3770385423"
2023-09-06T11:57:08.4732135Z 		},
2023-09-06T11:57:08.4732315Z 		{
2023-09-06T11:57:08.4732651Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4733124Z 			"size": 103,
2023-09-06T11:57:08.4733477Z 			"digest": "sha256:d980247356b07ade79cd4e6abc3b7197a1b20f1b52e5728f4d89d85b28e9bb27"
2023-09-06T11:57:08.4733790Z 		},
2023-09-06T11:57:08.4733986Z 		{
2023-09-06T11:57:08.4734326Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4734662Z 			"size": 1489,
2023-09-06T11:57:08.4735008Z 			"digest": "sha256:7034cfc3cf6fe6cb245155448cea0d010dad420c95f55ba46b51e83216730032"
2023-09-06T11:57:08.4735347Z 		},
2023-09-06T11:57:08.4735541Z 		{
2023-09-06T11:57:08.4735870Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4736200Z 			"size": 436228183,
2023-09-06T11:57:08.4736561Z 			"digest": "sha256:7ba56cc73c7b18414472f17201a04eb059e1ac5c999f2dd6cfd9a580003ef4ed"
2023-09-06T11:57:08.4736886Z 		},
2023-09-06T11:57:08.4737081Z 		{
2023-09-06T11:57:08.4737399Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4737744Z 			"size": 160,
2023-09-06T11:57:08.4738094Z 			"digest": "sha256:632e958b9436df6658af3e9a7dca2695b23069846dfe7432d8ec8bbec9b13e2f"
2023-09-06T11:57:08.4738456Z 		},
2023-09-06T11:57:08.4738639Z 		{
2023-09-06T11:57:08.4738971Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4739409Z 			"size": 452,
2023-09-06T11:57:08.4739782Z 			"digest": "sha256:4bb4afaa2a5fdf900a6f37e8b08ef173d3440650cfb001812186e535a23a0ad9"
2023-09-06T11:57:08.4740107Z 		},
2023-09-06T11:57:08.4740302Z 		{
2023-09-06T11:57:08.4740682Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4741026Z 			"size": 61499946,
2023-09-06T11:57:08.4741375Z 			"digest": "sha256:f9dcab60d1047e9f7253cb62584a5f34ce797217057a8e5d158bd8e8e223ad07"
2023-09-06T11:57:08.4741706Z 		},
2023-09-06T11:57:08.4741899Z 		{
2023-09-06T11:57:08.4742234Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4742558Z 			"size": 102,
2023-09-06T11:57:08.4742914Z 			"digest": "sha256:ef954792f06f8448d814505b8a13fab49e996ec966bdc05b726f03c2937744f7"
2023-09-06T11:57:08.4743230Z 		},
2023-09-06T11:57:08.4743425Z 		{
2023-09-06T11:57:08.4743741Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4744075Z 			"size": 402,
2023-09-06T11:57:08.4744441Z 			"digest": "sha256:d2bb74a53419c00c8dcc4a8c164d5ad2047a4a1690d2c1c7b396e0e1ee6c4bc6"
2023-09-06T11:57:08.4744782Z 		},
2023-09-06T11:57:08.4744964Z 		{
2023-09-06T11:57:08.4745294Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4745635Z 			"size": 20262060,
2023-09-06T11:57:08.4745979Z 			"digest": "sha256:777f0c7185cc93ed4ca49bb8c0988065afb31f8fa23173b9a9983976e11d5315"
2023-09-06T11:57:08.4746308Z 		},
2023-09-06T11:57:08.4746501Z 		{
2023-09-06T11:57:08.4746832Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4747160Z 			"size": 564,
2023-09-06T11:57:08.4747529Z 			"digest": "sha256:e3a06d78c00b73e55e687aff088f7bd83aa54c46da1c11dbb1caf2c19384c6a4"
2023-09-06T11:57:08.4747871Z 		},
2023-09-06T11:57:08.4748064Z 		{
2023-09-06T11:57:08.4748422Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4748762Z 			"size": 43166425,
2023-09-06T11:57:08.4749349Z 			"digest": "sha256:a5f28815538d0a4ccff434c455ea1bbdd44e1bd3d3fb4301dbde646fe610e900"
2023-09-06T11:57:08.4749768Z 		},
2023-09-06T11:57:08.4749950Z 		{
2023-09-06T11:57:08.4750282Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4750622Z 			"size": 106,
2023-09-06T11:57:08.4750995Z 			"digest": "sha256:3c227ecee3dac7c937f3634bf3fe544040a219fabeb46aa17165c84c48f6361b"
2023-09-06T11:57:08.4751323Z 		},
2023-09-06T11:57:08.4751518Z 		{
2023-09-06T11:57:08.4751849Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4752186Z 			"size": 409,
2023-09-06T11:57:08.4752514Z 			"digest": "sha256:4c1167c82e960855b376efeeb70779905c2178b26f9bd299ed25535c85a049a9"
2023-09-06T11:57:08.4752991Z 		},
2023-09-06T11:57:08.4753185Z 		{
2023-09-06T11:57:08.4753523Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4753848Z 			"size": 699,
2023-09-06T11:57:08.4754198Z 			"digest": "sha256:03870a46e125521f9df3ee2930c8b4984dd497a7e5f074d4e33f1e65e4b1b3ec"
2023-09-06T11:57:08.4754516Z 		},
2023-09-06T11:57:08.4754718Z 		{
2023-09-06T11:57:08.4755034Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4755380Z 			"size": 116,
2023-09-06T11:57:08.4755728Z 			"digest": "sha256:12dc7c97f6f36f1c05095df848775ee780386b42387f8aba393b92b2d3bc400e"
2023-09-06T11:57:08.4756052Z 		},
2023-09-06T11:57:08.4756235Z 		{
2023-09-06T11:57:08.4756571Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4756910Z 			"size": 137,
2023-09-06T11:57:08.4757242Z 			"digest": "sha256:5679b6795f95c41f08b76efcd70063a973f2361e4f7690168b078dffa21dd819"
2023-09-06T11:57:08.4757568Z 		},
2023-09-06T11:57:08.4757761Z 		{
2023-09-06T11:57:08.4758092Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4758468Z 			"size": 157396453,
2023-09-06T11:57:08.4758810Z 			"digest": "sha256:466e22347d9c8540039d3b67c566ba496b10dc0292cf23140bc906208aa7d23e"
2023-09-06T11:57:08.4759294Z 		},
2023-09-06T11:57:08.4759494Z 		{
2023-09-06T11:57:08.4759866Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4760249Z 			"size": 190,
2023-09-06T11:57:08.4760595Z 			"digest": "sha256:161ecab39a9315f990a21367e13e37ca4fde085e826c501730b5539840d4859d"
2023-09-06T11:57:08.4760945Z 		},
2023-09-06T11:57:08.4761126Z 		{
2023-09-06T11:57:08.4761463Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4761806Z 			"size": 1143,
2023-09-06T11:57:08.4762149Z 			"digest": "sha256:bf6703e5706053f56cb1910240b60435c9a7266985a56bdf1c16ae0878e1dfd7"
2023-09-06T11:57:08.4762446Z 		},
2023-09-06T11:57:08.4762652Z 		{
2023-09-06T11:57:08.4762985Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4763367Z 			"size": 699,
2023-09-06T11:57:08.4763698Z 			"digest": "sha256:03870a46e125521f9df3ee2930c8b4984dd497a7e5f074d4e33f1e65e4b1b3ec"
2023-09-06T11:57:08.4764017Z 		},
2023-09-06T11:57:08.4764210Z 		{
2023-09-06T11:57:08.4764549Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4764875Z 			"size": 136,
2023-09-06T11:57:08.4765220Z 			"digest": "sha256:d57103f8c6d07657d2a7fb7f13510388fd7963d0a03d204d506e2b7e0d4dd0a6"
2023-09-06T11:57:08.4765539Z 		},
2023-09-06T11:57:08.4765729Z 		{
2023-09-06T11:57:08.4766046Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4766381Z 			"size": 120,
2023-09-06T11:57:08.4766752Z 			"digest": "sha256:035aebedde1fe176b4f3348588e1f3acfbd167a813bc3ed08f72b06265255f1e"
2023-09-06T11:57:08.4767097Z 		},
2023-09-06T11:57:08.4767276Z 		{
2023-09-06T11:57:08.4767612Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4767953Z 			"size": 986074578,
2023-09-06T11:57:08.4768353Z 			"digest": "sha256:5ea8491b6bde0bd24b92c932a6016638552ccd1d384cc2592d295742180d8b2f"
2023-09-06T11:57:08.4768662Z 		},
2023-09-06T11:57:08.4768857Z 		{
2023-09-06T11:57:08.4769196Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4769537Z 			"size": 172,
2023-09-06T11:57:08.4769888Z 			"digest": "sha256:212b75bc4a9b47ebc7ed44324bf2ebf9967cd32271420ab73bb4f23d5773d3ed"
2023-09-06T11:57:08.4770220Z 		},
2023-09-06T11:57:08.4770415Z 		{
2023-09-06T11:57:08.4770731Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4771066Z 			"size": 1840,
2023-09-06T11:57:08.4771414Z 			"digest": "sha256:446a4d79ad326619fdfd58a0108351118602c607ac01a8f39663f22bcd03ab4f"
2023-09-06T11:57:08.4771731Z 		},
2023-09-06T11:57:08.4771909Z 		{
2023-09-06T11:57:08.4772240Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4772697Z 			"size": 7529786,
2023-09-06T11:57:08.4773056Z 			"digest": "sha256:52e739794ad976e23ca762e0fb3a814113ae74ef98edd248bf8e12c06d36eb25"
2023-09-06T11:57:08.4773366Z 		},
2023-09-06T11:57:08.4773565Z 		{
2023-09-06T11:57:08.4773904Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4774242Z 			"size": 164,
2023-09-06T11:57:08.4774578Z 			"digest": "sha256:e4877af63f8ec2721037f25eb3770ecd2bc342610482d75acec41faa40f522e7"
2023-09-06T11:57:08.4774901Z 		},
2023-09-06T11:57:08.4775094Z 		{
2023-09-06T11:57:08.4775424Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4775749Z 			"size": 7940,
2023-09-06T11:57:08.4776107Z 			"digest": "sha256:fe81fd286662acc75f945ed606480bcf48311be40bd6719bb86de4720eec4cbd"
2023-09-06T11:57:08.4776434Z 		},
2023-09-06T11:57:08.4776626Z 		{
2023-09-06T11:57:08.4776940Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4777279Z 			"size": 8075,
2023-09-06T11:57:08.4777647Z 			"digest": "sha256:ad35ee94afea921758db2b8ce9f9b09d1cd668819637151e9a8815ea9db78cb2"
2023-09-06T11:57:08.4777987Z 		},
2023-09-06T11:57:08.4778168Z 		{
2023-09-06T11:57:08.4778533Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4778991Z 			"size": 302,
2023-09-06T11:57:08.4779338Z 			"digest": "sha256:101e36e634cc62403f74ada770092d44c94772c1405d3d441b80cabc7a837861"
2023-09-06T11:57:08.4779643Z 		},
2023-09-06T11:57:08.4779837Z 		{
2023-09-06T11:57:08.4780173Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4780517Z 			"size": 7614587,
2023-09-06T11:57:08.4780863Z 			"digest": "sha256:48961d3f385f8f7c9cc560f9750d7fee6a5a476b19376b206dafff0cbbe401da"
2023-09-06T11:57:08.4781189Z 		},
2023-09-06T11:57:08.4781384Z 		{
2023-09-06T11:57:08.4781714Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4782038Z 			"size": 108,
2023-09-06T11:57:08.4782400Z 			"digest": "sha256:fe27d0a540bebc0e3687274c3c7646d3b5bf3325e6bb94647e9c544663ce0e1b"
2023-09-06T11:57:08.4782748Z 		},
2023-09-06T11:57:08.4782928Z 		{
2023-09-06T11:57:08.4783258Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4783599Z 			"size": 54145482,
2023-09-06T11:57:08.4783967Z 			"digest": "sha256:3f780253fec81b68c275ec1d59ee703f127c6c2912b03b5da26afb8ade63a2c1"
2023-09-06T11:57:08.4784285Z 		},
2023-09-06T11:57:08.4784478Z 		{
2023-09-06T11:57:08.4784810Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4785147Z 			"size": 513,
2023-09-06T11:57:08.4785484Z 			"digest": "sha256:f01e00261e76bc8555612715fe1db67ffcacf94e599710e4d941f75fbbadb61c"
2023-09-06T11:57:08.4785804Z 		},
2023-09-06T11:57:08.4785993Z 		{
2023-09-06T11:57:08.4786322Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4786649Z 			"size": 2993737168,
2023-09-06T11:57:08.4787000Z 			"digest": "sha256:422b9d97840f940a557d7917b4196dd85e0f4bad206d0a3458c5b445ced05adb"
2023-09-06T11:57:08.4787320Z 		},
2023-09-06T11:57:08.4787517Z 		{
2023-09-06T11:57:08.4787833Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-09-06T11:57:08.4788170Z 			"size": 106,
2023-09-06T11:57:08.4788555Z 			"digest": "sha256:9f7c9a5582ef648839d1029c5522d2be2a31122684541181eb904bb77055f376"
2023-09-06T11:57:08.4788870Z 		}
2023-09-06T11:57:08.4789047Z 	]
2023-09-06T11:57:08.4789525Z }
2023-09-06T11:57:08.4910902Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2023-09-06T11:57:08.4911244Z with:
2023-09-06T11:57:08.4911800Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:9dd361d1c04129f8eaa9d6b43335917800dd6d24
2023-09-06T11:57:08.4912336Z env:
2023-09-06T11:57:08.4912564Z   GIT_DEFAULT_BRANCH: main
2023-09-06T11:57:08.4912797Z ##[endgroup]
2023-09-06T11:57:08.4927439Z ##[group]Run retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2023-09-06T11:57:08.4928028Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2023-09-06T11:57:08.4928504Z # ignore output since only exit code is used for conditional
2023-09-06T11:57:08.4928895Z # only pull docker image if it's not available locally
2023-09-06T11:57:08.4929315Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2023-09-06T11:57:08.4929759Z   retry docker pull "${DOCKER_IMAGE}"
2023-09-06T11:57:08.4930017Z fi
2023-09-06T11:57:08.4948166Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-09-06T11:57:08.4948521Z env:
2023-09-06T11:57:08.4948755Z   GIT_DEFAULT_BRANCH: main
2023-09-06T11:57:08.4949798Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:9dd361d1c04129f8eaa9d6b43335917800dd6d24
2023-09-06T11:57:08.4950336Z ##[endgroup]
2023-09-06T11:57:09.0957015Z 9dd361d1c04129f8eaa9d6b43335917800dd6d24: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks
2023-09-06T11:57:09.0958124Z 99803d4b97f3: Pulling fs layer
2023-09-06T11:57:09.0958617Z 0b5414b58ed1: Pulling fs layer
2023-09-06T11:57:09.0959170Z 434ea07ac290: Pulling fs layer
2023-09-06T11:57:09.0959646Z 7bbb6f0efdde: Pulling fs layer
2023-09-06T11:57:09.0959955Z 93036f51a832: Pulling fs layer
2023-09-06T11:57:09.0960238Z d7fb9be2f79b: Pulling fs layer
2023-09-06T11:57:09.0960515Z b7deb82cec78: Pulling fs layer
2023-09-06T11:57:09.0960793Z ffe25f6234c7: Pulling fs layer
2023-09-06T11:57:09.0961046Z 9e5d1aba80d9: Pulling fs layer
2023-09-06T11:57:09.0961456Z 04430174d253: Pulling fs layer
2023-09-06T11:57:09.0961874Z 93036f51a832: Waiting
2023-09-06T11:57:09.0962284Z 845c662a61cb: Pulling fs layer
2023-09-06T11:57:09.0962707Z 8d60f017b61c: Pulling fs layer
2023-09-06T11:57:09.0963157Z 2c2cb8fdcb6b: Pulling fs layer
2023-09-06T11:57:09.0963574Z b4849290c554: Pulling fs layer
2023-09-06T11:57:09.0964039Z 9c1eadd32ad2: Pulling fs layer
2023-09-06T11:57:09.0964431Z d7fb9be2f79b: Waiting
2023-09-06T11:57:09.0964854Z daf57644add8: Pulling fs layer
2023-09-06T11:57:09.0965235Z b7deb82cec78: Waiting
2023-09-06T11:57:09.0965623Z 7c8d8836abad: Pulling fs layer
2023-09-06T11:57:09.0965989Z 9e5d1aba80d9: Waiting
2023-09-06T11:57:09.0966362Z ffe25f6234c7: Waiting
2023-09-06T11:57:09.0966737Z 580e0fea9066: Pulling fs layer
2023-09-06T11:57:09.0967116Z 845c662a61cb: Waiting
2023-09-06T11:57:09.0967472Z 95a486a5bceb: Pulling fs layer
2023-09-06T11:57:09.0967943Z 2c2cb8fdcb6b: Waiting
2023-09-06T11:57:09.0968343Z daf57644add8: Waiting
2023-09-06T11:57:09.0968700Z 7bbb6f0efdde: Waiting
2023-09-06T11:57:09.0969063Z b4849290c554: Waiting
2023-09-06T11:57:09.0969436Z 7c8d8836abad: Waiting
2023-09-06T11:57:09.0969793Z 580e0fea9066: Waiting
2023-09-06T11:57:09.0970183Z 03870a46e125: Pulling fs layer
2023-09-06T11:57:09.0970688Z 2f4342e3f8f3: Pulling fs layer
2023-09-06T11:57:09.0971158Z 95a486a5bceb: Waiting
2023-09-06T11:57:09.0971619Z aea5db51af84: Pulling fs layer
2023-09-06T11:57:09.0972094Z fe13553c5158: Pulling fs layer
2023-09-06T11:57:09.0972551Z 2f4342e3f8f3: Waiting
2023-09-06T11:57:09.0972978Z c16cf9a5795a: Pulling fs layer
2023-09-06T11:57:09.0973425Z 04430174d253: Waiting
2023-09-06T11:57:09.0973829Z 9c1eadd32ad2: Waiting
2023-09-06T11:57:09.0974298Z efb74f88ccdd: Pulling fs layer
2023-09-06T11:57:09.0974758Z aea5db51af84: Waiting
2023-09-06T11:57:09.0975683Z fe13553c5158: Waiting
2023-09-06T11:57:09.0976100Z e5fa01783c35: Pulling fs layer
2023-09-06T11:57:09.0976568Z d073ad6760c4: Pulling fs layer
2023-09-06T11:57:09.0976916Z 03870a46e125: Waiting
2023-09-06T11:57:09.0977300Z 3bc7d371ee77: Pulling fs layer
2023-09-06T11:57:09.0977689Z ba2991171fbc: Pulling fs layer
2023-09-06T11:57:09.0978235Z 9df7805d993b: Pulling fs layer
2023-09-06T11:57:09.0978635Z d980247356b0: Pulling fs layer
2023-09-06T11:57:09.0979028Z d073ad6760c4: Waiting
2023-09-06T11:57:09.0979398Z 7034cfc3cf6f: Pulling fs layer
2023-09-06T11:57:09.0980080Z 7ba56cc73c7b: Pulling fs layer
2023-09-06T11:57:09.0980461Z d980247356b0: Waiting
2023-09-06T11:57:09.0980798Z 9df7805d993b: Waiting
2023-09-06T11:57:09.0981179Z 632e958b9436: Pulling fs layer
2023-09-06T11:57:09.0982255Z ba2991171fbc: Waiting
2023-09-06T11:57:09.0985228Z 4bb4afaa2a5f: Pulling fs layer
2023-09-06T11:57:09.0985781Z f9dcab60d104: Pulling fs layer
2023-09-06T11:57:09.0986247Z 7ba56cc73c7b: Waiting
2023-09-06T11:57:09.0986699Z f9dcab60d104: Waiting
2023-09-06T11:57:09.0987120Z 632e958b9436: Waiting
2023-09-06T11:57:09.0987507Z 4bb4afaa2a5f: Waiting
2023-09-06T11:57:09.0987935Z ef954792f06f: Pulling fs layer
2023-09-06T11:57:09.0988519Z d2bb74a53419: Pulling fs layer
2023-09-06T11:57:09.0988809Z 777f0c7185cc: Pulling fs layer
2023-09-06T11:57:09.0989306Z d2bb74a53419: Waiting
2023-09-06T11:57:09.0989680Z e3a06d78c00b: Pulling fs layer
2023-09-06T11:57:09.0990080Z 777f0c7185cc: Waiting
2023-09-06T11:57:09.0990512Z a5f28815538d: Pulling fs layer
2023-09-06T11:57:09.0991037Z 3c227ecee3da: Pulling fs layer
2023-09-06T11:57:09.0991512Z 4c1167c82e96: Pulling fs layer
2023-09-06T11:57:09.0991945Z a5f28815538d: Waiting
2023-09-06T11:57:09.0992373Z e3a06d78c00b: Waiting
2023-09-06T11:57:09.0992845Z 12dc7c97f6f3: Pulling fs layer
2023-09-06T11:57:09.0993311Z 3c227ecee3da: Waiting
2023-09-06T11:57:09.0993579Z 5679b6795f95: Pulling fs layer
2023-09-06T11:57:09.0993814Z 12dc7c97f6f3: Waiting
2023-09-06T11:57:09.0994060Z 466e22347d9c: Pulling fs layer
2023-09-06T11:57:09.0994339Z 161ecab39a93: Pulling fs layer
2023-09-06T11:57:09.0994602Z bf6703e57060: Pulling fs layer
2023-09-06T11:57:09.0994851Z d57103f8c6d0: Pulling fs layer
2023-09-06T11:57:09.0995094Z 5679b6795f95: Waiting
2023-09-06T11:57:09.0995343Z 035aebedde1f: Pulling fs layer
2023-09-06T11:57:09.0995623Z 161ecab39a93: Waiting
2023-09-06T11:57:09.0995858Z 5ea8491b6bde: Pulling fs layer
2023-09-06T11:57:09.0996104Z bf6703e57060: Waiting
2023-09-06T11:57:09.0996330Z 466e22347d9c: Waiting
2023-09-06T11:57:09.0996580Z 212b75bc4a9b: Pulling fs layer
2023-09-06T11:57:09.0996813Z 5ea8491b6bde: Waiting
2023-09-06T11:57:09.0997062Z 446a4d79ad32: Pulling fs layer
2023-09-06T11:57:09.0997308Z 212b75bc4a9b: Waiting
2023-09-06T11:57:09.0997743Z 52e739794ad9: Pulling fs layer
2023-09-06T11:57:09.0998134Z 446a4d79ad32: Waiting
2023-09-06T11:57:09.0998562Z e4877af63f8e: Pulling fs layer
2023-09-06T11:57:09.0998810Z 52e739794ad9: Waiting
2023-09-06T11:57:09.0999027Z e4877af63f8e: Waiting
2023-09-06T11:57:09.0999287Z fe81fd286662: Pulling fs layer
2023-09-06T11:57:09.0999556Z ad35ee94afea: Pulling fs layer
2023-09-06T11:57:09.0999828Z 101e36e634cc: Pulling fs layer
2023-09-06T11:57:09.1000074Z 48961d3f385f: Pulling fs layer
2023-09-06T11:57:09.1000323Z ad35ee94afea: Waiting
2023-09-06T11:57:09.1000579Z fe27d0a540be: Pulling fs layer
2023-09-06T11:57:09.1000844Z 3f780253fec8: Pulling fs layer
2023-09-06T11:57:09.1001089Z f01e00261e76: Pulling fs layer
2023-09-06T11:57:09.1001331Z 3f780253fec8: Waiting
2023-09-06T11:57:09.1001562Z 48961d3f385f: Waiting
2023-09-06T11:57:09.1001798Z fe27d0a540be: Waiting
2023-09-06T11:57:09.1002031Z 422b9d97840f: Pulling fs layer
2023-09-06T11:57:09.1002302Z 9f7c9a5582ef: Pulling fs layer
2023-09-06T11:57:09.1002547Z 422b9d97840f: Waiting
2023-09-06T11:57:09.1002779Z f01e00261e76: Waiting
2023-09-06T11:57:09.7074232Z 0b5414b58ed1: Verifying Checksum
2023-09-06T11:57:09.7074844Z 0b5414b58ed1: Download complete
2023-09-06T11:57:09.9961518Z 99803d4b97f3: Verifying Checksum
2023-09-06T11:57:09.9962470Z 99803d4b97f3: Download complete
2023-09-06T11:57:10.0166552Z 7bbb6f0efdde: Verifying Checksum
2023-09-06T11:57:10.0167298Z 7bbb6f0efdde: Download complete
2023-09-06T11:57:10.2865299Z 93036f51a832: Download complete
2023-09-06T11:57:10.5201738Z 434ea07ac290: Verifying Checksum
2023-09-06T11:57:10.5202793Z 434ea07ac290: Download complete
2023-09-06T11:57:10.6531767Z b7deb82cec78: Verifying Checksum
2023-09-06T11:57:10.6532412Z b7deb82cec78: Download complete
2023-09-06T11:57:10.6981767Z 99803d4b97f3: Pull complete
2023-09-06T11:57:10.8120196Z ffe25f6234c7: Download complete
2023-09-06T11:57:10.9120399Z 0b5414b58ed1: Pull complete
2023-09-06T11:57:10.9592067Z 9e5d1aba80d9: Download complete
2023-09-06T11:57:11.3281781Z 845c662a61cb: Verifying Checksum
2023-09-06T11:57:11.3282132Z 845c662a61cb: Download complete
2023-09-06T11:57:11.6498603Z 8d60f017b61c: Verifying Checksum
2023-09-06T11:57:11.6498979Z 8d60f017b61c: Download complete
2023-09-06T11:57:11.9433380Z 434ea07ac290: Pull complete
2023-09-06T11:57:12.0012351Z 7bbb6f0efdde: Pull complete
2023-09-06T11:57:12.0594475Z 93036f51a832: Pull complete
2023-09-06T11:57:16.6257500Z 2c2cb8fdcb6b: Verifying Checksum
2023-09-06T11:57:16.6257958Z 2c2cb8fdcb6b: Download complete
2023-09-06T11:57:16.9817748Z b4849290c554: Verifying Checksum
2023-09-06T11:57:16.9818157Z b4849290c554: Download complete
2023-09-06T11:57:17.3545851Z 9c1eadd32ad2: Verifying Checksum
2023-09-06T11:57:17.3546195Z 9c1eadd32ad2: Download complete
2023-09-06T11:57:17.6760992Z daf57644add8: Download complete
2023-09-06T11:57:19.5475216Z 7c8d8836abad: Download complete
2023-09-06T11:57:19.8543759Z 580e0fea9066: Verifying Checksum
2023-09-06T11:57:19.8544214Z 580e0fea9066: Download complete
2023-09-06T11:57:20.1414252Z 95a486a5bceb: Verifying Checksum
2023-09-06T11:57:20.1414598Z 95a486a5bceb: Download complete
2023-09-06T11:57:20.4719034Z 03870a46e125: Verifying Checksum
2023-09-06T11:57:20.4719384Z 03870a46e125: Download complete
2023-09-06T11:57:48.2625578Z 04430174d253: Verifying Checksum
2023-09-06T11:57:48.2626143Z 04430174d253: Download complete
2023-09-06T11:57:48.6033451Z aea5db51af84: Verifying Checksum
2023-09-06T11:57:48.6033825Z aea5db51af84: Download complete
2023-09-06T11:57:48.8918002Z fe13553c5158: Verifying Checksum
2023-09-06T11:57:49.2061976Z c16cf9a5795a: Download complete
2023-09-06T11:57:49.5119518Z efb74f88ccdd: Verifying Checksum
2023-09-06T11:57:49.5120264Z efb74f88ccdd: Download complete
2023-09-06T11:57:51.9653722Z d073ad6760c4: Verifying Checksum
2023-09-06T11:57:51.9654076Z d073ad6760c4: Download complete
2023-09-06T11:57:52.2746839Z 3bc7d371ee77: Download complete
2023-09-06T11:57:52.5903261Z ba2991171fbc: Verifying Checksum
2023-09-06T11:57:52.5903849Z ba2991171fbc: Download complete
2023-09-06T11:57:53.0894878Z 9df7805d993b: Verifying Checksum
2023-09-06T11:57:53.0895480Z 9df7805d993b: Download complete
2023-09-06T11:57:53.3826610Z d980247356b0: Verifying Checksum
2023-09-06T11:57:53.3827084Z d980247356b0: Download complete
2023-09-06T11:57:53.7149353Z 7034cfc3cf6f: Verifying Checksum
2023-09-06T11:57:53.7149786Z 7034cfc3cf6f: Download complete
2023-09-06T11:57:59.1304909Z 2f4342e3f8f3: Verifying Checksum
2023-09-06T11:57:59.1305268Z 2f4342e3f8f3: Download complete
2023-09-06T11:57:59.4411545Z 632e958b9436: Verifying Checksum
2023-09-06T11:57:59.4411904Z 632e958b9436: Download complete
2023-09-06T11:57:59.7443778Z 4bb4afaa2a5f: Verifying Checksum
2023-09-06T11:57:59.7444316Z 4bb4afaa2a5f: Download complete
2023-09-06T11:58:00.6791952Z 7ba56cc73c7b: Verifying Checksum
2023-09-06T11:58:00.6792362Z 7ba56cc73c7b: Download complete
2023-09-06T11:58:01.0214488Z ef954792f06f: Download complete
2023-09-06T11:58:01.0258095Z f9dcab60d104: Verifying Checksum
2023-09-06T11:58:01.0258462Z f9dcab60d104: Download complete
2023-09-06T11:58:01.3403864Z d2bb74a53419: Verifying Checksum
2023-09-06T11:58:01.3404435Z d2bb74a53419: Download complete
2023-09-06T11:58:01.6522670Z e3a06d78c00b: Verifying Checksum
2023-09-06T11:58:01.6523021Z e3a06d78c00b: Download complete
2023-09-06T11:58:01.7130945Z d7fb9be2f79b: Verifying Checksum
2023-09-06T11:58:01.7131349Z d7fb9be2f79b: Download complete
2023-09-06T11:58:01.8030723Z 777f0c7185cc: Verifying Checksum
2023-09-06T11:58:01.8031260Z 777f0c7185cc: Download complete
2023-09-06T11:58:02.0431826Z 3c227ecee3da: Verifying Checksum
2023-09-06T11:58:02.0432436Z 3c227ecee3da: Download complete
2023-09-06T11:58:02.1031463Z 4c1167c82e96: Verifying Checksum
2023-09-06T11:58:02.1032005Z 4c1167c82e96: Download complete
2023-09-06T11:58:02.3595238Z 12dc7c97f6f3: Download complete
2023-09-06T11:58:02.4180064Z 5679b6795f95: Verifying Checksum
2023-09-06T11:58:02.4180645Z 5679b6795f95: Download complete
2023-09-06T11:58:02.7328094Z 161ecab39a93: Verifying Checksum
2023-09-06T11:58:02.7328726Z 161ecab39a93: Download complete
2023-09-06T11:58:02.9022497Z a5f28815538d: Verifying Checksum
2023-09-06T11:58:02.9023071Z a5f28815538d: Download complete
2023-09-06T11:58:03.0656893Z bf6703e57060: Download complete
2023-09-06T11:58:03.2025317Z d57103f8c6d0: Verifying Checksum
2023-09-06T11:58:03.3857248Z 035aebedde1f: Verifying Checksum
2023-09-06T11:58:03.3857618Z 035aebedde1f: Download complete
2023-09-06T11:58:03.7067511Z 212b75bc4a9b: Download complete
2023-09-06T11:58:04.6304521Z 52e739794ad9: Verifying Checksum
2023-09-06T11:58:04.6305089Z 52e739794ad9: Download complete
2023-09-06T11:58:04.9461026Z e4877af63f8e: Verifying Checksum
2023-09-06T11:58:04.9461559Z e4877af63f8e: Download complete
2023-09-06T11:58:05.2683496Z fe81fd286662: Verifying Checksum
2023-09-06T11:58:05.2684009Z fe81fd286662: Download complete
2023-09-06T11:58:05.6078952Z ad35ee94afea: Download complete
2023-09-06T11:58:05.9136184Z 101e36e634cc: Verifying Checksum
2023-09-06T11:58:05.9136552Z 101e36e634cc: Download complete
2023-09-06T11:58:05.9490703Z 466e22347d9c: Verifying Checksum
2023-09-06T11:58:05.9491120Z 466e22347d9c: Download complete
2023-09-06T11:58:06.2979928Z fe27d0a540be: Download complete
2023-09-06T11:58:06.5214284Z 48961d3f385f: Verifying Checksum
2023-09-06T11:58:06.5214674Z 48961d3f385f: Download complete
2023-09-06T11:58:06.9144277Z f01e00261e76: Verifying Checksum
2023-09-06T11:58:06.9144847Z f01e00261e76: Download complete
2023-09-06T11:58:07.6627796Z 3f780253fec8: Verifying Checksum
2023-09-06T11:58:07.6628241Z 3f780253fec8: Download complete
2023-09-06T11:58:07.9920251Z 9f7c9a5582ef: Download complete
2023-09-06T11:58:16.7387818Z d7fb9be2f79b: Pull complete
2023-09-06T11:58:16.7955664Z b7deb82cec78: Pull complete
2023-09-06T11:58:16.8512989Z ffe25f6234c7: Pull complete
2023-09-06T11:58:16.9094324Z 9e5d1aba80d9: Pull complete
2023-09-06T11:58:19.7392820Z 5ea8491b6bde: Verifying Checksum
2023-09-06T11:58:19.7393351Z 5ea8491b6bde: Download complete
2023-09-06T11:58:49.1567510Z 04430174d253: Pull complete
2023-09-06T11:58:49.2131389Z 845c662a61cb: Pull complete
2023-09-06T11:58:49.2654664Z 8d60f017b61c: Pull complete
2023-09-06T11:58:54.1842372Z 422b9d97840f: Verifying Checksum
2023-09-06T11:58:54.1842899Z 422b9d97840f: Download complete
2023-09-06T11:58:56.1542639Z 2c2cb8fdcb6b: Pull complete
2023-09-06T11:58:56.3124603Z b4849290c554: Pull complete
2023-09-06T11:58:56.3625813Z 9c1eadd32ad2: Pull complete
2023-09-06T11:58:56.4122240Z daf57644add8: Pull complete
2023-09-06T11:58:58.3306344Z 7c8d8836abad: Pull complete
2023-09-06T11:58:58.3812794Z 580e0fea9066: Pull complete
2023-09-06T11:58:58.4299890Z 95a486a5bceb: Pull complete
2023-09-06T11:58:58.4814979Z 03870a46e125: Pull complete
2023-09-06T11:59:34.7513624Z 2f4342e3f8f3: Pull complete
2023-09-06T11:59:34.8048538Z aea5db51af84: Pull complete
2023-09-06T11:59:34.8584137Z fe13553c5158: Pull complete
2023-09-06T11:59:34.9084115Z c16cf9a5795a: Pull complete
2023-09-06T11:59:34.9595990Z efb74f88ccdd: Pull complete
2023-09-06T11:59:35.0098202Z e5fa01783c35: Pull complete
2023-09-06T11:59:38.5383546Z d073ad6760c4: Pull complete
2023-09-06T11:59:38.5886113Z 3bc7d371ee77: Pull complete
2023-09-06T11:59:38.6373349Z ba2991171fbc: Pull complete
2023-09-06T11:59:38.7249415Z 9df7805d993b: Pull complete
2023-09-06T11:59:38.7775719Z d980247356b0: Pull complete
2023-09-06T11:59:38.8290479Z 7034cfc3cf6f: Pull complete
2023-09-06T11:59:45.3706905Z 7ba56cc73c7b: Pull complete
2023-09-06T11:59:45.4243415Z 632e958b9436: Pull complete
2023-09-06T11:59:45.4755345Z 4bb4afaa2a5f: Pull complete
2023-09-06T11:59:46.9979890Z f9dcab60d104: Pull complete
2023-09-06T11:59:47.0477628Z ef954792f06f: Pull complete
2023-09-06T11:59:47.0987797Z d2bb74a53419: Pull complete
2023-09-06T11:59:47.4175760Z 777f0c7185cc: Pull complete
2023-09-06T11:59:47.4675120Z e3a06d78c00b: Pull complete
2023-09-06T11:59:48.7615405Z a5f28815538d: Pull complete
2023-09-06T11:59:49.0318205Z 3c227ecee3da: Pull complete
2023-09-06T11:59:49.1987450Z 4c1167c82e96: Pull complete
2023-09-06T11:59:49.6046942Z 12dc7c97f6f3: Pull complete
2023-09-06T11:59:49.7543421Z 5679b6795f95: Pull complete
2023-09-06T11:59:52.8233892Z 466e22347d9c: Pull complete
2023-09-06T11:59:52.8745852Z 161ecab39a93: Pull complete
2023-09-06T11:59:52.9230281Z bf6703e57060: Pull complete
2023-09-06T11:59:53.0257634Z d57103f8c6d0: Pull complete
2023-09-06T11:59:53.0762718Z 035aebedde1f: Pull complete
2023-09-06T12:00:12.7832140Z 5ea8491b6bde: Pull complete
2023-09-06T12:00:12.8334089Z 212b75bc4a9b: Pull complete
2023-09-06T12:00:12.8835135Z 446a4d79ad32: Pull complete
2023-09-06T12:00:13.0936096Z 52e739794ad9: Pull complete
2023-09-06T12:00:13.1426822Z e4877af63f8e: Pull complete
2023-09-06T12:00:13.1909999Z fe81fd286662: Pull complete
2023-09-06T12:00:13.2402347Z ad35ee94afea: Pull complete
2023-09-06T12:00:13.2889845Z 101e36e634cc: Pull complete
2023-09-06T12:00:13.7318968Z 48961d3f385f: Pull complete
2023-09-06T12:00:13.7825727Z fe27d0a540be: Pull complete
2023-09-06T12:00:15.5647935Z 3f780253fec8: Pull complete
2023-09-06T12:00:15.6167306Z f01e00261e76: Pull complete
2023-09-06T12:00:50.0948381Z 422b9d97840f: Pull complete
2023-09-06T12:00:50.1454570Z 9f7c9a5582ef: Pull complete
2023-09-06T12:00:50.1554517Z Digest: sha256:ea376d3cb14a9400b57a5e0e5bc49830f955ac984dd12fde7eab8e45479d06ee
2023-09-06T12:00:50.1588172Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:9dd361d1c04129f8eaa9d6b43335917800dd6d24
2023-09-06T12:00:50.1612234Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:9dd361d1c04129f8eaa9d6b43335917800dd6d24
2023-09-06T12:00:50.1779428Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main
2023-09-06T12:00:50.1779764Z with:
2023-09-06T12:00:50.1779985Z   driver-version: 535.54.03
2023-09-06T12:00:50.1780245Z env:
2023-09-06T12:00:50.1780472Z   GIT_DEFAULT_BRANCH: main
2023-09-06T12:00:50.1780722Z ##[endgroup]
2023-09-06T12:00:50.1815716Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2023-09-06T12:00:50.1816017Z with:
2023-09-06T12:00:50.1816248Z   timeout_minutes: 10
2023-09-06T12:00:50.1816498Z   max_attempts: 3
2023-09-06T12:00:50.1827280Z   command: # Is it disgusting to have a full shell script here in this github action? Sure
# But is it the best way to make it so that this action relies on nothing else? Absolutely
set -eou pipefail

DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID)
DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run"
YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo"

install_nvidia_docker2_amzn2() {
    (
        set -x
        # Needed for yum-config-manager
        sudo yum install -y yum-utils
        sudo yum-config-manager --add-repo "${YUM_REPO_URL}"
        sudo yum install -y nvidia-docker2
        sudo systemctl restart docker
    )
}

install_nvidia_docker2_ubuntu20() {
    (
        set -x
        sudo apt-get install -y nvidia-docker2
        sudo systemctl restart docker
    )
}

pre_install_nvidia_driver_amzn2() {
    (
        # Purge any nvidia driver installed from RHEL repo
        sudo yum remove -y nvidia-driver-latest-dkms
    )
}

install_nvidia_driver_common() {
    (
        # Try to gather more information about the runner and its existing NVIDIA driver if any
        echo "Before installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        HAS_NVIDIA_DRIVER=0
        # Check if NVIDIA driver has already been installed
        if [ -x "$(command -v nvidia-smi)" ]; then
            set +e
            # The driver exists, check its version next. Also check only the first GPU if there are more than one of them
            # so that the same driver version is not print over multiple lines
            INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
            NVIDIA_SMI_STATUS=$?

            if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing"
            elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing"
            else
                HAS_NVIDIA_DRIVER=1
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation"
            fi
            set -e
        fi

        if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then
            # CAUTION: this may need to be updated in future
            if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then
                  sudo yum groupinstall -y "Development Tools"
                  # ensure our kernel install is the same as our underlying kernel,
                  # groupinstall "Development Tools" has a habit of mismatching kernel headers
                  sudo yum install -y "kernel-devel-uname-r == $(uname -r)"
                  sudo modprobe backlight
            fi
            sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN"

            set +e
            sudo /bin/bash /tmp/nvidia_driver -s --no-drm
            NVIDIA_INSTALLATION_STATUS=$?

            RESET_GPU=0
            if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then
                sudo cat /var/log/nvidia-installer.log
                # Fail to install NVIDIA driver, try to reset the GPU
                RESET_GPU=1
            elif [ -x "$(command -v nvidia-smi)" ]; then
                # Check again if nvidia-smi works even if the driver installation completes successfully
                INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
                NVIDIA_SMI_STATUS=$?

                if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                    RESET_GPU=1
                fi
            fi

            if [ "$RESET_GPU" -eq 1 ]; then
                NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1)
                # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this
                # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388
                for PCI_ID in $NVIDIA_DEVICES; do
                    DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable)

                    echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)"
                    # This requires sudo permission of course
                    echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset
                    sleep 1
                done
            fi

            sudo rm -fv /tmp/nvidia_driver
            set -e
        fi
    )
}

post_install_nvidia_driver_common() {
    (
        sudo modprobe nvidia || true
        echo "After installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        (
            set +e

            nvidia-smi
            # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in
            # the case where the driver has already crashed as it still can get the driver version
            # and some basic information like the bus ID.  However, the rest of the information
            # would be missing (ERR!), for example:
            #
            # +-----------------------------------------------------------------------------+
            # | NVIDIA-SMI 525.89.02    Driver Version: 525.89.02    CUDA Version: 12.0     |
            # |-------------------------------+----------------------+----------------------+
            # | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
            # | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
            # |                               |                      |               MIG M. |
            # |===============================+======================+======================|
            # |   0  ERR!                Off  | 00000000:00:1E.0 Off |                 ERR! |
            # |ERR!  ERR! ERR!    ERR! / ERR! |   4184MiB / 23028MiB |    ERR!      Default |
            # |                               |                      |                 ERR! |
            # +-------------------------------+----------------------+----------------------+
            #
            # +-----------------------------------------------------------------------------+
            # | Processes:                                                                  |
            # |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
            # |        ID   ID                                                   Usage      |
            # |=============================================================================|
            # +-----------------------------------------------------------------------------+
            #
            # This should be reported as a failure instead as it will guarantee to fail when
            # Docker tries to run with --gpus all
            #
            # So, the correct check here is to query one of the missing piece of info like
            # GPU name, so that the command can fail accordingly
            nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0
            NVIDIA_SMI_STATUS=$?

            # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285
            if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then
                echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}"
            else
                echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}"
                exit ${NVIDIA_SMI_STATUS}
            fi
            set -e
        )
    )
}

install_nvidia_driver_amzn2() {
    (
        set -x
        pre_install_nvidia_driver_amzn2
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

install_nvidia_driver_ubuntu20() {
    (
        set -x
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

echo "== Installing nvidia driver ${DRIVER_FN} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_driver_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_driver_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac

# Install container toolkit based on distribution
echo "== Installing nvidia container toolkit for ${DISTRIBUTION} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_docker2_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_docker2_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac
echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}"

2023-09-06T12:00:50.1838527Z   retry_wait_seconds: 10
2023-09-06T12:00:50.1838805Z   polling_interval_seconds: 1
2023-09-06T12:00:50.1839077Z   warning_on_retry: true
2023-09-06T12:00:50.1839336Z   continue_on_error: false
2023-09-06T12:00:50.1839558Z env:
2023-09-06T12:00:50.1839786Z   GIT_DEFAULT_BRANCH: main
2023-09-06T12:00:50.1840047Z   DRIVER_VERSION: 535.54.03
2023-09-06T12:00:50.1840290Z ##[endgroup]
2023-09-06T12:00:50.2623364Z == Installing nvidia driver NVIDIA-Linux-x86_64-535.54.03.run ==
2023-09-06T12:00:50.2624195Z + install_nvidia_driver_common
2023-09-06T12:00:50.2626596Z + echo 'Before installing NVIDIA driver'
2023-09-06T12:00:50.2630204Z Before installing NVIDIA driver
2023-09-06T12:00:50.2631286Z + lspci
2023-09-06T12:00:50.2741357Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
2023-09-06T12:00:50.2742014Z 00:01.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 03)
2023-09-06T12:00:50.2743302Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
2023-09-06T12:00:50.2744316Z 00:03.0 Non-VGA unclassified device: Red Hat, Inc. Virtio SCSI
2023-09-06T12:00:50.2746603Z 00:04.0 3D controller: NVIDIA Corporation Device 20b0 (rev a1)
2023-09-06T12:00:50.2747055Z 00:05.0 Ethernet controller: Red Hat, Inc. Virtio network device
2023-09-06T12:00:50.2747448Z 00:06.0 Unclassified device [00ff]: Red Hat, Inc. Virtio RNG
2023-09-06T12:00:50.2747742Z + lsmod
2023-09-06T12:00:50.2767749Z Module                  Size  Used by
2023-09-06T12:00:50.2768280Z ib_core               393216  0
2023-09-06T12:00:50.2768759Z btrfs                1536000  0
2023-09-06T12:00:50.2769020Z blake2b_generic        20480  0
2023-09-06T12:00:50.2769282Z xor                    24576  1 btrfs
2023-09-06T12:00:50.2771908Z zstd_compress         225280  1 btrfs
2023-09-06T12:00:50.2772358Z raid6_pq              122880  1 btrfs
2023-09-06T12:00:50.2772824Z ufs                   106496  0
2023-09-06T12:00:50.2773227Z msdos                  20480  0
2023-09-06T12:00:50.2773628Z xfs                  1753088  0
2023-09-06T12:00:50.2774057Z nvidia_modeset       1302528  0
2023-09-06T12:00:50.2774483Z veth                   32768  0
2023-09-06T12:00:50.2774996Z nvidia_uvm           1540096  0
2023-09-06T12:00:50.2775536Z nvidia              56455168  2 nvidia_uvm,nvidia_modeset
2023-09-06T12:00:50.2776010Z drm                   618496  1 nvidia
2023-09-06T12:00:50.2776426Z xt_conntrack           16384  1
2023-09-06T12:00:50.2776859Z xt_MASQUERADE          20480  1
2023-09-06T12:00:50.2777434Z xfrm_user              40960  1
2023-09-06T12:00:50.2777971Z xfrm_algo              16384  1 xfrm_user
2023-09-06T12:00:50.2778322Z xt_addrtype            16384  2
2023-09-06T12:00:50.2778582Z iptable_filter         16384  1
2023-09-06T12:00:50.2779129Z iptable_nat            16384  1
2023-09-06T12:00:50.2779514Z nf_nat                 49152  2 iptable_nat,xt_MASQUERADE
2023-09-06T12:00:50.2780031Z bpfilter               16384  0
2023-09-06T12:00:50.2780570Z br_netfilter           28672  0
2023-09-06T12:00:50.2781094Z bridge                307200  1 br_netfilter
2023-09-06T12:00:50.2781552Z stp                    16384  1 bridge
2023-09-06T12:00:50.2781828Z llc                    16384  2 bridge,stp
2023-09-06T12:00:50.2782083Z aufs                  270336  0
2023-09-06T12:00:50.2782332Z overlay               151552  0
2023-09-06T12:00:50.2782562Z nls_iso8859_1          16384  1
2023-09-06T12:00:50.2782810Z dm_multipath           40960  0
2023-09-06T12:00:50.2783080Z scsi_dh_rdac           16384  0
2023-09-06T12:00:50.2783332Z scsi_dh_emc            16384  0
2023-09-06T12:00:50.2783566Z scsi_dh_alua           20480  0
2023-09-06T12:00:50.2783812Z binfmt_misc            24576  1
2023-09-06T12:00:50.2784088Z crct10dif_pclmul       16384  1
2023-09-06T12:00:50.2784349Z crc32_pclmul           16384  0
2023-09-06T12:00:50.2784591Z ghash_clmulni_intel    16384  0
2023-09-06T12:00:50.2784854Z aesni_intel           376832  0
2023-09-06T12:00:50.2785110Z virtio_net             61440  0
2023-09-06T12:00:50.2785354Z psmouse               176128  0
2023-09-06T12:00:50.2785649Z crypto_simd            16384  1 aesni_intel
2023-09-06T12:00:50.2786148Z net_failover           20480  1 virtio_net
2023-09-06T12:00:50.2786669Z input_leds             16384  0
2023-09-06T12:00:50.2787262Z cryptd                 24576  2 crypto_simd,ghash_clmulni_intel
2023-09-06T12:00:50.2787862Z failover               16384  1 net_failover
2023-09-06T12:00:50.2788438Z serio_raw              20480  0
2023-09-06T12:00:50.2788879Z sch_fq_codel           24576  13
2023-09-06T12:00:50.2789596Z efi_pstore             16384  0
2023-09-06T12:00:50.2790108Z virtio_rng             16384  0
2023-09-06T12:00:50.2790588Z ip_tables              32768  2 iptable_filter,iptable_nat
2023-09-06T12:00:50.2791041Z x_tables               53248  6 xt_conntrack,iptable_filter,xt_addrtype,ip_tables,iptable_nat,xt_MASQUERADE
2023-09-06T12:00:50.2791419Z autofs4                49152  2
2023-09-06T12:00:50.2791885Z + modinfo nvidia
2023-09-06T12:00:50.2792451Z filename:       /lib/modules/5.15.0-1037-gcp/kernel/drivers/video/nvidia.ko
2023-09-06T12:00:50.2792802Z firmware:       nvidia/535.54.03/gsp_tu10x.bin
2023-09-06T12:00:50.2793108Z firmware:       nvidia/535.54.03/gsp_ga10x.bin
2023-09-06T12:00:50.2793471Z alias:          char-major-195-*
2023-09-06T12:00:50.2793721Z version:        535.54.03
2023-09-06T12:00:50.2793970Z supported:      external
2023-09-06T12:00:50.2794219Z license:        NVIDIA
2023-09-06T12:00:50.2794476Z srcversion:     415DC1001E7D8AD79B6207C
2023-09-06T12:00:50.2794790Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2023-09-06T12:00:50.2795098Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2023-09-06T12:00:50.2795398Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2023-09-06T12:00:50.2795658Z depends:        drm
2023-09-06T12:00:50.2795898Z retpoline:      Y
2023-09-06T12:00:50.2796130Z name:           nvidia
2023-09-06T12:00:50.2796515Z vermagic:       5.15.0-1037-gcp SMP mod_unload modversions 
2023-09-06T12:00:50.2796878Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2023-09-06T12:00:50.2797287Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2023-09-06T12:00:50.2797718Z parm:           NVreg_ResmanDebugLevel:int
2023-09-06T12:00:50.2798013Z parm:           NVreg_RmLogonRC:int
2023-09-06T12:00:50.2798301Z parm:           NVreg_ModifyDeviceFiles:int
2023-09-06T12:00:50.2798606Z parm:           NVreg_DeviceFileUID:int
2023-09-06T12:00:50.2798900Z parm:           NVreg_DeviceFileGID:int
2023-09-06T12:00:50.2799196Z parm:           NVreg_DeviceFileMode:int
2023-09-06T12:00:50.2799550Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2023-09-06T12:00:50.2799930Z parm:           NVreg_UsePageAttributeTable:int
2023-09-06T12:00:50.2800383Z parm:           NVreg_EnablePCIeGen3:int
2023-09-06T12:00:50.2800681Z parm:           NVreg_EnableMSI:int
2023-09-06T12:00:50.2800952Z parm:           NVreg_TCEBypassMode:int
2023-09-06T12:00:50.2801271Z parm:           NVreg_EnableStreamMemOPs:int
2023-09-06T12:00:50.2801637Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2023-09-06T12:00:50.2802037Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2023-09-06T12:00:50.2802407Z parm:           NVreg_EnableS0ixPowerManagement:int
2023-09-06T12:00:50.2802825Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2023-09-06T12:00:50.2803226Z parm:           NVreg_DynamicPowerManagement:int
2023-09-06T12:00:50.2803650Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2023-09-06T12:00:50.2804035Z parm:           NVreg_EnableGpuFirmware:int
2023-09-06T12:00:50.2804363Z parm:           NVreg_EnableGpuFirmwareLogs:int
2023-09-06T12:00:50.2804727Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2023-09-06T12:00:50.2805102Z parm:           NVreg_EnableUserNUMAManagement:int
2023-09-06T12:00:50.2805434Z parm:           NVreg_MemoryPoolSize:int
2023-09-06T12:00:50.2805736Z parm:           NVreg_KMallocHeapMaxSize:int
2023-09-06T12:00:50.2806065Z parm:           NVreg_VMallocHeapMaxSize:int
2023-09-06T12:00:50.2806380Z parm:           NVreg_IgnoreMMIOCheck:int
2023-09-06T12:00:50.2806664Z parm:           NVreg_NvLinkDisable:int
2023-09-06T12:00:50.2807011Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2023-09-06T12:00:50.2807368Z parm:           NVreg_RegisterPCIDriver:int
2023-09-06T12:00:50.2807733Z parm:           NVreg_EnableResizableBar:int
2023-09-06T12:00:50.2808060Z parm:           NVreg_EnableDbgBreakpoint:int
2023-09-06T12:00:50.2808382Z parm:           NVreg_RegistryDwords:charp
2023-09-06T12:00:50.2808717Z parm:           NVreg_RegistryDwordsPerDevice:charp
2023-09-06T12:00:50.2809028Z parm:           NVreg_RmMsg:charp
2023-09-06T12:00:50.2809300Z parm:           NVreg_GpuBlacklist:charp
2023-09-06T12:00:50.2809617Z parm:           NVreg_TemporaryFilePath:charp
2023-09-06T12:00:50.2809928Z parm:           NVreg_ExcludedGpus:charp
2023-09-06T12:00:50.2810232Z parm:           NVreg_DmaRemapPeerMmio:int
2023-09-06T12:00:50.2810654Z parm:           NVreg_RmNvlinkBandwidth:charp
2023-09-06T12:00:50.2810958Z parm:           rm_firmware_active:charp
2023-09-06T12:00:50.2811228Z + HAS_NVIDIA_DRIVER=0
2023-09-06T12:00:50.2811563Z ++ command -v nvidia-smi
2023-09-06T12:00:50.2811872Z + '[' -x /usr/bin/nvidia-smi ']'
2023-09-06T12:00:50.2812122Z + set +e
2023-09-06T12:00:50.2812522Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0
2023-09-06T12:00:50.3018531Z + INSTALLED_DRIVER_VERSION=535.54.03
2023-09-06T12:00:50.3018946Z + NVIDIA_SMI_STATUS=0
2023-09-06T12:00:50.3019538Z + '[' 0 -ne 0 ']'
2023-09-06T12:00:50.3019986Z + '[' 535.54.03 '!=' 535.54.03 ']'
2023-09-06T12:00:50.3020241Z + HAS_NVIDIA_DRIVER=1
2023-09-06T12:00:50.3020780Z + echo 'NVIDIA driver (535.54.03) has already been installed. Skipping NVIDIA driver installation'
2023-09-06T12:00:50.3021149Z + set -e
2023-09-06T12:00:50.3021408Z + '[' 1 -eq 0 ']'
2023-09-06T12:00:50.3021770Z NVIDIA driver (535.54.03) has already been installed. Skipping NVIDIA driver installation
2023-09-06T12:00:50.3022167Z + post_install_nvidia_driver_common
2023-09-06T12:00:50.3025290Z + sudo modprobe nvidia
2023-09-06T12:00:50.3163322Z + echo 'After installing NVIDIA driver'
2023-09-06T12:00:50.3164368Z + lspci
2023-09-06T12:00:50.3164776Z After installing NVIDIA driver
2023-09-06T12:00:50.3274030Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
2023-09-06T12:00:50.3274528Z 00:01.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 03)
2023-09-06T12:00:50.3275047Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
2023-09-06T12:00:50.3275540Z 00:03.0 Non-VGA unclassified device: Red Hat, Inc. Virtio SCSI
2023-09-06T12:00:50.3275927Z 00:04.0 3D controller: NVIDIA Corporation Device 20b0 (rev a1)
2023-09-06T12:00:50.3276591Z 00:05.0 Ethernet controller: Red Hat, Inc. Virtio network device
2023-09-06T12:00:50.3276981Z 00:06.0 Unclassified device [00ff]: Red Hat, Inc. Virtio RNG
2023-09-06T12:00:50.3277280Z + lsmod
2023-09-06T12:00:50.3299168Z Module                  Size  Used by
2023-09-06T12:00:50.3299527Z ib_core               393216  0
2023-09-06T12:00:50.3299912Z btrfs                1536000  0
2023-09-06T12:00:50.3300315Z blake2b_generic        20480  0
2023-09-06T12:00:50.3300743Z xor                    24576  1 btrfs
2023-09-06T12:00:50.3301190Z zstd_compress         225280  1 btrfs
2023-09-06T12:00:50.3301480Z raid6_pq              122880  1 btrfs
2023-09-06T12:00:50.3301854Z ufs                   106496  0
2023-09-06T12:00:50.3302396Z msdos                  20480  0
2023-09-06T12:00:50.3302862Z xfs                  1753088  0
2023-09-06T12:00:50.3303343Z nvidia_modeset       1302528  0
2023-09-06T12:00:50.3303636Z veth                   32768  0
2023-09-06T12:00:50.3303873Z nvidia_uvm           1540096  0
2023-09-06T12:00:50.3304329Z nvidia              56455168  2 nvidia_uvm,nvidia_modeset
2023-09-06T12:00:50.3304816Z drm                   618496  1 nvidia
2023-09-06T12:00:50.3305329Z xt_conntrack           16384  1
2023-09-06T12:00:50.3305809Z xt_MASQUERADE          20480  1
2023-09-06T12:00:50.3306302Z xfrm_user              40960  1
2023-09-06T12:00:50.3306686Z xfrm_algo              16384  1 xfrm_user
2023-09-06T12:00:50.3306963Z xt_addrtype            16384  2
2023-09-06T12:00:50.3307209Z iptable_filter         16384  1
2023-09-06T12:00:50.3307468Z iptable_nat            16384  1
2023-09-06T12:00:50.3307837Z nf_nat                 49152  2 iptable_nat,xt_MASQUERADE
2023-09-06T12:00:50.3308132Z bpfilter               16384  0
2023-09-06T12:00:50.3308388Z br_netfilter           28672  0
2023-09-06T12:00:50.3308693Z bridge                307200  1 br_netfilter
2023-09-06T12:00:50.3308966Z stp                    16384  1 bridge
2023-09-06T12:00:50.3309471Z llc                    16384  2 bridge,stp
2023-09-06T12:00:50.3309726Z aufs                  270336  0
2023-09-06T12:00:50.3309975Z overlay               151552  0
2023-09-06T12:00:50.3310223Z nls_iso8859_1          16384  1
2023-09-06T12:00:50.3310725Z dm_multipath           40960  0
2023-09-06T12:00:50.3310968Z scsi_dh_rdac           16384  0
2023-09-06T12:00:50.3311219Z scsi_dh_emc            16384  0
2023-09-06T12:00:50.3311468Z scsi_dh_alua           20480  0
2023-09-06T12:00:50.3311719Z binfmt_misc            24576  1
2023-09-06T12:00:50.3311962Z crct10dif_pclmul       16384  1
2023-09-06T12:00:50.3312219Z crc32_pclmul           16384  0
2023-09-06T12:00:50.3312482Z ghash_clmulni_intel    16384  0
2023-09-06T12:00:50.3312735Z aesni_intel           376832  0
2023-09-06T12:00:50.3312988Z virtio_net             61440  0
2023-09-06T12:00:50.3313238Z psmouse               176128  0
2023-09-06T12:00:50.3313509Z crypto_simd            16384  1 aesni_intel
2023-09-06T12:00:50.3313784Z net_failover           20480  1 virtio_net
2023-09-06T12:00:50.3314066Z input_leds             16384  0
2023-09-06T12:00:50.3314375Z cryptd                 24576  2 crypto_simd,ghash_clmulni_intel
2023-09-06T12:00:50.3314696Z failover               16384  1 net_failover
2023-09-06T12:00:50.3314960Z serio_raw              20480  0
2023-09-06T12:00:50.3315214Z sch_fq_codel           24576  13
2023-09-06T12:00:50.3315474Z efi_pstore             16384  0
2023-09-06T12:00:50.3315726Z virtio_rng             16384  0
2023-09-06T12:00:50.3316021Z ip_tables              32768  2 iptable_filter,iptable_nat
2023-09-06T12:00:50.3316477Z x_tables               53248  6 xt_conntrack,iptable_filter,xt_addrtype,ip_tables,iptable_nat,xt_MASQUERADE
2023-09-06T12:00:50.3317071Z autofs4                49152  2
2023-09-06T12:00:50.3317476Z + modinfo nvidia
2023-09-06T12:00:50.3318480Z filename:       /lib/modules/5.15.0-1037-gcp/kernel/drivers/video/nvidia.ko
2023-09-06T12:00:50.3319116Z firmware:       nvidia/535.54.03/gsp_tu10x.bin
2023-09-06T12:00:50.3319693Z firmware:       nvidia/535.54.03/gsp_ga10x.bin
2023-09-06T12:00:50.3320535Z alias:          char-major-195-*
2023-09-06T12:00:50.3320813Z version:        535.54.03
2023-09-06T12:00:50.3321053Z supported:      external
2023-09-06T12:00:50.3321309Z license:        NVIDIA
2023-09-06T12:00:50.3321582Z srcversion:     415DC1001E7D8AD79B6207C
2023-09-06T12:00:50.3321891Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2023-09-06T12:00:50.3322359Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2023-09-06T12:00:50.3322935Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2023-09-06T12:00:50.3323457Z depends:        drm
2023-09-06T12:00:50.3323760Z retpoline:      Y
2023-09-06T12:00:50.3324005Z name:           nvidia
2023-09-06T12:00:50.3324422Z vermagic:       5.15.0-1037-gcp SMP mod_unload modversions 
2023-09-06T12:00:50.3324794Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2023-09-06T12:00:50.3325202Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2023-09-06T12:00:50.3325560Z parm:           NVreg_ResmanDebugLevel:int
2023-09-06T12:00:50.3325861Z parm:           NVreg_RmLogonRC:int
2023-09-06T12:00:50.3326161Z parm:           NVreg_ModifyDeviceFiles:int
2023-09-06T12:00:50.3326463Z parm:           NVreg_DeviceFileUID:int
2023-09-06T12:00:50.3326748Z parm:           NVreg_DeviceFileGID:int
2023-09-06T12:00:50.3327043Z parm:           NVreg_DeviceFileMode:int
2023-09-06T12:00:50.3327407Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2023-09-06T12:00:50.3327843Z parm:           NVreg_UsePageAttributeTable:int
2023-09-06T12:00:50.3328151Z parm:           NVreg_EnablePCIeGen3:int
2023-09-06T12:00:50.3328441Z parm:           NVreg_EnableMSI:int
2023-09-06T12:00:50.3328732Z parm:           NVreg_TCEBypassMode:int
2023-09-06T12:00:50.3329048Z parm:           NVreg_EnableStreamMemOPs:int
2023-09-06T12:00:50.3329398Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2023-09-06T12:00:50.3329798Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2023-09-06T12:00:50.3330183Z parm:           NVreg_EnableS0ixPowerManagement:int
2023-09-06T12:00:50.3330599Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2023-09-06T12:00:50.3330990Z parm:           NVreg_DynamicPowerManagement:int
2023-09-06T12:00:50.3331586Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2023-09-06T12:00:50.3331992Z parm:           NVreg_EnableGpuFirmware:int
2023-09-06T12:00:50.3332326Z parm:           NVreg_EnableGpuFirmwareLogs:int
2023-09-06T12:00:50.3332675Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2023-09-06T12:00:50.3333047Z parm:           NVreg_EnableUserNUMAManagement:int
2023-09-06T12:00:50.3333375Z parm:           NVreg_MemoryPoolSize:int
2023-09-06T12:00:50.3333688Z parm:           NVreg_KMallocHeapMaxSize:int
2023-09-06T12:00:50.3334000Z parm:           NVreg_VMallocHeapMaxSize:int
2023-09-06T12:00:50.3334309Z parm:           NVreg_IgnoreMMIOCheck:int
2023-09-06T12:00:50.3334605Z parm:           NVreg_NvLinkDisable:int
2023-09-06T12:00:50.3334960Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2023-09-06T12:00:50.3335298Z parm:           NVreg_RegisterPCIDriver:int
2023-09-06T12:00:50.3335617Z parm:           NVreg_EnableResizableBar:int
2023-09-06T12:00:50.3335950Z parm:           NVreg_EnableDbgBreakpoint:int
2023-09-06T12:00:50.3336272Z parm:           NVreg_RegistryDwords:charp
2023-09-06T12:00:50.3336597Z parm:           NVreg_RegistryDwordsPerDevice:charp
2023-09-06T12:00:50.3336908Z parm:           NVreg_RmMsg:charp
2023-09-06T12:00:50.3337194Z parm:           NVreg_GpuBlacklist:charp
2023-09-06T12:00:50.3337507Z parm:           NVreg_TemporaryFilePath:charp
2023-09-06T12:00:50.3337850Z parm:           NVreg_ExcludedGpus:charp
2023-09-06T12:00:50.3338155Z parm:           NVreg_DmaRemapPeerMmio:int
2023-09-06T12:00:50.3338479Z parm:           NVreg_RmNvlinkBandwidth:charp
2023-09-06T12:00:50.3338787Z parm:           rm_firmware_active:charp
2023-09-06T12:00:50.3339043Z + set +e
2023-09-06T12:00:50.3339311Z + nvidia-smi
2023-09-06T12:00:50.3501678Z Wed Sep  6 12:00:50 2023       
2023-09-06T12:00:50.3502443Z +---------------------------------------------------------------------------------------+
2023-09-06T12:00:50.3502929Z | NVIDIA-SMI 535.54.03              Driver Version: 535.54.03    CUDA Version: 12.2     |
2023-09-06T12:00:50.3503412Z |-----------------------------------------+----------------------+----------------------+
2023-09-06T12:00:50.3503896Z | GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
2023-09-06T12:00:50.3504452Z | Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
2023-09-06T12:00:50.3504840Z |                                         |                      |               MIG M. |
2023-09-06T12:00:50.3505138Z |=========================================+======================+======================|
2023-09-06T12:00:50.3615683Z |   0  NVIDIA A100-SXM4-40GB          On  | 00000000:00:04.0 Off |                    0 |
2023-09-06T12:00:50.3616107Z | N/A   31C    P0              46W / 400W |      4MiB / 40960MiB |      0%      Default |
2023-09-06T12:00:50.3616461Z |                                         |                      |             Disabled |
2023-09-06T12:00:50.3616892Z +-----------------------------------------+----------------------+----------------------+
2023-09-06T12:00:50.3617306Z                                                                                          
2023-09-06T12:00:50.3617879Z +---------------------------------------------------------------------------------------+
2023-09-06T12:00:50.3618205Z | Processes:                                                                            |
2023-09-06T12:00:50.3618618Z |  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
2023-09-06T12:00:50.3618986Z |        ID   ID                                                             Usage      |
2023-09-06T12:00:50.3619287Z |=======================================================================================|
2023-09-06T12:00:50.3620977Z |  No running processes found                                                           |
2023-09-06T12:00:50.3621443Z +---------------------------------------------------------------------------------------+
2023-09-06T12:00:50.7348459Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0
2023-09-06T12:00:50.7515191Z NVIDIA A100-SXM4-40GB
2023-09-06T12:00:50.7547504Z + NVIDIA_SMI_STATUS=0
2023-09-06T12:00:50.7547932Z + '[' 0 -eq 0 ']'
2023-09-06T12:00:50.7548283Z + echo 'INFO: Ignoring allowed status 0'
2023-09-06T12:00:50.7548558Z + set -e
2023-09-06T12:00:50.7548835Z INFO: Ignoring allowed status 0
2023-09-06T12:00:50.7557710Z == Installing nvidia container toolkit for ubuntu20.04 ==
2023-09-06T12:00:50.7563404Z + sudo apt-get install -y nvidia-docker2
2023-09-06T12:00:50.8264920Z Reading package lists...
2023-09-06T12:00:51.0396067Z Building dependency tree...
2023-09-06T12:00:51.0401052Z Reading state information...
2023-09-06T12:00:51.2192181Z nvidia-docker2 is already the newest version (2.13.0-1).
2023-09-06T12:00:51.2192690Z The following packages were automatically installed and are no longer required:
2023-09-06T12:00:51.2193293Z   libatasmart4 libblockdev-fs2 libblockdev-loop2 libblockdev-part-err2
2023-09-06T12:00:51.2193869Z   libblockdev-part2 libblockdev-swap2 libblockdev-utils2 libblockdev2
2023-09-06T12:00:51.2194400Z   libexpat1-dev libmm-glib0 libnspr4 libnss3 libnuma1 libparted-fs-resize0
2023-09-06T12:00:51.2197685Z   libpython3-dev libpython3.8-dev libudisks2-0 usb-modeswitch
2023-09-06T12:00:51.2198109Z   usb-modeswitch-data
2023-09-06T12:00:51.2198467Z Use 'sudo apt autoremove' to remove them.
2023-09-06T12:00:51.2536279Z 0 upgraded, 0 newly installed, 0 to remove and 71 not upgraded.
2023-09-06T12:00:51.2549720Z + sudo systemctl restart docker
2023-09-06T12:01:02.2737601Z Command completed after 1 attempt(s).
2023-09-06T12:01:02.2814885Z ##[group]Run sudo nvidia-smi -pm 1
2023-09-06T12:01:02.2815231Z sudo nvidia-smi -pm 1
2023-09-06T12:01:02.2815530Z sudo nvidia-smi -ac 1215,1410
2023-09-06T12:01:02.2815783Z nvidia-smi
2023-09-06T12:01:02.2835095Z shell: /usr/bin/bash -e {0}
2023-09-06T12:01:02.2835371Z env:
2023-09-06T12:01:02.2835658Z   GIT_DEFAULT_BRANCH: main
2023-09-06T12:01:02.2835985Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T12:01:02.2836290Z ##[endgroup]
2023-09-06T12:01:02.3151199Z Persistence mode is already Enabled for GPU 00000000:00:04.0.
2023-09-06T12:01:02.3151842Z All done.
2023-09-06T12:01:02.3586948Z Applications clocks set to "(MEM 1215, SM 1410)" for GPU 00000000:00:04.0
2023-09-06T12:01:02.3587590Z All done.
2023-09-06T12:01:02.3771484Z Wed Sep  6 12:01:02 2023       
2023-09-06T12:01:02.3772589Z +---------------------------------------------------------------------------------------+
2023-09-06T12:01:02.3773364Z | NVIDIA-SMI 535.54.03              Driver Version: 535.54.03    CUDA Version: 12.2     |
2023-09-06T12:01:02.3773821Z |-----------------------------------------+----------------------+----------------------+
2023-09-06T12:01:02.3774319Z | GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
2023-09-06T12:01:02.3774909Z | Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
2023-09-06T12:01:02.3775298Z |                                         |                      |               MIG M. |
2023-09-06T12:01:02.3775607Z |=========================================+======================+======================|
2023-09-06T12:01:02.3876796Z |   0  NVIDIA A100-SXM4-40GB          On  | 00000000:00:04.0 Off |                    0 |
2023-09-06T12:01:02.3877406Z | N/A   31C    P0              46W / 400W |      4MiB / 40960MiB |      0%      Default |
2023-09-06T12:01:02.3877771Z |                                         |                      |             Disabled |
2023-09-06T12:01:02.3878443Z +-----------------------------------------+----------------------+----------------------+
2023-09-06T12:01:02.3878775Z                                                                                          
2023-09-06T12:01:02.3879165Z +---------------------------------------------------------------------------------------+
2023-09-06T12:01:02.3880019Z | Processes:                                                                            |
2023-09-06T12:01:02.3880398Z |  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
2023-09-06T12:01:02.3880767Z |        ID   ID                                                             Usage      |
2023-09-06T12:01:02.3881102Z |=======================================================================================|
2023-09-06T12:01:02.3881429Z |  No running processes found                                                           |
2023-09-06T12:01:02.3881881Z +---------------------------------------------------------------------------------------+
2023-09-06T12:01:02.7328722Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84
2023-09-06T12:01:02.7329193Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84
2023-09-06T12:01:02.7329695Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2023-09-06T12:01:02.7330295Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}"
2023-09-06T12:01:02.7348801Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-09-06T12:01:02.7349509Z env:
2023-09-06T12:01:02.7349989Z   GIT_DEFAULT_BRANCH: main
2023-09-06T12:01:02.7350407Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T12:01:02.7350787Z ##[endgroup]
2023-09-06T12:01:03.8787691Z Requirement already satisfied: psutil==5.9.1 in /home/ubuntu/.local/lib/python3.8/site-packages (5.9.1)
2023-09-06T12:01:03.8875742Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ubuntu/.local/lib/python3.8/site-packages (11.525.84)
2023-09-06T12:01:04.1016348Z Prepare all required actions
2023-09-06T12:01:04.1017168Z Getting action download info
2023-09-06T12:01:04.2800175Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6)
2023-09-06T12:01:04.8802066Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a)
2023-09-06T12:01:05.2488933Z ##[group]Run ./.github/actions/download-build-artifacts
2023-09-06T12:01:05.2489248Z with:
2023-09-06T12:01:05.2489518Z   name: linux-focal-cuda12.1-py3.10-gcc9-sm80
2023-09-06T12:01:05.2489795Z env:
2023-09-06T12:01:05.2490024Z   GIT_DEFAULT_BRANCH: main
2023-09-06T12:01:05.2490342Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T12:01:05.2490625Z ##[endgroup]
2023-09-06T12:01:05.2518744Z ##[group]Run seemethere/download-artifact-s3@v4
2023-09-06T12:01:05.2519034Z with:
2023-09-06T12:01:05.2519313Z   name: linux-focal-cuda12.1-py3.10-gcc9-sm80
2023-09-06T12:01:05.2519666Z   s3-bucket: gha-artifacts
2023-09-06T12:01:05.2519909Z   region: us-east-1
2023-09-06T12:01:05.2520128Z env:
2023-09-06T12:01:05.2520354Z   GIT_DEFAULT_BRANCH: main
2023-09-06T12:01:05.2520672Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T12:01:05.2520952Z ##[endgroup]
2023-09-06T12:01:05.8339918Z (node:1951848) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2023-09-06T12:01:05.8340380Z 
2023-09-06T12:01:05.8340572Z Please migrate your code to use AWS SDK for JavaScript (v3).
2023-09-06T12:01:05.8341023Z For more information, check the migration guide at https://a.co/7PzMCcy
2023-09-06T12:01:05.8341664Z (Use `node --trace-warnings ...` to show where the warning was created)
2023-09-06T12:01:06.0014905Z Found 1 objects with prefix pytorch/pytorch/6093795712/linux-focal-cuda12.1-py3.10-gcc9-sm80/
2023-09-06T12:01:06.0016261Z Starting download (1/1): /home/weiwangmeta/actions-runner/_work/pytorch/pytorch/artifacts.zip
2023-09-06T12:01:25.1443943Z Finished download (1/1): /home/weiwangmeta/actions-runner/_work/pytorch/pytorch/artifacts.zip
2023-09-06T12:01:25.1452174Z Artifact download has finished successfully
2023-09-06T12:01:25.1730284Z ##[group]Run unzip -o artifacts.zip
2023-09-06T12:01:25.1730602Z unzip -o artifacts.zip
2023-09-06T12:01:25.1750265Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-09-06T12:01:25.1750563Z env:
2023-09-06T12:01:25.1750800Z   GIT_DEFAULT_BRANCH: main
2023-09-06T12:01:25.1751125Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T12:01:25.1751496Z ##[endgroup]
2023-09-06T12:01:25.1807267Z Archive:  artifacts.zip
2023-09-06T12:01:25.1810313Z    creating: dist/
2023-09-06T12:01:27.3853456Z   inflating: dist/torch-2.2.0a0+git3fe8417-cp310-cp310-linux_x86_64.whl  
2023-09-06T12:01:27.3854231Z    creating: build/custom_test_artifacts/
2023-09-06T12:01:27.3854695Z    creating: build/custom_test_artifacts/custom-op-build/
2023-09-06T12:01:27.3855446Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2023-09-06T12:01:27.3857540Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/
2023-09-06T12:01:27.3860085Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml  
2023-09-06T12:01:27.3860929Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/
2023-09-06T12:01:27.3861539Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2023-09-06T12:01:27.3862130Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/
2023-09-06T12:01:27.3862992Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2023-09-06T12:01:27.3863813Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2023-09-06T12:01:27.3865389Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2023-09-06T12:01:27.3866409Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/
2023-09-06T12:01:27.3867029Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2023-09-06T12:01:27.3868258Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2023-09-06T12:01:27.3869913Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2023-09-06T12:01:27.3872265Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2023-09-06T12:01:27.3872943Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2023-09-06T12:01:27.3874273Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2023-09-06T12:01:27.3875557Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2023-09-06T12:01:27.3876192Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2023-09-06T12:01:27.3876790Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2023-09-06T12:01:27.3922657Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2023-09-06T12:01:27.3968380Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2023-09-06T12:01:27.3969176Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2023-09-06T12:01:27.4022414Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2023-09-06T12:01:27.4023874Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2023-09-06T12:01:27.4025311Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2023-09-06T12:01:27.4026590Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2023-09-06T12:01:27.4028479Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2023-09-06T12:01:27.4030879Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2023-09-06T12:01:27.4032101Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2023-09-06T12:01:27.4032885Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2023-09-06T12:01:27.4033623Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2023-09-06T12:01:27.4034326Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2023-09-06T12:01:27.4034999Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2023-09-06T12:01:27.4035677Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2023-09-06T12:01:27.4036348Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2023-09-06T12:01:27.4036988Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2023-09-06T12:01:27.4037680Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2023-09-06T12:01:27.4102938Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2023-09-06T12:01:27.4174601Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2023-09-06T12:01:27.4175355Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2023-09-06T12:01:27.4175972Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/
2023-09-06T12:01:27.4176597Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2023-09-06T12:01:27.4177166Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2023-09-06T12:01:27.4177757Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2023-09-06T12:01:27.4178355Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2023-09-06T12:01:27.4179006Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2023-09-06T12:01:27.4179641Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2023-09-06T12:01:27.4180252Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2023-09-06T12:01:27.4180878Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2023-09-06T12:01:27.4181490Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2023-09-06T12:01:27.4182115Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2023-09-06T12:01:27.4182737Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2023-09-06T12:01:27.4183444Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2023-09-06T12:01:27.4203852Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2023-09-06T12:01:27.4318248Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2023-09-06T12:01:27.4318872Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2023-09-06T12:01:27.4319519Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2023-09-06T12:01:27.4320459Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2023-09-06T12:01:27.4321117Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2023-09-06T12:01:27.4321730Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2023-09-06T12:01:27.4322371Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2023-09-06T12:01:27.4323060Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2023-09-06T12:01:27.4323719Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2023-09-06T12:01:27.4324357Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2023-09-06T12:01:27.4324976Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2023-09-06T12:01:27.4347314Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2023-09-06T12:01:27.4433810Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2023-09-06T12:01:27.4434512Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2023-09-06T12:01:27.4435163Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2023-09-06T12:01:27.4435773Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2023-09-06T12:01:27.4436557Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2023-09-06T12:01:27.4437134Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2023-09-06T12:01:27.4437696Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2023-09-06T12:01:27.4439735Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2023-09-06T12:01:27.4440624Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2023-09-06T12:01:27.4441202Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2023-09-06T12:01:27.4535499Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2023-09-06T12:01:27.4600242Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2023-09-06T12:01:27.4600739Z    creating: build/custom_test_artifacts/jit-hook-build/
2023-09-06T12:01:27.4601217Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2023-09-06T12:01:27.4601756Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/
2023-09-06T12:01:27.4609423Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml  
2023-09-06T12:01:27.4610001Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/
2023-09-06T12:01:27.4610587Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2023-09-06T12:01:27.4611151Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/
2023-09-06T12:01:27.4611729Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2023-09-06T12:01:27.4612715Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2023-09-06T12:01:27.4614597Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2023-09-06T12:01:27.4615187Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/
2023-09-06T12:01:27.4615754Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2023-09-06T12:01:27.4617510Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2023-09-06T12:01:27.4618839Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2023-09-06T12:01:27.4620728Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2023-09-06T12:01:27.4621384Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2023-09-06T12:01:27.4622636Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2023-09-06T12:01:27.4623542Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2023-09-06T12:01:27.4624163Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2023-09-06T12:01:27.4624754Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2023-09-06T12:01:27.4671399Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2023-09-06T12:01:27.4716983Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2023-09-06T12:01:27.4717753Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2023-09-06T12:01:27.4770560Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2023-09-06T12:01:27.4771325Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2023-09-06T12:01:27.4772326Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2023-09-06T12:01:27.4773184Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2023-09-06T12:01:27.4773935Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2023-09-06T12:01:27.4774694Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2023-09-06T12:01:27.4775616Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2023-09-06T12:01:27.4776462Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2023-09-06T12:01:27.4777202Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2023-09-06T12:01:27.4777893Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2023-09-06T12:01:27.4778550Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2023-09-06T12:01:27.4779190Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2023-09-06T12:01:27.4779855Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2023-09-06T12:01:27.4780499Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2023-09-06T12:01:27.4781165Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2023-09-06T12:01:27.4850612Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2023-09-06T12:01:27.4921632Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2023-09-06T12:01:27.4922311Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2023-09-06T12:01:27.4922899Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/
2023-09-06T12:01:27.4923509Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2023-09-06T12:01:27.4924314Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2023-09-06T12:01:27.4924870Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2023-09-06T12:01:27.4925502Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2023-09-06T12:01:27.4926155Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2023-09-06T12:01:27.4926788Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2023-09-06T12:01:27.4927395Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2023-09-06T12:01:27.4927999Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2023-09-06T12:01:27.4928625Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2023-09-06T12:01:27.4929261Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2023-09-06T12:01:27.4929881Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2023-09-06T12:01:27.4930499Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2023-09-06T12:01:27.4951499Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2023-09-06T12:01:27.5022573Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2023-09-06T12:01:27.5023387Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2023-09-06T12:01:27.5024024Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2023-09-06T12:01:27.5024823Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2023-09-06T12:01:27.5025405Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2023-09-06T12:01:27.5025940Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2023-09-06T12:01:27.5026494Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2023-09-06T12:01:27.5027498Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2023-09-06T12:01:27.5028155Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2023-09-06T12:01:27.5029291Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2023-09-06T12:01:27.5085054Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2023-09-06T12:01:27.5085588Z    creating: build/custom_test_artifacts/custom-backend-build/
2023-09-06T12:01:27.5086110Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2023-09-06T12:01:27.5086708Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/
2023-09-06T12:01:27.5093920Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml  
2023-09-06T12:01:27.5094521Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/
2023-09-06T12:01:27.5095124Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2023-09-06T12:01:27.5095742Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/
2023-09-06T12:01:27.5096368Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2023-09-06T12:01:27.5097131Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2023-09-06T12:01:27.5098941Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2023-09-06T12:01:27.5099869Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/
2023-09-06T12:01:27.5100487Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2023-09-06T12:01:27.5101954Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2023-09-06T12:01:27.5103437Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2023-09-06T12:01:27.5105261Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2023-09-06T12:01:27.5105960Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2023-09-06T12:01:27.5107228Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2023-09-06T12:01:27.5108031Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2023-09-06T12:01:27.5108682Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2023-09-06T12:01:27.5109437Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2023-09-06T12:01:27.5156090Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2023-09-06T12:01:27.5202180Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2023-09-06T12:01:27.5203200Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2023-09-06T12:01:27.5255979Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2023-09-06T12:01:27.5256808Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2023-09-06T12:01:27.5257616Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2023-09-06T12:01:27.5258451Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2023-09-06T12:01:27.5259213Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2023-09-06T12:01:27.5259976Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2023-09-06T12:01:27.5260742Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2023-09-06T12:01:27.5261515Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2023-09-06T12:01:27.5262274Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2023-09-06T12:01:27.5263073Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2023-09-06T12:01:27.5263813Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2023-09-06T12:01:27.5264503Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2023-09-06T12:01:27.5265200Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2023-09-06T12:01:27.5265877Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2023-09-06T12:01:27.5266583Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2023-09-06T12:01:27.5337116Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2023-09-06T12:01:27.5408243Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2023-09-06T12:01:27.5408947Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2023-09-06T12:01:27.5409569Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/
2023-09-06T12:01:27.5410153Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2023-09-06T12:01:27.5410741Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2023-09-06T12:01:27.5411347Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2023-09-06T12:01:27.5412008Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2023-09-06T12:01:27.5412706Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2023-09-06T12:01:27.5413384Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2023-09-06T12:01:27.5414087Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2023-09-06T12:01:27.5414811Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2023-09-06T12:01:27.5415475Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2023-09-06T12:01:27.5416388Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2023-09-06T12:01:27.5417061Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2023-09-06T12:01:27.5417701Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2023-09-06T12:01:27.5418969Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2023-09-06T12:01:27.5562032Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2023-09-06T12:01:27.5562691Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2023-09-06T12:01:27.5563364Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2023-09-06T12:01:27.5564128Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2023-09-06T12:01:27.5564809Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2023-09-06T12:01:27.5565472Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2023-09-06T12:01:27.5566157Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2023-09-06T12:01:27.5566839Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2023-09-06T12:01:27.5567521Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2023-09-06T12:01:27.5568181Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2023-09-06T12:01:27.5568861Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2023-09-06T12:01:27.5590081Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2023-09-06T12:01:27.5652720Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2023-09-06T12:01:27.5653785Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2023-09-06T12:01:27.5654461Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2023-09-06T12:01:27.5655076Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2023-09-06T12:01:27.5655678Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2023-09-06T12:01:27.5656304Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2023-09-06T12:01:27.5656907Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2023-09-06T12:01:27.5658910Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2023-09-06T12:01:27.5659438Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2023-09-06T12:01:27.5660089Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2023-09-06T12:01:27.5779113Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2023-09-06T12:01:27.5827193Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2023-09-06T12:01:27.5827564Z    creating: build/lib/
2023-09-06T12:01:27.5933003Z   inflating: build/lib/libprotobuf-lite.a  
2023-09-06T12:01:27.6474820Z   inflating: build/lib/libprotobuf.a  
2023-09-06T12:01:27.6483945Z   inflating: build/lib/libpthreadpool.a  
2023-09-06T12:01:27.6484679Z   inflating: build/lib/libclog.a     
2023-09-06T12:01:27.6487604Z   inflating: build/lib/libnnpack_reference_layers.a  
2023-09-06T12:01:27.6564033Z   inflating: build/lib/libgtest.a    
2023-09-06T12:01:27.6652577Z   inflating: build/lib/libbenchmark.a  
2023-09-06T12:01:27.6728193Z   inflating: build/lib/libasmjit.a   
2023-09-06T12:01:27.6736955Z   inflating: build/lib/libittnotify.a  
2023-09-06T12:01:27.6769994Z   inflating: build/lib/libtensorpipe_uv.a  
2023-09-06T12:01:27.6921013Z   inflating: build/lib/libgloo.a     
2023-09-06T12:01:27.6945844Z   inflating: build/lib/libfmt.a      
2023-09-06T12:01:27.7036689Z   inflating: build/lib/libc10.so     
2023-09-06T12:01:27.7038345Z   inflating: build/lib/libcaffe2_nvrtc.so  
2023-09-06T12:01:27.7039106Z   inflating: build/lib/libfoxi_loader.a  
2023-09-06T12:01:27.7040610Z   inflating: build/lib/libtorch_global_deps.so  
2023-09-06T12:01:27.7638962Z   inflating: build/lib/libprotoc.a   
2023-09-06T12:01:27.7647912Z   inflating: build/lib/libcpuinfo.a  
2023-09-06T12:01:27.7656702Z   inflating: build/lib/libcpuinfo_internals.a  
2023-09-06T12:01:27.7672717Z   inflating: build/lib/libqnnpack.a  
2023-09-06T12:01:27.7695419Z   inflating: build/lib/libpytorch_qnnpack.a  
2023-09-06T12:01:27.7716702Z   inflating: build/lib/libnnpack.a   
2023-09-06T12:01:27.7738125Z   inflating: build/lib/libgmock.a    
2023-09-06T12:01:27.7738476Z   inflating: build/lib/libgtest_main.a  
2023-09-06T12:01:27.7739139Z   inflating: build/lib/libbenchmark_main.a  
2023-09-06T12:01:27.8420073Z   inflating: build/lib/libtensorpipe.a  
2023-09-06T12:01:28.9036170Z   inflating: build/lib/libdnnl.a     
2023-09-06T12:01:28.9094981Z   inflating: build/lib/libc10_cuda.so  
2023-09-06T12:01:28.9095337Z   inflating: build/lib/libgmock_main.a  
2023-09-06T12:01:29.0584483Z   inflating: build/lib/libfbgemm.a   
2023-09-06T12:01:29.0887145Z   inflating: build/lib/libtensorpipe_cuda.a  
2023-09-06T12:01:29.1481182Z   inflating: build/lib/libkineto.a   
2023-09-06T12:01:29.1618421Z   inflating: build/lib/libXNNPACK.a  
2023-09-06T12:01:29.1664403Z   inflating: build/lib/libcaffe2_protos.a  
2023-09-06T12:01:29.1713800Z   inflating: build/lib/libonnx_proto.a  
2023-09-06T12:01:29.2174706Z   inflating: build/lib/libgloo_cuda.a  
2023-09-06T12:01:29.2937415Z   inflating: build/lib/libonnx.a     
2023-09-06T12:01:31.9659813Z   inflating: build/lib/libtorch_cpu.so  
2023-09-06T12:01:31.9663350Z   inflating: build/lib/libunbox_lib.a  
2023-09-06T12:01:34.1165434Z   inflating: build/lib/libtorch_cuda.so  
2023-09-06T12:01:34.1165831Z   inflating: build/lib/libtorch.so   
2023-09-06T12:01:35.1076091Z   inflating: build/lib/libtorch_cuda_linalg.so  
2023-09-06T12:01:35.1076680Z   inflating: build/lib/libc10d_cuda_test.so  
2023-09-06T12:01:35.1081295Z   inflating: build/lib/libshm.so     
2023-09-06T12:01:35.1798862Z   inflating: build/lib/libnvfuser_codegen.so  
2023-09-06T12:01:35.1859377Z   inflating: build/lib/libtorchbind_test.so  
2023-09-06T12:01:35.1883649Z   inflating: build/lib/libjitbackend_test.so  
2023-09-06T12:01:35.1913333Z   inflating: build/lib/libbackend_with_compiler.so  
2023-09-06T12:01:35.4000850Z   inflating: build/lib/libtorch_python.so  
2023-09-06T12:01:35.4040603Z   inflating: build/lib/libnnapi_backend.so  
2023-09-06T12:01:35.4040921Z    creating: build/bin/
2023-09-06T12:01:35.4100801Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2023-09-06T12:01:35.4162379Z   inflating: build/bin/c10_DeviceGuard_test  
2023-09-06T12:01:35.4222278Z   inflating: build/bin/c10_Device_test  
2023-09-06T12:01:35.4292507Z   inflating: build/bin/c10_DispatchKeySet_test  
2023-09-06T12:01:35.4352511Z   inflating: build/bin/c10_StreamGuard_test  
2023-09-06T12:01:35.4417408Z   inflating: build/bin/c10_SymInt_test  
2023-09-06T12:01:35.4482564Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2023-09-06T12:01:35.4548879Z   inflating: build/bin/c10_InlineStreamGuard_test  
2023-09-06T12:01:35.4616214Z   inflating: build/bin/c10_SizesAndStrides_test  
2023-09-06T12:01:35.4679169Z   inflating: build/bin/c10_context_test  
2023-09-06T12:01:35.4742236Z   inflating: build/bin/c10_Bitset_test  
2023-09-06T12:01:35.4800959Z   inflating: build/bin/c10_ConstexprCrc_test  
2023-09-06T12:01:35.4859871Z   inflating: build/bin/c10_DeadlockDetection_test  
2023-09-06T12:01:35.4919759Z   inflating: build/bin/c10_Half_test  
2023-09-06T12:01:35.4986319Z   inflating: build/bin/c10_LeftRight_test  
2023-09-06T12:01:35.5053655Z   inflating: build/bin/c10_Metaprogramming_test  
2023-09-06T12:01:35.5114251Z   inflating: build/bin/c10_Synchronized_test  
2023-09-06T12:01:35.5180083Z   inflating: build/bin/c10_ThreadLocal_test  
2023-09-06T12:01:35.5241453Z   inflating: build/bin/c10_TypeIndex_test  
2023-09-06T12:01:35.5302508Z   inflating: build/bin/c10_TypeList_test  
2023-09-06T12:01:35.5361440Z   inflating: build/bin/c10_TypeTraits_test  
2023-09-06T12:01:35.5423042Z   inflating: build/bin/c10_accumulate_test  
2023-09-06T12:01:35.5489239Z   inflating: build/bin/c10_bfloat16_test  
2023-09-06T12:01:35.5549266Z   inflating: build/bin/c10_bit_cast_test  
2023-09-06T12:01:35.5616797Z   inflating: build/bin/c10_complex_math_test  
2023-09-06T12:01:35.5683475Z   inflating: build/bin/c10_complex_test  
2023-09-06T12:01:35.5813842Z   inflating: build/bin/c10_either_test  
2023-09-06T12:01:35.5876845Z   inflating: build/bin/c10_exception_test  
2023-09-06T12:01:35.5936803Z   inflating: build/bin/c10_flags_test  
2023-09-06T12:01:35.6132922Z   inflating: build/bin/c10_intrusive_ptr_test  
2023-09-06T12:01:35.6194252Z   inflating: build/bin/c10_irange_test  
2023-09-06T12:01:35.6260356Z   inflating: build/bin/c10_logging_test  
2023-09-06T12:01:35.6351349Z   inflating: build/bin/c10_optional_test  
2023-09-06T12:01:35.6425921Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2023-09-06T12:01:35.6490498Z   inflating: build/bin/c10_registry_test  
2023-09-06T12:01:35.6671359Z   inflating: build/bin/c10_small_vector_test  
2023-09-06T12:01:35.6732224Z   inflating: build/bin/c10_ssize_test  
2023-09-06T12:01:35.6802100Z   inflating: build/bin/c10_string_view_test  
2023-09-06T12:01:35.6863059Z   inflating: build/bin/c10_tempfile_test  
2023-09-06T12:01:35.6929884Z   inflating: build/bin/c10_typeid_test  
2023-09-06T12:01:35.6987565Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2023-09-06T12:01:35.7521577Z   inflating: build/bin/protoc-3.13.0.0  
2023-09-06T12:01:35.8054546Z   inflating: build/bin/protoc        
2023-09-06T12:01:35.8117561Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test  
2023-09-06T12:01:35.8180707Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream  
2023-09-06T12:01:35.8243275Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device  
2023-09-06T12:01:35.8305741Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes  
2023-09-06T12:01:35.8368488Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads  
2023-09-06T12:01:35.8431504Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks  
2023-09-06T12:01:35.8494300Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block  
2023-09-06T12:01:35.8553045Z   inflating: build/bin/c10_cuda_CUDATest  
2023-09-06T12:01:35.8914080Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2023-09-06T12:01:35.9292712Z   inflating: build/bin/vec_test_all_types_AVX512  
2023-09-06T12:01:35.9689802Z   inflating: build/bin/vec_test_all_types_AVX2  
2023-09-06T12:01:35.9752607Z   inflating: build/bin/FileStoreTest  
2023-09-06T12:01:35.9819335Z   inflating: build/bin/TCPStoreTest  
2023-09-06T12:01:35.9883045Z   inflating: build/bin/HashStoreTest  
2023-09-06T12:01:35.9899427Z   inflating: build/bin/ProcessGroupMPITest  
2023-09-06T12:01:35.9963462Z   inflating: build/bin/test_edge_op_registration  
2023-09-06T12:01:35.9966470Z   inflating: build/bin/example_allreduce  
2023-09-06T12:01:36.0032409Z   inflating: build/bin/test_dist_autograd  
2023-09-06T12:01:36.0114956Z   inflating: build/bin/test_cpp_rpc  
2023-09-06T12:01:36.0117548Z   inflating: build/bin/parallel_benchmark  
2023-09-06T12:01:36.0199698Z   inflating: build/bin/test_mobile_nnc  
2023-09-06T12:01:36.0208552Z   inflating: build/bin/aot_model_compiler_test  
2023-09-06T12:01:36.0619631Z   inflating: build/bin/test_lazy     
2023-09-06T12:01:36.1993872Z   inflating: build/bin/test_api      
2023-09-06T12:01:36.2080873Z   inflating: build/bin/Dict_test     
2023-09-06T12:01:36.2142673Z   inflating: build/bin/Dimname_test  
2023-09-06T12:01:36.2219948Z   inflating: build/bin/MaybeOwned_test  
2023-09-06T12:01:36.2288292Z   inflating: build/bin/NamedTensor_test  
2023-09-06T12:01:36.2360220Z   inflating: build/bin/apply_utils_test  
2023-09-06T12:01:36.2429685Z   inflating: build/bin/atest         
2023-09-06T12:01:36.2504816Z   inflating: build/bin/basic         
2023-09-06T12:01:36.2570343Z   inflating: build/bin/broadcast_test  
2023-09-06T12:01:36.2639623Z   inflating: build/bin/cpu_generator_test  
2023-09-06T12:01:36.2702974Z   inflating: build/bin/cpu_profiling_allocator_test  
2023-09-06T12:01:36.2811120Z   inflating: build/bin/cpu_rng_test  
2023-09-06T12:01:36.2870902Z   inflating: build/bin/dispatch_key_set_test  
2023-09-06T12:01:36.2930792Z   inflating: build/bin/dlconvertor_test  
2023-09-06T12:01:36.2999798Z   inflating: build/bin/extension_backend_test  
2023-09-06T12:01:36.3065992Z   inflating: build/bin/half_test     
2023-09-06T12:01:36.3178782Z   inflating: build/bin/ivalue_test   
2023-09-06T12:01:36.3238216Z   inflating: build/bin/lazy_tensor_test  
2023-09-06T12:01:36.3303130Z   inflating: build/bin/math_kernel_test  
2023-09-06T12:01:36.3367406Z   inflating: build/bin/memory_format_test  
2023-09-06T12:01:36.3430632Z   inflating: build/bin/memory_overlapping_test  
2023-09-06T12:01:36.3493886Z   inflating: build/bin/mobile_memory_cleanup  
2023-09-06T12:01:36.3560756Z   inflating: build/bin/native_test   
2023-09-06T12:01:36.3620692Z   inflating: build/bin/operator_name_test  
2023-09-06T12:01:36.3682560Z   inflating: build/bin/operators_test  
2023-09-06T12:01:36.3744502Z   inflating: build/bin/packedtensoraccessor_test  
2023-09-06T12:01:36.3823068Z   inflating: build/bin/pow_test      
2023-09-06T12:01:36.3891375Z   inflating: build/bin/quantized_test  
2023-09-06T12:01:36.3950700Z   inflating: build/bin/reduce_ops_test  
2023-09-06T12:01:36.4012809Z   inflating: build/bin/reportMemoryUsage_test  
2023-09-06T12:01:36.4080837Z   inflating: build/bin/scalar_tensor_test  
2023-09-06T12:01:36.4149476Z   inflating: build/bin/scalar_test   
2023-09-06T12:01:36.4211366Z   inflating: build/bin/StorageUtils_test  
2023-09-06T12:01:36.4274238Z   inflating: build/bin/stride_properties_test  
2023-09-06T12:01:36.4366796Z   inflating: build/bin/tensor_iterator_test  
2023-09-06T12:01:36.4431543Z   inflating: build/bin/test_parallel  
2023-09-06T12:01:36.4434351Z   inflating: build/bin/thread_init_test  
2023-09-06T12:01:36.4500723Z   inflating: build/bin/type_ptr_test  
2023-09-06T12:01:36.4562957Z   inflating: build/bin/undefined_tensor_test  
2023-09-06T12:01:36.4635336Z   inflating: build/bin/type_test     
2023-09-06T12:01:36.4694614Z   inflating: build/bin/variant_test  
2023-09-06T12:01:36.4696191Z   inflating: build/bin/verify_api_visibility  
2023-09-06T12:01:36.4778891Z   inflating: build/bin/legacy_vmap_test  
2023-09-06T12:01:36.4840953Z   inflating: build/bin/weakref_test  
2023-09-06T12:01:36.4902166Z   inflating: build/bin/wrapdim_test  
2023-09-06T12:01:36.4961004Z   inflating: build/bin/xla_tensor_test  
2023-09-06T12:01:36.5032401Z   inflating: build/bin/IListRef_test  
2023-09-06T12:01:36.5159525Z   inflating: build/bin/List_test     
2023-09-06T12:01:36.5303727Z   inflating: build/bin/kernel_function_legacy_test  
2023-09-06T12:01:36.5381871Z   inflating: build/bin/KernelFunction_test  
2023-09-06T12:01:36.5496001Z   inflating: build/bin/kernel_function_test  
2023-09-06T12:01:36.5647530Z   inflating: build/bin/kernel_lambda_legacy_test  
2023-09-06T12:01:36.5769424Z   inflating: build/bin/kernel_lambda_test  
2023-09-06T12:01:36.5841106Z   inflating: build/bin/kernel_stackbased_test  
2023-09-06T12:01:36.5902347Z   inflating: build/bin/CppSignature_test  
2023-09-06T12:01:36.6017912Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2023-09-06T12:01:36.6075911Z   inflating: build/bin/op_allowlist_test  
2023-09-06T12:01:36.6143136Z   inflating: build/bin/backend_fallback_test  
2023-09-06T12:01:36.6497534Z   inflating: build/bin/op_registration_test  
2023-09-06T12:01:36.6566916Z   inflating: build/bin/inline_container_test  
2023-09-06T12:01:36.6630506Z   inflating: build/bin/cuda_apply_test  
2023-09-06T12:01:36.6713663Z   inflating: build/bin/cuda_atomic_ops_test  
2023-09-06T12:01:36.6778114Z   inflating: build/bin/cuda_caching_host_allocator_test  
2023-09-06T12:01:36.6861800Z   inflating: build/bin/cuda_complex_math_test  
2023-09-06T12:01:36.6932508Z   inflating: build/bin/cuda_complex_test  
2023-09-06T12:01:36.6991798Z   inflating: build/bin/cuda_device_test  
2023-09-06T12:01:36.7060727Z   inflating: build/bin/cuda_cub_test  
2023-09-06T12:01:36.7121444Z   inflating: build/bin/cuda_dlconvertor_test  
2023-09-06T12:01:36.7200785Z   inflating: build/bin/cuda_distributions_test  
2023-09-06T12:01:36.7269849Z   inflating: build/bin/cuda_generator_test  
2023-09-06T12:01:36.7328536Z   inflating: build/bin/cuda_half_test  
2023-09-06T12:01:36.7389544Z   inflating: build/bin/cuda_integer_divider_test  
2023-09-06T12:01:36.7448182Z   inflating: build/bin/cuda_optional_test  
2023-09-06T12:01:36.7510541Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2023-09-06T12:01:36.7573219Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2023-09-06T12:01:36.7646192Z   inflating: build/bin/cuda_stream_test  
2023-09-06T12:01:36.7704387Z   inflating: build/bin/cuda_cudnn_test  
2023-09-06T12:01:36.7766867Z   inflating: build/bin/cuda_vectorized_test  
2023-09-06T12:01:36.7784796Z   inflating: build/bin/tutorial_tensorexpr  
2023-09-06T12:01:36.7862817Z   inflating: build/bin/ProcessGroupGlooTest  
2023-09-06T12:01:36.7931078Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2023-09-06T12:01:36.8006085Z   inflating: build/bin/ProcessGroupNCCLTest  
2023-09-06T12:01:36.8073921Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2023-09-06T12:01:36.9063030Z   inflating: build/bin/test_tensorexpr  
2023-09-06T12:01:36.9067817Z   inflating: build/bin/torch_shm_manager  
2023-09-06T12:01:36.9745074Z   inflating: build/bin/nvfuser_tests  
2023-09-06T12:01:37.0411853Z   inflating: build/bin/test_jit      
2023-09-06T12:01:37.0434159Z   inflating: .pytorch-test-times.json  
2023-09-06T12:01:37.0970879Z   inflating: .pytorch-test-file-ratings.json  
2023-09-06T12:01:37.1005866Z ##[group]Run df -H
2023-09-06T12:01:37.1006203Z df -H
2023-09-06T12:01:37.1025141Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-09-06T12:01:37.1025442Z env:
2023-09-06T12:01:37.1025686Z   GIT_DEFAULT_BRANCH: main
2023-09-06T12:01:37.1026015Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T12:01:37.1026303Z ##[endgroup]
2023-09-06T12:01:37.1082687Z Filesystem      Size  Used Avail Use% Mounted on
2023-09-06T12:01:37.1083234Z /dev/root       1.1T  239G  827G  23% /
2023-09-06T12:01:37.1083692Z devtmpfs         45G     0   45G   0% /dev
2023-09-06T12:01:37.1084116Z tmpfs            45G     0   45G   0% /dev/shm
2023-09-06T12:01:37.1084460Z tmpfs           9.0G  1.1M  9.0G   1% /run
2023-09-06T12:01:37.1084751Z tmpfs           5.3M     0  5.3M   0% /run/lock
2023-09-06T12:01:37.1085077Z tmpfs            45G     0   45G   0% /sys/fs/cgroup
2023-09-06T12:01:37.1085828Z /dev/loop0       16M   16M     0 100% /snap/aws-cli/130
2023-09-06T12:01:37.1086286Z /dev/loop2      124M  124M     0 100% /snap/core/15511
2023-09-06T12:01:37.1086856Z /dev/loop3       59M   59M     0 100% /snap/core18/2785
2023-09-06T12:01:37.1087379Z /dev/loop5       67M   67M     0 100% /snap/core20/1974
2023-09-06T12:01:37.1087892Z /dev/loop10      97M   97M     0 100% /snap/lxd/24061
2023-09-06T12:01:37.1088330Z /dev/loop9       97M   97M     0 100% /snap/lxd/23991
2023-09-06T12:01:37.1088933Z /dev/sda15      110M  5.5M  104M   5% /boot/efi
2023-09-06T12:01:37.1090083Z /dev/loop8      366M  366M     0 100% /snap/google-cloud-sdk/374
2023-09-06T12:01:37.1090449Z /dev/loop13      67M   67M     0 100% /snap/core20/2015
2023-09-06T12:01:37.1090756Z /dev/loop6       59M   59M     0 100% /snap/core18/2790
2023-09-06T12:01:37.1091074Z /dev/loop4       43M   43M     0 100% /snap/snapd/19993
2023-09-06T12:01:37.1091533Z /dev/loop12     366M  366M     0 100% /snap/google-cloud-sdk/376
2023-09-06T12:01:37.1091876Z /dev/loop7      112M  112M     0 100% /snap/core/15925
2023-09-06T12:01:37.1092180Z /dev/loop1       43M   43M     0 100% /snap/snapd/20092
2023-09-06T12:01:37.1122719Z ##[group]Run .github/scripts/parse_ref.py
2023-09-06T12:01:37.1123034Z .github/scripts/parse_ref.py
2023-09-06T12:01:37.1140038Z shell: /usr/bin/bash -e {0}
2023-09-06T12:01:37.1140283Z env:
2023-09-06T12:01:37.1140519Z   GIT_DEFAULT_BRANCH: main
2023-09-06T12:01:37.1140835Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T12:01:37.1141134Z ##[endgroup]
2023-09-06T12:01:37.1479071Z Prepare all required actions
2023-09-06T12:01:37.1479817Z Getting action download info
2023-09-06T12:01:37.3433023Z ##[group]Run ./.github/actions/filter-test-configs
2023-09-06T12:01:37.3433300Z with:
2023-09-06T12:01:37.3433898Z   github-token: ***
2023-09-06T12:01:37.3435690Z   test-matrix: {"include": [{"config": "inductor_huggingface_perf", "shard": 1, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_huggingface_perf", "shard": 2, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_huggingface_perf", "shard": 3, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 1, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 2, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 3, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 4, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 5, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 1, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 2, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 3, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 4, "num_shards": 4, "runner": "linux.gcp.a100.large"}]}
2023-09-06T12:01:37.3437867Z env:
2023-09-06T12:01:37.3438102Z   GIT_DEFAULT_BRANCH: main
2023-09-06T12:01:37.3438422Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T12:01:37.3438721Z ##[endgroup]
2023-09-06T12:01:37.3490719Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2023-09-06T12:01:37.3491038Z with:
2023-09-06T12:01:37.3491250Z   shell: bash
2023-09-06T12:01:37.3491471Z   timeout_minutes: 10
2023-09-06T12:01:37.3491716Z   max_attempts: 5
2023-09-06T12:01:37.3491975Z   retry_wait_seconds: 30
2023-09-06T12:01:37.3492317Z   command: set -eux
python3 -m pip install requests==2.26.0 pyyaml==6.0

2023-09-06T12:01:37.3492652Z   polling_interval_seconds: 1
2023-09-06T12:01:37.3492911Z   warning_on_retry: true
2023-09-06T12:01:37.3493163Z   continue_on_error: false
2023-09-06T12:01:37.3493403Z env:
2023-09-06T12:01:37.3493614Z   GIT_DEFAULT_BRANCH: main
2023-09-06T12:01:37.3493927Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T12:01:37.3494394Z   GITHUB_TOKEN: ***
2023-09-06T12:01:37.3494629Z ##[endgroup]
2023-09-06T12:01:37.4184505Z + python3 -m pip install requests==2.26.0 pyyaml==6.0
2023-09-06T12:01:38.5615701Z Requirement already satisfied: requests==2.26.0 in /home/ubuntu/.local/lib/python3.8/site-packages (2.26.0)
2023-09-06T12:01:38.5738415Z Requirement already satisfied: pyyaml==6.0 in /home/ubuntu/.local/lib/python3.8/site-packages (6.0)
2023-09-06T12:01:38.5759704Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ubuntu/.local/lib/python3.8/site-packages (from requests==2.26.0) (2.0.12)
2023-09-06T12:01:38.5791054Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /usr/lib/python3/dist-packages (from requests==2.26.0) (2.8)
2023-09-06T12:01:38.5809671Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3/dist-packages (from requests==2.26.0) (1.25.8)
2023-09-06T12:01:38.5864899Z Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests==2.26.0) (2019.11.28)
2023-09-06T12:01:39.4189693Z Command completed after 1 attempt(s).
2023-09-06T12:01:39.4244017Z ##[group]Run set -x
2023-09-06T12:01:39.4244290Z set -x
2023-09-06T12:01:39.4244515Z 
2023-09-06T12:01:39.4244863Z # Use relative path here as this could be checked out anywhere, not necessarily
2023-09-06T12:01:39.4245231Z # in runner workspace
2023-09-06T12:01:39.4245755Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py"
2023-09-06T12:01:39.4264134Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-09-06T12:01:39.4264419Z env:
2023-09-06T12:01:39.4264664Z   GIT_DEFAULT_BRANCH: main
2023-09-06T12:01:39.4264992Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T12:01:39.4265295Z ##[endgroup]
2023-09-06T12:01:39.4304085Z + python3 /home/weiwangmeta/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py
2023-09-06T12:01:39.4557784Z ##[group]Run set -x
2023-09-06T12:01:39.4558057Z set -x
2023-09-06T12:01:39.4558278Z 
2023-09-06T12:01:39.4558707Z # TODO: This is a very hacky way to get the job name. GitHub runner has the info
2023-09-06T12:01:39.4559163Z # but doesn't expose it in anyway. The job name is part of the job message the
2023-09-06T12:01:39.4559627Z # runner receives, so it's there and printed out to the diag log. Below is the
2023-09-06T12:01:39.4560105Z # code responsible for printing it. Need to check with GitHub to see if they can
2023-09-06T12:01:39.4560529Z # expose this variable as part of GitHub context.
2023-09-06T12:01:39.4560975Z # https://github.com/actions/runner/blob/main/src/Runner.Worker/JobExtension.cs#L345
2023-09-06T12:01:39.4561418Z pushd "/home/weiwangmeta/actions-runner/_work/pytorch/../../_diag"
2023-09-06T12:01:39.4561957Z pwd
2023-09-06T12:01:39.4562170Z 
2023-09-06T12:01:39.4562491Z LOG_FILE=$(grep -l -r "3fe8417643c8d6c2b3d95552cd90321d141b5d54" *.log | tail -n 1)
2023-09-06T12:01:39.4562844Z if [ -n "${LOG_FILE}" ]; then
2023-09-06T12:01:39.4563222Z   # For some reasons, awk {print $2} on Linux and Windows (bash) work correctly while it
2023-09-06T12:01:39.4563624Z   # needs to be awk {print $3} on MacOS
2023-09-06T12:01:39.4563920Z   case ${RUNNER_OS} in
2023-09-06T12:01:39.4564169Z     macOS)
2023-09-06T12:01:39.4564545Z       JOB_NAME=$(grep -r "\"jobDisplayName\"" "${LOG_FILE}" | awk -F '[:]' '{print $3}' | sed 's/"//g' | xargs)
2023-09-06T12:01:39.4564911Z       ;;
2023-09-06T12:01:39.4565132Z     *)
2023-09-06T12:01:39.4565494Z       JOB_NAME=$(grep -r "\"jobDisplayName\"" "${LOG_FILE}" | awk -F '[:]' '{print $2}' | sed 's/"//g' | xargs)
2023-09-06T12:01:39.4565842Z       ;;
2023-09-06T12:01:39.4566060Z   esac
2023-09-06T12:01:39.4566348Z   echo "job-name=${JOB_NAME}" >> "${GITHUB_OUTPUT}"
2023-09-06T12:01:39.4566629Z fi
2023-09-06T12:01:39.4566821Z 
2023-09-06T12:01:39.4567032Z popd
2023-09-06T12:01:39.4584340Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-09-06T12:01:39.4584636Z env:
2023-09-06T12:01:39.4584873Z   GIT_DEFAULT_BRANCH: main
2023-09-06T12:01:39.4585197Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T12:01:39.4585482Z ##[endgroup]
2023-09-06T12:01:39.4621453Z + pushd /home/weiwangmeta/actions-runner/_work/pytorch/../../_diag
2023-09-06T12:01:39.4622048Z /home/weiwangmeta/actions-runner/_diag /home/weiwangmeta/actions-runner/_work/pytorch/pytorch
2023-09-06T12:01:39.4622511Z /home/weiwangmeta/actions-runner/_diag
2023-09-06T12:01:39.4622766Z + pwd
2023-09-06T12:01:39.4628834Z ++ tail -n 1
2023-09-06T12:01:39.4637421Z ++ grep -l -r 3fe8417643c8d6c2b3d95552cd90321d141b5d54 Runner_20230712-233544-utc.log Runner_20230728-044108-utc.log Runner_20230817-002357-utc.log Worker_20230807-074928-utc.log Worker_20230808-001245-utc.log Worker_20230808-082236-utc.log Worker_20230808-180931-utc.log Worker_20230808-205905-utc.log Worker_20230809-045247-utc.log Worker_20230809-055449-utc.log Worker_20230809-072732-utc.log Worker_20230809-094816-utc.log Worker_20230809-221034-utc.log Worker_20230809-230656-utc.log Worker_20230810-001744-utc.log Worker_20230810-025957-utc.log Worker_20230810-072649-utc.log Worker_20230810-130632-utc.log Worker_20230810-200840-utc.log Worker_20230811-040523-utc.log Worker_20230811-070837-utc.log Worker_20230811-103131-utc.log Worker_20230811-122424-utc.log Worker_20230812-072651-utc.log Worker_20230812-104219-utc.log Worker_20230812-172749-utc.log Worker_20230813-072704-utc.log Worker_20230813-105726-utc.log Worker_20230814-072733-utc.log Worker_20230814-200923-utc.log Worker_20230815-072808-utc.log Worker_20230815-193205-utc.log Worker_20230815-220049-utc.log Worker_20230816-072742-utc.log Worker_20230816-192814-utc.log Worker_20230816-215444-utc.log Worker_20230817-072803-utc.log Worker_20230817-195408-utc.log Worker_20230817-203320-utc.log Worker_20230818-015030-utc.log Worker_20230818-072837-utc.log Worker_20230818-155807-utc.log Worker_20230819-020519-utc.log Worker_20230819-033141-utc.log Worker_20230819-044305-utc.log Worker_20230819-072715-utc.log Worker_20230820-072708-utc.log Worker_20230821-032925-utc.log Worker_20230821-072800-utc.log Worker_20230822-072734-utc.log Worker_20230823-072717-utc.log Worker_20230824-072930-utc.log Worker_20230825-052236-utc.log Worker_20230825-084154-utc.log Worker_20230825-183119-utc.log Worker_20230825-203243-utc.log Worker_20230825-225333-utc.log Worker_20230826-072800-utc.log Worker_20230827-072801-utc.log Worker_20230828-072749-utc.log Worker_20230828-162216-utc.log Worker_20230829-023124-utc.log Worker_20230829-052444-utc.log Worker_20230829-074444-utc.log Worker_20230829-100918-utc.log Worker_20230829-121149-utc.log Worker_20230829-155512-utc.log Worker_20230829-204020-utc.log Worker_20230829-230024-utc.log Worker_20230830-011801-utc.log Worker_20230830-055427-utc.log Worker_20230830-111248-utc.log Worker_20230830-170127-utc.log Worker_20230830-225553-utc.log Worker_20230831-073216-utc.log Worker_20230831-130524-utc.log Worker_20230831-151058-utc.log Worker_20230831-184514-utc.log Worker_20230901-053519-utc.log Worker_20230901-075822-utc.log Worker_20230901-172225-utc.log Worker_20230902-010030-utc.log Worker_20230902-032054-utc.log Worker_20230902-050332-utc.log Worker_20230902-080246-utc.log Worker_20230902-113445-utc.log Worker_20230902-204932-utc.log Worker_20230903-074741-utc.log Worker_20230904-074447-utc.log Worker_20230904-112449-utc.log Worker_20230904-192229-utc.log Worker_20230905-075720-utc.log Worker_20230905-151557-utc.log Worker_20230905-182645-utc.log Worker_20230905-210803-utc.log Worker_20230906-061651-utc.log Worker_20230906-075055-utc.log Worker_20230906-105137-utc.log Worker_20230906-115310-utc.log
2023-09-06T12:01:39.5229625Z + LOG_FILE=Worker_20230906-115310-utc.log
2023-09-06T12:01:39.5230039Z + '[' -n Worker_20230906-115310-utc.log ']'
2023-09-06T12:01:39.5230315Z + case ${RUNNER_OS} in
2023-09-06T12:01:39.5235573Z ++ grep -r '"jobDisplayName"' Worker_20230906-115310-utc.log
2023-09-06T12:01:39.5236364Z ++ awk -F '[:]' '{print $2}'
2023-09-06T12:01:39.5238826Z ++ sed 's/"//g'
2023-09-06T12:01:39.5239281Z ++ xargs
2023-09-06T12:01:39.5269932Z + JOB_NAME='cuda12.1-py3.10-gcc9-sm80 / test (inductor_torchbench_perf, 1, 4, linux.gcp.a100.large),'
2023-09-06T12:01:39.5270816Z + echo 'job-name=cuda12.1-py3.10-gcc9-sm80 / test (inductor_torchbench_perf, 1, 4, linux.gcp.a100.large),'
2023-09-06T12:01:39.5271165Z + popd
2023-09-06T12:01:39.5271534Z /home/weiwangmeta/actions-runner/_work/pytorch/pytorch
2023-09-06T12:01:39.5294641Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}"
2023-09-06T12:01:39.5295022Z echo "Workflow: ${GITHUB_WORKFLOW}"
2023-09-06T12:01:39.5295334Z echo "Job name: ${JOB_NAME}"
2023-09-06T12:01:39.5295568Z 
2023-09-06T12:01:39.5295918Z # Use relative path here as this could be checked out anywhere, not necessarily
2023-09-06T12:01:39.5296296Z # in runner workspace
2023-09-06T12:01:39.5296647Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \
2023-09-06T12:01:39.5297000Z   --workflow "${GITHUB_WORKFLOW}" \
2023-09-06T12:01:39.5297291Z   --job-name "${JOB_NAME}" \
2023-09-06T12:01:39.5299199Z   --test-matrix "{"include": [{"config": "inductor_huggingface_perf", "shard": 1, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_huggingface_perf", "shard": 2, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_huggingface_perf", "shard": 3, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 1, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 2, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 3, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 4, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 5, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 1, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 2, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 3, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 4, "num_shards": 4, "runner": "linux.gcp.a100.large"}]}" \
2023-09-06T12:01:39.5301123Z   --pr-number "${PR_NUMBER}" \
2023-09-06T12:01:39.5301401Z   --tag "${TAG}" \
2023-09-06T12:01:39.5301676Z   --event-name "${EVENT_NAME}" \
2023-09-06T12:01:39.5302094Z   --schedule "${SCHEDULE}" \
2023-09-06T12:01:39.5302362Z   --branch "${HEAD_BRANCH}"
2023-09-06T12:01:39.5319833Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-09-06T12:01:39.5320141Z env:
2023-09-06T12:01:39.5320380Z   GIT_DEFAULT_BRANCH: main
2023-09-06T12:01:39.5320706Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T12:01:39.5321201Z   GITHUB_TOKEN: ***
2023-09-06T12:01:39.5321570Z   JOB_NAME: cuda12.1-py3.10-gcc9-sm80 / test (inductor_torchbench_perf, 1, 4, linux.gcp.a100.large),
2023-09-06T12:01:39.5321939Z   PR_NUMBER: 
2023-09-06T12:01:39.5322157Z   TAG: 
2023-09-06T12:01:39.5322387Z   EVENT_NAME: schedule
2023-09-06T12:01:39.5322627Z   SCHEDULE: 0 7 * * 1-6
2023-09-06T12:01:39.5322870Z   HEAD_BRANCH: 
2023-09-06T12:01:39.5323095Z ##[endgroup]
2023-09-06T12:01:39.5358067Z Workflow: inductor-A100-perf-nightly
2023-09-06T12:01:39.5358678Z Job name: cuda12.1-py3.10-gcc9-sm80 / test (inductor_torchbench_perf, 1, 4, linux.gcp.a100.large),
2023-09-06T12:01:39.9867546Z fatal: unknown commit main
2023-09-06T12:01:39.9892245Z /home/weiwangmeta/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/filter_test_configs.py:471: UserWarning: failed to get commit messages: Command '['git', 'cherry', '-v', 'main']' returned non-zero exit status 128.
2023-09-06T12:01:39.9893017Z   warnings.warn(f"failed to get commit messages: {e}")
2023-09-06T12:01:40.0027971Z ##[group]Run echo "Filtered matrix:"
2023-09-06T12:01:40.0028317Z echo "Filtered matrix:"
2023-09-06T12:01:40.0030485Z echo "{"include": [{"config": "inductor_huggingface_perf", "shard": 1, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_huggingface_perf", "shard": 2, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_huggingface_perf", "shard": 3, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 1, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 2, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 3, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 4, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 5, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 1, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 2, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 3, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 4, "num_shards": 4, "runner": "linux.gcp.a100.large"}]}"
2023-09-06T12:01:40.0032408Z 
2023-09-06T12:01:40.0032680Z echo
2023-09-06T12:01:40.0032970Z echo "Is the current job unstable? False"
2023-09-06T12:01:40.0033260Z 
2023-09-06T12:01:40.0033471Z echo
2023-09-06T12:01:40.0033734Z echo "Is keep-going label set? False"
2023-09-06T12:01:40.0034004Z 
2023-09-06T12:01:40.0034214Z echo
2023-09-06T12:01:40.0034465Z echo "Renabled issues? "
2023-09-06T12:01:40.0052886Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-09-06T12:01:40.0053190Z env:
2023-09-06T12:01:40.0053429Z   GIT_DEFAULT_BRANCH: main
2023-09-06T12:01:40.0053755Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T12:01:40.0054050Z ##[endgroup]
2023-09-06T12:01:40.0091757Z Filtered matrix:
2023-09-06T12:01:40.0095031Z {include: [{config: inductor_huggingface_perf, shard: 1, num_shards: 3, runner: linux.gcp.a100.large}, {config: inductor_huggingface_perf, shard: 2, num_shards: 3, runner: linux.gcp.a100.large}, {config: inductor_huggingface_perf, shard: 3, num_shards: 3, runner: linux.gcp.a100.large}, {config: inductor_timm_perf, shard: 1, num_shards: 5, runner: linux.gcp.a100.large}, {config: inductor_timm_perf, shard: 2, num_shards: 5, runner: linux.gcp.a100.large}, {config: inductor_timm_perf, shard: 3, num_shards: 5, runner: linux.gcp.a100.large}, {config: inductor_timm_perf, shard: 4, num_shards: 5, runner: linux.gcp.a100.large}, {config: inductor_timm_perf, shard: 5, num_shards: 5, runner: linux.gcp.a100.large}, {config: inductor_torchbench_perf, shard: 1, num_shards: 4, runner: linux.gcp.a100.large}, {config: inductor_torchbench_perf, shard: 2, num_shards: 4, runner: linux.gcp.a100.large}, {config: inductor_torchbench_perf, shard: 3, num_shards: 4, runner: linux.gcp.a100.large}, {config: inductor_torchbench_perf, shard: 4, num_shards: 4, runner: linux.gcp.a100.large}]}
2023-09-06T12:01:40.0097141Z 
2023-09-06T12:01:40.0097278Z Is the current job unstable? False
2023-09-06T12:01:40.0097463Z 
2023-09-06T12:01:40.0097787Z Is keep-going label set? False
2023-09-06T12:01:40.0097949Z 
2023-09-06T12:01:40.0098050Z Renabled issues? 
2023-09-06T12:01:40.0129575Z Prepare all required actions
2023-09-06T12:01:40.0129931Z Getting action download info
2023-09-06T12:01:40.2425985Z ##[group]Run ./.github/actions/pytest-cache-download
2023-09-06T12:01:40.2426290Z with:
2023-09-06T12:01:40.2426511Z   cache_dir: .pytest_cache
2023-09-06T12:01:40.2426948Z   job_identifier: inductor-A100-perf-nightly_linux-focal-cuda12.1-py3.10-gcc9-sm80_test_inductor_torchbench_perf
2023-09-06T12:01:40.2427343Z env:
2023-09-06T12:01:40.2427567Z   GIT_DEFAULT_BRANCH: main
2023-09-06T12:01:40.2427873Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T12:01:40.2428165Z ##[endgroup]
2023-09-06T12:01:40.2460006Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2023-09-06T12:01:40.2460327Z with:
2023-09-06T12:01:40.2460538Z   shell: bash
2023-09-06T12:01:40.2460770Z   timeout_minutes: 5
2023-09-06T12:01:40.2460996Z   max_attempts: 5
2023-09-06T12:01:40.2461239Z   retry_wait_seconds: 30
2023-09-06T12:01:40.2461612Z   command: set -eu
python3 -m pip install boto3==1.19.12

2023-09-06T12:01:40.2461939Z   polling_interval_seconds: 1
2023-09-06T12:01:40.2462199Z   warning_on_retry: true
2023-09-06T12:01:40.2462453Z   continue_on_error: false
2023-09-06T12:01:40.2462808Z env:
2023-09-06T12:01:40.2463022Z   GIT_DEFAULT_BRANCH: main
2023-09-06T12:01:40.2463334Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T12:01:40.2463629Z ##[endgroup]
2023-09-06T12:01:41.4375717Z Requirement already satisfied: boto3==1.19.12 in /home/ubuntu/.local/lib/python3.8/site-packages (1.19.12)
2023-09-06T12:01:41.4435697Z Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/ubuntu/.local/lib/python3.8/site-packages (from boto3==1.19.12) (0.10.0)
2023-09-06T12:01:41.4456033Z Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /home/ubuntu/.local/lib/python3.8/site-packages (from boto3==1.19.12) (0.5.2)
2023-09-06T12:01:41.4489050Z Requirement already satisfied: botocore<1.23.0,>=1.22.12 in /home/ubuntu/.local/lib/python3.8/site-packages (from boto3==1.19.12) (1.22.12)
2023-09-06T12:01:41.4538379Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ubuntu/.local/lib/python3.8/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (2.8.2)
2023-09-06T12:01:41.4563402Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3/dist-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.25.8)
2023-09-06T12:01:41.4621400Z Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.14.0)
2023-09-06T12:01:42.3118035Z Command completed after 1 attempt(s).
2023-09-06T12:01:42.3171373Z ##[group]Run python3 .github/scripts/pytest_cache.py \
2023-09-06T12:01:42.3171778Z python3 .github/scripts/pytest_cache.py \
2023-09-06T12:01:42.3172078Z   --download \
2023-09-06T12:01:42.3172380Z   --cache_dir $GITHUB_WORKSPACE/$CACHE_DIR \
2023-09-06T12:01:42.3172781Z   --pr_identifier $GITHUB_REF \
2023-09-06T12:01:42.3173311Z   --job_identifier $JOB_IDENTIFIER \
2023-09-06T12:01:42.3173611Z   --temp_dir $RUNNER_TEMP \
2023-09-06T12:01:42.3173880Z   --repo $REPO \
2023-09-06T12:01:42.3192323Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-09-06T12:01:42.3192684Z env:
2023-09-06T12:01:42.3192911Z   GIT_DEFAULT_BRANCH: main
2023-09-06T12:01:42.3193234Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T12:01:42.3193547Z   CACHE_DIR: .pytest_cache
2023-09-06T12:01:42.3193989Z   JOB_IDENTIFIER: inductor-A100-perf-nightly_linux-focal-cuda12.1-py3.10-gcc9-sm80_test_inductor_torchbench_perf
2023-09-06T12:01:42.3194404Z   REPO: pytorch/pytorch
2023-09-06T12:01:42.3194644Z ##[endgroup]
2023-09-06T12:01:42.7307322Z PR identifier for `refs/heads/main` is `96e092540d6b3c4076e3d2bc6f1f9013`
2023-09-06T12:01:42.7309709Z Downloading cache with args Namespace(bucket=None, cache_dir='/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.pytest_cache', download=True, job_identifier='inductor-A100-perf-nightly_linux-focal-cuda12.1-py3.10-gcc9-sm80_test_inductor_torchbench_perf', pr_identifier='refs/heads/main', repo='pytorch/pytorch', shard=None, temp_dir='/home/weiwangmeta/actions-runner/_work/_temp', upload=False)
2023-09-06T12:01:42.7311211Z There were no files matching the prefix `pytest_cache/pytorch/pytorch/96e092540d6b3c4076e3d2bc6f1f9013/inductor-A100-perf-nightly_linux-focal-cuda12_1-py3_10-gcc9-sm80_test_inductor_torchbench_perf` in bucket `gha-artifacts`
2023-09-06T12:01:42.7815950Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2023-09-06T12:01:42.7816371Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2023-09-06T12:01:42.7834677Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-09-06T12:01:42.7834983Z env:
2023-09-06T12:01:42.7835206Z   GIT_DEFAULT_BRANCH: main
2023-09-06T12:01:42.7835526Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T12:01:42.7835826Z   JOB_TIMEOUT: 720
2023-09-06T12:01:42.7836075Z ##[endgroup]
2023-09-06T12:01:42.7930270Z ##[group]Run set -x
2023-09-06T12:01:42.7930611Z set -x
2023-09-06T12:01:42.7930834Z 
2023-09-06T12:01:42.7931105Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2023-09-06T12:01:42.7931451Z   TEST_COMMAND=.ci/pytorch/multigpu-test.sh
2023-09-06T12:01:42.7931785Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2023-09-06T12:01:42.7932110Z   TEST_COMMAND=.ci/onnx/test.sh
2023-09-06T12:01:42.7932371Z else
2023-09-06T12:01:42.7932631Z   TEST_COMMAND=.ci/pytorch/test.sh
2023-09-06T12:01:42.7932876Z fi
2023-09-06T12:01:42.7933169Z 
2023-09-06T12:01:42.7933492Z # detached container should get cleaned up by teardown_ec2_linux
2023-09-06T12:01:42.7933917Z # TODO: Stop building test binaries as part of the build phase
2023-09-06T12:01:42.7934293Z # Used for GPU_FLAG since that doesn't play nice
2023-09-06T12:01:42.7934694Z # shellcheck disable=SC2086,SC2090
2023-09-06T12:01:42.7934999Z container_name=$(docker run \
2023-09-06T12:01:42.7935258Z   ${GPU_FLAG:-} \
2023-09-06T12:01:42.7935524Z   -e BUILD_ENVIRONMENT \
2023-09-06T12:01:42.7935795Z   -e PR_NUMBER \
2023-09-06T12:01:42.7936054Z   -e GITHUB_ACTIONS \
2023-09-06T12:01:42.7936312Z   -e GITHUB_REPOSITORY \
2023-09-06T12:01:42.7936587Z   -e GITHUB_WORKFLOW \
2023-09-06T12:01:42.7936845Z   -e GITHUB_JOB \
2023-09-06T12:01:42.7937094Z   -e GITHUB_RUN_ID \
2023-09-06T12:01:42.7937344Z   -e GITHUB_RUN_NUMBER \
2023-09-06T12:01:42.7937617Z   -e GITHUB_RUN_ATTEMPT \
2023-09-06T12:01:42.7937878Z   -e BASE_SHA \
2023-09-06T12:01:42.7938119Z   -e BRANCH \
2023-09-06T12:01:42.7938344Z   -e SHA1 \
2023-09-06T12:01:42.7938596Z   -e AWS_DEFAULT_REGION \
2023-09-06T12:01:42.7938862Z   -e IN_WHEEL_TEST \
2023-09-06T12:01:42.7939324Z   -e SHARD_NUMBER \
2023-09-06T12:01:42.7939579Z   -e TEST_CONFIG \
2023-09-06T12:01:42.7939840Z   -e NUM_TEST_SHARDS \
2023-09-06T12:01:42.7940106Z   -e REENABLED_ISSUES \
2023-09-06T12:01:42.7940377Z   -e CONTINUE_THROUGH_ERROR \
2023-09-06T12:01:42.7940674Z   -e PYTORCH_RETRY_TEST_CASES \
2023-09-06T12:01:42.7940983Z   -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \
2023-09-06T12:01:42.7941266Z   -e PR_LABELS \
2023-09-06T12:01:42.7941547Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2023-09-06T12:01:42.7941819Z   -e SCCACHE_BUCKET \
2023-09-06T12:01:42.7942091Z   -e SCCACHE_S3_KEY_PREFIX \
2023-09-06T12:01:42.7942354Z   -e XLA_CUDA \
2023-09-06T12:01:42.7942630Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2023-09-06T12:01:42.7942932Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2023-09-06T12:01:42.7943317Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2023-09-06T12:01:42.7943655Z   -e SKIP_SCCACHE_INITIALIZATION=1 \
2023-09-06T12:01:42.7943960Z   -e HUGGING_FACE_HUB_TOKEN \
2023-09-06T12:01:42.7944219Z   -e DASHBOARD_TAG \
2023-09-06T12:01:42.7944528Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2023-09-06T12:01:42.7944839Z   --ulimit stack=10485760:83886080 \
2023-09-06T12:01:42.7945144Z   --security-opt seccomp=unconfined \
2023-09-06T12:01:42.7945426Z   --cap-add=SYS_PTRACE \
2023-09-06T12:01:42.7945679Z   --ipc=host \
2023-09-06T12:01:42.7945936Z   --shm-size="${SHM_SIZE}" \
2023-09-06T12:01:42.7946183Z   --tty \
2023-09-06T12:01:42.7946402Z   --detach \
2023-09-06T12:01:42.7946692Z   --name="${container_name}" \
2023-09-06T12:01:42.7946961Z   --user jenkins \
2023-09-06T12:01:42.7947281Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2023-09-06T12:01:42.7947605Z   -w /var/lib/jenkins/workspace \
2023-09-06T12:01:42.7948025Z   "${DOCKER_IMAGE}"
2023-09-06T12:01:42.7948263Z )
2023-09-06T12:01:42.7948564Z # Propagate download.pytorch.org IP to container
2023-09-06T12:01:42.7949044Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts"
2023-09-06T12:01:42.7949728Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}"
2023-09-06T12:01:42.7950189Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"
2023-09-06T12:01:42.7967084Z shell: /usr/bin/bash -e {0}
2023-09-06T12:01:42.7967329Z env:
2023-09-06T12:01:42.7967547Z   GIT_DEFAULT_BRANCH: main
2023-09-06T12:01:42.7967873Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T12:01:42.7968266Z   BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm80
2023-09-06T12:01:42.7968579Z   PR_NUMBER: 
2023-09-06T12:01:42.7968829Z   GITHUB_REPOSITORY: pytorch/pytorch
2023-09-06T12:01:42.7969185Z   GITHUB_WORKFLOW: inductor-A100-perf-nightly
2023-09-06T12:01:42.7969477Z   GITHUB_JOB: test
2023-09-06T12:01:42.7969721Z   GITHUB_RUN_ID: 6093795712
2023-09-06T12:01:42.7969973Z   GITHUB_RUN_NUMBER: 1033
2023-09-06T12:01:42.7970228Z   GITHUB_RUN_ATTEMPT: 1
2023-09-06T12:01:42.7970470Z   BRANCH: main
2023-09-06T12:01:42.7970739Z   SHA1: 3fe8417643c8d6c2b3d95552cd90321d141b5d54
2023-09-06T12:01:42.7971045Z   BASE_SHA: 3fe8417643c8d6c2b3d95552cd90321d141b5d54
2023-09-06T12:01:42.7971341Z   PYTORCH_RETRY_TEST_CASES: 1
2023-09-06T12:01:42.7971626Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2023-09-06T12:01:42.7971964Z   TEST_CONFIG: inductor_torchbench_perf
2023-09-06T12:01:42.7972223Z   SHARD_NUMBER: 1
2023-09-06T12:01:42.7972456Z   NUM_TEST_SHARDS: 4
2023-09-06T12:01:42.7972699Z   REENABLED_ISSUES: 
2023-09-06T12:01:42.7972959Z   CONTINUE_THROUGH_ERROR: False
2023-09-06T12:01:42.7973319Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2023-09-06T12:01:42.7973905Z   SCCACHE_S3_KEY_PREFIX: inductor-A100-perf-nightly
2023-09-06T12:01:42.7974197Z   SHM_SIZE: 2g
2023-09-06T12:01:42.7974750Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:9dd361d1c04129f8eaa9d6b43335917800dd6d24
2023-09-06T12:01:42.7975285Z   XLA_CUDA: 
2023-09-06T12:01:42.7975621Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2023-09-06T12:01:42.7976002Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2023-09-06T12:01:42.7976298Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2023-09-06T12:01:42.7977032Z   DASHBOARD_TAG: training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true
2023-09-06T12:01:42.7977855Z   HUGGING_FACE_HUB_TOKEN: ***
2023-09-06T12:01:42.7978093Z ##[endgroup]
2023-09-06T12:01:42.8014374Z + [[ inductor_torchbench_perf == \m\u\l\t\i\g\p\u ]]
2023-09-06T12:01:42.8015090Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *onnx* ]]
2023-09-06T12:01:42.8015454Z + TEST_COMMAND=.ci/pytorch/test.sh
2023-09-06T12:01:42.8026799Z +++ nproc --ignore=2
2023-09-06T12:01:42.8040750Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=10 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_6093795712 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/weiwangmeta/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:9dd361d1c04129f8eaa9d6b43335917800dd6d24
2023-09-06T12:01:46.3651874Z + container_name=36c47af667b537196146162b92493090315b5cd7ff3f07f8f64ee4bb99365a29
2023-09-06T12:01:46.3653619Z + grep download.pytorch.org /etc/hosts
2023-09-06T12:01:46.3656686Z + docker exec -i 36c47af667b537196146162b92493090315b5cd7ff3f07f8f64ee4bb99365a29 sudo bash -c '/bin/cat >> /etc/hosts'
2023-09-06T12:01:46.4542134Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted
2023-09-06T12:01:46.4603493Z + echo DOCKER_CONTAINER_ID=36c47af667b537196146162b92493090315b5cd7ff3f07f8f64ee4bb99365a29
2023-09-06T12:01:46.4609558Z ++ echo dist/torch-2.2.0a0+git3fe8417-cp310-cp310-linux_x86_64.whl
2023-09-06T12:01:46.4612291Z + docker exec -t 36c47af667b537196146162b92493090315b5cd7ff3f07f8f64ee4bb99365a29 sh -c 'pip install dist/torch-2.2.0a0+git3fe8417-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh'
2023-09-06T12:01:46.9672118Z Processing ./dist/torch-2.2.0a0+git3fe8417-cp310-cp310-linux_x86_64.whl
2023-09-06T12:01:47.9578627Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.2.0a0+git3fe8417) (3.9.0)
2023-09-06T12:01:47.9579516Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.2.0a0+git3fe8417) (4.7.1)
2023-09-06T12:01:47.9585070Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.2.0a0+git3fe8417) (1.12)
2023-09-06T12:01:47.9590499Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.2.0a0+git3fe8417) (2.8.8)
2023-09-06T12:01:47.9594039Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.2.0a0+git3fe8417) (3.1.2)
2023-09-06T12:01:47.9598524Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.2.0a0+git3fe8417) (2023.4.0)
2023-09-06T12:01:47.9610940Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.2.0a0+git3fe8417) (3.3.0)
2023-09-06T12:01:47.9682255Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.2.0a0+git3fe8417) (1.21.2)
2023-09-06T12:01:48.0159084Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.2.0a0+git3fe8417) (2.1.3)
2023-09-06T12:01:48.0346943Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.2.0a0+git3fe8417) (1.3.0)
2023-09-06T12:01:49.4466025Z Installing collected packages: torch
2023-09-06T12:01:59.6615211Z Successfully installed torch-2.2.0a0+git3fe8417
2023-09-06T12:01:59.7491696Z + echo 'Environment variables:'
2023-09-06T12:01:59.7492093Z Environment variables:
2023-09-06T12:01:59.7492562Z + env
2023-09-06T12:01:59.7501432Z INSTALLED_DB=yes
2023-09-06T12:01:59.7502105Z NV_LIBCUBLAS_VERSION=12.1.3.1-1
2023-09-06T12:01:59.7502546Z NVIDIA_VISIBLE_DEVICES=all
2023-09-06T12:01:59.7503032Z NV_NVML_DEV_VERSION=12.1.105-1
2023-09-06T12:01:59.7503925Z GITHUB_WORKSPACE=/home/weiwangmeta/actions-runner/_work/pytorch/pytorch
2023-09-06T12:01:59.7504438Z CONTINUE_THROUGH_ERROR=False
2023-09-06T12:01:59.7505388Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1
2023-09-06T12:01:59.7506034Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1
2023-09-06T12:01:59.7506771Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm80
2023-09-06T12:01:59.7507310Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2023-09-06T12:01:59.7507789Z HOSTNAME=36c47af667b5
2023-09-06T12:01:59.7508793Z GITHUB_PATH=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/add_path_8d26c93d-f8fc-4def-a019-0a44d1061b4c
2023-09-06T12:01:59.7510175Z GITHUB_ACTION=__self
2023-09-06T12:01:59.7510649Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2023-09-06T12:01:59.7514779Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=510,driver<511 brand=unknown,driver>=510,driver<511 brand=nvidia,driver>=510,driver<511 brand=nvidiartx,driver>=510,driver<511 brand=geforce,driver>=510,driver<511 brand=geforcertx,driver>=510,driver<511 brand=quadro,driver>=510,driver<511 brand=quadrortx,driver>=510,driver<511 brand=titan,driver>=510,driver<511 brand=titanrtx,driver>=510,driver<511 brand=tesla,driver>=515,driver<516 brand=unknown,driver>=515,driver<516 brand=nvidia,driver>=515,driver<516 brand=nvidiartx,driver>=515,driver<516 brand=geforce,driver>=515,driver<516 brand=geforcertx,driver>=515,driver<516 brand=quadro,driver>=515,driver<516 brand=quadrortx,driver>=515,driver<516 brand=titan,driver>=515,driver<516 brand=titanrtx,driver>=515,driver<516 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526
2023-09-06T12:01:59.7519325Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1
2023-09-06T12:01:59.7519951Z NV_NVTX_VERSION=12.1.105-1
2023-09-06T12:01:59.7520374Z GITHUB_RUN_NUMBER=1033
2023-09-06T12:01:59.7520897Z TEST_CONFIG=inductor_torchbench_perf
2023-09-06T12:01:59.7521506Z GITHUB_REPOSITORY_OWNER_ID=21003710
2023-09-06T12:01:59.7522479Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2023-09-06T12:01:59.7523110Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1
2023-09-06T12:01:59.7523694Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1
2023-09-06T12:01:59.7524201Z NV_LIBNPP_VERSION=12.1.0.40-1
2023-09-06T12:01:59.7524751Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot
2023-09-06T12:01:59.7525318Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2023-09-06T12:01:59.7525816Z GITHUB_REF_TYPE=branch
2023-09-06T12:01:59.7526292Z TORCH_CUDA_ARCH_LIST=Maxwell
2023-09-06T12:01:59.7526767Z NCCL_VERSION=2.17.1-1
2023-09-06T12:01:59.7527297Z BASE_SHA=3fe8417643c8d6c2b3d95552cd90321d141b5d54
2023-09-06T12:01:59.7527735Z XLA_CUDA=
2023-09-06T12:01:59.7528968Z HUGGING_FACE_HUB_TOKEN=***
2023-09-06T12:01:59.7534261Z ***
2023-09-06T12:01:59.7534748Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2023-09-06T12:01:59.7535217Z GITHUB_REPOSITORY_ID=65600975
2023-09-06T12:01:59.7535702Z GITHUB_ACTIONS=true
2023-09-06T12:01:59.7536141Z NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T12:01:59.7536969Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1
2023-09-06T12:01:59.7537680Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1
2023-09-06T12:01:59.7538199Z SHA1=3fe8417643c8d6c2b3d95552cd90321d141b5d54
2023-09-06T12:01:59.7538874Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2023-09-06T12:01:59.7539457Z GITHUB_SHA=3fe8417643c8d6c2b3d95552cd90321d141b5d54
2023-09-06T12:01:59.7540503Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-perf-test-nightly.yml@refs/heads/main
2023-09-06T12:01:59.7541160Z UCC_HOME=/usr
2023-09-06T12:01:59.7541659Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1
2023-09-06T12:01:59.7542164Z NVIDIA_PRODUCT_NAME=CUDA
2023-09-06T12:01:59.7542803Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1
2023-09-06T12:01:59.7543435Z GITHUB_REF=refs/heads/main
2023-09-06T12:01:59.7543937Z NV_NSIGHT_COMPUTE_VERSION=12.1.1-1
2023-09-06T12:01:59.7544492Z NV_CUDA_CUDART_VERSION=12.1.105-1
2023-09-06T12:01:59.7544930Z SHARD_NUMBER=1
2023-09-06T12:01:59.7545381Z GITHUB_REF_PROTECTED=true
2023-09-06T12:01:59.7546112Z HOME=/var/lib/jenkins
2023-09-06T12:01:59.7546608Z GITHUB_API_URL=https://api.github.com
2023-09-06T12:01:59.7547181Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2023-09-06T12:01:59.7547563Z UCX_COMMIT=00bcc6bb18fc282eb160623b4c0d300147f579af
2023-09-06T12:01:59.7548372Z SCCACHE_S3_KEY_PREFIX=inductor-A100-perf-nightly
2023-09-06T12:01:59.7548674Z CUDA_VERSION=12.1.1
2023-09-06T12:01:59.7549032Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1
2023-09-06T12:01:59.7549739Z NUM_TEST_SHARDS=4
2023-09-06T12:01:59.7550088Z UCX_HOME=/usr
2023-09-06T12:01:59.7550671Z GITHUB_STATE=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/save_state_8d26c93d-f8fc-4def-a019-0a44d1061b4c
2023-09-06T12:01:59.7551071Z PYTORCH_RETRY_TEST_CASES=1
2023-09-06T12:01:59.7551814Z GITHUB_ENV=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/set_env_8d26c93d-f8fc-4def-a019-0a44d1061b4c
2023-09-06T12:01:59.7552454Z GITHUB_EVENT_PATH=/home/weiwangmeta/actions-runner/_work/_temp/_github_workflow/event.json
2023-09-06T12:01:59.7552832Z GITHUB_EVENT_NAME=schedule
2023-09-06T12:01:59.7553674Z DASHBOARD_TAG=training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true
2023-09-06T12:01:59.7554318Z GITHUB_RUN_ID=6093795712
2023-09-06T12:01:59.7554690Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1
2023-09-06T12:01:59.7555080Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1
2023-09-06T12:01:59.7555942Z GITHUB_STEP_SUMMARY=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/step_summary_8d26c93d-f8fc-4def-a019-0a44d1061b4c
2023-09-06T12:01:59.7556388Z GITHUB_ACTOR=pytorchmergebot
2023-09-06T12:01:59.7556708Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1
2023-09-06T12:01:59.7556957Z PR_NUMBER=
2023-09-06T12:01:59.7557175Z GITHUB_RUN_ATTEMPT=1
2023-09-06T12:01:59.7557588Z NV_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1
2023-09-06T12:01:59.7557919Z ANACONDA_PYTHON_VERSION=3.10
2023-09-06T12:01:59.7558247Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2023-09-06T12:01:59.7558751Z TERM=xterm
2023-09-06T12:01:59.7559044Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1
2023-09-06T12:01:59.7559316Z INSTALLED_VISION=yes
2023-09-06T12:01:59.7559551Z BRANCH=main
2023-09-06T12:01:59.7559778Z OPENSSL_ROOT_DIR=/opt/openssl
2023-09-06T12:01:59.7560064Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2023-09-06T12:01:59.7560343Z CUDA_PATH=/usr/local/cuda
2023-09-06T12:01:59.7560859Z GITHUB_ACTION_PATH=/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2023-09-06T12:01:59.7561259Z GITHUB_SERVER_URL=https://github.com
2023-09-06T12:01:59.7561600Z UCC_COMMIT=7cb07a76ccedad7e56ceb136b865eb9319c258ea
2023-09-06T12:01:59.7561898Z REENABLED_ISSUES=
2023-09-06T12:01:59.7562121Z SHLVL=1
2023-09-06T12:01:59.7562313Z MAX_JOBS=10
2023-09-06T12:01:59.7562611Z NV_CUDA_LIB_VERSION=12.1.1-1
2023-09-06T12:01:59.7563027Z NVARCH=x86_64
2023-09-06T12:01:59.7563318Z GITHUB_ACTOR_ID=97764156
2023-09-06T12:01:59.7563616Z GITHUB_WORKFLOW_SHA=3fe8417643c8d6c2b3d95552cd90321d141b5d54
2023-09-06T12:01:59.7563928Z GITHUB_REF_NAME=main
2023-09-06T12:01:59.7564284Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1
2023-09-06T12:01:59.7564768Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2023-09-06T12:01:59.7565107Z GITHUB_JOB=test
2023-09-06T12:01:59.7565449Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1
2023-09-06T12:01:59.7565798Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2023-09-06T12:01:59.7566128Z GITHUB_REPOSITORY=pytorch/pytorch
2023-09-06T12:01:59.7566419Z NV_NVPROF_VERSION=12.1.105-1
2023-09-06T12:01:59.7566680Z GITHUB_RETENTION_DAYS=90
2023-09-06T12:01:59.7566939Z OPENSSL_DIR=/opt/openssl
2023-09-06T12:01:59.7567200Z GITHUB_ACTION_REPOSITORY=
2023-09-06T12:01:59.7567654Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-09-06T12:01:59.7568077Z GITHUB_BASE_REF=
2023-09-06T12:01:59.7568334Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2023-09-06T12:01:59.7568754Z CI=true
2023-09-06T12:01:59.7569026Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1
2023-09-06T12:01:59.7569312Z GITHUB_REPOSITORY_OWNER=pytorch
2023-09-06T12:01:59.7569581Z INSTALLED_PROTOBUF=yes
2023-09-06T12:01:59.7569812Z GITHUB_HEAD_REF=
2023-09-06T12:01:59.7570050Z GITHUB_ACTION_REF=
2023-09-06T12:01:59.7570431Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2023-09-06T12:01:59.7570855Z GITHUB_WORKFLOW=inductor-A100-perf-nightly
2023-09-06T12:01:59.7571149Z DEBIAN_FRONTEND=noninteractive
2023-09-06T12:01:59.7571741Z GITHUB_OUTPUT=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/set_output_8d26c93d-f8fc-4def-a019-0a44d1061b4c
2023-09-06T12:01:59.7572167Z SKIP_SCCACHE_INITIALIZATION=1
2023-09-06T12:01:59.7572422Z _=/usr/bin/env
2023-09-06T12:01:59.7572803Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2023-09-06T12:01:59.7712974Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch
2023-09-06T12:01:59.7713947Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2023-09-06T12:01:59.7714975Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib
2023-09-06T12:01:59.7715954Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test
2023-09-06T12:01:59.7716478Z + BUILD_DIR=build
2023-09-06T12:01:59.7716738Z + BUILD_RENAMED_DIR=build_renamed
2023-09-06T12:01:59.7717033Z + BUILD_BIN_DIR=build/bin
2023-09-06T12:01:59.7717480Z + export VALGRIND=ON
2023-09-06T12:01:59.7717857Z + VALGRIND=ON
2023-09-06T12:01:59.7718695Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *clang9* ]]
2023-09-06T12:01:59.7719222Z + [[ 0 == \1 ]]
2023-09-06T12:01:59.7719650Z + [[ False == \1 ]]
2023-09-06T12:01:59.7720331Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 != *bazel* ]]
2023-09-06T12:01:59.7723345Z ++ realpath build/custom_test_artifacts
2023-09-06T12:01:59.7734611Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2023-09-06T12:01:59.7738414Z ++ dirname .ci/pytorch/test.sh
2023-09-06T12:01:59.7747547Z + source .ci/pytorch/common.sh
2023-09-06T12:01:59.7752758Z +++ dirname .ci/pytorch/common.sh
2023-09-06T12:01:59.7761847Z ++ source .ci/pytorch/common_utils.sh
2023-09-06T12:01:59.7764542Z +++ declare -f -t trap_add
2023-09-06T12:01:59.7772642Z ++ set -ex
2023-09-06T12:01:59.7773062Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *rocm* ]]
2023-09-06T12:01:59.7773453Z ++ BUILD_TEST_LIBTORCH=0
2023-09-06T12:01:59.7773784Z + echo 'Environment variables'
2023-09-06T12:01:59.7774054Z Environment variables
2023-09-06T12:01:59.7774270Z + env
2023-09-06T12:01:59.7781916Z INSTALLED_DB=yes
2023-09-06T12:01:59.7782589Z NV_LIBCUBLAS_VERSION=12.1.3.1-1
2023-09-06T12:01:59.7783114Z NVIDIA_VISIBLE_DEVICES=all
2023-09-06T12:01:59.7783729Z NV_NVML_DEV_VERSION=12.1.105-1
2023-09-06T12:01:59.7784551Z GITHUB_WORKSPACE=/home/weiwangmeta/actions-runner/_work/pytorch/pytorch
2023-09-06T12:01:59.7785223Z CONTINUE_THROUGH_ERROR=False
2023-09-06T12:01:59.7785980Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1
2023-09-06T12:01:59.7786625Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1
2023-09-06T12:01:59.7787396Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm80
2023-09-06T12:01:59.7787854Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2023-09-06T12:01:59.7788120Z HOSTNAME=36c47af667b5
2023-09-06T12:01:59.7788693Z GITHUB_PATH=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/add_path_8d26c93d-f8fc-4def-a019-0a44d1061b4c
2023-09-06T12:01:59.7789255Z GITHUB_ACTION=__self
2023-09-06T12:01:59.7789607Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2023-09-06T12:01:59.7793799Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=510,driver<511 brand=unknown,driver>=510,driver<511 brand=nvidia,driver>=510,driver<511 brand=nvidiartx,driver>=510,driver<511 brand=geforce,driver>=510,driver<511 brand=geforcertx,driver>=510,driver<511 brand=quadro,driver>=510,driver<511 brand=quadrortx,driver>=510,driver<511 brand=titan,driver>=510,driver<511 brand=titanrtx,driver>=510,driver<511 brand=tesla,driver>=515,driver<516 brand=unknown,driver>=515,driver<516 brand=nvidia,driver>=515,driver<516 brand=nvidiartx,driver>=515,driver<516 brand=geforce,driver>=515,driver<516 brand=geforcertx,driver>=515,driver<516 brand=quadro,driver>=515,driver<516 brand=quadrortx,driver>=515,driver<516 brand=titan,driver>=515,driver<516 brand=titanrtx,driver>=515,driver<516 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526
2023-09-06T12:01:59.7796239Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1
2023-09-06T12:01:59.7796576Z NV_NVTX_VERSION=12.1.105-1
2023-09-06T12:01:59.7796834Z GITHUB_RUN_NUMBER=1033
2023-09-06T12:01:59.7797097Z TEST_CONFIG=inductor_torchbench_perf
2023-09-06T12:01:59.7797395Z GITHUB_REPOSITORY_OWNER_ID=21003710
2023-09-06T12:01:59.7797754Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2023-09-06T12:01:59.7798087Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1
2023-09-06T12:01:59.7798391Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1
2023-09-06T12:01:59.7798691Z NV_LIBNPP_VERSION=12.1.0.40-1
2023-09-06T12:01:59.7798981Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot
2023-09-06T12:01:59.7799315Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2023-09-06T12:01:59.7799595Z GITHUB_REF_TYPE=branch
2023-09-06T12:01:59.7799850Z TORCH_CUDA_ARCH_LIST=Maxwell
2023-09-06T12:01:59.7800128Z NCCL_VERSION=2.17.1-1
2023-09-06T12:01:59.7800573Z BASE_SHA=3fe8417643c8d6c2b3d95552cd90321d141b5d54
2023-09-06T12:01:59.7800827Z XLA_CUDA=
2023-09-06T12:01:59.7801245Z HUGGING_FACE_HUB_TOKEN=***
2023-09-06T12:01:59.7801551Z ***
2023-09-06T12:01:59.7801786Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2023-09-06T12:01:59.7802049Z GITHUB_REPOSITORY_ID=65600975
2023-09-06T12:01:59.7802306Z GITHUB_ACTIONS=true
2023-09-06T12:01:59.7802566Z NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T12:01:59.7803057Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1
2023-09-06T12:01:59.7803517Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1
2023-09-06T12:01:59.7803829Z SHA1=3fe8417643c8d6c2b3d95552cd90321d141b5d54
2023-09-06T12:01:59.7804197Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2023-09-06T12:01:59.7804519Z GITHUB_SHA=3fe8417643c8d6c2b3d95552cd90321d141b5d54
2023-09-06T12:01:59.7805089Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-perf-test-nightly.yml@refs/heads/main
2023-09-06T12:01:59.7805471Z UCC_HOME=/usr
2023-09-06T12:01:59.7805753Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1
2023-09-06T12:01:59.7806038Z NVIDIA_PRODUCT_NAME=CUDA
2023-09-06T12:01:59.7806399Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1
2023-09-06T12:01:59.7806700Z GITHUB_REF=refs/heads/main
2023-09-06T12:01:59.7806998Z NV_NSIGHT_COMPUTE_VERSION=12.1.1-1
2023-09-06T12:01:59.7807304Z NV_CUDA_CUDART_VERSION=12.1.105-1
2023-09-06T12:01:59.7807543Z SHARD_NUMBER=1
2023-09-06T12:01:59.7807790Z GITHUB_REF_PROTECTED=true
2023-09-06T12:01:59.7808041Z HOME=/var/lib/jenkins
2023-09-06T12:01:59.7808326Z GITHUB_API_URL=https://api.github.com
2023-09-06T12:01:59.7808622Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2023-09-06T12:01:59.7808945Z UCX_COMMIT=00bcc6bb18fc282eb160623b4c0d300147f579af
2023-09-06T12:01:59.7809372Z SCCACHE_S3_KEY_PREFIX=inductor-A100-perf-nightly
2023-09-06T12:01:59.7809670Z CUDA_VERSION=12.1.1
2023-09-06T12:01:59.7810010Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1
2023-09-06T12:01:59.7810286Z NUM_TEST_SHARDS=4
2023-09-06T12:01:59.7810519Z UCX_HOME=/usr
2023-09-06T12:01:59.7811213Z GITHUB_STATE=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/save_state_8d26c93d-f8fc-4def-a019-0a44d1061b4c
2023-09-06T12:01:59.7811627Z PYTORCH_RETRY_TEST_CASES=1
2023-09-06T12:01:59.7812195Z GITHUB_ENV=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/set_env_8d26c93d-f8fc-4def-a019-0a44d1061b4c
2023-09-06T12:01:59.7812814Z GITHUB_EVENT_PATH=/home/weiwangmeta/actions-runner/_work/_temp/_github_workflow/event.json
2023-09-06T12:01:59.7813229Z GITHUB_EVENT_NAME=schedule
2023-09-06T12:01:59.7814026Z DASHBOARD_TAG=training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true
2023-09-06T12:01:59.7814673Z GITHUB_RUN_ID=6093795712
2023-09-06T12:01:59.7815087Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1
2023-09-06T12:01:59.7815479Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1
2023-09-06T12:01:59.7816111Z GITHUB_STEP_SUMMARY=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/step_summary_8d26c93d-f8fc-4def-a019-0a44d1061b4c
2023-09-06T12:01:59.7816561Z GITHUB_ACTOR=pytorchmergebot
2023-09-06T12:01:59.7816860Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1
2023-09-06T12:01:59.7817110Z PR_NUMBER=
2023-09-06T12:01:59.7817339Z GITHUB_RUN_ATTEMPT=1
2023-09-06T12:01:59.7817558Z VALGRIND=ON
2023-09-06T12:01:59.7817953Z NV_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1
2023-09-06T12:01:59.7818284Z ANACONDA_PYTHON_VERSION=3.10
2023-09-06T12:01:59.7818609Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2023-09-06T12:01:59.7818882Z TERM=xterm
2023-09-06T12:01:59.7819167Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1
2023-09-06T12:01:59.7819437Z INSTALLED_VISION=yes
2023-09-06T12:01:59.7819669Z BRANCH=main
2023-09-06T12:01:59.7819896Z OPENSSL_ROOT_DIR=/opt/openssl
2023-09-06T12:01:59.7820183Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2023-09-06T12:01:59.7820460Z CUDA_PATH=/usr/local/cuda
2023-09-06T12:01:59.7820973Z GITHUB_ACTION_PATH=/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2023-09-06T12:01:59.7821545Z GITHUB_SERVER_URL=https://github.com
2023-09-06T12:01:59.7821885Z UCC_COMMIT=7cb07a76ccedad7e56ceb136b865eb9319c258ea
2023-09-06T12:01:59.7822181Z REENABLED_ISSUES=
2023-09-06T12:01:59.7822403Z SHLVL=1
2023-09-06T12:01:59.7822596Z MAX_JOBS=10
2023-09-06T12:01:59.7822863Z NV_CUDA_LIB_VERSION=12.1.1-1
2023-09-06T12:01:59.7823107Z NVARCH=x86_64
2023-09-06T12:01:59.7823396Z GITHUB_ACTOR_ID=97764156
2023-09-06T12:01:59.7823695Z GITHUB_WORKFLOW_SHA=3fe8417643c8d6c2b3d95552cd90321d141b5d54
2023-09-06T12:01:59.7823996Z GITHUB_REF_NAME=main
2023-09-06T12:01:59.7824339Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1
2023-09-06T12:01:59.7824826Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2023-09-06T12:01:59.7825160Z GITHUB_JOB=test
2023-09-06T12:01:59.7825500Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1
2023-09-06T12:01:59.7825846Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2023-09-06T12:01:59.7826177Z GITHUB_REPOSITORY=pytorch/pytorch
2023-09-06T12:01:59.7826477Z NV_NVPROF_VERSION=12.1.105-1
2023-09-06T12:01:59.7826743Z GITHUB_RETENTION_DAYS=90
2023-09-06T12:01:59.7826998Z OPENSSL_DIR=/opt/openssl
2023-09-06T12:01:59.7827243Z GITHUB_ACTION_REPOSITORY=
2023-09-06T12:01:59.7827706Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-09-06T12:01:59.7828126Z GITHUB_BASE_REF=
2023-09-06T12:01:59.7828382Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2023-09-06T12:01:59.7828624Z CI=true
2023-09-06T12:01:59.7828883Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1
2023-09-06T12:01:59.7829451Z GITHUB_REPOSITORY_OWNER=pytorch
2023-09-06T12:01:59.7829851Z INSTALLED_PROTOBUF=yes
2023-09-06T12:01:59.7830085Z GITHUB_HEAD_REF=
2023-09-06T12:01:59.7830318Z GITHUB_ACTION_REF=
2023-09-06T12:01:59.7830716Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2023-09-06T12:01:59.7831139Z GITHUB_WORKFLOW=inductor-A100-perf-nightly
2023-09-06T12:01:59.7831437Z DEBIAN_FRONTEND=noninteractive
2023-09-06T12:01:59.7832255Z GITHUB_OUTPUT=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/set_output_8d26c93d-f8fc-4def-a019-0a44d1061b4c
2023-09-06T12:01:59.7832688Z SKIP_SCCACHE_INITIALIZATION=1
2023-09-06T12:01:59.7832945Z _=/usr/bin/env
2023-09-06T12:01:59.7833276Z + echo 'Testing pytorch'
2023-09-06T12:01:59.7833527Z Testing pytorch
2023-09-06T12:01:59.7833794Z + export LANG=C.UTF-8
2023-09-06T12:01:59.7834054Z + LANG=C.UTF-8
2023-09-06T12:01:59.7834261Z + PR_NUMBER=
2023-09-06T12:01:59.7834542Z + [[ inductor_torchbench_perf == \d\e\f\a\u\l\t ]]
2023-09-06T12:01:59.7834890Z + [[ inductor_torchbench_perf == \d\i\s\t\r\i\b\u\t\e\d ]]
2023-09-06T12:01:59.7835224Z + [[ inductor_torchbench_perf == \s\l\o\w ]]
2023-09-06T12:01:59.7835679Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *slow-gradcheck* ]]
2023-09-06T12:01:59.7836161Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *cuda* ]]
2023-09-06T12:01:59.7836519Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2023-09-06T12:01:59.7836832Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2023-09-06T12:01:59.7837148Z + [[ inductor_torchbench_perf == *crossref* ]]
2023-09-06T12:01:59.7837582Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *rocm* ]]
2023-09-06T12:01:59.7838040Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 != *-bazel-* ]]
2023-09-06T12:01:59.7838442Z + pip_install --user ninja==1.10.2
2023-09-06T12:01:59.7838827Z + pip install --progress-bar off --user ninja==1.10.2
2023-09-06T12:02:00.3506669Z Collecting ninja==1.10.2
2023-09-06T12:02:00.4372051Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2023-09-06T12:02:01.8705166Z Installing collected packages: ninja
2023-09-06T12:02:01.8801862Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2023-09-06T12:02:01.8802627Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2023-09-06T12:02:01.8861616Z Successfully installed ninja-1.10.2
2023-09-06T12:02:01.9760431Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-09-06T12:02:01.9761380Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-09-06T12:02:01.9762190Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *asan* ]]
2023-09-06T12:02:01.9762808Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *-debug* ]]
2023-09-06T12:02:01.9763411Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 != *-bazel-* ]]
2023-09-06T12:02:01.9764197Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm80. Expect the assertion to pass'
2023-09-06T12:02:01.9765069Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm80. Expect the assertion to pass
2023-09-06T12:02:01.9765722Z + cd test
2023-09-06T12:02:01.9766439Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)'
2023-09-06T12:02:03.8394953Z + [[ inductor_torchbench_perf == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2023-09-06T12:02:03.8395451Z + [[ inductor_torchbench_perf == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2023-09-06T12:02:03.8396290Z + DYNAMO_BENCHMARK_FLAGS=()
2023-09-06T12:02:03.8396830Z + [[ inductor_torchbench_perf == *dynamo_eager* ]]
2023-09-06T12:02:03.8397283Z + [[ inductor_torchbench_perf == *aot_eager* ]]
2023-09-06T12:02:03.8397606Z + [[ inductor_torchbench_perf == *inductor* ]]
2023-09-06T12:02:03.8397933Z + [[ inductor_torchbench_perf != *perf* ]]
2023-09-06T12:02:03.8398263Z + [[ inductor_torchbench_perf == *dynamic* ]]
2023-09-06T12:02:03.8398598Z + [[ inductor_torchbench_perf == *cpu_accuracy* ]]
2023-09-06T12:02:03.8399177Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda)
2023-09-06T12:02:03.8410284Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *tbb* ]]
2023-09-06T12:02:03.8425606Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *libtorch* ]]
2023-09-06T12:02:03.8426460Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *-bazel-* ]]
2023-09-06T12:02:03.8430276Z + cd test
2023-09-06T12:02:03.8430746Z + python -c 'import torch; print(torch.__config__.show())'
2023-09-06T12:02:05.4348604Z PyTorch built with:
2023-09-06T12:02:05.4349411Z   - GCC 9.4
2023-09-06T12:02:05.4349883Z   - C++ Version: 201703
2023-09-06T12:02:05.4350477Z   - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2023-09-06T12:02:05.4351104Z   - Intel(R) MKL-DNN v3.1.1 (Git Hash 64f6bcbcbab628e96f33a62c3e975f8535a7bde4)
2023-09-06T12:02:05.4351539Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2023-09-06T12:02:05.4351945Z   - LAPACK is enabled (usually provided by MKL)
2023-09-06T12:02:05.4352286Z   - NNPACK is enabled
2023-09-06T12:02:05.4352617Z   - CPU capability usage: AVX512
2023-09-06T12:02:05.4353016Z   - CUDA Runtime 12.1
2023-09-06T12:02:05.4353415Z   - NVCC architecture flags: -gencode;arch=compute_80,code=sm_80
2023-09-06T12:02:05.4353810Z   - CuDNN 8.9.2
2023-09-06T12:02:05.4354080Z   - Magma 2.6.1
2023-09-06T12:02:05.4357717Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.9.2, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-invalid-partial-specialization -Wno-unused-private-field -Wno-aligned-allocation-unavailable -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.2.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 
2023-09-06T12:02:05.4360462Z 
2023-09-06T12:02:05.7509796Z + cd test
2023-09-06T12:02:05.7510514Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2023-09-06T12:02:07.2616672Z ATen/Parallel:
2023-09-06T12:02:07.2618026Z 	at::get_num_threads() : 6
2023-09-06T12:02:07.2618319Z 	at::get_num_interop_threads() : 6
2023-09-06T12:02:07.2618639Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2023-09-06T12:02:07.2618916Z 	omp_get_max_threads() : 6
2023-09-06T12:02:07.2619692Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2023-09-06T12:02:07.2620142Z 	mkl_get_max_threads() : 6
2023-09-06T12:02:07.2620615Z Intel(R) MKL-DNN v3.1.1 (Git Hash 64f6bcbcbab628e96f33a62c3e975f8535a7bde4)
2023-09-06T12:02:07.2620994Z std::thread::hardware_concurrency() : 12
2023-09-06T12:02:07.2621272Z Environment variables:
2023-09-06T12:02:07.2621542Z 	OMP_NUM_THREADS : [not set]
2023-09-06T12:02:07.2621815Z 	MKL_NUM_THREADS : [not set]
2023-09-06T12:02:07.2622093Z ATen parallel backend: OpenMP
2023-09-06T12:02:07.2622271Z 
2023-09-06T12:02:07.5385295Z + [[ inductor_torchbench_perf == *backward* ]]
2023-09-06T12:02:07.5386111Z + [[ inductor_torchbench_perf == *xla* ]]
2023-09-06T12:02:07.5386693Z + [[ inductor_torchbench_perf == \j\i\t\_\l\e\g\a\c\y ]]
2023-09-06T12:02:07.5387426Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *libtorch* ]]
2023-09-06T12:02:07.5387871Z + [[ inductor_torchbench_perf == distributed ]]
2023-09-06T12:02:07.5388513Z + [[ inductor_torchbench_perf == deploy ]]
2023-09-06T12:02:07.5390028Z + [[ inductor_torchbench_perf == *inductor_distributed* ]]
2023-09-06T12:02:07.5390756Z + [[ inductor_torchbench_perf == *huggingface* ]]
2023-09-06T12:02:07.5391882Z + [[ inductor_torchbench_perf == *timm* ]]
2023-09-06T12:02:07.5392238Z + [[ inductor_torchbench_perf == *torchbench* ]]
2023-09-06T12:02:07.5392592Z + [[ inductor_torchbench_perf == *cpu_accuracy* ]]
2023-09-06T12:02:07.5392902Z + install_torchaudio cuda
2023-09-06T12:02:07.5393221Z + local commit
2023-09-06T12:02:07.5393455Z ++ get_pinned_commit audio
2023-09-06T12:02:07.5393743Z ++ cat .github/ci_commit_pins/audio.txt
2023-09-06T12:02:07.5409423Z + commit=a8f4e97bd5356a7a77510cdf6a3a62e25a5dc602
2023-09-06T12:02:07.5409968Z + [[ cuda == \c\u\d\a ]]
2023-09-06T12:02:07.5410579Z + TORCH_CUDA_ARCH_LIST='8.0;8.6'
2023-09-06T12:02:07.5411207Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@a8f4e97bd5356a7a77510cdf6a3a62e25a5dc602
2023-09-06T12:02:07.5412016Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@a8f4e97bd5356a7a77510cdf6a3a62e25a5dc602
2023-09-06T12:02:07.9839471Z Collecting git+https://github.com/pytorch/audio.git@a8f4e97bd5356a7a77510cdf6a3a62e25a5dc602
2023-09-06T12:02:07.9844265Z   Cloning https://github.com/pytorch/audio.git (to revision a8f4e97bd5356a7a77510cdf6a3a62e25a5dc602) to /tmp/pip-req-build-1ubp68m_
2023-09-06T12:02:07.9877994Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-1ubp68m_
2023-09-06T12:02:10.1585066Z   Running command git rev-parse -q --verify 'sha^a8f4e97bd5356a7a77510cdf6a3a62e25a5dc602'
2023-09-06T12:02:10.1611734Z   Running command git fetch -q https://github.com/pytorch/audio.git a8f4e97bd5356a7a77510cdf6a3a62e25a5dc602
2023-09-06T12:02:10.8235026Z   Running command git checkout -q a8f4e97bd5356a7a77510cdf6a3a62e25a5dc602
2023-09-06T12:02:11.4902764Z   Resolved https://github.com/pytorch/audio.git to commit a8f4e97bd5356a7a77510cdf6a3a62e25a5dc602
2023-09-06T12:02:32.9750726Z   Running command git submodule update --init --recursive -q
2023-09-06T12:02:32.9751439Z   Preparing metadata (setup.py) ... [?25l- done
2023-09-06T12:02:32.9788366Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.0.0a0+a8f4e97) (2.2.0a0+git3fe8417)
2023-09-06T12:02:32.9865527Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.0.0a0+a8f4e97) (3.9.0)
2023-09-06T12:02:32.9868871Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.0.0a0+a8f4e97) (4.7.1)
2023-09-06T12:02:32.9873416Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.0.0a0+a8f4e97) (1.12)
2023-09-06T12:02:32.9877277Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.0.0a0+a8f4e97) (2.8.8)
2023-09-06T12:02:32.9881512Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.0.0a0+a8f4e97) (3.1.2)
2023-09-06T12:02:32.9885670Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.0.0a0+a8f4e97) (2023.4.0)
2023-09-06T12:02:33.0374657Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.0.0a0+a8f4e97) (2.1.3)
2023-09-06T12:02:33.0559814Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchaudio==2.0.0a0+a8f4e97) (1.3.0)
2023-09-06T12:02:33.0649526Z Building wheels for collected packages: torchaudio
2023-09-06T12:06:14.0013496Z   Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done
2023-09-06T12:06:14.0153197Z [?25h  Created wheel for torchaudio: filename=torchaudio-2.0.0a0+a8f4e97-cp310-cp310-linux_x86_64.whl size=3889551 sha256=fa69c0d8280a81802796d1a567162d160850a0dad700fe5d7325f49d1808afcd
2023-09-06T12:06:14.0153981Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/34/3a/3f/9b303b7cc5d4e7d824fe266e4f875e08f15eb13882f11f305c
2023-09-06T12:06:14.0190412Z Successfully built torchaudio
2023-09-06T12:06:15.3414695Z Installing collected packages: torchaudio
2023-09-06T12:06:15.6023714Z Successfully installed torchaudio-2.0.0a0+a8f4e97
2023-09-06T12:06:16.1713349Z + install_torchtext
2023-09-06T12:06:16.1713661Z + local data_commit
2023-09-06T12:06:16.1713952Z + local text_commit
2023-09-06T12:06:16.1719602Z ++ get_pinned_commit data
2023-09-06T12:06:16.1720109Z ++ cat .github/ci_commit_pins/data.txt
2023-09-06T12:06:16.1737805Z + data_commit=11bb5b847ea8b9e0d9bb82db3304daf383008d3f
2023-09-06T12:06:16.1743097Z ++ get_pinned_commit text
2023-09-06T12:06:16.1743594Z ++ cat .github/ci_commit_pins/text.txt
2023-09-06T12:06:16.1759131Z + text_commit=b0ebddc648d279826089db91775375221777a2db
2023-09-06T12:06:16.1760217Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f
2023-09-06T12:06:16.1761029Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f
2023-09-06T12:06:16.6127026Z Collecting git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f
2023-09-06T12:06:16.6133535Z   Cloning https://github.com/pytorch/data.git (to revision 11bb5b847ea8b9e0d9bb82db3304daf383008d3f) to /tmp/pip-req-build-oajp7_1r
2023-09-06T12:06:16.6168874Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/data.git /tmp/pip-req-build-oajp7_1r
2023-09-06T12:06:17.6770364Z   Running command git rev-parse -q --verify 'sha^11bb5b847ea8b9e0d9bb82db3304daf383008d3f'
2023-09-06T12:06:17.6936901Z   Running command git fetch -q https://github.com/pytorch/data.git 11bb5b847ea8b9e0d9bb82db3304daf383008d3f
2023-09-06T12:06:18.1149713Z   Running command git checkout -q 11bb5b847ea8b9e0d9bb82db3304daf383008d3f
2023-09-06T12:06:18.4747093Z   Resolved https://github.com/pytorch/data.git to commit 11bb5b847ea8b9e0d9bb82db3304daf383008d3f
2023-09-06T12:06:18.4748303Z   Running command git submodule update --init --recursive -q
2023-09-06T12:07:37.5607038Z   Preparing metadata (setup.py) ... [?25l- done
2023-09-06T12:07:37.5664702Z [?25hRequirement already satisfied: urllib3>=1.25 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (1.26.16)
2023-09-06T12:07:37.5667441Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.31.0)
2023-09-06T12:07:37.5674771Z Requirement already satisfied: torch>2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.2.0a0+git3fe8417)
2023-09-06T12:07:37.5753230Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.9.0)
2023-09-06T12:07:37.5757249Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (4.7.1)
2023-09-06T12:07:37.5761678Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.12)
2023-09-06T12:07:37.5765492Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.8.8)
2023-09-06T12:07:37.5768370Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.1.2)
2023-09-06T12:07:37.5772058Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2023.4.0)
2023-09-06T12:07:37.5984447Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.2.0)
2023-09-06T12:07:37.5989794Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.4)
2023-09-06T12:07:37.5998201Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (2023.7.22)
2023-09-06T12:07:37.6524585Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.1.3)
2023-09-06T12:07:37.6720111Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.3.0)
2023-09-06T12:07:37.6826509Z Building wheels for collected packages: torchdata
2023-09-06T12:07:39.9331001Z   Building wheel for torchdata (setup.py) ... [?25l- \ | / done
2023-09-06T12:07:39.9342021Z [?25h  Created wheel for torchdata: filename=torchdata-0.7.0a0+11bb5b8-py3-none-any.whl size=182989 sha256=80fc5c3a895eabd81046a356a75005a8823bb34ce72acf57a2983a401b3f6f1a
2023-09-06T12:07:39.9343645Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/ca/59/a3/c8250bfc8d3d4d639498d4beb2e0f0e70b9a508ac61fde85ce
2023-09-06T12:07:39.9381087Z Successfully built torchdata
2023-09-06T12:07:41.2402173Z Installing collected packages: torchdata
2023-09-06T12:07:41.3499551Z Successfully installed torchdata-0.7.0a0+11bb5b8
2023-09-06T12:07:44.2538166Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db
2023-09-06T12:07:44.2539026Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db
2023-09-06T12:07:44.6928366Z Collecting git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db
2023-09-06T12:07:44.6935134Z   Cloning https://github.com/pytorch/text.git (to revision b0ebddc648d279826089db91775375221777a2db) to /tmp/pip-req-build-f4amfz1z
2023-09-06T12:07:44.6961250Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-f4amfz1z
2023-09-06T12:07:46.2990517Z   Running command git rev-parse -q --verify 'sha^b0ebddc648d279826089db91775375221777a2db'
2023-09-06T12:07:46.3016521Z   Running command git fetch -q https://github.com/pytorch/text.git b0ebddc648d279826089db91775375221777a2db
2023-09-06T12:07:47.0376637Z   Running command git checkout -q b0ebddc648d279826089db91775375221777a2db
2023-09-06T12:07:47.4680606Z   Resolved https://github.com/pytorch/text.git to commit b0ebddc648d279826089db91775375221777a2db
2023-09-06T12:07:47.4681983Z   Running command git submodule update --init --recursive -q
2023-09-06T12:07:55.8090902Z   Preparing metadata (setup.py) ... [?25l- done
2023-09-06T12:07:55.8151605Z [?25hRequirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (4.66.1)
2023-09-06T12:07:55.8156656Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.31.0)
2023-09-06T12:07:55.8161145Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.2.0a0+git3fe8417)
2023-09-06T12:07:55.8166471Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (1.21.2)
2023-09-06T12:07:55.8170376Z Requirement already satisfied: torchdata in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (0.7.0a0+11bb5b8)
2023-09-06T12:07:55.8254750Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.2.0)
2023-09-06T12:07:55.8263256Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.4)
2023-09-06T12:07:55.8271415Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (1.26.16)
2023-09-06T12:07:55.8278651Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (2023.7.22)
2023-09-06T12:07:55.8346701Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.9.0)
2023-09-06T12:07:55.8351705Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (4.7.1)
2023-09-06T12:07:55.8355886Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (1.12)
2023-09-06T12:07:55.8360863Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2.8.8)
2023-09-06T12:07:55.8364943Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.1.2)
2023-09-06T12:07:55.8369799Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2023.4.0)
2023-09-06T12:07:55.9152044Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchtext==0.16.0a0+b0ebddc) (2.1.3)
2023-09-06T12:07:55.9355065Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchtext==0.16.0a0+b0ebddc) (1.3.0)
2023-09-06T12:07:55.9464447Z Building wheels for collected packages: torchtext
2023-09-06T12:08:48.9308899Z   Building wheel for torchtext (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done
2023-09-06T12:08:48.9377947Z [?25h  Created wheel for torchtext: filename=torchtext-0.16.0a0+b0ebddc-cp310-cp310-linux_x86_64.whl size=2041323 sha256=fcc990e83513a1ebfec13e458b54acfb3eeb938e530a15d58d96ecc59e91eb45
2023-09-06T12:08:48.9378766Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/e5/3d/d7/faafad098ec9437ecdf4495c0f5e72b817fffbb06063e32cbf
2023-09-06T12:08:48.9412811Z Successfully built torchtext
2023-09-06T12:08:50.2279645Z Installing collected packages: torchtext
2023-09-06T12:08:50.3660013Z Successfully installed torchtext-0.16.0a0+b0ebddc
2023-09-06T12:08:50.5213939Z + install_torchvision
2023-09-06T12:08:50.5214303Z + local orig_preload
2023-09-06T12:08:50.5217342Z + local commit
2023-09-06T12:08:50.5220175Z ++ get_pinned_commit vision
2023-09-06T12:08:50.5222859Z ++ cat .github/ci_commit_pins/vision.txt
2023-09-06T12:08:50.5243416Z + commit=1f94320d8db8d102214a7dc02c22fa65ee9ac58a
2023-09-06T12:08:50.5243878Z + orig_preload=
2023-09-06T12:08:50.5244560Z + '[' -n '' ']'
2023-09-06T12:08:50.5245176Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@1f94320d8db8d102214a7dc02c22fa65ee9ac58a
2023-09-06T12:08:50.5246357Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@1f94320d8db8d102214a7dc02c22fa65ee9ac58a
2023-09-06T12:08:50.9622956Z Collecting git+https://github.com/pytorch/vision.git@1f94320d8db8d102214a7dc02c22fa65ee9ac58a
2023-09-06T12:08:50.9630882Z   Cloning https://github.com/pytorch/vision.git (to revision 1f94320d8db8d102214a7dc02c22fa65ee9ac58a) to /tmp/pip-req-build-6hxdc7mj
2023-09-06T12:08:50.9662803Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-6hxdc7mj
2023-09-06T12:08:53.0890648Z   Running command git rev-parse -q --verify 'sha^1f94320d8db8d102214a7dc02c22fa65ee9ac58a'
2023-09-06T12:08:53.0917229Z   Running command git fetch -q https://github.com/pytorch/vision.git 1f94320d8db8d102214a7dc02c22fa65ee9ac58a
2023-09-06T12:08:54.6442293Z   Running command git checkout -q 1f94320d8db8d102214a7dc02c22fa65ee9ac58a
2023-09-06T12:08:55.1057139Z   Resolved https://github.com/pytorch/vision.git to commit 1f94320d8db8d102214a7dc02c22fa65ee9ac58a
2023-09-06T12:08:57.6743630Z   Preparing metadata (setup.py) ... [?25l- \ done
2023-09-06T12:08:57.6806725Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.16.0a0+1f94320) (1.21.2)
2023-09-06T12:08:57.6811146Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.16.0a0+1f94320) (2.31.0)
2023-09-06T12:08:57.6815390Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.16.0a0+1f94320) (2.2.0a0+git3fe8417)
2023-09-06T12:08:57.6823491Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.16.0a0+1f94320) (9.5.0)
2023-09-06T12:08:57.7031961Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchvision==0.16.0a0+1f94320) (3.2.0)
2023-09-06T12:08:57.7039086Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchvision==0.16.0a0+1f94320) (3.4)
2023-09-06T12:08:57.7048296Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchvision==0.16.0a0+1f94320) (1.26.16)
2023-09-06T12:08:57.7054690Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchvision==0.16.0a0+1f94320) (2023.7.22)
2023-09-06T12:08:57.7124138Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.16.0a0+1f94320) (3.9.0)
2023-09-06T12:08:57.7127782Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.16.0a0+1f94320) (4.7.1)
2023-09-06T12:08:57.7130833Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.16.0a0+1f94320) (1.12)
2023-09-06T12:08:57.7134856Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.16.0a0+1f94320) (2.8.8)
2023-09-06T12:08:57.7140322Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.16.0a0+1f94320) (3.1.2)
2023-09-06T12:08:57.7144250Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.16.0a0+1f94320) (2023.4.0)
2023-09-06T12:08:57.7814362Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.16.0a0+1f94320) (2.1.3)
2023-09-06T12:08:57.8011178Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.16.0a0+1f94320) (1.3.0)
2023-09-06T12:08:57.8118977Z Building wheels for collected packages: torchvision
2023-09-06T12:10:35.2675384Z   Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done
2023-09-06T12:10:35.2743112Z [?25h  Created wheel for torchvision: filename=torchvision-0.16.0a0+1f94320-cp310-cp310-linux_x86_64.whl size=1919393 sha256=b1ddd622f306fec9094fe11f9f412f2e8364ea7ca35d1a025ea056b33162ad1f
2023-09-06T12:10:35.2743996Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/e8/0d/df/fbb13cc562d2f580963bdc47f8e6f952797d4fe1506fe5db13
2023-09-06T12:10:35.2793380Z Successfully built torchvision
2023-09-06T12:10:36.5980434Z Installing collected packages: torchvision
2023-09-06T12:10:37.1059231Z Successfully installed torchvision-0.16.0a0+1f94320
2023-09-06T12:10:37.2396130Z + '[' -n '' ']'
2023-09-06T12:10:37.2396517Z + id=0
2023-09-06T12:10:37.2397062Z + pip_install opencv-python==4.8.0.74
2023-09-06T12:10:37.2397833Z + pip install --progress-bar off opencv-python==4.8.0.74
2023-09-06T12:10:37.9527109Z Collecting opencv-python==4.8.0.74
2023-09-06T12:10:37.9528209Z   Obtaining dependency information for opencv-python==4.8.0.74 from https://files.pythonhosted.org/packages/34/7c/8a5043f362b0a55f07812a0db3f86092cdbd0fe41b933d7bc6fce3ab6c15/opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
2023-09-06T12:10:38.0324292Z   Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB)
2023-09-06T12:10:38.0547402Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.21.2)
2023-09-06T12:10:38.0732189Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB)
2023-09-06T12:10:40.2899238Z Installing collected packages: opencv-python
2023-09-06T12:10:41.3672951Z Successfully installed opencv-python-4.8.0.74
2023-09-06T12:10:41.4777043Z + [[ inductor_torchbench_perf == *inductor_torchbench_smoketest_perf* ]]
2023-09-06T12:10:41.4777468Z + checkout_install_torchbench
2023-09-06T12:10:41.4777733Z + local commit
2023-09-06T12:10:41.4784306Z ++ get_pinned_commit torchbench
2023-09-06T12:10:41.4784635Z ++ cat .github/ci_commit_pins/torchbench.txt
2023-09-06T12:10:41.4805004Z + commit=9371b9e13c826f3930e54346b4d619cb59182f68
2023-09-06T12:10:41.4805474Z + git clone https://github.com/pytorch/benchmark torchbench
2023-09-06T12:10:41.4828653Z Cloning into 'torchbench'...
2023-09-06T12:10:41.8952927Z remote: Enumerating objects: 24307, done.
2023-09-06T12:10:41.8955335Z remote: Counting objects:   0% (1/4131)
2023-09-06T12:10:41.8956042Z remote: Counting objects:   1% (42/4131)
2023-09-06T12:10:41.8956736Z remote: Counting objects:   2% (83/4131)
2023-09-06T12:10:41.8957396Z remote: Counting objects:   3% (124/4131)
2023-09-06T12:10:41.8958194Z remote: Counting objects:   4% (166/4131)
2023-09-06T12:10:41.8958839Z remote: Counting objects:   5% (207/4131)
2023-09-06T12:10:41.8959452Z remote: Counting objects:   6% (248/4131)
2023-09-06T12:10:41.8960057Z remote: Counting objects:   7% (290/4131)
2023-09-06T12:10:41.8960746Z remote: Counting objects:   8% (331/4131)
2023-09-06T12:10:41.8961420Z remote: Counting objects:   9% (372/4131)
2023-09-06T12:10:41.8962061Z remote: Counting objects:  10% (414/4131)
2023-09-06T12:10:41.8962738Z remote: Counting objects:  11% (455/4131)
2023-09-06T12:10:41.8963430Z remote: Counting objects:  12% (496/4131)
2023-09-06T12:10:41.8964021Z remote: Counting objects:  13% (538/4131)
2023-09-06T12:10:41.8964390Z remote: Counting objects:  14% (579/4131)
2023-09-06T12:10:41.8964758Z remote: Counting objects:  15% (620/4131)
2023-09-06T12:10:41.8965122Z remote: Counting objects:  16% (661/4131)
2023-09-06T12:10:41.8965472Z remote: Counting objects:  17% (703/4131)
2023-09-06T12:10:41.8966001Z remote: Counting objects:  18% (744/4131)
2023-09-06T12:10:41.8966932Z remote: Counting objects:  19% (785/4131)
2023-09-06T12:10:41.8967634Z remote: Counting objects:  20% (827/4131)
2023-09-06T12:10:41.8968371Z remote: Counting objects:  21% (868/4131)
2023-09-06T12:10:41.8969019Z remote: Counting objects:  22% (909/4131)
2023-09-06T12:10:41.8969695Z remote: Counting objects:  23% (951/4131)
2023-09-06T12:10:41.8970370Z remote: Counting objects:  24% (992/4131)
2023-09-06T12:10:41.8970994Z remote: Counting objects:  25% (1033/4131)
2023-09-06T12:10:41.8971679Z remote: Counting objects:  26% (1075/4131)
2023-09-06T12:10:41.8972345Z remote: Counting objects:  27% (1116/4131)
2023-09-06T12:10:41.8973030Z remote: Counting objects:  28% (1157/4131)
2023-09-06T12:10:41.8973703Z remote: Counting objects:  29% (1198/4131)
2023-09-06T12:10:41.8974394Z remote: Counting objects:  30% (1240/4131)
2023-09-06T12:10:41.8975072Z remote: Counting objects:  31% (1281/4131)
2023-09-06T12:10:41.8975755Z remote: Counting objects:  32% (1322/4131)
2023-09-06T12:10:41.8976474Z remote: Counting objects:  33% (1364/4131)
2023-09-06T12:10:41.8977168Z remote: Counting objects:  34% (1405/4131)
2023-09-06T12:10:41.8977836Z remote: Counting objects:  35% (1446/4131)
2023-09-06T12:10:41.8978596Z remote: Counting objects:  36% (1488/4131)
2023-09-06T12:10:41.8979282Z remote: Counting objects:  37% (1529/4131)
2023-09-06T12:10:41.8979848Z remote: Counting objects:  38% (1570/4131)
2023-09-06T12:10:41.8980542Z remote: Counting objects:  39% (1612/4131)
2023-09-06T12:10:41.8981233Z remote: Counting objects:  40% (1653/4131)
2023-09-06T12:10:41.8981909Z remote: Counting objects:  41% (1694/4131)
2023-09-06T12:10:41.8982560Z remote: Counting objects:  42% (1736/4131)
2023-09-06T12:10:41.8983274Z remote: Counting objects:  43% (1777/4131)
2023-09-06T12:10:41.8983966Z remote: Counting objects:  44% (1818/4131)
2023-09-06T12:10:41.8984351Z remote: Counting objects:  45% (1859/4131)
2023-09-06T12:10:41.8985068Z remote: Counting objects:  46% (1901/4131)
2023-09-06T12:10:41.8985750Z remote: Counting objects:  47% (1942/4131)
2023-09-06T12:10:41.8986469Z remote: Counting objects:  48% (1983/4131)
2023-09-06T12:10:41.8987120Z remote: Counting objects:  49% (2025/4131)
2023-09-06T12:10:41.8987790Z remote: Counting objects:  50% (2066/4131)
2023-09-06T12:10:41.8988552Z remote: Counting objects:  51% (2107/4131)
2023-09-06T12:10:41.8989370Z remote: Counting objects:  52% (2149/4131)
2023-09-06T12:10:41.8990073Z remote: Counting objects:  53% (2190/4131)
2023-09-06T12:10:41.8990743Z remote: Counting objects:  54% (2231/4131)
2023-09-06T12:10:41.8991441Z remote: Counting objects:  55% (2273/4131)
2023-09-06T12:10:41.8992138Z remote: Counting objects:  56% (2314/4131)
2023-09-06T12:10:41.8992826Z remote: Counting objects:  57% (2355/4131)
2023-09-06T12:10:41.9032215Z remote: Counting objects:  58% (2396/4131)
2023-09-06T12:10:41.9032643Z remote: Counting objects:  59% (2438/4131)
2023-09-06T12:10:41.9033041Z remote: Counting objects:  60% (2479/4131)
2023-09-06T12:10:41.9033415Z remote: Counting objects:  61% (2520/4131)
2023-09-06T12:10:41.9033792Z remote: Counting objects:  62% (2562/4131)
2023-09-06T12:10:41.9034161Z remote: Counting objects:  63% (2603/4131)
2023-09-06T12:10:41.9034511Z remote: Counting objects:  64% (2644/4131)
2023-09-06T12:10:41.9034876Z remote: Counting objects:  65% (2686/4131)
2023-09-06T12:10:41.9035250Z remote: Counting objects:  66% (2727/4131)
2023-09-06T12:10:41.9035611Z remote: Counting objects:  67% (2768/4131)
2023-09-06T12:10:41.9035968Z remote: Counting objects:  68% (2810/4131)
2023-09-06T12:10:41.9036325Z remote: Counting objects:  69% (2851/4131)
2023-09-06T12:10:41.9036695Z remote: Counting objects:  70% (2892/4131)
2023-09-06T12:10:41.9037094Z remote: Counting objects:  71% (2934/4131)
2023-09-06T12:10:41.9037449Z remote: Counting objects:  72% (2975/4131)
2023-09-06T12:10:41.9038125Z remote: Counting objects:  73% (3016/4131)
2023-09-06T12:10:41.9038530Z remote: Counting objects:  74% (3057/4131)
2023-09-06T12:10:41.9038903Z remote: Counting objects:  75% (3099/4131)
2023-09-06T12:10:41.9039262Z remote: Counting objects:  76% (3140/4131)
2023-09-06T12:10:41.9039626Z remote: Counting objects:  77% (3181/4131)
2023-09-06T12:10:41.9039996Z remote: Counting objects:  78% (3223/4131)
2023-09-06T12:10:41.9040362Z remote: Counting objects:  79% (3264/4131)
2023-09-06T12:10:41.9040716Z remote: Counting objects:  80% (3305/4131)
2023-09-06T12:10:41.9041087Z remote: Counting objects:  81% (3347/4131)
2023-09-06T12:10:41.9041460Z remote: Counting objects:  82% (3388/4131)
2023-09-06T12:10:41.9041829Z remote: Counting objects:  83% (3429/4131)
2023-09-06T12:10:41.9042187Z remote: Counting objects:  84% (3471/4131)
2023-09-06T12:10:41.9042559Z remote: Counting objects:  85% (3512/4131)
2023-09-06T12:10:41.9042930Z remote: Counting objects:  86% (3553/4131)
2023-09-06T12:10:41.9043312Z remote: Counting objects:  87% (3594/4131)
2023-09-06T12:10:41.9043666Z remote: Counting objects:  88% (3636/4131)
2023-09-06T12:10:41.9044030Z remote: Counting objects:  89% (3677/4131)
2023-09-06T12:10:41.9044400Z remote: Counting objects:  90% (3718/4131)
2023-09-06T12:10:41.9044766Z remote: Counting objects:  91% (3760/4131)
2023-09-06T12:10:41.9045134Z remote: Counting objects:  92% (3801/4131)
2023-09-06T12:10:41.9045487Z remote: Counting objects:  93% (3842/4131)
2023-09-06T12:10:41.9045854Z remote: Counting objects:  94% (3884/4131)
2023-09-06T12:10:41.9046225Z remote: Counting objects:  95% (3925/4131)
2023-09-06T12:10:41.9046594Z remote: Counting objects:  96% (3966/4131)
2023-09-06T12:10:41.9046946Z remote: Counting objects:  97% (4008/4131)
2023-09-06T12:10:41.9047315Z remote: Counting objects:  98% (4049/4131)
2023-09-06T12:10:41.9047676Z remote: Counting objects:  99% (4090/4131)
2023-09-06T12:10:41.9048080Z remote: Counting objects: 100% (4131/4131)
2023-09-06T12:10:41.9048611Z remote: Counting objects: 100% (4131/4131), done.
2023-09-06T12:10:41.9049013Z remote: Compressing objects:   0% (1/1087)
2023-09-06T12:10:41.9049398Z remote: Compressing objects:   1% (11/1087)
2023-09-06T12:10:41.9084438Z remote: Compressing objects:   2% (22/1087)
2023-09-06T12:10:41.9097484Z remote: Compressing objects:   3% (33/1087)
2023-09-06T12:10:41.9136790Z remote: Compressing objects:   4% (44/1087)
2023-09-06T12:10:41.9162899Z remote: Compressing objects:   5% (55/1087)
2023-09-06T12:10:41.9192829Z remote: Compressing objects:   6% (66/1087)
2023-09-06T12:10:41.9214290Z remote: Compressing objects:   7% (77/1087)
2023-09-06T12:10:41.9315977Z remote: Compressing objects:   8% (87/1087)
2023-09-06T12:10:41.9415897Z remote: Compressing objects:   9% (98/1087)
2023-09-06T12:10:41.9526027Z remote: Compressing objects:  10% (109/1087)
2023-09-06T12:10:41.9594825Z remote: Compressing objects:  11% (120/1087)
2023-09-06T12:10:41.9676666Z remote: Compressing objects:  12% (131/1087)
2023-09-06T12:10:41.9738310Z remote: Compressing objects:  13% (142/1087)
2023-09-06T12:10:41.9789840Z remote: Compressing objects:  14% (153/1087)
2023-09-06T12:10:41.9834148Z remote: Compressing objects:  15% (164/1087)
2023-09-06T12:10:41.9876622Z remote: Compressing objects:  16% (174/1087)
2023-09-06T12:10:41.9929811Z remote: Compressing objects:  17% (185/1087)
2023-09-06T12:10:41.9958448Z remote: Compressing objects:  18% (196/1087)
2023-09-06T12:10:42.0002338Z remote: Compressing objects:  19% (207/1087)
2023-09-06T12:10:42.0026677Z remote: Compressing objects:  20% (218/1087)
2023-09-06T12:10:42.0064287Z remote: Compressing objects:  21% (229/1087)
2023-09-06T12:10:42.0091981Z remote: Compressing objects:  22% (240/1087)
2023-09-06T12:10:42.0114366Z remote: Compressing objects:  23% (251/1087)
2023-09-06T12:10:42.0139202Z remote: Compressing objects:  24% (261/1087)
2023-09-06T12:10:42.0155775Z remote: Compressing objects:  25% (272/1087)
2023-09-06T12:10:42.0172092Z remote: Compressing objects:  26% (283/1087)
2023-09-06T12:10:42.0183437Z remote: Compressing objects:  27% (294/1087)
2023-09-06T12:10:42.0200347Z remote: Compressing objects:  28% (305/1087)
2023-09-06T12:10:42.0207770Z remote: Compressing objects:  29% (316/1087)
2023-09-06T12:10:42.0211972Z remote: Compressing objects:  30% (327/1087)
2023-09-06T12:10:42.0212371Z remote: Compressing objects:  31% (337/1087)
2023-09-06T12:10:42.0212765Z remote: Compressing objects:  32% (348/1087)
2023-09-06T12:10:42.0213152Z remote: Compressing objects:  33% (359/1087)
2023-09-06T12:10:42.0218244Z remote: Compressing objects:  34% (370/1087)
2023-09-06T12:10:42.0223903Z remote: Compressing objects:  35% (381/1087)
2023-09-06T12:10:42.0224296Z remote: Compressing objects:  36% (392/1087)
2023-09-06T12:10:42.0226632Z remote: Compressing objects:  37% (403/1087)
2023-09-06T12:10:42.0228327Z remote: Compressing objects:  38% (414/1087)
2023-09-06T12:10:42.0242877Z remote: Compressing objects:  39% (424/1087)
2023-09-06T12:10:42.0249050Z remote: Compressing objects:  40% (435/1087)
2023-09-06T12:10:42.0255622Z remote: Compressing objects:  41% (446/1087)
2023-09-06T12:10:42.0259026Z remote: Compressing objects:  42% (457/1087)
2023-09-06T12:10:42.0266519Z remote: Compressing objects:  43% (468/1087)
2023-09-06T12:10:42.0278439Z remote: Compressing objects:  44% (479/1087)
2023-09-06T12:10:42.0279091Z remote: Compressing objects:  45% (490/1087)
2023-09-06T12:10:42.0280446Z remote: Compressing objects:  46% (501/1087)
2023-09-06T12:10:42.0285268Z remote: Compressing objects:  47% (511/1087)
2023-09-06T12:10:42.0295518Z remote: Compressing objects:  48% (522/1087)
2023-09-06T12:10:42.0296077Z remote: Compressing objects:  49% (533/1087)
2023-09-06T12:10:42.0299008Z remote: Compressing objects:  50% (544/1087)
2023-09-06T12:10:42.0302010Z remote: Compressing objects:  51% (555/1087)
2023-09-06T12:10:42.0305600Z remote: Compressing objects:  52% (566/1087)
2023-09-06T12:10:42.0306803Z remote: Compressing objects:  53% (577/1087)
2023-09-06T12:10:42.0308860Z remote: Compressing objects:  54% (587/1087)
2023-09-06T12:10:42.0309837Z remote: Compressing objects:  55% (598/1087)
2023-09-06T12:10:42.0312412Z remote: Compressing objects:  56% (609/1087)
2023-09-06T12:10:42.0313087Z remote: Compressing objects:  57% (620/1087)
2023-09-06T12:10:42.0316657Z remote: Compressing objects:  58% (631/1087)
2023-09-06T12:10:42.0317149Z remote: Compressing objects:  59% (642/1087)
2023-09-06T12:10:42.0317891Z remote: Compressing objects:  60% (653/1087)
2023-09-06T12:10:42.0318817Z remote: Compressing objects:  61% (664/1087)
2023-09-06T12:10:42.0319541Z remote: Compressing objects:  62% (674/1087)
2023-09-06T12:10:42.0320206Z remote: Compressing objects:  63% (685/1087)
2023-09-06T12:10:42.0320670Z remote: Compressing objects:  64% (696/1087)
2023-09-06T12:10:42.0321388Z remote: Compressing objects:  65% (707/1087)
2023-09-06T12:10:42.0322116Z remote: Compressing objects:  66% (718/1087)
2023-09-06T12:10:42.0323004Z remote: Compressing objects:  67% (729/1087)
2023-09-06T12:10:42.0323672Z remote: Compressing objects:  68% (740/1087)
2023-09-06T12:10:42.0324403Z remote: Compressing objects:  69% (751/1087)
2023-09-06T12:10:42.0325067Z remote: Compressing objects:  70% (761/1087)
2023-09-06T12:10:42.0325664Z remote: Compressing objects:  71% (772/1087)
2023-09-06T12:10:42.0326034Z remote: Compressing objects:  72% (783/1087)
2023-09-06T12:10:42.0326421Z remote: Compressing objects:  73% (794/1087)
2023-09-06T12:10:42.0326810Z remote: Compressing objects:  74% (805/1087)
2023-09-06T12:10:42.0327190Z remote: Compressing objects:  75% (816/1087)
2023-09-06T12:10:42.0327560Z remote: Compressing objects:  76% (827/1087)
2023-09-06T12:10:42.0327999Z remote: Compressing objects:  77% (837/1087)
2023-09-06T12:10:42.0333861Z remote: Compressing objects:  78% (848/1087)
2023-09-06T12:10:42.0341907Z remote: Compressing objects:  79% (859/1087)
2023-09-06T12:10:42.0347570Z remote: Compressing objects:  80% (870/1087)
2023-09-06T12:10:42.0351476Z remote: Compressing objects:  81% (881/1087)
2023-09-06T12:10:42.0354082Z remote: Compressing objects:  82% (892/1087)
2023-09-06T12:10:42.0355469Z remote: Compressing objects:  83% (903/1087)
2023-09-06T12:10:42.0357762Z remote: Compressing objects:  84% (914/1087)
2023-09-06T12:10:42.0363286Z remote: Compressing objects:  85% (924/1087)
2023-09-06T12:10:42.0368682Z remote: Compressing objects:  86% (935/1087)
2023-09-06T12:10:42.0372947Z remote: Compressing objects:  87% (946/1087)
2023-09-06T12:10:42.0377252Z remote: Compressing objects:  88% (957/1087)
2023-09-06T12:10:42.0380050Z remote: Compressing objects:  89% (968/1087)
2023-09-06T12:10:42.0383130Z remote: Compressing objects:  90% (979/1087)
2023-09-06T12:10:42.0386814Z remote: Compressing objects:  91% (990/1087)
2023-09-06T12:10:42.0390081Z remote: Compressing objects:  92% (1001/1087)
2023-09-06T12:10:42.0390880Z remote: Compressing objects:  93% (1011/1087)
2023-09-06T12:10:42.0393725Z remote: Compressing objects:  94% (1022/1087)
2023-09-06T12:10:42.0394117Z remote: Compressing objects:  95% (1033/1087)
2023-09-06T12:10:42.0397887Z remote: Compressing objects:  96% (1044/1087)
2023-09-06T12:10:42.0398573Z remote: Compressing objects:  97% (1055/1087)
2023-09-06T12:10:42.0402199Z remote: Compressing objects:  98% (1066/1087)
2023-09-06T12:10:42.0404397Z remote: Compressing objects:  99% (1077/1087)
2023-09-06T12:10:42.0404794Z remote: Compressing objects: 100% (1087/1087)
2023-09-06T12:10:42.0405288Z remote: Compressing objects: 100% (1087/1087), done.
2023-09-06T12:10:42.0470100Z Receiving objects:   0% (1/24307)
2023-09-06T12:10:42.0679500Z Receiving objects:   1% (244/24307)
2023-09-06T12:10:42.0727784Z Receiving objects:   2% (487/24307)
2023-09-06T12:10:42.0922132Z Receiving objects:   3% (730/24307)
2023-09-06T12:10:42.0959015Z Receiving objects:   4% (973/24307)
2023-09-06T12:10:42.1175700Z Receiving objects:   5% (1216/24307)
2023-09-06T12:10:42.1200844Z Receiving objects:   6% (1459/24307)
2023-09-06T12:10:42.1419302Z Receiving objects:   7% (1702/24307)
2023-09-06T12:10:42.1518588Z Receiving objects:   8% (1945/24307)
2023-09-06T12:10:42.1709314Z Receiving objects:   9% (2188/24307)
2023-09-06T12:10:42.1933415Z Receiving objects:  10% (2431/24307)
2023-09-06T12:10:42.2071742Z Receiving objects:  11% (2674/24307)
2023-09-06T12:10:42.2468485Z Receiving objects:  12% (2917/24307)
2023-09-06T12:10:42.3179033Z Receiving objects:  13% (3160/24307)
2023-09-06T12:10:42.3261974Z Receiving objects:  14% (3403/24307)
2023-09-06T12:10:42.3346256Z Receiving objects:  15% (3647/24307)
2023-09-06T12:10:42.3428090Z Receiving objects:  16% (3890/24307)
2023-09-06T12:10:42.3447730Z Receiving objects:  17% (4133/24307)
2023-09-06T12:10:42.3459972Z Receiving objects:  18% (4376/24307)
2023-09-06T12:10:42.4219704Z Receiving objects:  19% (4619/24307)
2023-09-06T12:10:42.4334736Z Receiving objects:  20% (4862/24307)
2023-09-06T12:10:42.4452171Z Receiving objects:  21% (5105/24307)
2023-09-06T12:10:43.0438645Z Receiving objects:  22% (5348/24307)
2023-09-06T12:10:43.2886822Z Receiving objects:  22% (5504/24307), 19.07 MiB | 19.06 MiB/s
2023-09-06T12:10:44.0440310Z Receiving objects:  23% (5591/24307), 19.07 MiB | 19.06 MiB/s
2023-09-06T12:10:44.5893696Z Receiving objects:  23% (5720/24307), 86.22 MiB | 43.13 MiB/s
2023-09-06T12:10:44.6538318Z Receiving objects:  24% (5834/24307), 118.20 MiB | 47.30 MiB/s
2023-09-06T12:10:44.7149480Z Receiving objects:  25% (6077/24307), 118.20 MiB | 47.30 MiB/s
2023-09-06T12:10:44.7754057Z Receiving objects:  26% (6320/24307), 118.20 MiB | 47.30 MiB/s
2023-09-06T12:10:44.8370999Z Receiving objects:  27% (6563/24307), 118.20 MiB | 47.30 MiB/s
2023-09-06T12:10:44.8975812Z Receiving objects:  28% (6806/24307), 118.20 MiB | 47.30 MiB/s
2023-09-06T12:10:44.9638176Z Receiving objects:  29% (7050/24307), 118.20 MiB | 47.30 MiB/s
2023-09-06T12:10:45.0246006Z Receiving objects:  30% (7293/24307), 118.20 MiB | 47.30 MiB/s
2023-09-06T12:10:45.0437566Z Receiving objects:  31% (7536/24307), 118.20 MiB | 47.30 MiB/s
2023-09-06T12:10:45.0855085Z Receiving objects:  31% (7613/24307), 154.30 MiB | 51.45 MiB/s
2023-09-06T12:10:45.1463831Z Receiving objects:  32% (7779/24307), 154.30 MiB | 51.45 MiB/s
2023-09-06T12:10:45.2068769Z Receiving objects:  33% (8022/24307), 154.30 MiB | 51.45 MiB/s
2023-09-06T12:10:45.4467886Z Receiving objects:  34% (8265/24307), 154.30 MiB | 51.45 MiB/s
2023-09-06T12:10:45.5305915Z Receiving objects:  35% (8508/24307), 154.30 MiB | 51.45 MiB/s
2023-09-06T12:10:45.5422597Z Receiving objects:  36% (8751/24307), 154.30 MiB | 51.45 MiB/s
2023-09-06T12:10:45.7509969Z Receiving objects:  37% (8994/24307), 154.30 MiB | 51.45 MiB/s
2023-09-06T12:10:45.9355674Z Receiving objects:  38% (9237/24307), 184.45 MiB | 52.71 MiB/s
2023-09-06T12:10:46.0439942Z Receiving objects:  39% (9480/24307), 184.45 MiB | 52.71 MiB/s
2023-09-06T12:10:46.1080063Z Receiving objects:  39% (9536/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.1376352Z Receiving objects:  40% (9723/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.1441321Z Receiving objects:  41% (9966/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.1593089Z Receiving objects:  42% (10209/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.1601038Z Receiving objects:  43% (10453/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.1607103Z Receiving objects:  44% (10696/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.1616276Z Receiving objects:  45% (10939/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.1624461Z Receiving objects:  46% (11182/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.1631983Z Receiving objects:  47% (11425/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.1643583Z Receiving objects:  48% (11668/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.2114451Z Receiving objects:  49% (11911/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.2122657Z Receiving objects:  50% (12154/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.2134007Z Receiving objects:  51% (12397/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.2142332Z Receiving objects:  52% (12640/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.2156164Z Receiving objects:  53% (12883/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.2164318Z Receiving objects:  54% (13126/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.2171830Z Receiving objects:  55% (13369/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.2201681Z Receiving objects:  56% (13612/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.2207124Z Receiving objects:  57% (13855/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.2214399Z Receiving objects:  58% (14099/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.2221479Z Receiving objects:  59% (14342/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.4484261Z Receiving objects:  60% (14585/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.4733622Z Receiving objects:  61% (14828/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.4741903Z Receiving objects:  62% (15071/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.4748336Z Receiving objects:  63% (15314/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.4757649Z Receiving objects:  64% (15557/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.4769749Z Receiving objects:  65% (15800/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.4777237Z Receiving objects:  66% (16043/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.4926424Z Receiving objects:  67% (16286/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.4931194Z Receiving objects:  68% (16529/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.4936381Z Receiving objects:  69% (16772/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.4941660Z Receiving objects:  70% (17015/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.4946469Z Receiving objects:  71% (17258/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.4953206Z Receiving objects:  72% (17502/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.4957574Z Receiving objects:  73% (17745/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.4964751Z Receiving objects:  74% (17988/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5100477Z Receiving objects:  75% (18231/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5112389Z Receiving objects:  76% (18474/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5121479Z Receiving objects:  77% (18717/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5138654Z Receiving objects:  78% (18960/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5160959Z Receiving objects:  79% (19203/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5170220Z Receiving objects:  80% (19446/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5196245Z Receiving objects:  81% (19689/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5215648Z Receiving objects:  82% (19932/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5244769Z Receiving objects:  83% (20175/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5249102Z Receiving objects:  84% (20418/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5251388Z Receiving objects:  85% (20661/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5253625Z Receiving objects:  86% (20905/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5256432Z Receiving objects:  87% (21148/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5259440Z Receiving objects:  88% (21391/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5263580Z Receiving objects:  89% (21634/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5268972Z Receiving objects:  90% (21877/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5276272Z Receiving objects:  91% (22120/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5280621Z Receiving objects:  92% (22363/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5306332Z Receiving objects:  93% (22606/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5338500Z Receiving objects:  94% (22849/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5353088Z Receiving objects:  95% (23092/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5374133Z Receiving objects:  96% (23335/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5429356Z Receiving objects:  97% (23578/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5469561Z Receiving objects:  98% (23821/24307), 215.91 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5509432Z Receiving objects:  99% (24064/24307), 242.89 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5510181Z remote: Total 24307 (delta 3339), reused 3581 (delta 3037), pack-reused 20176
2023-09-06T12:10:46.5532571Z Receiving objects: 100% (24307/24307), 242.89 MiB | 53.99 MiB/s
2023-09-06T12:10:46.5532956Z Receiving objects: 100% (24307/24307), 243.15 MiB | 53.92 MiB/s, done.
2023-09-06T12:10:46.5560067Z Resolving deltas:   0% (0/12383)
2023-09-06T12:10:46.5568384Z Resolving deltas:   1% (124/12383)
2023-09-06T12:10:46.5577753Z Resolving deltas:   2% (254/12383)
2023-09-06T12:10:46.5587758Z Resolving deltas:   3% (374/12383)
2023-09-06T12:10:46.5605835Z Resolving deltas:   4% (505/12383)
2023-09-06T12:10:46.5616671Z Resolving deltas:   5% (620/12383)
2023-09-06T12:10:46.5624935Z Resolving deltas:   6% (746/12383)
2023-09-06T12:10:46.5634736Z Resolving deltas:   7% (867/12383)
2023-09-06T12:10:46.5644128Z Resolving deltas:   8% (1002/12383)
2023-09-06T12:10:46.5651152Z Resolving deltas:   9% (1147/12383)
2023-09-06T12:10:46.5658117Z Resolving deltas:  10% (1241/12383)
2023-09-06T12:10:46.5668010Z Resolving deltas:  11% (1365/12383)
2023-09-06T12:10:46.5678299Z Resolving deltas:  12% (1487/12383)
2023-09-06T12:10:46.5684658Z Resolving deltas:  13% (1638/12383)
2023-09-06T12:10:46.5691705Z Resolving deltas:  14% (1768/12383)
2023-09-06T12:10:46.5700545Z Resolving deltas:  15% (1859/12383)
2023-09-06T12:10:46.5712546Z Resolving deltas:  16% (1982/12383)
2023-09-06T12:10:46.5721383Z Resolving deltas:  17% (2107/12383)
2023-09-06T12:10:46.5727542Z Resolving deltas:  18% (2259/12383)
2023-09-06T12:10:46.5747963Z Resolving deltas:  19% (2354/12383)
2023-09-06T12:10:46.5761625Z Resolving deltas:  20% (2478/12383)
2023-09-06T12:10:46.5769259Z Resolving deltas:  21% (2630/12383)
2023-09-06T12:10:46.5779910Z Resolving deltas:  22% (2729/12383)
2023-09-06T12:10:46.5794561Z Resolving deltas:  23% (2850/12383)
2023-09-06T12:10:46.5806343Z Resolving deltas:  25% (3100/12383)
2023-09-06T12:10:46.5817121Z Resolving deltas:  26% (3234/12383)
2023-09-06T12:10:46.5830052Z Resolving deltas:  27% (3344/12383)
2023-09-06T12:10:46.5836137Z Resolving deltas:  28% (3468/12383)
2023-09-06T12:10:46.5841150Z Resolving deltas:  29% (3594/12383)
2023-09-06T12:10:46.5851557Z Resolving deltas:  30% (3715/12383)
2023-09-06T12:10:46.5858017Z Resolving deltas:  31% (3857/12383)
2023-09-06T12:10:46.5861862Z Resolving deltas:  32% (3978/12383)
2023-09-06T12:10:46.5877372Z Resolving deltas:  33% (4089/12383)
2023-09-06T12:10:46.5888432Z Resolving deltas:  34% (4212/12383)
2023-09-06T12:10:46.5896615Z Resolving deltas:  35% (4338/12383)
2023-09-06T12:10:46.5902148Z Resolving deltas:  36% (4487/12383)
2023-09-06T12:10:46.5920849Z Resolving deltas:  37% (4584/12383)
2023-09-06T12:10:46.5932528Z Resolving deltas:  38% (4706/12383)
2023-09-06T12:10:46.5939998Z Resolving deltas:  39% (4855/12383)
2023-09-06T12:10:46.5947542Z Resolving deltas:  40% (4954/12383)
2023-09-06T12:10:46.5953897Z Resolving deltas:  41% (5085/12383)
2023-09-06T12:10:46.5958812Z Resolving deltas:  42% (5233/12383)
2023-09-06T12:10:46.5963699Z Resolving deltas:  43% (5327/12383)
2023-09-06T12:10:46.5970439Z Resolving deltas:  44% (5454/12383)
2023-09-06T12:10:46.5975428Z Resolving deltas:  45% (5573/12383)
2023-09-06T12:10:46.5980465Z Resolving deltas:  46% (5698/12383)
2023-09-06T12:10:46.5985791Z Resolving deltas:  47% (5824/12383)
2023-09-06T12:10:46.5992323Z Resolving deltas:  48% (5944/12383)
2023-09-06T12:10:46.5998194Z Resolving deltas:  49% (6073/12383)
2023-09-06T12:10:46.6004408Z Resolving deltas:  50% (6194/12383)
2023-09-06T12:10:46.6010029Z Resolving deltas:  51% (6316/12383)
2023-09-06T12:10:46.6015155Z Resolving deltas:  52% (6444/12383)
2023-09-06T12:10:46.6020771Z Resolving deltas:  53% (6563/12383)
2023-09-06T12:10:46.6028264Z Resolving deltas:  54% (6688/12383)
2023-09-06T12:10:46.6036224Z Resolving deltas:  55% (6842/12383)
2023-09-06T12:10:46.6041309Z Resolving deltas:  56% (6984/12383)
2023-09-06T12:10:46.6046941Z Resolving deltas:  57% (7064/12383)
2023-09-06T12:10:46.6052476Z Resolving deltas:  58% (7184/12383)
2023-09-06T12:10:46.6062091Z Resolving deltas:  59% (7315/12383)
2023-09-06T12:10:46.6074107Z Resolving deltas:  60% (7435/12383)
2023-09-06T12:10:46.6079355Z Resolving deltas:  61% (7554/12383)
2023-09-06T12:10:46.6084889Z Resolving deltas:  62% (7687/12383)
2023-09-06T12:10:46.6090258Z Resolving deltas:  63% (7802/12383)
2023-09-06T12:10:46.6097383Z Resolving deltas:  64% (7926/12383)
2023-09-06T12:10:46.6100540Z Resolving deltas:  65% (8104/12383)
2023-09-06T12:10:46.6106587Z Resolving deltas:  66% (8183/12383)
2023-09-06T12:10:46.6112281Z Resolving deltas:  67% (8297/12383)
2023-09-06T12:10:46.6117716Z Resolving deltas:  68% (8424/12383)
2023-09-06T12:10:46.6122367Z Resolving deltas:  69% (8551/12383)
2023-09-06T12:10:46.6127652Z Resolving deltas:  70% (8670/12383)
2023-09-06T12:10:46.6133601Z Resolving deltas:  71% (8794/12383)
2023-09-06T12:10:46.6140344Z Resolving deltas:  72% (8919/12383)
2023-09-06T12:10:46.6153145Z Resolving deltas:  73% (9042/12383)
2023-09-06T12:10:46.6167777Z Resolving deltas:  74% (9252/12383)
2023-09-06T12:10:46.6169147Z Resolving deltas:  76% (9499/12383)
2023-09-06T12:10:46.6201263Z Resolving deltas:  77% (9537/12383)
2023-09-06T12:10:46.6207111Z Resolving deltas:  81% (10088/12383)
2023-09-06T12:10:46.6217707Z Resolving deltas:  82% (10208/12383)
2023-09-06T12:10:46.6222958Z Resolving deltas:  83% (10400/12383)
2023-09-06T12:10:46.6226438Z Resolving deltas:  84% (10498/12383)
2023-09-06T12:10:46.6232682Z Resolving deltas:  85% (10552/12383)
2023-09-06T12:10:46.6237050Z Resolving deltas:  86% (10678/12383)
2023-09-06T12:10:46.6241795Z Resolving deltas:  87% (10793/12383)
2023-09-06T12:10:46.6247535Z Resolving deltas:  88% (10898/12383)
2023-09-06T12:10:46.6252479Z Resolving deltas:  89% (11025/12383)
2023-09-06T12:10:46.6261542Z Resolving deltas:  90% (11159/12383)
2023-09-06T12:10:46.6302033Z Resolving deltas:  91% (11293/12383)
2023-09-06T12:10:46.6313219Z Resolving deltas:  92% (11509/12383)
2023-09-06T12:10:46.6322308Z Resolving deltas:  93% (11570/12383)
2023-09-06T12:10:46.6332062Z Resolving deltas:  94% (11657/12383)
2023-09-06T12:10:46.6345414Z Resolving deltas:  95% (11776/12383)
2023-09-06T12:10:46.6356036Z Resolving deltas:  96% (11901/12383)
2023-09-06T12:10:46.6374259Z Resolving deltas:  97% (12012/12383)
2023-09-06T12:10:46.6383856Z Resolving deltas:  98% (12147/12383)
2023-09-06T12:10:46.6403920Z Resolving deltas:  99% (12261/12383)
2023-09-06T12:10:46.6404228Z Resolving deltas: 100% (12383/12383)
2023-09-06T12:10:46.6404543Z Resolving deltas: 100% (12383/12383), done.
2023-09-06T12:10:47.2668907Z + pushd torchbench
2023-09-06T12:10:47.2669751Z ~/workspace/torchbench ~/workspace
2023-09-06T12:10:47.2670110Z + git checkout 9371b9e13c826f3930e54346b4d619cb59182f68
2023-09-06T12:10:47.3164598Z Note: switching to '9371b9e13c826f3930e54346b4d619cb59182f68'.
2023-09-06T12:10:47.3165001Z 
2023-09-06T12:10:47.3165540Z You are in 'detached HEAD' state. You can look around, make experimental
2023-09-06T12:10:47.3166249Z changes and commit them, and you can discard any commits you make in this
2023-09-06T12:10:47.3167012Z state without impacting any branches by switching back to a branch.
2023-09-06T12:10:47.3167469Z 
2023-09-06T12:10:47.3167706Z If you want to create a new branch to retain commits you create, you may
2023-09-06T12:10:47.3168260Z do so (now or later) by using -c with the switch command. Example:
2023-09-06T12:10:47.3168494Z 
2023-09-06T12:10:47.3168677Z   git switch -c <new-branch-name>
2023-09-06T12:10:47.3168864Z 
2023-09-06T12:10:47.3169374Z Or undo this operation with:
2023-09-06T12:10:47.3169551Z 
2023-09-06T12:10:47.3169647Z   git switch -
2023-09-06T12:10:47.3169795Z 
2023-09-06T12:10:47.3170032Z Turn off this advice by setting config variable advice.detachedHead to false
2023-09-06T12:10:47.3170293Z 
2023-09-06T12:10:47.3170578Z HEAD is now at 9371b9e1 Don't shard torchrec_dlrm by default (#1822)
2023-09-06T12:10:47.3170907Z + '[' '' ']'
2023-09-06T12:10:47.3171342Z + python install.py --continue_on_fail
2023-09-06T12:10:49.6180491Z checking packages torch, torchvision, torchtext, torchaudio are installed...OK
2023-09-06T12:11:17.7084513Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK
2023-09-06T12:11:21.2038566Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK
2023-09-06T12:11:54.8095171Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/DALLE2_pytorch...OK
2023-09-06T12:12:02.0123591Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK
2023-09-06T12:12:04.8570009Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK
2023-09-06T12:12:04.8783364Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK
2023-09-06T12:12:41.3025794Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/attention_is_all_you_need_pytorch...OK
2023-09-06T12:12:53.6253293Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK
2023-09-06T12:12:58.2236268Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK
2023-09-06T12:13:02.7865081Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK
2023-09-06T12:13:07.3914342Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK
2023-09-06T12:13:17.8222132Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/clip...OK
2023-09-06T12:13:17.8224140Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found
2023-09-06T12:13:20.4584649Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK
2023-09-06T12:13:25.7436953Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK
2023-09-06T12:13:25.7656199Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK
2023-09-06T12:14:34.0611963Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK
2023-09-06T12:14:47.9090662Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK
2023-09-06T12:14:59.3614373Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK
2023-09-06T12:15:10.2314368Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK
2023-09-06T12:15:22.2750758Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK
2023-09-06T12:15:32.7619705Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK
2023-09-06T12:15:42.3562017Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK
2023-09-06T12:16:02.9986090Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK
2023-09-06T12:16:25.8356550Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK
2023-09-06T12:16:37.8930845Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK
2023-09-06T12:16:48.7924499Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK
2023-09-06T12:17:00.0213159Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK
2023-09-06T12:17:05.1748991Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK
2023-09-06T12:18:35.2740879Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK
2023-09-06T12:19:11.6533354Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK
2023-09-06T12:19:14.9334948Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK
2023-09-06T12:22:30.3001790Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fambench_xlmr...OK
2023-09-06T12:22:37.0863276Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK
2023-09-06T12:22:39.9729557Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK
2023-09-06T12:22:42.8455482Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK
2023-09-06T12:22:50.8912082Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK
2023-09-06T12:23:01.5857023Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK
2023-09-06T12:23:10.1683703Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK
2023-09-06T12:23:21.4677136Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK
2023-09-06T12:23:30.4778875Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK
2023-09-06T12:23:38.4906793Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK
2023-09-06T12:23:47.9734103Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK
2023-09-06T12:24:10.3702338Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK
2023-09-06T12:24:19.9817804Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK
2023-09-06T12:24:26.5123496Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK
2023-09-06T12:24:34.1978647Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK
2023-09-06T12:24:44.4611187Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK
2023-09-06T12:24:44.4613029Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found
2023-09-06T12:25:02.8964938Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK
2023-09-06T12:25:12.6704943Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK
2023-09-06T12:25:15.6322540Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK
2023-09-06T12:25:18.6076720Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK
2023-09-06T12:26:15.4423745Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK
2023-09-06T12:26:15.4671635Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK
2023-09-06T12:26:19.4693902Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK
2023-09-06T12:26:19.4924657Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK
2023-09-06T12:26:19.5154987Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK
2023-09-06T12:26:19.5381260Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK
2023-09-06T12:26:19.5608924Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK
2023-09-06T12:26:19.5835648Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK
2023-09-06T12:26:19.5840664Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt_generate...SKIP - No install.py is found
2023-09-06T12:26:22.5540740Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK
2023-09-06T12:26:27.5101295Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK
2023-09-06T12:26:27.5336675Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK
2023-09-06T12:26:27.5569694Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK
2023-09-06T12:26:27.5797700Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK
2023-09-06T12:26:27.6026963Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK
2023-09-06T12:26:27.6257187Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK
2023-09-06T12:26:37.7842370Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK
2023-09-06T12:26:41.2756577Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK
2023-09-06T12:26:52.9889659Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_struct...OK
2023-09-06T12:26:56.1681390Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK
2023-09-06T12:26:56.1916398Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK
2023-09-06T12:26:56.2149627Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK
2023-09-06T12:26:56.2381121Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK
2023-09-06T12:26:56.2615187Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK
2023-09-06T12:26:56.2842154Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK
2023-09-06T12:29:28.6034012Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK
2023-09-06T12:29:28.6284559Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK
2023-09-06T12:29:39.2569791Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK
2023-09-06T12:29:43.1899611Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK
2023-09-06T12:29:43.2136029Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK
2023-09-06T12:30:20.1247188Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion...OK
2023-09-06T12:30:25.7414830Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK
2023-09-06T12:30:48.4891102Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK
2023-09-06T12:30:48.5129637Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK
2023-09-06T12:30:48.5359571Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK
2023-09-06T12:30:48.5587514Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK
2023-09-06T12:30:48.5809837Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK
2023-09-06T12:30:48.6043309Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK
2023-09-06T12:30:48.6269261Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK
2023-09-06T12:30:48.6491203Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK
2023-09-06T12:30:59.8677473Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK
2023-09-06T12:30:59.8910949Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK
2023-09-06T12:31:03.4663968Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK
2023-09-06T12:31:07.2131299Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK
2023-09-06T12:31:07.6212110Z + popd
2023-09-06T12:31:07.6212606Z ~/workspace
2023-09-06T12:31:07.6213307Z + [[ inductor_torchbench_perf != *cpu_accuracy* ]]
2023-09-06T12:31:07.6214169Z + install_torchrec_and_fbgemm
2023-09-06T12:31:07.6214431Z + local torchrec_commit
2023-09-06T12:31:07.6223371Z ++ get_pinned_commit torchrec
2023-09-06T12:31:07.6223748Z ++ cat .github/ci_commit_pins/torchrec.txt
2023-09-06T12:31:07.6243844Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2023-09-06T12:31:07.6244429Z + local fbgemm_commit
2023-09-06T12:31:07.6248506Z ++ get_pinned_commit fbgemm
2023-09-06T12:31:07.6248958Z ++ cat .github/ci_commit_pins/fbgemm.txt
2023-09-06T12:31:07.6265621Z + fbgemm_commit=1b2746f642cc2c99fe9d1a0c34359c0de45341c2
2023-09-06T12:31:07.6267757Z + pip_uninstall torchrec-nightly
2023-09-06T12:31:07.6268149Z + pip uninstall -y torchrec-nightly
2023-09-06T12:31:08.1314032Z WARNING: Skipping torchrec-nightly as it is not installed.
2023-09-06T12:31:08.1874759Z + pip_uninstall fbgemm-gpu-nightly
2023-09-06T12:31:08.1875210Z + pip uninstall -y fbgemm-gpu-nightly
2023-09-06T12:31:08.6741663Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed.
2023-09-06T12:31:08.7378869Z + pip_install setuptools-git-versioning scikit-build pyre-extensions
2023-09-06T12:31:08.7379561Z + pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions
2023-09-06T12:31:09.3374894Z Collecting setuptools-git-versioning
2023-09-06T12:31:09.3375996Z   Obtaining dependency information for setuptools-git-versioning from https://files.pythonhosted.org/packages/1c/6b/9ab18da9ef59b242221449e6bec9901d554628454e6e6983be8cc231dbd6/setuptools_git_versioning-1.13.5-py3-none-any.whl.metadata
2023-09-06T12:31:09.4153973Z   Downloading setuptools_git_versioning-1.13.5-py3-none-any.whl.metadata (5.7 kB)
2023-09-06T12:31:09.4610755Z Collecting scikit-build
2023-09-06T12:31:09.4613926Z   Obtaining dependency information for scikit-build from https://files.pythonhosted.org/packages/fa/af/b3ef8fe0bb96bf7308e1f9d196fc069f0c75d9c74cfaad851e418cc704f4/scikit_build-0.17.6-py3-none-any.whl.metadata
2023-09-06T12:31:09.4761972Z   Downloading scikit_build-0.17.6-py3-none-any.whl.metadata (14 kB)
2023-09-06T12:31:09.5145110Z Collecting pyre-extensions
2023-09-06T12:31:09.5290919Z   Downloading pyre_extensions-0.0.30-py3-none-any.whl (12 kB)
2023-09-06T12:31:09.5389702Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (23.1)
2023-09-06T12:31:09.5393509Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (68.0.0)
2023-09-06T12:31:09.5660630Z Collecting toml>=0.10.2 (from setuptools-git-versioning)
2023-09-06T12:31:09.5794126Z   Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)
2023-09-06T12:31:09.6063732Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.8.0)
2023-09-06T12:31:09.6073408Z Requirement already satisfied: tomli in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (2.0.1)
2023-09-06T12:31:09.6080576Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.38.4)
2023-09-06T12:31:09.6105563Z Requirement already satisfied: typing-inspect in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (0.9.0)
2023-09-06T12:31:09.6110661Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.7.1)
2023-09-06T12:31:09.6355638Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0)
2023-09-06T12:31:09.6515853Z Downloading setuptools_git_versioning-1.13.5-py3-none-any.whl (10 kB)
2023-09-06T12:31:09.6695899Z Downloading scikit_build-0.17.6-py3-none-any.whl (84 kB)
2023-09-06T12:31:12.3616556Z Installing collected packages: toml, scikit-build, setuptools-git-versioning, pyre-extensions
2023-09-06T12:31:12.4981140Z Successfully installed pyre-extensions-0.0.30 scikit-build-0.17.6 setuptools-git-versioning-1.13.5 toml-0.10.2
2023-09-06T12:31:12.6227149Z + CUDA_PATH=/usr/local/cuda-12.1
2023-09-06T12:31:12.6228488Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@1b2746f642cc2c99fe9d1a0c34359c0de45341c2#egg=fbgemm-gpu&subdirectory=fbgemm_gpu'
2023-09-06T12:31:12.6230399Z + pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@1b2746f642cc2c99fe9d1a0c34359c0de45341c2#egg=fbgemm-gpu&subdirectory=fbgemm_gpu'
2023-09-06T12:31:13.1099728Z Collecting fbgemm-gpu
2023-09-06T12:31:13.1103075Z   Cloning https://github.com/pytorch/FBGEMM.git (to revision 1b2746f642cc2c99fe9d1a0c34359c0de45341c2) to /tmp/pip-install-aeiael76/fbgemm-gpu_afe94badecf746229e3ef63a23902078
2023-09-06T12:31:13.1135159Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-aeiael76/fbgemm-gpu_afe94badecf746229e3ef63a23902078
2023-09-06T12:31:15.4623908Z   Running command git rev-parse -q --verify 'sha^1b2746f642cc2c99fe9d1a0c34359c0de45341c2'
2023-09-06T12:31:15.4650698Z   Running command git fetch -q https://github.com/pytorch/FBGEMM.git 1b2746f642cc2c99fe9d1a0c34359c0de45341c2
2023-09-06T12:31:15.9781610Z   Running command git checkout -q 1b2746f642cc2c99fe9d1a0c34359c0de45341c2
2023-09-06T12:31:16.4930433Z   Resolved https://github.com/pytorch/FBGEMM.git to commit 1b2746f642cc2c99fe9d1a0c34359c0de45341c2
2023-09-06T12:31:16.4932364Z   Running command git submodule update --init --recursive -q
2023-09-06T12:31:26.0067145Z   Preparing metadata (setup.py) ... [?25l- done
2023-09-06T12:31:26.0087851Z [?25hBuilding wheels for collected packages: fbgemm-gpu
2023-09-06T13:33:56.9524235Z   Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done
2023-09-06T13:33:57.6616448Z [?25h  Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post222-cp310-cp310-linux_x86_64.whl size=240406363 sha256=30fef09557f9ffa5ad87340425fda115890fb715b353090485f9e2884368f6cc
2023-09-06T13:33:57.6617175Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/ff/05/73/e79cb64731953a7b9a891826597aab1899e1265baa4c70a191
2023-09-06T13:33:57.6648588Z Successfully built fbgemm-gpu
2023-09-06T13:34:00.4937669Z Installing collected packages: fbgemm-gpu
2023-09-06T13:34:04.5583332Z Successfully installed fbgemm-gpu-0.4.1rc0.post222
2023-09-06T13:34:05.4601664Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2023-09-06T13:34:05.4602763Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2023-09-06T13:34:05.9965826Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2023-09-06T13:34:05.9976209Z   Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-tdtjqbh4
2023-09-06T13:34:06.0012364Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-tdtjqbh4
2023-09-06T13:34:07.1821874Z   Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2'
2023-09-06T13:34:07.1855514Z   Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2023-09-06T13:34:07.6369487Z   Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2023-09-06T13:34:08.1233727Z   Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2023-09-06T13:34:08.4520897Z   Preparing metadata (setup.py) ... [?25l- done
2023-09-06T13:34:08.4602430Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9)
2023-09-06T13:34:08.4606575Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.1.0)
2023-09-06T13:34:08.4612885Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0)
2023-09-06T13:34:08.4617547Z Requirement already satisfied: torchmetrics in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (1.1.1)
2023-09-06T13:34:08.4621576Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.66.1)
2023-09-06T13:34:08.4626944Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post222)
2023-09-06T13:34:08.4661901Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (2.7.0)
2023-09-06T13:34:08.5469763Z Requirement already satisfied: numpy>=1.22.4 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (1.22.4)
2023-09-06T13:34:08.5478651Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.8.2)
2023-09-06T13:34:08.5486193Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2023.3)
2023-09-06T13:34:08.5494429Z Requirement already satisfied: tzdata>=2022.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2023.3)
2023-09-06T13:34:08.6719654Z Requirement already satisfied: torch>=1.8.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.2.0a0+git3fe8417)
2023-09-06T13:34:08.6726931Z Requirement already satisfied: lightning-utilities>=0.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (0.9.0)
2023-09-06T13:34:08.6882546Z Requirement already satisfied: packaging>=17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (23.1)
2023-09-06T13:34:08.6884496Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.7.1)
2023-09-06T13:34:08.6917827Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.16.0)
2023-09-06T13:34:08.7002205Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.8.1->torchmetrics->torchrec==0.3.2) (3.9.0)
2023-09-06T13:34:08.7005772Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.8.1->torchmetrics->torchrec==0.3.2) (1.12)
2023-09-06T13:34:08.7011269Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.8.1->torchmetrics->torchrec==0.3.2) (2.8.8)
2023-09-06T13:34:08.7015998Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.8.1->torchmetrics->torchrec==0.3.2) (3.1.2)
2023-09-06T13:34:08.7020377Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.8.1->torchmetrics->torchrec==0.3.2) (2023.4.0)
2023-09-06T13:34:08.7653175Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=1.8.1->torchmetrics->torchrec==0.3.2) (2.1.3)
2023-09-06T13:34:08.7872929Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>=1.8.1->torchmetrics->torchrec==0.3.2) (1.3.0)
2023-09-06T13:34:08.8262706Z Building wheels for collected packages: torchrec
2023-09-06T13:34:09.5282425Z   Building wheel for torchrec (setup.py) ... [?25l- \ | / done
2023-09-06T13:34:09.5302442Z [?25h  Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374487 sha256=058b5348db66d0c85bc7f2f8cfa4cc87083eb5d23160a2259f34eb99bfa869af
2023-09-06T13:34:09.5303131Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8
2023-09-06T13:34:09.5345344Z Successfully built torchrec
2023-09-06T13:34:12.0510548Z Installing collected packages: torchrec
2023-09-06T13:34:12.3513030Z Successfully installed torchrec-0.3.2
2023-09-06T13:34:12.5823174Z ++ pwd
2023-09-06T13:34:12.5823605Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench
2023-09-06T13:34:12.5823954Z + test_dynamo_benchmark torchbench 0
2023-09-06T13:34:12.5831474Z ++ pwd
2023-09-06T13:34:12.5834528Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2023-09-06T13:34:12.5835118Z + local suite=torchbench
2023-09-06T13:34:12.5835510Z + shift
2023-09-06T13:34:12.5835871Z + local shard_id=0
2023-09-06T13:34:12.5836257Z + shift
2023-09-06T13:34:12.5837419Z + [[ inductor_torchbench_perf == *perf_compare* ]]
2023-09-06T13:34:12.5838054Z + [[ inductor_torchbench_perf == *perf* ]]
2023-09-06T13:34:12.5838540Z + test_single_dynamo_benchmark dashboard torchbench 0
2023-09-06T13:34:12.5845194Z ++ pwd
2023-09-06T13:34:12.5847627Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2023-09-06T13:34:12.5848115Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports
2023-09-06T13:34:12.5870282Z + local name=dashboard
2023-09-06T13:34:12.5870719Z + shift
2023-09-06T13:34:12.5871103Z + local suite=torchbench
2023-09-06T13:34:12.5871520Z + shift
2023-09-06T13:34:12.5871856Z + local shard_id=0
2023-09-06T13:34:12.5872492Z + shift
2023-09-06T13:34:12.5872721Z + partition_flags=()
2023-09-06T13:34:12.5873104Z + local partition_flags
2023-09-06T13:34:12.5873816Z + [[ -n 4 ]]
2023-09-06T13:34:12.5874250Z + [[ -n 0 ]]
2023-09-06T13:34:12.5875764Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id")
2023-09-06T13:34:12.5876345Z + [[ inductor_torchbench_perf == *perf_compare* ]]
2023-09-06T13:34:12.5876786Z + [[ inductor_torchbench_perf == *perf* ]]
2023-09-06T13:34:12.5877614Z + test_perf_for_dashboard torchbench --device cuda --total-partitions 4 --partition-id 0
2023-09-06T13:34:12.5878220Z ++ pwd
2023-09-06T13:34:12.5881489Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2023-09-06T13:34:12.5882072Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports
2023-09-06T13:34:12.5898943Z + local suite=torchbench
2023-09-06T13:34:12.5899359Z + shift
2023-09-06T13:34:12.5899726Z + local backend=inductor
2023-09-06T13:34:12.5900095Z + modes=()
2023-09-06T13:34:12.5900459Z + local modes
2023-09-06T13:34:12.5901991Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *training-true* ]]
2023-09-06T13:34:12.5903398Z + modes+=(training)
2023-09-06T13:34:12.5904665Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *inference-true* ]]
2023-09-06T13:34:12.5905348Z + modes+=(inference)
2023-09-06T13:34:12.5905626Z + targets=(accuracy performance)
2023-09-06T13:34:12.5905879Z + local targets
2023-09-06T13:34:12.5906126Z + for mode in "${modes[@]}"
2023-09-06T13:34:12.5906408Z + [[ training == \i\n\f\e\r\e\n\c\e ]]
2023-09-06T13:34:12.5906701Z + [[ training == \t\r\a\i\n\i\n\g ]]
2023-09-06T13:34:12.5906940Z + dtype=amp
2023-09-06T13:34:12.5907193Z + for target in "${targets[@]}"
2023-09-06T13:34:12.5907539Z + target_flag=("--${target}")
2023-09-06T13:34:12.5907783Z + local target_flag
2023-09-06T13:34:12.5908065Z + [[ accuracy == \p\e\r\f\o\r\m\a\n\c\e ]]
2023-09-06T13:34:12.5908726Z + [[ accuracy == \a\c\c\u\r\a\c\y ]]
2023-09-06T13:34:12.5909372Z + target_flag+=(--no-translation-validation)
2023-09-06T13:34:12.5910307Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *default-true* ]]
2023-09-06T13:34:12.5911771Z + python benchmarks/dynamo/torchbench.py --accuracy --no-translation-validation --training --amp --backend inductor --disable-cudagraphs --device cuda --total-partitions 4 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inductor_no_cudagraphs_torchbench_amp_training_cuda_accuracy.csv
2023-09-06T13:34:19.8249173Z 
2023-09-06T13:34:21.1584912Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:34:21.1585274Z loading model: 0it [00:01, ?it/s]
2023-09-06T13:34:21.1585581Z cuda train torchrec_dlrm                      
2023-09-06T13:34:21.1814120Z WARNING:common:fp64 golden ref were not generated for torchrec_dlrm. Setting accuracy check to cosine
2023-09-06T13:34:38.7716576Z pass
2023-09-06T13:34:43.8220740Z 
2023-09-06T13:34:47.0438435Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:34:47.0438813Z loading model: 0it [00:03, ?it/s]
2023-09-06T13:34:47.0439118Z cuda train BERT_pytorch                       
2023-09-06T13:35:41.4048408Z pass
2023-09-06T13:35:47.1185749Z 
2023-09-06T13:35:53.0627136Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:35:53.0628706Z loading model: 0it [00:05, ?it/s]
2023-09-06T13:35:53.0629022Z cuda train Background_Matting                 
2023-09-06T13:35:53.0629634Z pass_due_to_skip
2023-09-06T13:35:57.7423753Z 
2023-09-06T13:35:59.2558795Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:35:59.2559434Z 
2023-09-06T13:35:59.3566907Z   0%|                                               | 0.00/354M [00:00<?, ?iB/s]
2023-09-06T13:35:59.3567177Z 
2023-09-06T13:35:59.4575801Z   0%|▏                                     | 1.56M/354M [00:00<00:22, 15.5MiB/s]
2023-09-06T13:35:59.4576264Z 
2023-09-06T13:35:59.5574021Z   5%|█▊                                    | 16.5M/354M [00:00<00:03, 94.1MiB/s]
2023-09-06T13:35:59.5574845Z 
2023-09-06T13:35:59.6574256Z   9%|███▍                                   | 30.8M/354M [00:00<00:02, 116MiB/s]
2023-09-06T13:35:59.6574526Z 
2023-09-06T13:35:59.7574373Z  13%|█████                                  | 45.9M/354M [00:00<00:02, 130MiB/s]
2023-09-06T13:35:59.7574669Z 
2023-09-06T13:35:59.8574906Z  17%|██████▊                                | 61.8M/354M [00:00<00:02, 140MiB/s]
2023-09-06T13:35:59.8575205Z 
2023-09-06T13:35:59.9605785Z  22%|████████▌                              | 78.3M/354M [00:00<00:01, 148MiB/s]
2023-09-06T13:35:59.9606196Z 
2023-09-06T13:36:00.0602056Z  26%|██████████▎                            | 93.1M/354M [00:00<00:01, 147MiB/s]
2023-09-06T13:36:00.0602322Z 
2023-09-06T13:36:00.1604624Z  31%|████████████▏                           | 108M/354M [00:00<00:01, 149MiB/s]
2023-09-06T13:36:00.1605072Z 
2023-09-06T13:36:00.2608489Z  35%|█████████████▉                          | 124M/354M [00:00<00:01, 150MiB/s]
2023-09-06T13:36:00.2608799Z 
2023-09-06T13:36:00.3613814Z  39%|███████████████▋                        | 139M/354M [00:01<00:01, 150MiB/s]
2023-09-06T13:36:00.3614279Z 
2023-09-06T13:36:00.4621088Z  43%|█████████████████▍                      | 154M/354M [00:01<00:01, 150MiB/s]
2023-09-06T13:36:00.4621440Z 
2023-09-06T13:36:00.5625164Z  48%|███████████████████                     | 169M/354M [00:01<00:01, 150MiB/s]
2023-09-06T13:36:00.5625455Z 
2023-09-06T13:36:00.6663872Z  52%|████████████████████▊                   | 184M/354M [00:01<00:01, 150MiB/s]
2023-09-06T13:36:00.6664167Z 
2023-09-06T13:36:00.7666923Z  56%|██████████████████████▍                 | 199M/354M [00:01<00:01, 148MiB/s]
2023-09-06T13:36:00.7667219Z 
2023-09-06T13:36:00.8673580Z  61%|████████████████████████▎               | 215M/354M [00:01<00:00, 153MiB/s]
2023-09-06T13:36:00.8673880Z 
2023-09-06T13:36:00.9804491Z  65%|██████████████████████████              | 231M/354M [00:01<00:00, 153MiB/s]
2023-09-06T13:36:00.9805991Z 
2023-09-06T13:36:01.0803710Z  70%|███████████████████████████▊            | 246M/354M [00:01<00:00, 147MiB/s]
2023-09-06T13:36:01.0804035Z 
2023-09-06T13:36:01.1805212Z  74%|█████████████████████████████▍          | 261M/354M [00:01<00:00, 148MiB/s]
2023-09-06T13:36:01.1805493Z 
2023-09-06T13:36:01.2808960Z  78%|███████████████████████████████▏        | 276M/354M [00:01<00:00, 149MiB/s]
2023-09-06T13:36:01.2809337Z 
2023-09-06T13:36:01.3903044Z  82%|████████████████████████████████▉       | 291M/354M [00:02<00:00, 149MiB/s]
2023-09-06T13:36:01.3903390Z 
2023-09-06T13:36:01.4907006Z  86%|██████████████████████████████████▌     | 306M/354M [00:02<00:00, 145MiB/s]
2023-09-06T13:36:01.4907311Z 
2023-09-06T13:36:01.5930296Z  91%|████████████████████████████████████▏   | 321M/354M [00:02<00:00, 145MiB/s]
2023-09-06T13:36:01.5930782Z 
2023-09-06T13:36:01.6989476Z  95%|█████████████████████████████████████▊  | 335M/354M [00:02<00:00, 144MiB/s]
2023-09-06T13:36:01.6990038Z 
2023-09-06T13:36:01.7329882Z  99%|███████████████████████████████████████▌| 350M/354M [00:02<00:00, 142MiB/s]
2023-09-06T13:36:01.7330537Z 100%|████████████████████████████████████████| 354M/354M [00:02<00:00, 143MiB/s]
2023-09-06T13:36:11.3568306Z 
2023-09-06T13:36:11.3570214Z loading model: 0it [00:13, ?it/s]
2023-09-06T13:36:11.3570561Z Eager model failed to run
2023-09-06T13:36:11.3574553Z Traceback (most recent call last):
2023-09-06T13:36:11.3575356Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T13:36:11.3575984Z     self.model_iter_fn(model, example_inputs)
2023-09-06T13:36:11.3581706Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_and_backward_pass
2023-09-06T13:36:11.3582358Z     self.grad_scaler.scale(loss).backward()
2023-09-06T13:36:11.3583694Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
2023-09-06T13:36:11.3585256Z     torch.autograd.backward(
2023-09-06T13:36:11.3586026Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
2023-09-06T13:36:11.3586556Z     Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2023-09-06T13:36:11.3587053Z RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
2023-09-06T13:36:11.3587466Z 
2023-09-06T13:36:11.3587841Z The above exception was the direct cause of the following exception:
2023-09-06T13:36:11.3588360Z 
2023-09-06T13:36:11.3588588Z Traceback (most recent call last):
2023-09-06T13:36:11.3588996Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T13:36:11.3589797Z     ) = runner.load_model(
2023-09-06T13:36:11.3590177Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T13:36:11.3590589Z     self.validate_model(model, example_inputs)
2023-09-06T13:36:11.3591265Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T13:36:11.3591743Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T13:36:11.3592125Z NotImplementedError: Eager model failed to run
2023-09-06T13:36:11.3592337Z 
2023-09-06T13:36:11.3592475Z WARNING:root:DALLE2_pytorch failed to load
2023-09-06T13:36:16.1608912Z 
2023-09-06T13:36:20.4166791Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:36:20.4167129Z loading model: 0it [00:04, ?it/s]
2023-09-06T13:36:20.4167452Z cuda train LearningToPaint                    
2023-09-06T13:36:42.6844818Z pass
2023-09-06T13:36:48.2814793Z 
2023-09-06T13:36:50.0907549Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/vgg16-397923af.pth
2023-09-06T13:36:50.1748845Z 
2023-09-06T13:36:50.1749399Z 
2023-09-06T13:36:50.2755822Z   0% 0.00/528M [00:00<?, ?B/s]
2023-09-06T13:36:50.2756790Z 
2023-09-06T13:36:50.3758059Z   1% 7.05M/528M [00:00<00:07, 73.7MB/s]
2023-09-06T13:36:50.3758304Z 
2023-09-06T13:36:50.4759703Z   3% 16.4M/528M [00:00<00:06, 88.0MB/s]
2023-09-06T13:36:50.4759933Z 
2023-09-06T13:36:50.5764587Z   5% 25.9M/528M [00:00<00:05, 92.9MB/s]
2023-09-06T13:36:50.5764961Z 
2023-09-06T13:36:50.6765138Z   7% 35.2M/528M [00:00<00:05, 94.6MB/s]
2023-09-06T13:36:50.6765529Z 
2023-09-06T13:36:50.7766254Z   8% 44.5M/528M [00:00<00:05, 95.8MB/s]
2023-09-06T13:36:50.7766652Z 
2023-09-06T13:36:50.8766239Z  10% 54.1M/528M [00:00<00:05, 97.2MB/s]
2023-09-06T13:36:50.8766608Z 
2023-09-06T13:36:50.9769288Z  12% 63.6M/528M [00:00<00:04, 98.3MB/s]
2023-09-06T13:36:50.9769679Z 
2023-09-06T13:36:51.0768902Z  14% 73.0M/528M [00:00<00:04, 98.2MB/s]
2023-09-06T13:36:51.0769277Z 
2023-09-06T13:36:51.1826377Z  16% 82.6M/528M [00:00<00:04, 98.8MB/s]
2023-09-06T13:36:51.1826637Z 
2023-09-06T13:36:51.2827869Z  17% 92.0M/528M [00:01<00:04, 97.1MB/s]
2023-09-06T13:36:51.2828227Z 
2023-09-06T13:36:51.3827372Z  19% 102M/528M [00:01<00:04, 98.0MB/s] 
2023-09-06T13:36:51.3827737Z 
2023-09-06T13:36:51.4935017Z  21% 111M/528M [00:01<00:04, 98.4MB/s]
2023-09-06T13:36:51.4935269Z 
2023-09-06T13:36:51.5935613Z  23% 120M/528M [00:01<00:04, 95.4MB/s]
2023-09-06T13:36:51.5935875Z 
2023-09-06T13:36:51.6939030Z  25% 130M/528M [00:01<00:04, 95.7MB/s]
2023-09-06T13:36:51.6939275Z 
2023-09-06T13:36:51.8017685Z  26% 139M/528M [00:01<00:04, 96.9MB/s]
2023-09-06T13:36:51.8017985Z 
2023-09-06T13:36:51.9033790Z  28% 148M/528M [00:01<00:04, 94.7MB/s]
2023-09-06T13:36:51.9034027Z 
2023-09-06T13:36:52.0034575Z  30% 157M/528M [00:01<00:04, 94.3MB/s]
2023-09-06T13:36:52.0034835Z 
2023-09-06T13:36:52.1084480Z  32% 167M/528M [00:01<00:03, 95.8MB/s]
2023-09-06T13:36:52.1084701Z 
2023-09-06T13:36:52.2086020Z  33% 176M/528M [00:01<00:03, 94.5MB/s]
2023-09-06T13:36:52.2086264Z 
2023-09-06T13:36:52.3084614Z  35% 186M/528M [00:02<00:03, 95.7MB/s]
2023-09-06T13:36:52.3084847Z 
2023-09-06T13:36:52.4132936Z  37% 195M/528M [00:02<00:03, 96.1MB/s]
2023-09-06T13:36:52.4133243Z 
2023-09-06T13:36:52.5134387Z  39% 204M/528M [00:02<00:03, 94.8MB/s]
2023-09-06T13:36:52.5134625Z 
2023-09-06T13:36:52.6136773Z  40% 213M/528M [00:02<00:03, 95.6MB/s]
2023-09-06T13:36:52.6137007Z 
2023-09-06T13:36:52.7144267Z  42% 222M/528M [00:02<00:03, 95.5MB/s]
2023-09-06T13:36:52.7144527Z 
2023-09-06T13:36:52.8144340Z  44% 232M/528M [00:02<00:03, 96.8MB/s]
2023-09-06T13:36:52.8144688Z 
2023-09-06T13:36:52.9161467Z  46% 242M/528M [00:02<00:03, 97.9MB/s]
2023-09-06T13:36:52.9161921Z 
2023-09-06T13:36:53.0161228Z  48% 251M/528M [00:02<00:02, 97.5MB/s]
2023-09-06T13:36:53.0161611Z 
2023-09-06T13:36:53.1161142Z  49% 260M/528M [00:02<00:02, 98.2MB/s]
2023-09-06T13:36:53.1161496Z 
2023-09-06T13:36:53.2197038Z  51% 270M/528M [00:02<00:02, 98.3MB/s]
2023-09-06T13:36:53.2198305Z 
2023-09-06T13:36:53.3195740Z  53% 279M/528M [00:03<00:02, 97.4MB/s]
2023-09-06T13:36:53.3196135Z 
2023-09-06T13:36:53.4196897Z  55% 289M/528M [00:03<00:02, 97.4MB/s]
2023-09-06T13:36:53.4197161Z 
2023-09-06T13:36:53.5284735Z  56% 298M/528M [00:03<00:02, 98.2MB/s]
2023-09-06T13:36:53.5284961Z 
2023-09-06T13:36:53.6286057Z  58% 307M/528M [00:03<00:02, 95.7MB/s]
2023-09-06T13:36:53.6286326Z 
2023-09-06T13:36:53.7290929Z  60% 317M/528M [00:03<00:02, 96.3MB/s]
2023-09-06T13:36:53.7291224Z 
2023-09-06T13:36:53.8292714Z  62% 326M/528M [00:03<00:02, 96.9MB/s]
2023-09-06T13:36:53.8292927Z 
2023-09-06T13:36:53.9292847Z  64% 336M/528M [00:03<00:02, 97.3MB/s]
2023-09-06T13:36:53.9293124Z 
2023-09-06T13:36:54.0293980Z  65% 345M/528M [00:03<00:01, 98.1MB/s]
2023-09-06T13:36:54.0294211Z 
2023-09-06T13:36:54.1295314Z  67% 355M/528M [00:03<00:01, 99.0MB/s]
2023-09-06T13:36:54.1295540Z 
2023-09-06T13:36:54.2299104Z  69% 364M/528M [00:03<00:01, 99.1MB/s]
2023-09-06T13:36:54.2299380Z 
2023-09-06T13:36:54.3326831Z  71% 374M/528M [00:04<00:01, 99.1MB/s]
2023-09-06T13:36:54.3327067Z 
2023-09-06T13:36:54.4329781Z  73% 383M/528M [00:04<00:01, 98.2MB/s]
2023-09-06T13:36:54.4330337Z 
2023-09-06T13:36:54.5441814Z  74% 393M/528M [00:04<00:01, 98.8MB/s]
2023-09-06T13:36:54.5442046Z 
2023-09-06T13:36:54.6446939Z  76% 402M/528M [00:04<00:01, 95.6MB/s]
2023-09-06T13:36:54.6447172Z 
2023-09-06T13:36:54.7504634Z  78% 411M/528M [00:04<00:01, 95.9MB/s]
2023-09-06T13:36:54.7504868Z 
2023-09-06T13:36:54.8503188Z  80% 421M/528M [00:04<00:01, 94.4MB/s]
2023-09-06T13:36:54.9510402Z 
2023-09-06T13:36:54.9511247Z  82% 430M/528M [00:04<00:01, 96.2MB/s]
2023-09-06T13:36:54.9511475Z 
2023-09-06T13:36:55.0511425Z  83% 440M/528M [00:04<00:00, 97.1MB/s]
2023-09-06T13:36:55.0511664Z 
2023-09-06T13:36:55.1544638Z  85% 449M/528M [00:04<00:00, 98.4MB/s]
2023-09-06T13:36:55.1544853Z 
2023-09-06T13:36:55.2545885Z  87% 459M/528M [00:04<00:00, 97.5MB/s]
2023-09-06T13:36:55.2546137Z 
2023-09-06T13:36:55.3627753Z  89% 468M/528M [00:05<00:00, 98.3MB/s]
2023-09-06T13:36:55.3627986Z 
2023-09-06T13:36:55.4629025Z  91% 478M/528M [00:05<00:00, 96.0MB/s]
2023-09-06T13:36:55.4630722Z 
2023-09-06T13:36:55.5633486Z  92% 487M/528M [00:05<00:00, 97.0MB/s]
2023-09-06T13:36:55.5633715Z 
2023-09-06T13:36:55.6634295Z  94% 497M/528M [00:05<00:00, 97.4MB/s]
2023-09-06T13:36:55.6634524Z 
2023-09-06T13:36:55.7635578Z  96% 506M/528M [00:05<00:00, 97.5MB/s]
2023-09-06T13:36:55.7635822Z 
2023-09-06T13:36:55.8635207Z  98% 516M/528M [00:05<00:00, 98.7MB/s]
2023-09-06T13:36:55.8635466Z 
2023-09-06T13:36:55.8884510Z 100% 526M/528M [00:05<00:00, 99.8MB/s]
2023-09-06T13:36:55.8884847Z 100% 528M/528M [00:05<00:00, 96.9MB/s]
2023-09-06T13:36:57.4375769Z 
2023-09-06T13:36:57.4376591Z loading model: 0it [00:09, ?it/s]
2023-09-06T13:36:57.4377137Z cuda train Super_SloMo                        
2023-09-06T13:38:20.2723434Z pass
2023-09-06T13:38:25.8571715Z 
2023-09-06T13:38:26.4957341Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/alexnet-owt-7be5be79.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/alexnet-owt-7be5be79.pth
2023-09-06T13:38:26.6145831Z 
2023-09-06T13:38:26.6146244Z 
2023-09-06T13:38:26.7149685Z   0% 0.00/233M [00:00<?, ?B/s]
2023-09-06T13:38:26.7149955Z 
2023-09-06T13:38:26.8150664Z   3% 5.97M/233M [00:00<00:03, 62.5MB/s]
2023-09-06T13:38:26.8150960Z 
2023-09-06T13:38:26.9149605Z   7% 16.6M/233M [00:00<00:02, 91.3MB/s]
2023-09-06T13:38:26.9149845Z 
2023-09-06T13:38:27.0149772Z  14% 33.7M/233M [00:00<00:01, 131MB/s] 
2023-09-06T13:38:27.0150051Z 
2023-09-06T13:38:27.1151007Z  22% 52.2M/233M [00:00<00:01, 156MB/s]
2023-09-06T13:38:27.1151257Z 
2023-09-06T13:38:27.2150742Z  30% 70.8M/233M [00:00<00:01, 170MB/s]
2023-09-06T13:38:27.2151004Z 
2023-09-06T13:38:27.3150805Z  38% 89.3M/233M [00:00<00:00, 178MB/s]
2023-09-06T13:38:27.3151056Z 
2023-09-06T13:38:27.4419811Z  46% 108M/233M [00:00<00:00, 183MB/s] 
2023-09-06T13:38:27.4420221Z 
2023-09-06T13:38:27.5418577Z  54% 125M/233M [00:00<00:00, 169MB/s]
2023-09-06T13:38:27.5418968Z 
2023-09-06T13:38:27.6418323Z  62% 144M/233M [00:00<00:00, 177MB/s]
2023-09-06T13:38:27.6419223Z 
2023-09-06T13:38:27.7484670Z  70% 163M/233M [00:01<00:00, 183MB/s]
2023-09-06T13:38:27.7485064Z 
2023-09-06T13:38:27.9427230Z  77% 180M/233M [00:01<00:00, 180MB/s]
2023-09-06T13:38:27.9427796Z 
2023-09-06T13:38:28.0427579Z  85% 197M/233M [00:01<00:00, 141MB/s]
2023-09-06T13:38:28.0427997Z 
2023-09-06T13:38:28.1355406Z  93% 216M/233M [00:01<00:00, 154MB/s]
2023-09-06T13:38:28.1355878Z 100% 233M/233M [00:01<00:00, 161MB/s]
2023-09-06T13:38:29.4311634Z 
2023-09-06T13:38:29.4312121Z loading model: 0it [00:03, ?it/s]
2023-09-06T13:38:29.4315314Z cuda train alexnet                            
2023-09-06T13:38:44.8072193Z pass
2023-09-06T13:38:49.9346215Z 
2023-09-06T13:38:59.3929072Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:38:59.3929436Z loading model: 0it [00:09, ?it/s]
2023-09-06T13:38:59.3929762Z cuda train attention_is_all_you_need_pytorch  
2023-09-06T13:39:55.5230160Z pass
2023-09-06T13:40:01.5882585Z 
2023-09-06T13:40:04.6977174Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:40:04.6978138Z loading model: 0it [00:03, ?it/s]
2023-09-06T13:40:04.6978453Z cuda train basic_gnn_edgecnn                  
2023-09-06T13:40:05.6406059Z [2023-09-06 13:40:05,639] [4/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T13:40:18.1644272Z pass
2023-09-06T13:40:23.1391967Z 
2023-09-06T13:40:26.0392085Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:40:26.0392867Z loading model: 0it [00:02, ?it/s]
2023-09-06T13:40:26.0393173Z cuda train basic_gnn_gcn                      
2023-09-06T13:40:34.8338319Z [2023-09-06 13:40:34,832] [14/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T13:40:39.3773360Z pass
2023-09-06T13:40:44.2944301Z 
2023-09-06T13:40:47.2418546Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:40:47.2419058Z loading model: 0it [00:02, ?it/s]
2023-09-06T13:40:47.2419360Z cuda train basic_gnn_gin                      
2023-09-06T13:40:48.2386276Z [2023-09-06 13:40:48,237] [4/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T13:40:57.0081667Z pass
2023-09-06T13:41:01.9193389Z 
2023-09-06T13:41:04.9482813Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:41:04.9483260Z loading model: 0it [00:03, ?it/s]
2023-09-06T13:41:04.9483563Z cuda train basic_gnn_sage                     
2023-09-06T13:41:05.9237427Z [2023-09-06 13:41:05,922] [4/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T13:41:13.6556810Z pass
2023-09-06T13:41:18.5455735Z 
2023-09-06T13:41:24.1627891Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:41:24.1628249Z loading model: 0it [00:05, ?it/s]
2023-09-06T13:41:24.1628531Z Eager model failed to run
2023-09-06T13:41:24.1636884Z Traceback (most recent call last):
2023-09-06T13:41:24.1637657Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T13:41:24.1639320Z     self.model_iter_fn(model, example_inputs)
2023-09-06T13:41:24.1640164Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_and_backward_pass
2023-09-06T13:41:24.1640806Z     self.grad_scaler.scale(loss).backward()
2023-09-06T13:41:24.1644534Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
2023-09-06T13:41:24.1645332Z     torch.autograd.backward(
2023-09-06T13:41:24.1646403Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
2023-09-06T13:41:24.1647330Z     Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2023-09-06T13:41:24.1648179Z RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
2023-09-06T13:41:24.1648567Z 
2023-09-06T13:41:24.1648971Z The above exception was the direct cause of the following exception:
2023-09-06T13:41:24.1649361Z 
2023-09-06T13:41:24.1649598Z Traceback (most recent call last):
2023-09-06T13:41:24.1650239Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T13:41:24.1650882Z     ) = runner.load_model(
2023-09-06T13:41:24.1651506Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T13:41:24.1652271Z     self.validate_model(model, example_inputs)
2023-09-06T13:41:24.1652981Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T13:41:24.1653774Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T13:41:24.1654486Z NotImplementedError: Eager model failed to run
2023-09-06T13:41:24.1654797Z 
2023-09-06T13:41:24.1655043Z WARNING:root:cm3leon_generate failed to load
2023-09-06T13:41:28.7181992Z 
2023-09-06T13:41:29.7395717Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:41:29.7396608Z loading model: 0it [00:01, ?it/s]
2023-09-06T13:41:29.7396890Z cuda train dcgan                              
2023-09-06T13:41:40.1873698Z pass
2023-09-06T13:41:45.2172589Z 
2023-09-06T13:41:49.4260527Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:41:49.4260893Z loading model: 0it [00:04, ?it/s]
2023-09-06T13:41:49.4261177Z cuda train demucs                             
2023-09-06T13:42:22.9571307Z pass
2023-09-06T13:42:28.2160890Z 
2023-09-06T13:42:28.5139685Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/densenet121-a639ec97.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/densenet121-a639ec97.pth
2023-09-06T13:42:28.6080604Z 
2023-09-06T13:42:28.6080631Z 
2023-09-06T13:42:28.7083060Z   0% 0.00/30.8M [00:00<?, ?B/s]
2023-09-06T13:42:28.7083313Z 
2023-09-06T13:42:28.8084666Z  22% 6.74M/30.8M [00:00<00:00, 70.7MB/s]
2023-09-06T13:42:28.8084914Z 
2023-09-06T13:42:28.8817859Z  63% 19.3M/30.8M [00:00<00:00, 107MB/s] 
2023-09-06T13:42:28.8818203Z 100% 30.8M/30.8M [00:00<00:00, 118MB/s]
2023-09-06T13:42:32.6486118Z 
2023-09-06T13:42:32.6487207Z loading model: 0it [00:04, ?it/s]
2023-09-06T13:44:31.0417320Z cuda train densenet121                        
2023-09-06T13:44:31.0417658Z pass
2023-09-06T13:44:37.9910555Z 
2023-09-06T13:44:39.5234295Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:44:39.5234655Z loading model: 0it [00:01, ?it/s]
2023-09-06T13:44:39.5236045Z FCOS train is not supported by upstream detectron2. See GH Issue: https://github.com/facebookresearch/detectron2/issues/4369.
2023-09-06T13:44:39.5239108Z Traceback (most recent call last):
2023-09-06T13:44:39.5239819Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T13:44:39.5240222Z     ) = runner.load_model(
2023-09-06T13:44:39.5240611Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 368, in load_model
2023-09-06T13:44:39.5245104Z     benchmark = benchmark_cls(
2023-09-06T13:44:39.5245616Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 25, in __call__
2023-09-06T13:44:39.5246046Z     obj = type.__call__(cls, *args, **kwargs)
2023-09-06T13:44:39.5246514Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn/__init__.py", line 15, in __init__
2023-09-06T13:44:39.5247953Z     super().__init__(variant="COCO-Detection/fcos_R_50_FPN_1x.py", test=test, device=device,
2023-09-06T13:44:39.5248951Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/detectron2/model_factory.py", line 105, in __init__
2023-09-06T13:44:39.5249520Z     raise NotImplementedError("FCOS train is not supported by upstream detectron2. " \
2023-09-06T13:44:39.5250155Z NotImplementedError: FCOS train is not supported by upstream detectron2. See GH Issue: https://github.com/facebookresearch/detectron2/issues/4369.
2023-09-06T13:44:39.5250536Z 
2023-09-06T13:44:39.5250698Z WARNING:root:detectron2_fcos_r_50_fpn failed to load
2023-09-06T13:44:40.5031118Z accuracy            pass_rate=92.86%
2023-09-06T13:44:40.5031518Z calls_captured      gmean=0.00x mean=176.643x
2023-09-06T13:44:40.5032939Z unique_graphs       gmean=0.00x mean=3.071x
2023-09-06T13:44:40.5033789Z graph_breaks        gmean=0.00x mean=9.143x
2023-09-06T13:44:40.5035407Z unique_graph_breaks gmean=0.00x mean=5.786x
2023-09-06T13:44:41.2960553Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *cudagraphs-true* ]]
2023-09-06T13:44:41.2962129Z + python benchmarks/dynamo/torchbench.py --accuracy --no-translation-validation --training --amp --backend inductor --device cuda --total-partitions 4 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inductor_with_cudagraphs_torchbench_amp_training_cuda_accuracy.csv
2023-09-06T13:44:48.6442923Z 
2023-09-06T13:44:49.9889167Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:44:49.9890034Z loading model: 0it [00:01, ?it/s]
2023-09-06T13:44:49.9890351Z cuda train torchrec_dlrm                      
2023-09-06T13:44:50.0123821Z WARNING:common:fp64 golden ref were not generated for torchrec_dlrm. Setting accuracy check to cosine
2023-09-06T13:45:06.2192758Z pass
2023-09-06T13:45:11.2337553Z 
2023-09-06T13:45:14.3531281Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:45:14.3531636Z loading model: 0it [00:03, ?it/s]
2023-09-06T13:45:14.3531951Z cuda train BERT_pytorch                       
2023-09-06T13:46:06.3434647Z pass
2023-09-06T13:46:12.1302184Z 
2023-09-06T13:46:17.9003497Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:46:17.9004805Z loading model: 0it [00:05, ?it/s]
2023-09-06T13:46:17.9005293Z cuda train Background_Matting                 
2023-09-06T13:46:17.9005787Z pass_due_to_skip
2023-09-06T13:46:22.5679729Z 
2023-09-06T13:46:33.3994354Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:46:33.3994721Z loading model: 0it [00:10, ?it/s]
2023-09-06T13:46:33.3994994Z Eager model failed to run
2023-09-06T13:46:33.4001337Z Traceback (most recent call last):
2023-09-06T13:46:33.4001813Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T13:46:33.4004957Z     self.model_iter_fn(model, example_inputs)
2023-09-06T13:46:33.4005448Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_and_backward_pass
2023-09-06T13:46:33.4005891Z     self.grad_scaler.scale(loss).backward()
2023-09-06T13:46:33.4008199Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
2023-09-06T13:46:33.4008603Z     torch.autograd.backward(
2023-09-06T13:46:33.4009164Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
2023-09-06T13:46:33.4011786Z     Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2023-09-06T13:46:33.4012488Z RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
2023-09-06T13:46:33.4012846Z 
2023-09-06T13:46:33.4013053Z The above exception was the direct cause of the following exception:
2023-09-06T13:46:33.4013302Z 
2023-09-06T13:46:33.4013460Z Traceback (most recent call last):
2023-09-06T13:46:33.4013881Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T13:46:33.4014240Z     ) = runner.load_model(
2023-09-06T13:46:33.4014612Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T13:46:33.4015800Z     self.validate_model(model, example_inputs)
2023-09-06T13:46:33.4016414Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T13:46:33.4016866Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T13:46:33.4017243Z NotImplementedError: Eager model failed to run
2023-09-06T13:46:33.4017452Z 
2023-09-06T13:46:33.4017592Z WARNING:root:DALLE2_pytorch failed to load
2023-09-06T13:46:38.2542170Z 
2023-09-06T13:46:42.5273946Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:46:42.5274295Z loading model: 0it [00:04, ?it/s]
2023-09-06T13:46:42.5274624Z cuda train LearningToPaint                    
2023-09-06T13:47:02.9330672Z pass
2023-09-06T13:47:08.5230926Z 
2023-09-06T13:47:11.9399252Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:47:11.9399594Z loading model: 0it [00:03, ?it/s]
2023-09-06T13:47:11.9399912Z cuda train Super_SloMo                        
2023-09-06T13:48:13.3002418Z pass
2023-09-06T13:48:19.1717852Z 
2023-09-06T13:48:21.0976514Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:48:21.0977039Z loading model: 0it [00:01, ?it/s]
2023-09-06T13:48:21.0977336Z cuda train alexnet                            
2023-09-06T13:48:32.6812441Z pass
2023-09-06T13:48:37.7457243Z 
2023-09-06T13:48:46.9539936Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:48:46.9540308Z loading model: 0it [00:09, ?it/s]
2023-09-06T13:48:46.9540639Z cuda train attention_is_all_you_need_pytorch  
2023-09-06T13:49:42.3751986Z pass
2023-09-06T13:49:48.0736595Z 
2023-09-06T13:49:50.9136622Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:49:50.9137762Z loading model: 0it [00:02, ?it/s]
2023-09-06T13:49:50.9138075Z cuda train basic_gnn_edgecnn                  
2023-09-06T13:49:51.8660647Z [2023-09-06 13:49:51,864] [4/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T13:49:59.1253098Z skipping cudagraphs for unknown reason
2023-09-06T13:50:03.4790619Z pass
2023-09-06T13:50:08.3835086Z 
2023-09-06T13:50:11.3455336Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:50:11.3455686Z loading model: 0it [00:02, ?it/s]
2023-09-06T13:50:11.3455989Z cuda train basic_gnn_gcn                      
2023-09-06T13:50:19.4008759Z skipping cudagraphs for unknown reason
2023-09-06T13:50:19.4211039Z [2023-09-06 13:50:19,420] [14/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T13:50:20.8434389Z skipping cudagraphs for unknown reason
2023-09-06T13:50:22.1736096Z skipping cudagraphs for unknown reason
2023-09-06T13:50:22.5906493Z skipping cudagraphs for unknown reason
2023-09-06T13:50:24.2653608Z pass
2023-09-06T13:50:28.9074213Z 
2023-09-06T13:50:31.8253466Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:50:31.8255271Z loading model: 0it [00:02, ?it/s]
2023-09-06T13:50:31.8255877Z cuda train basic_gnn_gin                      
2023-09-06T13:50:32.7951087Z [2023-09-06 13:50:32,794] [4/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T13:50:39.2436129Z skipping cudagraphs for unknown reason
2023-09-06T13:50:41.1633548Z pass
2023-09-06T13:50:46.0205201Z 
2023-09-06T13:50:49.0017759Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:50:49.0018296Z loading model: 0it [00:02, ?it/s]
2023-09-06T13:50:49.0018902Z cuda train basic_gnn_sage                     
2023-09-06T13:50:49.9842168Z [2023-09-06 13:50:49,983] [4/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T13:50:56.4139054Z skipping cudagraphs for unknown reason
2023-09-06T13:50:57.0577256Z pass
2023-09-06T13:51:01.8476579Z 
2023-09-06T13:51:07.4035426Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:51:07.4035771Z loading model: 0it [00:05, ?it/s]
2023-09-06T13:51:07.4036049Z Eager model failed to run
2023-09-06T13:51:07.4045991Z Traceback (most recent call last):
2023-09-06T13:51:07.4046442Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T13:51:07.4046895Z     self.model_iter_fn(model, example_inputs)
2023-09-06T13:51:07.4047427Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_and_backward_pass
2023-09-06T13:51:07.4048122Z     self.grad_scaler.scale(loss).backward()
2023-09-06T13:51:07.4049738Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
2023-09-06T13:51:07.4052472Z     torch.autograd.backward(
2023-09-06T13:51:07.4053244Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
2023-09-06T13:51:07.4053785Z     Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2023-09-06T13:51:07.4054309Z RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
2023-09-06T13:51:07.4054583Z 
2023-09-06T13:51:07.4054814Z The above exception was the direct cause of the following exception:
2023-09-06T13:51:07.4055141Z 
2023-09-06T13:51:07.4055280Z Traceback (most recent call last):
2023-09-06T13:51:07.4055673Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T13:51:07.4056052Z     ) = runner.load_model(
2023-09-06T13:51:07.4056479Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T13:51:07.4057518Z     self.validate_model(model, example_inputs)
2023-09-06T13:51:07.4058073Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T13:51:07.4058539Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T13:51:07.4058987Z NotImplementedError: Eager model failed to run
2023-09-06T13:51:07.4059302Z 
2023-09-06T13:51:07.4059453Z WARNING:root:cm3leon_generate failed to load
2023-09-06T13:51:11.9606221Z 
2023-09-06T13:51:12.9371625Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:51:12.9372188Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:51:12.9372493Z cuda train dcgan                              
2023-09-06T13:51:23.3210850Z pass
2023-09-06T13:51:28.2025565Z 
2023-09-06T13:51:32.4037063Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:51:32.4037608Z loading model: 0it [00:04, ?it/s]
2023-09-06T13:51:32.4037908Z cuda train demucs                             
2023-09-06T13:52:05.1017524Z pass
2023-09-06T13:52:10.2448587Z 
2023-09-06T13:52:14.2639121Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:52:14.2639658Z loading model: 0it [00:04, ?it/s]
2023-09-06T13:52:14.2639960Z cuda train densenet121                        
2023-09-06T13:54:04.0779891Z pass
2023-09-06T13:54:11.0091674Z 
2023-09-06T13:54:12.5174372Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:54:12.5174923Z loading model: 0it [00:01, ?it/s]
2023-09-06T13:54:12.5175410Z FCOS train is not supported by upstream detectron2. See GH Issue: https://github.com/facebookresearch/detectron2/issues/4369.
2023-09-06T13:54:12.5178942Z Traceback (most recent call last):
2023-09-06T13:54:12.5179357Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T13:54:12.5179722Z     ) = runner.load_model(
2023-09-06T13:54:12.5180112Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 368, in load_model
2023-09-06T13:54:12.5184958Z     benchmark = benchmark_cls(
2023-09-06T13:54:12.5186221Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 25, in __call__
2023-09-06T13:54:12.5186691Z     obj = type.__call__(cls, *args, **kwargs)
2023-09-06T13:54:12.5187166Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn/__init__.py", line 15, in __init__
2023-09-06T13:54:12.5188512Z     super().__init__(variant="COCO-Detection/fcos_R_50_FPN_1x.py", test=test, device=device,
2023-09-06T13:54:12.5189061Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/detectron2/model_factory.py", line 105, in __init__
2023-09-06T13:54:12.5189757Z     raise NotImplementedError("FCOS train is not supported by upstream detectron2. " \
2023-09-06T13:54:12.5190638Z NotImplementedError: FCOS train is not supported by upstream detectron2. See GH Issue: https://github.com/facebookresearch/detectron2/issues/4369.
2023-09-06T13:54:12.5191030Z 
2023-09-06T13:54:12.5191210Z WARNING:root:detectron2_fcos_r_50_fpn failed to load
2023-09-06T13:54:13.4811686Z accuracy            pass_rate=92.86%
2023-09-06T13:54:13.4814717Z calls_captured      gmean=0.00x mean=176.643x
2023-09-06T13:54:13.4815747Z unique_graphs       gmean=0.00x mean=3.071x
2023-09-06T13:54:13.4817517Z graph_breaks        gmean=0.00x mean=9.143x
2023-09-06T13:54:13.4820336Z unique_graph_breaks gmean=0.00x mean=5.786x
2023-09-06T13:54:14.2479025Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *dynamic-true* ]]
2023-09-06T13:54:14.2480807Z + python benchmarks/dynamo/torchbench.py --accuracy --no-translation-validation --training --amp --backend inductor --dynamic-shapes --dynamic-batch-only --device cuda --total-partitions 4 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inductor_dynamic_torchbench_amp_training_cuda_accuracy.csv
2023-09-06T13:54:21.4520515Z 
2023-09-06T13:54:22.7746775Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:54:22.7748198Z loading model: 0it [00:01, ?it/s]
2023-09-06T13:54:22.7748492Z Traceback (most recent call last):
2023-09-06T13:54:22.7748911Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 495, in <module>
2023-09-06T13:54:22.7749456Z     torchbench_main()
2023-09-06T13:54:22.7749835Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 491, in torchbench_main
2023-09-06T13:54:22.7750261Z     main(TorchBenchmarkRunner(), original_dir)
2023-09-06T13:54:22.7750667Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3031, in main
2023-09-06T13:54:22.7754934Z     process_entry(0, runner, original_dir, args)
2023-09-06T13:54:22.7755767Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2988, in process_entry
2023-09-06T13:54:22.7761380Z     return maybe_fresh_cache(
2023-09-06T13:54:22.7762136Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1659, in inner
2023-09-06T13:54:22.7765953Z     return fn(*args, **kwargs)
2023-09-06T13:54:22.7766735Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3477, in run
2023-09-06T13:54:22.7774522Z     assert marked, f"nothing in example_inputs had a dim with {batch_size}"
2023-09-06T13:54:22.7775266Z AssertionError: nothing in example_inputs had a dim with 4
2023-09-06T13:54:23.7088536Z ERROR
2023-09-06T13:54:27.2537214Z 
2023-09-06T13:54:30.4539324Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:54:30.4539734Z loading model: 0it [00:03, ?it/s]
2023-09-06T13:54:30.4540044Z cuda train BERT_pytorch                       
2023-09-06T13:55:49.4251490Z pass
2023-09-06T13:55:55.1586254Z 
2023-09-06T13:56:00.9244680Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:56:00.9245241Z loading model: 0it [00:05, ?it/s]
2023-09-06T13:56:00.9245795Z cuda train Background_Matting                 
2023-09-06T13:56:00.9246340Z pass_due_to_skip
2023-09-06T13:56:05.6787008Z 
2023-09-06T13:56:16.4836089Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:56:16.4837317Z loading model: 0it [00:10, ?it/s]
2023-09-06T13:56:16.4838151Z Eager model failed to run
2023-09-06T13:56:16.4847627Z Traceback (most recent call last):
2023-09-06T13:56:16.4848360Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T13:56:16.4848781Z     self.model_iter_fn(model, example_inputs)
2023-09-06T13:56:16.4849228Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_and_backward_pass
2023-09-06T13:56:16.4853652Z     self.grad_scaler.scale(loss).backward()
2023-09-06T13:56:16.4855417Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
2023-09-06T13:56:16.4855821Z     torch.autograd.backward(
2023-09-06T13:56:16.4856390Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
2023-09-06T13:56:16.4856927Z     Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2023-09-06T13:56:16.4857441Z RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
2023-09-06T13:56:16.4857945Z 
2023-09-06T13:56:16.4858162Z The above exception was the direct cause of the following exception:
2023-09-06T13:56:16.4858451Z 
2023-09-06T13:56:16.4858660Z Traceback (most recent call last):
2023-09-06T13:56:16.4859113Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T13:56:16.4859474Z     ) = runner.load_model(
2023-09-06T13:56:16.4859836Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T13:56:16.4860244Z     self.validate_model(model, example_inputs)
2023-09-06T13:56:16.4862074Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T13:56:16.4862690Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T13:56:16.4863094Z NotImplementedError: Eager model failed to run
2023-09-06T13:56:16.4863363Z 
2023-09-06T13:56:16.4863912Z WARNING:root:DALLE2_pytorch failed to load
2023-09-06T13:56:21.2709549Z 
2023-09-06T13:56:25.5386671Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:56:25.5387014Z loading model: 0it [00:04, ?it/s]
2023-09-06T13:56:25.5387336Z cuda train LearningToPaint                    
2023-09-06T13:56:50.4273008Z pass
2023-09-06T13:56:55.9705104Z 
2023-09-06T13:56:59.3548891Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:56:59.3549458Z loading model: 0it [00:03, ?it/s]
2023-09-06T13:56:59.3549838Z cuda train Super_SloMo                        
2023-09-06T13:58:41.8413478Z pass
2023-09-06T13:58:47.6179527Z 
2023-09-06T13:58:49.5676613Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:58:49.5677804Z loading model: 0it [00:01, ?it/s]
2023-09-06T13:58:49.5678109Z cuda train alexnet                            
2023-09-06T13:59:14.4163870Z pass
2023-09-06T13:59:19.6243936Z 
2023-09-06T13:59:28.8124787Z loading model: 0it [00:00, ?it/s]
2023-09-06T13:59:28.8125131Z loading model: 0it [00:09, ?it/s]
2023-09-06T13:59:28.8125507Z cuda train attention_is_all_you_need_pytorch  
2023-09-06T14:00:57.2220689Z pass
2023-09-06T14:01:03.2237259Z 
2023-09-06T14:01:06.1643495Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:01:06.1644015Z loading model: 0it [00:02, ?it/s]
2023-09-06T14:01:06.1644495Z cuda train basic_gnn_edgecnn                  
2023-09-06T14:01:07.1246205Z [2023-09-06 14:01:07,123] [4/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T14:01:14.5931295Z skipping cudagraphs for unknown reason
2023-09-06T14:01:19.0756663Z pass
2023-09-06T14:01:24.0346733Z 
2023-09-06T14:01:26.9309437Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:01:26.9310209Z loading model: 0it [00:02, ?it/s]
2023-09-06T14:01:26.9313673Z cuda train basic_gnn_gcn                      
2023-09-06T14:01:35.0160754Z skipping cudagraphs for unknown reason
2023-09-06T14:01:35.0356522Z [2023-09-06 14:01:35,034] [14/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T14:01:36.4760448Z skipping cudagraphs for unknown reason
2023-09-06T14:01:37.8146228Z skipping cudagraphs for unknown reason
2023-09-06T14:01:38.2539164Z skipping cudagraphs for unknown reason
2023-09-06T14:01:40.0093746Z pass
2023-09-06T14:01:44.8783850Z 
2023-09-06T14:01:47.9586409Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:01:47.9586758Z loading model: 0it [00:03, ?it/s]
2023-09-06T14:01:47.9587063Z cuda train basic_gnn_gin                      
2023-09-06T14:01:48.9472857Z [2023-09-06 14:01:48,946] [4/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T14:01:55.5970462Z skipping cudagraphs for unknown reason
2023-09-06T14:01:57.4546201Z pass
2023-09-06T14:02:02.3301246Z 
2023-09-06T14:02:05.2658340Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:02:05.2658699Z loading model: 0it [00:02, ?it/s]
2023-09-06T14:02:05.2658990Z cuda train basic_gnn_sage                     
2023-09-06T14:02:06.2344905Z [2023-09-06 14:02:06,233] [4/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T14:02:12.7318444Z skipping cudagraphs for unknown reason
2023-09-06T14:02:13.3856313Z pass
2023-09-06T14:02:18.2457944Z 
2023-09-06T14:02:23.7399428Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:02:23.7400526Z loading model: 0it [00:05, ?it/s]
2023-09-06T14:02:23.7400810Z Eager model failed to run
2023-09-06T14:02:23.7406888Z Traceback (most recent call last):
2023-09-06T14:02:23.7407631Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T14:02:23.7408274Z     self.model_iter_fn(model, example_inputs)
2023-09-06T14:02:23.7408775Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_and_backward_pass
2023-09-06T14:02:23.7409203Z     self.grad_scaler.scale(loss).backward()
2023-09-06T14:02:23.7410262Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
2023-09-06T14:02:23.7410955Z     torch.autograd.backward(
2023-09-06T14:02:23.7411724Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
2023-09-06T14:02:23.7412833Z     Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2023-09-06T14:02:23.7413561Z RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
2023-09-06T14:02:23.7414071Z 
2023-09-06T14:02:23.7414541Z The above exception was the direct cause of the following exception:
2023-09-06T14:02:23.7414993Z 
2023-09-06T14:02:23.7415138Z Traceback (most recent call last):
2023-09-06T14:02:23.7415523Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T14:02:23.7415875Z     ) = runner.load_model(
2023-09-06T14:02:23.7416238Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T14:02:23.7416644Z     self.validate_model(model, example_inputs)
2023-09-06T14:02:23.7417054Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T14:02:23.7417498Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T14:02:23.7417885Z NotImplementedError: Eager model failed to run
2023-09-06T14:02:23.7418436Z 
2023-09-06T14:02:23.7418589Z WARNING:root:cm3leon_generate failed to load
2023-09-06T14:02:28.3297449Z 
2023-09-06T14:02:29.3133176Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:02:29.3133524Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:02:29.3133829Z cuda train dcgan                              
2023-09-06T14:02:41.3302974Z pass
2023-09-06T14:02:46.3470615Z 
2023-09-06T14:02:50.6031798Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:02:50.6032673Z loading model: 0it [00:04, ?it/s]
2023-09-06T14:02:50.6033250Z cuda train demucs                             
2023-09-06T14:03:12.5322702Z ERROR:common:Constraints violated!
2023-09-06T14:03:12.5325427Z   1. Not all values of RelaxedUnspecConstraint(L['streams'].size()[0]) are valid because L['streams'].size()[0] was inferred to be a constant (4). For more information about why it is constant, run with TORCH_LOGS=dynamic.
2023-09-06T14:03:12.5325902Z 
2023-09-06T14:03:12.5325909Z 
2023-09-06T14:03:12.5326133Z You can suppress this exception and fall back to eager by setting:
2023-09-06T14:03:12.5332364Z     import torch._dynamo
2023-09-06T14:03:12.5333518Z     torch._dynamo.config.suppress_errors = True
2023-09-06T14:03:12.5333992Z Traceback (most recent call last):
2023-09-06T14:03:12.5334762Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2135, in check_accuracy
2023-09-06T14:03:12.5335426Z     new_result = optimized_model_iter_fn(model_copy, example_inputs)
2023-09-06T14:03:12.5336198Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T14:03:12.5336577Z     return fn(*args, **kwargs)
2023-09-06T14:03:12.5336971Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1899, in run_n_iterations
2023-09-06T14:03:12.5337402Z     self.model_iter_fn(mod, inputs, collect_outputs=False)
2023-09-06T14:03:12.5337871Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 475, in forward_and_backward_pass
2023-09-06T14:03:12.5338859Z     cloned_inputs = clone_inputs(inputs)
2023-09-06T14:03:12.5339298Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 476, in <resume in forward_and_backward_pass>
2023-09-06T14:03:12.5339722Z     self.optimizer_zero_grad(mod)
2023-09-06T14:03:12.5340167Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 478, in <resume in forward_and_backward_pass>
2023-09-06T14:03:12.5340583Z     pred = mod(*cloned_inputs)
2023-09-06T14:03:12.5341177Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T14:03:12.5341602Z     return self._call_impl(*args, **kwargs)
2023-09-06T14:03:12.5342173Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T14:03:12.5342582Z     return forward_call(*args, **kwargs)
2023-09-06T14:03:12.5343145Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T14:03:12.5343588Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T14:03:12.5344216Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 634, in _convert_frame
2023-09-06T14:03:12.5344681Z     result = inner_convert(frame, cache_entry, hooks, frame_state)
2023-09-06T14:03:12.5345271Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T14:03:12.5345660Z     return fn(*args, **kwargs)
2023-09-06T14:03:12.5346217Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T14:03:12.5346618Z     return _compile(
2023-09-06T14:03:12.5347148Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T14:03:12.5347617Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T14:03:12.5348417Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T14:03:12.5348806Z     r = func(*args, **kwargs)
2023-09-06T14:03:12.5349567Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 542, in compile_inner
2023-09-06T14:03:12.5349996Z     check_fn = CheckFunctionManager(
2023-09-06T14:03:12.5350553Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 958, in __init__
2023-09-06T14:03:12.5350958Z     guard.create(local_builder, global_builder)
2023-09-06T14:03:12.5351496Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_guards.py", line 243, in create
2023-09-06T14:03:12.5351967Z     return self.create_fn(self.source.select(local_builder, global_builder), self)
2023-09-06T14:03:12.5352585Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 621, in SHAPE_ENV
2023-09-06T14:03:12.5353021Z     guards = output_graph.shape_env.produce_guards(
2023-09-06T14:03:12.5353650Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 3022, in produce_guards
2023-09-06T14:03:12.5354161Z     raise ConstraintViolationError(f"Constraints violated!\n{err}")
2023-09-06T14:03:12.5354667Z torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated!
2023-09-06T14:03:12.5355621Z   1. Not all values of RelaxedUnspecConstraint(L['streams'].size()[0]) are valid because L['streams'].size()[0] was inferred to be a constant (4). For more information about why it is constant, run with TORCH_LOGS=dynamic.
2023-09-06T14:03:12.5356076Z 
2023-09-06T14:03:12.5356082Z 
2023-09-06T14:03:12.5356264Z You can suppress this exception and fall back to eager by setting:
2023-09-06T14:03:12.5356605Z     import torch._dynamo
2023-09-06T14:03:12.5356919Z     torch._dynamo.config.suppress_errors = True
2023-09-06T14:03:12.5357129Z 
2023-09-06T14:03:12.5357490Z TorchDynamo optimized model failed to run because of following error
2023-09-06T14:03:12.5381423Z fail_to_run
2023-09-06T14:03:17.5331301Z 
2023-09-06T14:03:21.4915914Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:03:21.4916271Z loading model: 0it [00:03, ?it/s]
2023-09-06T14:03:21.4916635Z cuda train densenet121                        
2023-09-06T14:05:59.4521059Z pass
2023-09-06T14:06:06.5383406Z 
2023-09-06T14:06:08.0409258Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:06:08.0409647Z loading model: 0it [00:01, ?it/s]
2023-09-06T14:06:08.0410157Z FCOS train is not supported by upstream detectron2. See GH Issue: https://github.com/facebookresearch/detectron2/issues/4369.
2023-09-06T14:06:08.0415573Z Traceback (most recent call last):
2023-09-06T14:06:08.0416231Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T14:06:08.0419444Z     ) = runner.load_model(
2023-09-06T14:06:08.0420146Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 368, in load_model
2023-09-06T14:06:08.0420575Z     benchmark = benchmark_cls(
2023-09-06T14:06:08.0420985Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 25, in __call__
2023-09-06T14:06:08.0421400Z     obj = type.__call__(cls, *args, **kwargs)
2023-09-06T14:06:08.0425090Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn/__init__.py", line 15, in __init__
2023-09-06T14:06:08.0426692Z     super().__init__(variant="COCO-Detection/fcos_R_50_FPN_1x.py", test=test, device=device,
2023-09-06T14:06:08.0427234Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/detectron2/model_factory.py", line 105, in __init__
2023-09-06T14:06:08.0427787Z     raise NotImplementedError("FCOS train is not supported by upstream detectron2. " \
2023-09-06T14:06:08.0428394Z NotImplementedError: FCOS train is not supported by upstream detectron2. See GH Issue: https://github.com/facebookresearch/detectron2/issues/4369.
2023-09-06T14:06:08.0429505Z 
2023-09-06T14:06:08.0429684Z WARNING:root:detectron2_fcos_r_50_fpn failed to load
2023-09-06T14:06:09.0572041Z accuracy            pass_rate=78.57%
2023-09-06T14:06:09.0572436Z calls_captured      gmean=0.00x mean=166.357x
2023-09-06T14:06:09.0574849Z unique_graphs       gmean=0.00x mean=2.571x
2023-09-06T14:06:09.0575882Z graph_breaks        gmean=0.00x mean=8.143x
2023-09-06T14:06:09.0577996Z unique_graph_breaks gmean=0.00x mean=5.071x
2023-09-06T14:06:09.8251490Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *cppwrapper-true* ]]
2023-09-06T14:06:09.8253729Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *freezing_cudagraphs-true* ]]
2023-09-06T14:06:09.8254431Z + [[ training == \i\n\f\e\r\e\n\c\e ]]
2023-09-06T14:06:09.8255324Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *aotinductor-true* ]]
2023-09-06T14:06:09.8256029Z + [[ training == \i\n\f\e\r\e\n\c\e ]]
2023-09-06T14:06:09.8256886Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *maxautotune-true* ]]
2023-09-06T14:06:09.8257564Z + for target in "${targets[@]}"
2023-09-06T14:06:09.8257883Z + target_flag=("--${target}")
2023-09-06T14:06:09.8258157Z + local target_flag
2023-09-06T14:06:09.8258450Z + [[ performance == \p\e\r\f\o\r\m\a\n\c\e ]]
2023-09-06T14:06:09.8258833Z + target_flag+=(--cold-start-latency)
2023-09-06T14:06:09.8259672Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *default-true* ]]
2023-09-06T14:06:09.8261128Z + python benchmarks/dynamo/torchbench.py --performance --cold-start-latency --training --amp --backend inductor --disable-cudagraphs --device cuda --total-partitions 4 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inductor_no_cudagraphs_torchbench_amp_training_cuda_performance.csv
2023-09-06T14:06:17.1377524Z 
2023-09-06T14:06:18.4585699Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:06:18.4586211Z loading model: 0it [00:01, ?it/s]
2023-09-06T14:06:18.4586514Z cuda train torchrec_dlrm                      
2023-09-06T14:06:39.4515600Z 
2023-09-06T14:06:39.5577910Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:06:39.6631468Z running benchmark:  13% 4/30 [00:00<00:00, 37.61it/s]
2023-09-06T14:06:39.7665603Z running benchmark:  27% 8/30 [00:00<00:00, 37.77it/s]
2023-09-06T14:06:39.8910268Z running benchmark:  40% 12/30 [00:00<00:00, 38.18it/s]
2023-09-06T14:06:39.9917724Z running benchmark:  57% 17/30 [00:00<00:00, 39.05it/s]
2023-09-06T14:06:40.1175175Z running benchmark:  70% 21/30 [00:00<00:00, 39.26it/s]
2023-09-06T14:06:40.2237248Z running benchmark:  83% 25/30 [00:00<00:00, 36.51it/s]
2023-09-06T14:06:40.2505565Z running benchmark:  97% 29/30 [00:00<00:00, 36.83it/s]
2023-09-06T14:06:40.2505986Z running benchmark: 100% 30/30 [00:00<00:00, 37.53it/s]
2023-09-06T14:06:40.2519970Z 0.954x
2023-09-06T14:06:45.2845039Z 
2023-09-06T14:06:48.7675111Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:06:48.7675835Z loading model: 0it [00:03, ?it/s]
2023-09-06T14:06:48.7676151Z cuda train BERT_pytorch                       
2023-09-06T14:07:53.5413317Z 
2023-09-06T14:07:53.7266066Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:07:53.9270829Z running benchmark:   7% 2/30 [00:00<00:02, 10.78it/s]
2023-09-06T14:07:54.1045762Z running benchmark:  13% 4/30 [00:00<00:02, 10.29it/s]
2023-09-06T14:07:54.2823013Z running benchmark:  20% 6/30 [00:00<00:02, 10.71it/s]
2023-09-06T14:07:54.4595258Z running benchmark:  27% 8/30 [00:00<00:02, 10.92it/s]
2023-09-06T14:07:54.6369465Z running benchmark:  33% 10/30 [00:00<00:01, 11.05it/s]
2023-09-06T14:07:54.8253884Z running benchmark:  40% 12/30 [00:01<00:01, 11.12it/s]
2023-09-06T14:07:54.9994413Z running benchmark:  47% 14/30 [00:01<00:01, 10.95it/s]
2023-09-06T14:07:55.1771396Z running benchmark:  53% 16/30 [00:01<00:01, 11.12it/s]
2023-09-06T14:07:55.3553120Z running benchmark:  60% 18/30 [00:01<00:01, 11.16it/s]
2023-09-06T14:07:55.5363279Z running benchmark:  67% 20/30 [00:01<00:00, 11.18it/s]
2023-09-06T14:07:55.7124168Z running benchmark:  73% 22/30 [00:01<00:00, 11.14it/s]
2023-09-06T14:07:55.9000102Z running benchmark:  80% 24/30 [00:02<00:00, 11.21it/s]
2023-09-06T14:07:56.0758438Z running benchmark:  87% 26/30 [00:02<00:00, 11.04it/s]
2023-09-06T14:07:56.2493310Z running benchmark:  93% 28/30 [00:02<00:00, 11.13it/s]
2023-09-06T14:07:56.2493967Z running benchmark: 100% 30/30 [00:02<00:00, 11.25it/s]
2023-09-06T14:07:56.2494497Z running benchmark: 100% 30/30 [00:02<00:00, 11.08it/s]
2023-09-06T14:07:56.2522090Z 1.518x
2023-09-06T14:08:02.1785240Z 
2023-09-06T14:08:07.8997955Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:08:07.8998583Z loading model: 0it [00:05, ?it/s]
2023-09-06T14:08:07.8998881Z cuda train Background_Matting                 
2023-09-06T14:09:26.7355059Z 
2023-09-06T14:09:27.0479078Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:09:27.2914671Z running benchmark:   3% 1/30 [00:00<00:09,  3.20it/s]
2023-09-06T14:09:27.5350290Z running benchmark:   7% 2/30 [00:00<00:07,  3.68it/s]
2023-09-06T14:09:27.7783188Z running benchmark:  10% 3/30 [00:00<00:06,  3.86it/s]
2023-09-06T14:09:28.0215073Z running benchmark:  13% 4/30 [00:01<00:06,  3.95it/s]
2023-09-06T14:09:28.2649343Z running benchmark:  17% 5/30 [00:01<00:06,  4.01it/s]
2023-09-06T14:09:28.5079955Z running benchmark:  20% 6/30 [00:01<00:05,  4.04it/s]
2023-09-06T14:09:28.7511787Z running benchmark:  23% 7/30 [00:01<00:05,  4.07it/s]
2023-09-06T14:09:28.9946810Z running benchmark:  27% 8/30 [00:02<00:05,  4.08it/s]
2023-09-06T14:09:29.2380786Z running benchmark:  30% 9/30 [00:02<00:05,  4.09it/s]
2023-09-06T14:09:29.4811358Z running benchmark:  33% 10/30 [00:02<00:04,  4.09it/s]
2023-09-06T14:09:29.7242611Z running benchmark:  37% 11/30 [00:02<00:04,  4.10it/s]
2023-09-06T14:09:29.9676020Z running benchmark:  40% 12/30 [00:02<00:04,  4.10it/s]
2023-09-06T14:09:30.2114645Z running benchmark:  43% 13/30 [00:03<00:04,  4.11it/s]
2023-09-06T14:09:30.4547415Z running benchmark:  47% 14/30 [00:03<00:03,  4.11it/s]
2023-09-06T14:09:30.6981103Z running benchmark:  50% 15/30 [00:03<00:03,  4.11it/s]
2023-09-06T14:09:30.9415483Z running benchmark:  53% 16/30 [00:03<00:03,  4.11it/s]
2023-09-06T14:09:31.1854440Z running benchmark:  57% 17/30 [00:04<00:03,  4.11it/s]
2023-09-06T14:09:31.4286230Z running benchmark:  60% 18/30 [00:04<00:02,  4.11it/s]
2023-09-06T14:09:31.6723052Z running benchmark:  63% 19/30 [00:04<00:02,  4.11it/s]
2023-09-06T14:09:31.9155592Z running benchmark:  67% 20/30 [00:04<00:02,  4.11it/s]
2023-09-06T14:09:32.1724962Z running benchmark:  70% 21/30 [00:05<00:02,  4.11it/s]
2023-09-06T14:09:32.4158624Z running benchmark:  73% 22/30 [00:05<00:01,  4.04it/s]
2023-09-06T14:09:32.6595296Z running benchmark:  77% 23/30 [00:05<00:01,  4.06it/s]
2023-09-06T14:09:32.9030352Z running benchmark:  80% 24/30 [00:05<00:01,  4.07it/s]
2023-09-06T14:09:33.1464264Z running benchmark:  83% 25/30 [00:06<00:01,  4.08it/s]
2023-09-06T14:09:33.3896428Z running benchmark:  87% 26/30 [00:06<00:00,  4.09it/s]
2023-09-06T14:09:33.6331161Z running benchmark:  90% 27/30 [00:06<00:00,  4.10it/s]
2023-09-06T14:09:33.8761871Z running benchmark:  93% 28/30 [00:06<00:00,  4.10it/s]
2023-09-06T14:09:34.1200301Z running benchmark:  97% 29/30 [00:07<00:00,  4.10it/s]
2023-09-06T14:09:34.1200918Z running benchmark: 100% 30/30 [00:07<00:00,  4.10it/s]
2023-09-06T14:09:34.1201487Z running benchmark: 100% 30/30 [00:07<00:00,  4.06it/s]
2023-09-06T14:09:34.1225263Z 1.156x
2023-09-06T14:09:40.1874881Z 
2023-09-06T14:09:51.3861845Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:09:51.3862348Z loading model: 0it [00:11, ?it/s]
2023-09-06T14:09:51.3862625Z Eager model failed to run
2023-09-06T14:09:51.3867981Z Traceback (most recent call last):
2023-09-06T14:09:51.3869034Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T14:09:51.3869788Z     self.model_iter_fn(model, example_inputs)
2023-09-06T14:09:51.3873202Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_and_backward_pass
2023-09-06T14:09:51.3873966Z     self.grad_scaler.scale(loss).backward()
2023-09-06T14:09:51.3875719Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
2023-09-06T14:09:51.3876122Z     torch.autograd.backward(
2023-09-06T14:09:51.3876668Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
2023-09-06T14:09:51.3877183Z     Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2023-09-06T14:09:51.3877676Z RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
2023-09-06T14:09:51.3878143Z 
2023-09-06T14:09:51.3878355Z The above exception was the direct cause of the following exception:
2023-09-06T14:09:51.3878802Z 
2023-09-06T14:09:51.3879048Z Traceback (most recent call last):
2023-09-06T14:09:51.3879637Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T14:09:51.3880223Z     ) = runner.load_model(
2023-09-06T14:09:51.3880976Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T14:09:51.3881558Z     self.validate_model(model, example_inputs)
2023-09-06T14:09:51.3881957Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T14:09:51.3882473Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T14:09:51.3882858Z NotImplementedError: Eager model failed to run
2023-09-06T14:09:51.3883073Z 
2023-09-06T14:09:51.3883213Z WARNING:root:DALLE2_pytorch failed to load
2023-09-06T14:09:56.2016248Z 
2023-09-06T14:10:00.9313152Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:10:00.9313871Z loading model: 0it [00:04, ?it/s]
2023-09-06T14:10:00.9315659Z cuda train LearningToPaint                    
2023-09-06T14:10:34.4989365Z 
2023-09-06T14:10:34.6033606Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:10:34.7223621Z running benchmark:  10% 3/30 [00:00<00:00, 28.64it/s]
2023-09-06T14:10:34.8412278Z running benchmark:  23% 7/30 [00:00<00:00, 31.65it/s]
2023-09-06T14:10:34.9664111Z running benchmark:  37% 11/30 [00:00<00:00, 32.59it/s]
2023-09-06T14:10:35.0893856Z running benchmark:  50% 15/30 [00:00<00:00, 32.33it/s]
2023-09-06T14:10:35.2189404Z running benchmark:  63% 19/30 [00:00<00:00, 32.40it/s]
2023-09-06T14:10:35.3429068Z running benchmark:  77% 23/30 [00:00<00:00, 31.86it/s]
2023-09-06T14:10:35.4261005Z running benchmark:  90% 27/30 [00:00<00:00, 31.99it/s]
2023-09-06T14:10:35.4261402Z running benchmark: 100% 30/30 [00:00<00:00, 32.33it/s]
2023-09-06T14:10:35.4282753Z 0.884x
2023-09-06T14:10:41.1640546Z 
2023-09-06T14:10:44.5399979Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:10:44.5400581Z loading model: 0it [00:03, ?it/s]
2023-09-06T14:10:44.5404390Z cuda train Super_SloMo                        
2023-09-06T14:12:05.6014249Z 
2023-09-06T14:12:05.8054882Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:12:05.9491156Z running benchmark:   3% 1/30 [00:00<00:05,  4.90it/s]
2023-09-06T14:12:06.0925109Z running benchmark:   7% 2/30 [00:00<00:04,  5.93it/s]
2023-09-06T14:12:06.2361264Z running benchmark:  10% 3/30 [00:00<00:04,  6.36it/s]
2023-09-06T14:12:06.3797982Z running benchmark:  13% 4/30 [00:00<00:03,  6.59it/s]
2023-09-06T14:12:06.5239330Z running benchmark:  17% 5/30 [00:00<00:03,  6.72it/s]
2023-09-06T14:12:06.6675293Z running benchmark:  20% 6/30 [00:00<00:03,  6.79it/s]
2023-09-06T14:12:06.8228404Z running benchmark:  23% 7/30 [00:01<00:03,  6.85it/s]
2023-09-06T14:12:06.9665044Z running benchmark:  27% 8/30 [00:01<00:03,  6.71it/s]
2023-09-06T14:12:07.1103081Z running benchmark:  30% 9/30 [00:01<00:03,  6.79it/s]
2023-09-06T14:12:07.2542023Z running benchmark:  33% 10/30 [00:01<00:02,  6.84it/s]
2023-09-06T14:12:07.3982983Z running benchmark:  37% 11/30 [00:01<00:02,  6.87it/s]
2023-09-06T14:12:07.5419411Z running benchmark:  40% 12/30 [00:01<00:02,  6.89it/s]
2023-09-06T14:12:07.6854064Z running benchmark:  43% 13/30 [00:01<00:02,  6.91it/s]
2023-09-06T14:12:07.8315455Z running benchmark:  47% 14/30 [00:02<00:02,  6.93it/s]
2023-09-06T14:12:07.9751901Z running benchmark:  50% 15/30 [00:02<00:02,  6.91it/s]
2023-09-06T14:12:08.1189262Z running benchmark:  53% 16/30 [00:02<00:02,  6.92it/s]
2023-09-06T14:12:08.2627582Z running benchmark:  57% 17/30 [00:02<00:01,  6.93it/s]
2023-09-06T14:12:08.4066581Z running benchmark:  60% 18/30 [00:02<00:01,  6.94it/s]
2023-09-06T14:12:08.5501909Z running benchmark:  63% 19/30 [00:02<00:01,  6.94it/s]
2023-09-06T14:12:08.6938184Z running benchmark:  67% 20/30 [00:02<00:01,  6.95it/s]
2023-09-06T14:12:08.8373262Z running benchmark:  70% 21/30 [00:03<00:01,  6.95it/s]
2023-09-06T14:12:08.9810022Z running benchmark:  73% 22/30 [00:03<00:01,  6.96it/s]
2023-09-06T14:12:09.1245356Z running benchmark:  77% 23/30 [00:03<00:01,  6.96it/s]
2023-09-06T14:12:09.2682637Z running benchmark:  80% 24/30 [00:03<00:00,  6.96it/s]
2023-09-06T14:12:09.4121705Z running benchmark:  83% 25/30 [00:03<00:00,  6.96it/s]
2023-09-06T14:12:09.5559548Z running benchmark:  87% 26/30 [00:03<00:00,  6.96it/s]
2023-09-06T14:12:09.6998180Z running benchmark:  90% 27/30 [00:03<00:00,  6.96it/s]
2023-09-06T14:12:09.8440061Z running benchmark:  93% 28/30 [00:04<00:00,  6.96it/s]
2023-09-06T14:12:09.9876422Z running benchmark:  97% 29/30 [00:04<00:00,  6.95it/s]
2023-09-06T14:12:09.9876822Z running benchmark: 100% 30/30 [00:04<00:00,  6.95it/s]
2023-09-06T14:12:09.9877130Z running benchmark: 100% 30/30 [00:04<00:00,  6.84it/s]
2023-09-06T14:12:09.9900865Z 1.256x
2023-09-06T14:12:15.7579287Z 
2023-09-06T14:12:18.0846691Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:12:18.0847041Z loading model: 0it [00:02, ?it/s]
2023-09-06T14:12:18.0847324Z cuda train alexnet                            
2023-09-06T14:12:34.3018023Z 
2023-09-06T14:12:34.4031583Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:12:34.5098348Z running benchmark:  13% 4/30 [00:00<00:00, 39.47it/s]
2023-09-06T14:12:34.6187164Z running benchmark:  33% 10/30 [00:00<00:00, 49.50it/s]
2023-09-06T14:12:34.7280016Z running benchmark:  53% 16/30 [00:00<00:00, 52.11it/s]
2023-09-06T14:12:34.8375330Z running benchmark:  73% 22/30 [00:00<00:00, 53.23it/s]
2023-09-06T14:12:34.8746061Z running benchmark:  93% 28/30 [00:00<00:00, 53.80it/s]
2023-09-06T14:12:34.8746455Z running benchmark: 100% 30/30 [00:00<00:00, 52.35it/s]
2023-09-06T14:12:34.8765159Z 1.326x
2023-09-06T14:12:40.0868017Z 
2023-09-06T14:12:47.7334962Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:12:47.7335314Z loading model: 0it [00:07, ?it/s]
2023-09-06T14:12:47.7335644Z cuda train attention_is_all_you_need_pytorch  
2023-09-06T14:14:04.1767244Z 
2023-09-06T14:14:04.2762285Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:14:04.4751101Z running benchmark:   3% 1/30 [00:00<00:02, 10.00it/s]
2023-09-06T14:14:04.6590224Z running benchmark:  10% 3/30 [00:00<00:02, 10.04it/s]
2023-09-06T14:14:04.8433721Z running benchmark:  17% 5/30 [00:00<00:02, 10.45it/s]
2023-09-06T14:14:05.0309513Z running benchmark:  23% 7/30 [00:00<00:02, 10.61it/s]
2023-09-06T14:14:05.2164011Z running benchmark:  30% 9/30 [00:00<00:01, 10.63it/s]
2023-09-06T14:14:05.3994115Z running benchmark:  37% 11/30 [00:01<00:01, 10.69it/s]
2023-09-06T14:14:05.6044402Z running benchmark:  43% 13/30 [00:01<00:01, 10.76it/s]
2023-09-06T14:14:05.7892574Z running benchmark:  50% 15/30 [00:01<00:01, 10.42it/s]
2023-09-06T14:14:05.9732534Z running benchmark:  57% 17/30 [00:01<00:01, 10.54it/s]
2023-09-06T14:14:06.1578474Z running benchmark:  63% 19/30 [00:01<00:01, 10.64it/s]
2023-09-06T14:14:06.3408015Z running benchmark:  70% 21/30 [00:01<00:00, 10.70it/s]
2023-09-06T14:14:06.5415632Z running benchmark:  77% 23/30 [00:02<00:00, 10.77it/s]
2023-09-06T14:14:06.7255196Z running benchmark:  83% 25/30 [00:02<00:00, 10.51it/s]
2023-09-06T14:14:06.9084460Z running benchmark:  90% 27/30 [00:02<00:00, 10.62it/s]
2023-09-06T14:14:07.0018719Z running benchmark:  97% 29/30 [00:02<00:00, 10.71it/s]
2023-09-06T14:14:07.0019124Z running benchmark: 100% 30/30 [00:02<00:00, 10.62it/s]
2023-09-06T14:14:07.0045685Z 1.424x
2023-09-06T14:14:12.9413811Z 
2023-09-06T14:14:15.7694638Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:14:15.7694980Z loading model: 0it [00:02, ?it/s]
2023-09-06T14:14:15.7695283Z cuda train basic_gnn_edgecnn                  
2023-09-06T14:14:16.5044457Z [2023-09-06 14:14:16,503] [3/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T14:14:32.7202570Z 
2023-09-06T14:14:32.8201546Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:14:32.9203531Z running benchmark:  23% 7/30 [00:00<00:00, 69.86it/s]
2023-09-06T14:14:33.0284547Z running benchmark:  47% 14/30 [00:00<00:00, 69.82it/s]
2023-09-06T14:14:33.1666175Z running benchmark:  70% 21/30 [00:00<00:00, 67.40it/s]
2023-09-06T14:14:33.1960856Z running benchmark:  93% 28/30 [00:00<00:00, 59.66it/s]
2023-09-06T14:14:33.1961286Z running benchmark: 100% 30/30 [00:00<00:00, 62.97it/s]
2023-09-06T14:14:33.1978237Z 1.200x
2023-09-06T14:14:38.1184792Z 
2023-09-06T14:14:40.9168347Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:14:40.9169239Z loading model: 0it [00:02, ?it/s]
2023-09-06T14:14:40.9169543Z cuda train basic_gnn_gcn                      
2023-09-06T14:14:51.2690006Z [2023-09-06 14:14:51,267] [13/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T14:14:58.0593181Z 
2023-09-06T14:14:58.1639281Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:14:58.2735250Z running benchmark:  30% 9/30 [00:00<00:00, 85.80it/s]
2023-09-06T14:14:58.3788623Z running benchmark:  63% 19/30 [00:00<00:00, 89.06it/s]
2023-09-06T14:14:58.3895122Z running benchmark:  97% 29/30 [00:00<00:00, 91.66it/s]
2023-09-06T14:14:58.3895855Z running benchmark: 100% 30/30 [00:00<00:00, 90.72it/s]
2023-09-06T14:14:58.3911357Z 0.904x
2023-09-06T14:15:03.3782592Z 
2023-09-06T14:15:06.3105210Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:15:06.3105879Z loading model: 0it [00:02, ?it/s]
2023-09-06T14:15:06.3106185Z cuda train basic_gnn_gin                      
2023-09-06T14:15:07.0700403Z [2023-09-06 14:15:07,068] [3/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T14:15:18.0891703Z 
2023-09-06T14:15:18.1928135Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:15:18.2969530Z running benchmark:  27% 8/30 [00:00<00:00, 77.40it/s]
2023-09-06T14:15:18.3661003Z running benchmark:  70% 21/30 [00:00<00:00, 105.23it/s]
2023-09-06T14:15:18.3661352Z running benchmark: 100% 30/30 [00:00<00:00, 108.29it/s]
2023-09-06T14:15:18.3679727Z 1.154x
2023-09-06T14:15:23.3497465Z 
2023-09-06T14:15:26.1778110Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:15:26.1778457Z loading model: 0it [00:02, ?it/s]
2023-09-06T14:15:26.1778764Z cuda train basic_gnn_sage                     
2023-09-06T14:15:26.9409657Z [2023-09-06 14:15:26,939] [3/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T14:15:37.7263680Z 
2023-09-06T14:15:37.8347660Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:15:37.9385204Z running benchmark:  47% 14/30 [00:00<00:00, 129.15it/s]
2023-09-06T14:15:37.9592284Z running benchmark:  90% 27/30 [00:00<00:00, 126.83it/s]
2023-09-06T14:15:37.9592639Z running benchmark: 100% 30/30 [00:00<00:00, 128.60it/s]
2023-09-06T14:15:37.9611598Z 1.070x
2023-09-06T14:15:42.8539977Z 
2023-09-06T14:15:43.3485187Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:15:43.3485555Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:15:43.3485841Z Test train is not implemented.
2023-09-06T14:15:43.3491794Z Traceback (most recent call last):
2023-09-06T14:15:43.3492245Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T14:15:43.3492614Z     ) = runner.load_model(
2023-09-06T14:15:43.3493002Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 368, in load_model
2023-09-06T14:15:43.3493547Z     benchmark = benchmark_cls(
2023-09-06T14:15:43.3498741Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 25, in __call__
2023-09-06T14:15:43.3499238Z     obj = type.__call__(cls, *args, **kwargs)
2023-09-06T14:15:43.3499696Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate/__init__.py", line 12, in __init__
2023-09-06T14:15:43.3500428Z     super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args)
2023-09-06T14:15:43.3501220Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 117, in __init__
2023-09-06T14:15:43.3501923Z     self.determine_batch_size(batch_size)
2023-09-06T14:15:43.3502807Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 175, in determine_batch_size
2023-09-06T14:15:43.3503359Z     raise NotImplementedError(f"Test {self.test} is not implemented.")
2023-09-06T14:15:43.3503771Z NotImplementedError: Test train is not implemented.
2023-09-06T14:15:43.3503999Z 
2023-09-06T14:15:43.3504142Z WARNING:root:cm3leon_generate failed to load
2023-09-06T14:15:47.7723442Z 
2023-09-06T14:15:48.7912955Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:15:48.7913299Z loading model: 0it [00:01, ?it/s]
2023-09-06T14:15:48.7913593Z cuda train dcgan                              
2023-09-06T14:16:02.3851217Z 
2023-09-06T14:16:02.4868288Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:16:02.5978251Z running benchmark:  47% 14/30 [00:00<00:00, 137.38it/s]
2023-09-06T14:16:02.6125881Z running benchmark:  93% 28/30 [00:00<00:00, 130.35it/s]
2023-09-06T14:16:02.6126266Z running benchmark: 100% 30/30 [00:00<00:00, 131.53it/s]
2023-09-06T14:16:02.6146493Z 0.780x
2023-09-06T14:16:07.6100650Z 
2023-09-06T14:16:12.2476273Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:16:12.2476812Z loading model: 0it [00:04, ?it/s]
2023-09-06T14:16:12.2477094Z cuda train demucs                             
2023-09-06T14:16:50.0228949Z 
2023-09-06T14:16:50.2014388Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:16:50.3060725Z running benchmark:   3% 1/30 [00:00<00:05,  5.59it/s]
2023-09-06T14:16:50.4100868Z running benchmark:   7% 2/30 [00:00<00:03,  7.39it/s]
2023-09-06T14:16:50.5154509Z running benchmark:  10% 3/30 [00:00<00:03,  8.26it/s]
2023-09-06T14:16:50.6194294Z running benchmark:  13% 4/30 [00:00<00:02,  8.70it/s]
2023-09-06T14:16:50.7240532Z running benchmark:  17% 5/30 [00:00<00:02,  9.01it/s]
2023-09-06T14:16:50.8282725Z running benchmark:  20% 6/30 [00:00<00:02,  9.19it/s]
2023-09-06T14:16:50.9325526Z running benchmark:  23% 7/30 [00:00<00:02,  9.32it/s]
2023-09-06T14:16:51.0373217Z running benchmark:  27% 8/30 [00:00<00:02,  9.40it/s]
2023-09-06T14:16:51.1418632Z running benchmark:  30% 9/30 [00:01<00:02,  9.45it/s]
2023-09-06T14:16:51.2470067Z running benchmark:  33% 10/30 [00:01<00:02,  9.49it/s]
2023-09-06T14:16:51.3563241Z running benchmark:  37% 11/30 [00:01<00:02,  9.49it/s]
2023-09-06T14:16:51.4621371Z running benchmark:  40% 12/30 [00:01<00:01,  9.40it/s]
2023-09-06T14:16:51.5665121Z running benchmark:  43% 13/30 [00:01<00:01,  9.40it/s]
2023-09-06T14:16:51.6722364Z running benchmark:  47% 14/30 [00:01<00:01,  9.46it/s]
2023-09-06T14:16:51.7768399Z running benchmark:  50% 15/30 [00:01<00:01,  9.46it/s]
2023-09-06T14:16:51.8811456Z running benchmark:  53% 16/30 [00:01<00:01,  9.49it/s]
2023-09-06T14:16:51.9856782Z running benchmark:  57% 17/30 [00:01<00:01,  9.51it/s]
2023-09-06T14:16:52.0899854Z running benchmark:  60% 18/30 [00:01<00:01,  9.53it/s]
2023-09-06T14:16:52.1945324Z running benchmark:  63% 19/30 [00:02<00:01,  9.55it/s]
2023-09-06T14:16:52.2994661Z running benchmark:  67% 20/30 [00:02<00:01,  9.56it/s]
2023-09-06T14:16:52.4038885Z running benchmark:  70% 21/30 [00:02<00:00,  9.55it/s]
2023-09-06T14:16:52.5084974Z running benchmark:  73% 22/30 [00:02<00:00,  9.56it/s]
2023-09-06T14:16:52.6126302Z running benchmark:  77% 23/30 [00:02<00:00,  9.56it/s]
2023-09-06T14:16:52.7175272Z running benchmark:  80% 24/30 [00:02<00:00,  9.57it/s]
2023-09-06T14:16:52.8217005Z running benchmark:  83% 25/30 [00:02<00:00,  9.56it/s]
2023-09-06T14:16:52.9266667Z running benchmark:  87% 26/30 [00:02<00:00,  9.57it/s]
2023-09-06T14:16:53.0305436Z running benchmark:  90% 27/30 [00:02<00:00,  9.56it/s]
2023-09-06T14:16:53.1350663Z running benchmark:  93% 28/30 [00:03<00:00,  9.58it/s]
2023-09-06T14:16:53.2393827Z running benchmark:  97% 29/30 [00:03<00:00,  9.58it/s]
2023-09-06T14:16:53.2394401Z running benchmark: 100% 30/30 [00:03<00:00,  9.58it/s]
2023-09-06T14:16:53.2394875Z running benchmark: 100% 30/30 [00:03<00:00,  9.32it/s]
2023-09-06T14:16:53.2418046Z 1.037x
2023-09-06T14:16:58.5991390Z 
2023-09-06T14:17:02.8286526Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:17:02.8287540Z loading model: 0it [00:04, ?it/s]
2023-09-06T14:17:02.8287846Z cuda train densenet121                        
2023-09-06T14:19:50.0051750Z 
2023-09-06T14:19:50.1711544Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:19:50.3282014Z running benchmark:   3% 1/30 [00:00<00:04,  6.02it/s]
2023-09-06T14:19:50.4907919Z running benchmark:   7% 2/30 [00:00<00:04,  6.21it/s]
2023-09-06T14:19:50.6567404Z running benchmark:  10% 3/30 [00:00<00:04,  6.18it/s]
2023-09-06T14:19:50.8149832Z running benchmark:  13% 4/30 [00:00<00:04,  6.12it/s]
2023-09-06T14:19:50.9715559Z running benchmark:  17% 5/30 [00:00<00:04,  6.19it/s]
2023-09-06T14:19:51.1334882Z running benchmark:  20% 6/30 [00:00<00:03,  6.25it/s]
2023-09-06T14:19:51.2948532Z running benchmark:  23% 7/30 [00:01<00:03,  6.23it/s]
2023-09-06T14:19:51.4555293Z running benchmark:  27% 8/30 [00:01<00:03,  6.22it/s]
2023-09-06T14:19:51.6180073Z running benchmark:  30% 9/30 [00:01<00:03,  6.22it/s]
2023-09-06T14:19:51.7794049Z running benchmark:  33% 10/30 [00:01<00:03,  6.20it/s]
2023-09-06T14:19:51.9350172Z running benchmark:  37% 11/30 [00:01<00:03,  6.20it/s]
2023-09-06T14:19:52.0920579Z running benchmark:  40% 12/30 [00:01<00:02,  6.27it/s]
2023-09-06T14:19:52.2520474Z running benchmark:  43% 13/30 [00:02<00:02,  6.30it/s]
2023-09-06T14:19:52.4105782Z running benchmark:  47% 14/30 [00:02<00:02,  6.28it/s]
2023-09-06T14:19:52.5710805Z running benchmark:  50% 15/30 [00:02<00:02,  6.29it/s]
2023-09-06T14:19:52.7390407Z running benchmark:  53% 16/30 [00:02<00:02,  6.27it/s]
2023-09-06T14:19:52.8969775Z running benchmark:  57% 17/30 [00:02<00:02,  6.17it/s]
2023-09-06T14:19:53.0570617Z running benchmark:  60% 18/30 [00:02<00:01,  6.22it/s]
2023-09-06T14:19:53.2160342Z running benchmark:  63% 19/30 [00:03<00:01,  6.23it/s]
2023-09-06T14:19:53.3746105Z running benchmark:  67% 20/30 [00:03<00:01,  6.25it/s]
2023-09-06T14:19:53.5314891Z running benchmark:  70% 21/30 [00:03<00:01,  6.26it/s]
2023-09-06T14:19:53.7062688Z running benchmark:  73% 22/30 [00:03<00:01,  6.30it/s]
2023-09-06T14:19:53.8676249Z running benchmark:  77% 23/30 [00:03<00:01,  6.11it/s]
2023-09-06T14:19:54.0305213Z running benchmark:  80% 24/30 [00:03<00:00,  6.14it/s]
2023-09-06T14:19:54.1906127Z running benchmark:  83% 25/30 [00:04<00:00,  6.14it/s]
2023-09-06T14:19:54.3505725Z running benchmark:  87% 26/30 [00:04<00:00,  6.17it/s]
2023-09-06T14:19:54.5105036Z running benchmark:  90% 27/30 [00:04<00:00,  6.19it/s]
2023-09-06T14:19:54.6842261Z running benchmark:  93% 28/30 [00:04<00:00,  6.21it/s]
2023-09-06T14:19:54.8426054Z running benchmark:  97% 29/30 [00:04<00:00,  6.07it/s]
2023-09-06T14:19:54.8426471Z running benchmark: 100% 30/30 [00:04<00:00,  6.14it/s]
2023-09-06T14:19:54.8426798Z running benchmark: 100% 30/30 [00:04<00:00,  6.20it/s]
2023-09-06T14:19:54.8453118Z 0.843x
2023-09-06T14:20:01.9945050Z 
2023-09-06T14:20:03.5169016Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:20:03.5169382Z loading model: 0it [00:01, ?it/s]
2023-09-06T14:20:03.5170082Z FCOS train is not supported by upstream detectron2. See GH Issue: https://github.com/facebookresearch/detectron2/issues/4369.
2023-09-06T14:20:03.5175585Z Traceback (most recent call last):
2023-09-06T14:20:03.5176193Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T14:20:03.5176687Z     ) = runner.load_model(
2023-09-06T14:20:03.5177610Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 368, in load_model
2023-09-06T14:20:03.5178072Z     benchmark = benchmark_cls(
2023-09-06T14:20:03.5178605Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 25, in __call__
2023-09-06T14:20:03.5182908Z     obj = type.__call__(cls, *args, **kwargs)
2023-09-06T14:20:03.5183480Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn/__init__.py", line 15, in __init__
2023-09-06T14:20:03.5185268Z     super().__init__(variant="COCO-Detection/fcos_R_50_FPN_1x.py", test=test, device=device,
2023-09-06T14:20:03.5185809Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/detectron2/model_factory.py", line 105, in __init__
2023-09-06T14:20:03.5186346Z     raise NotImplementedError("FCOS train is not supported by upstream detectron2. " \
2023-09-06T14:20:03.5186968Z NotImplementedError: FCOS train is not supported by upstream detectron2. See GH Issue: https://github.com/facebookresearch/detectron2/issues/4369.
2023-09-06T14:20:03.5189394Z 
2023-09-06T14:20:03.5189544Z WARNING:root:detectron2_fcos_r_50_fpn failed to load
2023-09-06T14:20:04.5503137Z speedup             gmean=1.09x mean=1.108x
2023-09-06T14:20:04.5503595Z abs_latency         gmean=16.61x mean=31.834x
2023-09-06T14:20:04.5504035Z compilation_latency mean=45.861 seconds
2023-09-06T14:20:04.5504566Z compression_ratio   mean=0.722x
2023-09-06T14:20:04.5505698Z eager_peak_mem      gmean=0.81x mean=1.894x
2023-09-06T14:20:04.5507140Z dynamo_peak_mem     gmean=1.27x mean=2.101x
2023-09-06T14:20:04.5510807Z calls_captured      gmean=110.41x mean=190.000x
2023-09-06T14:20:04.5512967Z unique_graphs       gmean=2.66x mean=3.214x
2023-09-06T14:20:04.5514939Z graph_breaks        gmean=10.22x mean=10.714x
2023-09-06T14:20:04.5517046Z unique_graph_breaks gmean=7.20x mean=7.214x
2023-09-06T14:20:05.3393438Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *cudagraphs-true* ]]
2023-09-06T14:20:05.3395158Z + python benchmarks/dynamo/torchbench.py --performance --cold-start-latency --training --amp --backend inductor --device cuda --total-partitions 4 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inductor_with_cudagraphs_torchbench_amp_training_cuda_performance.csv
2023-09-06T14:20:12.5918433Z 
2023-09-06T14:20:13.8929669Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:20:13.8930706Z loading model: 0it [00:01, ?it/s]
2023-09-06T14:20:13.8931013Z cuda train torchrec_dlrm                      
2023-09-06T14:20:36.1176232Z 
2023-09-06T14:20:36.2372735Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:20:36.3481112Z running benchmark:  17% 5/30 [00:00<00:00, 41.71it/s]
2023-09-06T14:20:36.4586567Z running benchmark:  33% 10/30 [00:00<00:00, 43.63it/s]
2023-09-06T14:20:36.5655319Z running benchmark:  50% 15/30 [00:00<00:00, 44.35it/s]
2023-09-06T14:20:36.6947219Z running benchmark:  67% 20/30 [00:00<00:00, 45.29it/s]
2023-09-06T14:20:36.8038469Z running benchmark:  83% 25/30 [00:00<00:00, 42.68it/s]
2023-09-06T14:20:36.8038890Z running benchmark: 100% 30/30 [00:00<00:00, 43.68it/s]
2023-09-06T14:20:36.8039216Z running benchmark: 100% 30/30 [00:00<00:00, 43.69it/s]
2023-09-06T14:20:36.8057027Z 1.607x
2023-09-06T14:20:41.9131696Z 
2023-09-06T14:20:45.2881008Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:20:45.2881364Z loading model: 0it [00:03, ?it/s]
2023-09-06T14:20:45.2881899Z cuda train BERT_pytorch                       
2023-09-06T14:21:51.9858269Z 
2023-09-06T14:21:52.0952524Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:21:52.2323753Z running benchmark:   3% 1/30 [00:00<00:03,  9.12it/s]
2023-09-06T14:21:52.3739258Z running benchmark:  10% 3/30 [00:00<00:02, 12.62it/s]
2023-09-06T14:21:52.5406418Z running benchmark:  17% 5/30 [00:00<00:01, 13.35it/s]
2023-09-06T14:21:52.6817910Z running benchmark:  23% 7/30 [00:00<00:01, 12.74it/s]
2023-09-06T14:21:52.8309003Z running benchmark:  30% 9/30 [00:00<00:01, 13.24it/s]
2023-09-06T14:21:52.9840921Z running benchmark:  37% 11/30 [00:00<00:01, 13.30it/s]
2023-09-06T14:21:53.1217746Z running benchmark:  43% 13/30 [00:00<00:01, 13.22it/s]
2023-09-06T14:21:53.2633395Z running benchmark:  50% 15/30 [00:01<00:01, 13.61it/s]
2023-09-06T14:21:53.4071178Z running benchmark:  57% 17/30 [00:01<00:00, 13.77it/s]
2023-09-06T14:21:53.5601480Z running benchmark:  63% 19/30 [00:01<00:00, 13.82it/s]
2023-09-06T14:21:53.7041073Z running benchmark:  70% 21/30 [00:01<00:00, 13.58it/s]
2023-09-06T14:21:53.8487553Z running benchmark:  77% 23/30 [00:01<00:00, 13.67it/s]
2023-09-06T14:21:53.9917355Z running benchmark:  83% 25/30 [00:01<00:00, 13.72it/s]
2023-09-06T14:21:54.1324745Z running benchmark:  90% 27/30 [00:02<00:00, 13.80it/s]
2023-09-06T14:21:54.2020525Z running benchmark:  97% 29/30 [00:02<00:00, 13.92it/s]
2023-09-06T14:21:54.2020915Z running benchmark: 100% 30/30 [00:02<00:00, 13.53it/s]
2023-09-06T14:21:54.2044746Z 3.250x
2023-09-06T14:21:59.8724489Z 
2023-09-06T14:22:05.6345159Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:22:05.6346111Z loading model: 0it [00:05, ?it/s]
2023-09-06T14:22:05.6347721Z cuda train Background_Matting                 
2023-09-06T14:23:25.6414554Z 
2023-09-06T14:23:26.0617439Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:23:26.3038760Z running benchmark:   3% 1/30 [00:00<00:12,  2.38it/s]
2023-09-06T14:23:26.5464501Z running benchmark:   7% 2/30 [00:00<00:08,  3.17it/s]
2023-09-06T14:23:26.7889480Z running benchmark:  10% 3/30 [00:00<00:07,  3.54it/s]
2023-09-06T14:23:27.0313326Z running benchmark:  13% 4/30 [00:01<00:06,  3.75it/s]
2023-09-06T14:23:27.2734583Z running benchmark:  17% 5/30 [00:01<00:06,  3.88it/s]
2023-09-06T14:23:27.5156129Z running benchmark:  20% 6/30 [00:01<00:06,  3.96it/s]
2023-09-06T14:23:27.7585897Z running benchmark:  23% 7/30 [00:01<00:05,  4.01it/s]
2023-09-06T14:23:28.0007529Z running benchmark:  27% 8/30 [00:02<00:05,  4.05it/s]
2023-09-06T14:23:28.2428139Z running benchmark:  30% 9/30 [00:02<00:05,  4.07it/s]
2023-09-06T14:23:28.4853895Z running benchmark:  33% 10/30 [00:02<00:04,  4.09it/s]
2023-09-06T14:23:28.7278675Z running benchmark:  37% 11/30 [00:02<00:04,  4.10it/s]
2023-09-06T14:23:28.9703969Z running benchmark:  40% 12/30 [00:03<00:04,  4.11it/s]
2023-09-06T14:23:29.2125586Z running benchmark:  43% 13/30 [00:03<00:04,  4.11it/s]
2023-09-06T14:23:29.4551609Z running benchmark:  47% 14/30 [00:03<00:03,  4.12it/s]
2023-09-06T14:23:29.6977251Z running benchmark:  50% 15/30 [00:03<00:03,  4.12it/s]
2023-09-06T14:23:29.9402321Z running benchmark:  53% 16/30 [00:04<00:03,  4.12it/s]
2023-09-06T14:23:30.1822644Z running benchmark:  57% 17/30 [00:04<00:03,  4.12it/s]
2023-09-06T14:23:30.4246295Z running benchmark:  60% 18/30 [00:04<00:02,  4.12it/s]
2023-09-06T14:23:30.6671472Z running benchmark:  63% 19/30 [00:04<00:02,  4.12it/s]
2023-09-06T14:23:30.9099196Z running benchmark:  67% 20/30 [00:05<00:02,  4.12it/s]
2023-09-06T14:23:31.1518380Z running benchmark:  70% 21/30 [00:05<00:02,  4.12it/s]
2023-09-06T14:23:31.3938706Z running benchmark:  73% 22/30 [00:05<00:01,  4.13it/s]
2023-09-06T14:23:31.6359185Z running benchmark:  77% 23/30 [00:05<00:01,  4.13it/s]
2023-09-06T14:23:31.8783631Z running benchmark:  80% 24/30 [00:05<00:01,  4.13it/s]
2023-09-06T14:23:32.1206097Z running benchmark:  83% 25/30 [00:06<00:01,  4.13it/s]
2023-09-06T14:23:32.3626496Z running benchmark:  87% 26/30 [00:06<00:00,  4.13it/s]
2023-09-06T14:23:32.6049325Z running benchmark:  90% 27/30 [00:06<00:00,  4.13it/s]
2023-09-06T14:23:32.8473298Z running benchmark:  93% 28/30 [00:06<00:00,  4.13it/s]
2023-09-06T14:23:33.0895461Z running benchmark:  97% 29/30 [00:07<00:00,  4.13it/s]
2023-09-06T14:23:33.0896197Z running benchmark: 100% 30/30 [00:07<00:00,  4.13it/s]
2023-09-06T14:23:33.0896779Z running benchmark: 100% 30/30 [00:07<00:00,  4.03it/s]
2023-09-06T14:23:33.0921250Z 1.165x
2023-09-06T14:23:39.2445738Z 
2023-09-06T14:23:50.5995606Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:23:50.5996692Z loading model: 0it [00:11, ?it/s]
2023-09-06T14:23:50.5997521Z Eager model failed to run
2023-09-06T14:23:50.6008249Z Traceback (most recent call last):
2023-09-06T14:23:50.6008989Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T14:23:50.6009651Z     self.model_iter_fn(model, example_inputs)
2023-09-06T14:23:50.6010331Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_and_backward_pass
2023-09-06T14:23:50.6011001Z     self.grad_scaler.scale(loss).backward()
2023-09-06T14:23:50.6013464Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
2023-09-06T14:23:50.6018168Z     torch.autograd.backward(
2023-09-06T14:23:50.6020033Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
2023-09-06T14:23:50.6020995Z     Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2023-09-06T14:23:50.6021917Z RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
2023-09-06T14:23:50.6023529Z 
2023-09-06T14:23:50.6023850Z The above exception was the direct cause of the following exception:
2023-09-06T14:23:50.6024319Z 
2023-09-06T14:23:50.6024567Z Traceback (most recent call last):
2023-09-06T14:23:50.6025309Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T14:23:50.6025917Z     ) = runner.load_model(
2023-09-06T14:23:50.6026543Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T14:23:50.6027312Z     self.validate_model(model, example_inputs)
2023-09-06T14:23:50.6028008Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T14:23:50.6028815Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T14:23:50.6029770Z NotImplementedError: Eager model failed to run
2023-09-06T14:23:50.6030062Z 
2023-09-06T14:23:50.6030312Z WARNING:root:DALLE2_pytorch failed to load
2023-09-06T14:23:55.4456909Z 
2023-09-06T14:24:00.1406397Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:24:00.1406786Z loading model: 0it [00:04, ?it/s]
2023-09-06T14:24:00.1407109Z cuda train LearningToPaint                    
2023-09-06T14:24:35.2730077Z 
2023-09-06T14:24:35.3750045Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:24:35.4890115Z running benchmark:  13% 4/30 [00:00<00:00, 39.15it/s]
2023-09-06T14:24:35.6071890Z running benchmark:  30% 9/30 [00:00<00:00, 42.01it/s]
2023-09-06T14:24:35.7260712Z running benchmark:  47% 14/30 [00:00<00:00, 42.13it/s]
2023-09-06T14:24:35.8362870Z running benchmark:  63% 19/30 [00:00<00:00, 42.09it/s]
2023-09-06T14:24:35.9490465Z running benchmark:  80% 24/30 [00:00<00:00, 43.24it/s]
2023-09-06T14:24:35.9726333Z running benchmark:  97% 29/30 [00:00<00:00, 43.62it/s]
2023-09-06T14:24:35.9726921Z running benchmark: 100% 30/30 [00:00<00:00, 42.85it/s]
2023-09-06T14:24:35.9746101Z 1.648x
2023-09-06T14:24:41.6066643Z 
2023-09-06T14:24:45.0048242Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:24:45.0049109Z loading model: 0it [00:03, ?it/s]
2023-09-06T14:24:45.0049418Z cuda train Super_SloMo                        
2023-09-06T14:26:07.4135446Z 
2023-09-06T14:26:07.7200833Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:26:07.8622511Z running benchmark:   3% 1/30 [00:00<00:08,  3.26it/s]
2023-09-06T14:26:08.0047661Z running benchmark:   7% 2/30 [00:00<00:05,  4.76it/s]
2023-09-06T14:26:08.1470383Z running benchmark:  10% 3/30 [00:00<00:04,  5.58it/s]
2023-09-06T14:26:08.2891413Z running benchmark:  13% 4/30 [00:00<00:04,  6.07it/s]
2023-09-06T14:26:08.4312214Z running benchmark:  17% 5/30 [00:00<00:03,  6.39it/s]
2023-09-06T14:26:08.5736012Z running benchmark:  20% 6/30 [00:01<00:03,  6.60it/s]
2023-09-06T14:26:08.7157403Z running benchmark:  23% 7/30 [00:01<00:03,  6.73it/s]
2023-09-06T14:26:08.8577042Z running benchmark:  27% 8/30 [00:01<00:03,  6.82it/s]
2023-09-06T14:26:08.9998192Z running benchmark:  30% 9/30 [00:01<00:03,  6.89it/s]
2023-09-06T14:26:09.1417945Z running benchmark:  33% 10/30 [00:01<00:02,  6.94it/s]
2023-09-06T14:26:09.2838015Z running benchmark:  37% 11/30 [00:01<00:02,  6.97it/s]
2023-09-06T14:26:09.4303094Z running benchmark:  40% 12/30 [00:01<00:02,  6.99it/s]
2023-09-06T14:26:09.5726401Z running benchmark:  43% 13/30 [00:02<00:02,  7.01it/s]
2023-09-06T14:26:09.7147626Z running benchmark:  47% 14/30 [00:02<00:02,  6.95it/s]
2023-09-06T14:26:09.8569939Z running benchmark:  50% 15/30 [00:02<00:02,  6.97it/s]
2023-09-06T14:26:09.9992490Z running benchmark:  53% 16/30 [00:02<00:02,  6.99it/s]
2023-09-06T14:26:10.1413019Z running benchmark:  57% 17/30 [00:02<00:01,  7.00it/s]
2023-09-06T14:26:10.2832002Z running benchmark:  60% 18/30 [00:02<00:01,  7.01it/s]
2023-09-06T14:26:10.4253950Z running benchmark:  63% 19/30 [00:02<00:01,  7.02it/s]
2023-09-06T14:26:10.5677572Z running benchmark:  67% 20/30 [00:03<00:01,  7.03it/s]
2023-09-06T14:26:10.7098638Z running benchmark:  70% 21/30 [00:03<00:01,  7.03it/s]
2023-09-06T14:26:10.8521121Z running benchmark:  73% 22/30 [00:03<00:01,  7.03it/s]
2023-09-06T14:26:10.9942407Z running benchmark:  77% 23/30 [00:03<00:00,  7.03it/s]
2023-09-06T14:26:11.1362851Z running benchmark:  80% 24/30 [00:03<00:00,  7.03it/s]
2023-09-06T14:26:11.2782884Z running benchmark:  83% 25/30 [00:03<00:00,  7.03it/s]
2023-09-06T14:26:11.4202611Z running benchmark:  87% 26/30 [00:03<00:00,  7.04it/s]
2023-09-06T14:26:11.5628553Z running benchmark:  90% 27/30 [00:04<00:00,  7.04it/s]
2023-09-06T14:26:11.7050733Z running benchmark:  93% 28/30 [00:04<00:00,  7.03it/s]
2023-09-06T14:26:11.8473964Z running benchmark:  97% 29/30 [00:04<00:00,  7.03it/s]
2023-09-06T14:26:11.8474457Z running benchmark: 100% 30/30 [00:04<00:00,  7.03it/s]
2023-09-06T14:26:11.8474780Z running benchmark: 100% 30/30 [00:04<00:00,  6.76it/s]
2023-09-06T14:26:11.8499307Z 1.288x
2023-09-06T14:26:17.6017942Z 
2023-09-06T14:26:19.9626723Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:26:19.9627323Z loading model: 0it [00:02, ?it/s]
2023-09-06T14:26:19.9627671Z cuda train alexnet                            
2023-09-06T14:26:37.3127677Z 
2023-09-06T14:26:37.4200285Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:26:37.5295567Z running benchmark:  17% 5/30 [00:00<00:00, 46.43it/s]
2023-09-06T14:26:37.6368762Z running benchmark:  37% 11/30 [00:00<00:00, 51.32it/s]
2023-09-06T14:26:37.7451926Z running benchmark:  57% 17/30 [00:00<00:00, 53.40it/s]
2023-09-06T14:26:37.8539856Z running benchmark:  77% 23/30 [00:00<00:00, 54.20it/s]
2023-09-06T14:26:37.8722877Z running benchmark:  97% 29/30 [00:00<00:00, 54.55it/s]
2023-09-06T14:26:37.8723323Z running benchmark: 100% 30/30 [00:00<00:00, 53.55it/s]
2023-09-06T14:26:37.8739949Z 1.352x
2023-09-06T14:26:43.0275866Z 
2023-09-06T14:26:50.8325886Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:26:50.8326839Z loading model: 0it [00:07, ?it/s]
2023-09-06T14:26:50.8327294Z cuda train attention_is_all_you_need_pytorch  
2023-09-06T14:28:07.8792619Z 
2023-09-06T14:28:08.0692121Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:28:08.2552630Z running benchmark:   3% 1/30 [00:00<00:05,  5.26it/s]
2023-09-06T14:28:08.4320215Z running benchmark:  10% 3/30 [00:00<00:03,  8.46it/s]
2023-09-06T14:28:08.6099135Z running benchmark:  17% 5/30 [00:00<00:02,  9.72it/s]
2023-09-06T14:28:08.7870996Z running benchmark:  23% 7/30 [00:00<00:02, 10.31it/s]
2023-09-06T14:28:08.9643389Z running benchmark:  30% 9/30 [00:00<00:01, 10.66it/s]
2023-09-06T14:28:09.1380763Z running benchmark:  37% 11/30 [00:01<00:01, 10.87it/s]
2023-09-06T14:28:09.3294601Z running benchmark:  43% 13/30 [00:01<00:01, 11.08it/s]
2023-09-06T14:28:09.5072337Z running benchmark:  50% 15/30 [00:01<00:01, 10.87it/s]
2023-09-06T14:28:09.6833204Z running benchmark:  57% 17/30 [00:01<00:01, 10.98it/s]
2023-09-06T14:28:09.8607846Z running benchmark:  63% 19/30 [00:01<00:00, 11.10it/s]
2023-09-06T14:28:10.0360512Z running benchmark:  70% 21/30 [00:01<00:00, 11.15it/s]
2023-09-06T14:28:10.2108389Z running benchmark:  77% 23/30 [00:02<00:00, 11.23it/s]
2023-09-06T14:28:10.3873969Z running benchmark:  83% 25/30 [00:02<00:00, 11.29it/s]
2023-09-06T14:28:10.5654801Z running benchmark:  90% 27/30 [00:02<00:00, 11.30it/s]
2023-09-06T14:28:10.6541402Z running benchmark:  97% 29/30 [00:02<00:00, 11.28it/s]
2023-09-06T14:28:10.6541770Z running benchmark: 100% 30/30 [00:02<00:00, 10.81it/s]
2023-09-06T14:28:10.6566261Z 1.625x
2023-09-06T14:28:16.5374885Z 
2023-09-06T14:28:19.3555026Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:28:19.3555385Z loading model: 0it [00:02, ?it/s]
2023-09-06T14:28:19.3555674Z cuda train basic_gnn_edgecnn                  
2023-09-06T14:28:20.0884913Z [2023-09-06 14:28:20,087] [3/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T14:28:38.8276412Z 
2023-09-06T14:28:38.9679832Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:28:39.0756895Z running benchmark:  27% 8/30 [00:00<00:00, 57.06it/s]
2023-09-06T14:28:39.1858173Z running benchmark:  53% 16/30 [00:00<00:00, 65.94it/s]
2023-09-06T14:28:39.2686853Z running benchmark:  80% 24/30 [00:00<00:00, 68.85it/s]
2023-09-06T14:28:39.2687246Z running benchmark: 100% 30/30 [00:00<00:00, 67.98it/s]
2023-09-06T14:28:39.2706577Z 1.450x
2023-09-06T14:28:44.2698994Z 
2023-09-06T14:28:47.0318560Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:28:47.0318925Z loading model: 0it [00:02, ?it/s]
2023-09-06T14:28:47.0319239Z cuda train basic_gnn_gcn                      
2023-09-06T14:28:57.6322026Z [2023-09-06 14:28:57,630] [13/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T14:29:07.2403346Z 
2023-09-06T14:29:07.3410643Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:29:07.4463951Z running benchmark:  30% 9/30 [00:00<00:00, 89.38it/s]
2023-09-06T14:29:07.5508497Z running benchmark:  63% 19/30 [00:00<00:00, 92.65it/s]
2023-09-06T14:29:07.5630678Z running benchmark:  97% 29/30 [00:00<00:00, 94.07it/s]
2023-09-06T14:29:07.5636445Z running benchmark: 100% 30/30 [00:00<00:00, 92.93it/s]
2023-09-06T14:29:07.5654101Z 1.043x
2023-09-06T14:29:12.5200223Z 
2023-09-06T14:29:15.3818631Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:29:15.3819028Z loading model: 0it [00:02, ?it/s]
2023-09-06T14:29:15.3819331Z cuda train basic_gnn_gin                      
2023-09-06T14:29:16.1211096Z [2023-09-06 14:29:16,120] [3/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T14:29:27.6167472Z 
2023-09-06T14:29:27.7180430Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:29:27.8120964Z running benchmark:  50% 15/30 [00:00<00:00, 148.24it/s]
2023-09-06T14:29:27.8121389Z running benchmark: 100% 30/30 [00:00<00:00, 153.40it/s]
2023-09-06T14:29:27.8138633Z 1.980x
2023-09-06T14:29:32.7546960Z 
2023-09-06T14:29:35.5860227Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:29:35.5860811Z loading model: 0it [00:02, ?it/s]
2023-09-06T14:29:35.5861161Z cuda train basic_gnn_sage                     
2023-09-06T14:29:36.3302924Z [2023-09-06 14:29:36,329] [3/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T14:29:47.7148297Z 
2023-09-06T14:29:47.8167051Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:29:47.8803661Z running benchmark:  60% 18/30 [00:00<00:00, 176.80it/s]
2023-09-06T14:29:47.8804108Z running benchmark: 100% 30/30 [00:00<00:00, 181.02it/s]
2023-09-06T14:29:47.8819439Z 2.044x
2023-09-06T14:29:52.8003727Z 
2023-09-06T14:29:53.2860606Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:29:53.2860963Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:29:53.2861265Z Test train is not implemented.
2023-09-06T14:29:53.2868432Z Traceback (most recent call last):
2023-09-06T14:29:53.2868879Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T14:29:53.2869538Z     ) = runner.load_model(
2023-09-06T14:29:53.2870146Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 368, in load_model
2023-09-06T14:29:53.2870841Z     benchmark = benchmark_cls(
2023-09-06T14:29:53.2871385Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 25, in __call__
2023-09-06T14:29:53.2871953Z     obj = type.__call__(cls, *args, **kwargs)
2023-09-06T14:29:53.2872763Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate/__init__.py", line 12, in __init__
2023-09-06T14:29:53.2873794Z     super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args)
2023-09-06T14:29:53.2874504Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 117, in __init__
2023-09-06T14:29:53.2875357Z     self.determine_batch_size(batch_size)
2023-09-06T14:29:53.2875931Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 175, in determine_batch_size
2023-09-06T14:29:53.2876432Z     raise NotImplementedError(f"Test {self.test} is not implemented.")
2023-09-06T14:29:53.2876859Z NotImplementedError: Test train is not implemented.
2023-09-06T14:29:53.2877097Z 
2023-09-06T14:29:53.2877242Z WARNING:root:cm3leon_generate failed to load
2023-09-06T14:29:57.6663217Z 
2023-09-06T14:29:58.7277486Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:29:58.7278016Z loading model: 0it [00:01, ?it/s]
2023-09-06T14:29:58.7278557Z cuda train dcgan                              
2023-09-06T14:30:13.2474572Z 
2023-09-06T14:30:13.3474813Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:30:13.4215204Z running benchmark:  57% 17/30 [00:00<00:00, 169.57it/s]
2023-09-06T14:30:13.4215572Z running benchmark: 100% 30/30 [00:00<00:00, 171.91it/s]
2023-09-06T14:30:13.4233026Z 1.386x
2023-09-06T14:30:18.4948323Z 
2023-09-06T14:30:23.0942022Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:30:23.0944810Z loading model: 0it [00:04, ?it/s]
2023-09-06T14:30:23.0945169Z cuda train demucs                             
2023-09-06T14:31:03.1357560Z 
2023-09-06T14:31:03.3350443Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:31:03.4387191Z running benchmark:   3% 1/30 [00:00<00:05,  5.02it/s]
2023-09-06T14:31:03.5428351Z running benchmark:   7% 2/30 [00:00<00:04,  6.98it/s]
2023-09-06T14:31:03.6466339Z running benchmark:  10% 3/30 [00:00<00:03,  7.97it/s]
2023-09-06T14:31:03.7504780Z running benchmark:  13% 4/30 [00:00<00:03,  8.56it/s]
2023-09-06T14:31:03.8542433Z running benchmark:  17% 5/30 [00:00<00:02,  8.91it/s]
2023-09-06T14:31:03.9580713Z running benchmark:  20% 6/30 [00:00<00:02,  9.15it/s]
2023-09-06T14:31:04.0618599Z running benchmark:  23% 7/30 [00:00<00:02,  9.30it/s]
2023-09-06T14:31:04.1658007Z running benchmark:  27% 8/30 [00:00<00:02,  9.40it/s]
2023-09-06T14:31:04.2699816Z running benchmark:  30% 9/30 [00:01<00:02,  9.47it/s]
2023-09-06T14:31:04.3740972Z running benchmark:  33% 10/30 [00:01<00:02,  9.51it/s]
2023-09-06T14:31:04.4778516Z running benchmark:  37% 11/30 [00:01<00:01,  9.54it/s]
2023-09-06T14:31:04.5819571Z running benchmark:  40% 12/30 [00:01<00:01,  9.57it/s]
2023-09-06T14:31:04.6857368Z running benchmark:  43% 13/30 [00:01<00:01,  9.58it/s]
2023-09-06T14:31:04.7898683Z running benchmark:  47% 14/30 [00:01<00:01,  9.60it/s]
2023-09-06T14:31:04.8936672Z running benchmark:  50% 15/30 [00:01<00:01,  9.60it/s]
2023-09-06T14:31:04.9976120Z running benchmark:  53% 16/30 [00:01<00:01,  9.61it/s]
2023-09-06T14:31:05.1016468Z running benchmark:  57% 17/30 [00:01<00:01,  9.61it/s]
2023-09-06T14:31:05.2055299Z running benchmark:  60% 18/30 [00:01<00:01,  9.61it/s]
2023-09-06T14:31:05.3092041Z running benchmark:  63% 19/30 [00:02<00:01,  9.62it/s]
2023-09-06T14:31:05.4159500Z running benchmark:  67% 20/30 [00:02<00:01,  9.62it/s]
2023-09-06T14:31:05.5200012Z running benchmark:  70% 21/30 [00:02<00:00,  9.54it/s]
2023-09-06T14:31:05.6238451Z running benchmark:  73% 22/30 [00:02<00:00,  9.56it/s]
2023-09-06T14:31:05.7278208Z running benchmark:  77% 23/30 [00:02<00:00,  9.59it/s]
2023-09-06T14:31:05.8315959Z running benchmark:  80% 24/30 [00:02<00:00,  9.59it/s]
2023-09-06T14:31:05.9353767Z running benchmark:  83% 25/30 [00:02<00:00,  9.61it/s]
2023-09-06T14:31:06.0394906Z running benchmark:  87% 26/30 [00:02<00:00,  9.61it/s]
2023-09-06T14:31:06.1435814Z running benchmark:  90% 27/30 [00:02<00:00,  9.61it/s]
2023-09-06T14:31:06.2477192Z running benchmark:  93% 28/30 [00:03<00:00,  9.61it/s]
2023-09-06T14:31:06.3516544Z running benchmark:  97% 29/30 [00:03<00:00,  9.61it/s]
2023-09-06T14:31:06.3517865Z running benchmark: 100% 30/30 [00:03<00:00,  9.61it/s]
2023-09-06T14:31:06.3518418Z running benchmark: 100% 30/30 [00:03<00:00,  9.33it/s]
2023-09-06T14:31:06.3541508Z 1.054x
2023-09-06T14:31:11.6665525Z 
2023-09-06T14:31:15.9642878Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:31:15.9643315Z loading model: 0it [00:04, ?it/s]
2023-09-06T14:31:15.9643619Z cuda train densenet121                        
2023-09-06T14:34:06.9322782Z 
2023-09-06T14:34:07.0521785Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:34:07.1532431Z running benchmark:   3% 1/30 [00:00<00:03,  8.32it/s]
2023-09-06T14:34:07.2537282Z running benchmark:   7% 2/30 [00:00<00:03,  9.17it/s]
2023-09-06T14:34:07.3557607Z running benchmark:  10% 3/30 [00:00<00:02,  9.51it/s]
2023-09-06T14:34:07.4582054Z running benchmark:  13% 4/30 [00:00<00:02,  9.62it/s]
2023-09-06T14:34:07.5588070Z running benchmark:  17% 5/30 [00:00<00:02,  9.69it/s]
2023-09-06T14:34:07.6740306Z running benchmark:  20% 6/30 [00:00<00:02,  9.76it/s]
2023-09-06T14:34:07.7754866Z running benchmark:  23% 7/30 [00:00<00:02,  9.38it/s]
2023-09-06T14:34:07.8758270Z running benchmark:  27% 8/30 [00:00<00:02,  9.52it/s]
2023-09-06T14:34:07.9758527Z running benchmark:  30% 9/30 [00:00<00:02,  9.66it/s]
2023-09-06T14:34:08.0765404Z running benchmark:  33% 10/30 [00:01<00:02,  9.76it/s]
2023-09-06T14:34:08.1772360Z running benchmark:  37% 11/30 [00:01<00:01,  9.81it/s]
2023-09-06T14:34:08.2784425Z running benchmark:  40% 12/30 [00:01<00:01,  9.85it/s]
2023-09-06T14:34:08.3797225Z running benchmark:  43% 13/30 [00:01<00:01,  9.86it/s]
2023-09-06T14:34:08.4808166Z running benchmark:  47% 14/30 [00:01<00:01,  9.86it/s]
2023-09-06T14:34:08.5817325Z running benchmark:  50% 15/30 [00:01<00:01,  9.87it/s]
2023-09-06T14:34:08.6925898Z running benchmark:  53% 16/30 [00:01<00:01,  9.88it/s]
2023-09-06T14:34:08.7927652Z running benchmark:  57% 17/30 [00:01<00:01,  9.61it/s]
2023-09-06T14:34:08.9891738Z running benchmark:  60% 18/30 [00:01<00:01,  9.71it/s]
2023-09-06T14:34:09.1848519Z running benchmark:  67% 20/30 [00:02<00:01,  9.93it/s]
2023-09-06T14:34:09.3800026Z running benchmark:  73% 22/30 [00:02<00:00, 10.04it/s]
2023-09-06T14:34:09.5769039Z running benchmark:  80% 24/30 [00:02<00:00, 10.12it/s]
2023-09-06T14:34:09.7907604Z running benchmark:  87% 26/30 [00:02<00:00, 10.13it/s]
2023-09-06T14:34:09.8907933Z running benchmark:  93% 28/30 [00:02<00:00,  9.86it/s]
2023-09-06T14:34:09.9913590Z running benchmark:  97% 29/30 [00:02<00:00,  9.89it/s]
2023-09-06T14:34:09.9914347Z running benchmark: 100% 30/30 [00:03<00:00,  9.90it/s]
2023-09-06T14:34:09.9914832Z running benchmark: 100% 30/30 [00:03<00:00,  9.80it/s]
2023-09-06T14:34:09.9941150Z 2.399x
2023-09-06T14:34:16.5749280Z 
2023-09-06T14:34:18.0684638Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:34:18.0684976Z loading model: 0it [00:01, ?it/s]
2023-09-06T14:34:18.0685685Z FCOS train is not supported by upstream detectron2. See GH Issue: https://github.com/facebookresearch/detectron2/issues/4369.
2023-09-06T14:34:18.0691870Z Traceback (most recent call last):
2023-09-06T14:34:18.0692503Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T14:34:18.0693207Z     ) = runner.load_model(
2023-09-06T14:34:18.0693914Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 368, in load_model
2023-09-06T14:34:18.0694996Z     benchmark = benchmark_cls(
2023-09-06T14:34:18.0695727Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 25, in __call__
2023-09-06T14:34:18.0696401Z     obj = type.__call__(cls, *args, **kwargs)
2023-09-06T14:34:18.0697165Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn/__init__.py", line 15, in __init__
2023-09-06T14:34:18.0699595Z     super().__init__(variant="COCO-Detection/fcos_R_50_FPN_1x.py", test=test, device=device,
2023-09-06T14:34:18.0700554Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/detectron2/model_factory.py", line 105, in __init__
2023-09-06T14:34:18.0702110Z     raise NotImplementedError("FCOS train is not supported by upstream detectron2. " \
2023-09-06T14:34:18.0703256Z NotImplementedError: FCOS train is not supported by upstream detectron2. See GH Issue: https://github.com/facebookresearch/detectron2/issues/4369.
2023-09-06T14:34:18.0703935Z 
2023-09-06T14:34:18.0704208Z WARNING:root:detectron2_fcos_r_50_fpn failed to load
2023-09-06T14:34:18.9730548Z speedup             gmean=1.58x mean=1.664x
2023-09-06T14:34:18.9731070Z abs_latency         gmean=11.39x mean=24.538x
2023-09-06T14:34:18.9731557Z compilation_latency mean=47.472 seconds
2023-09-06T14:34:18.9732159Z compression_ratio   mean=0.729x
2023-09-06T14:34:18.9732722Z eager_peak_mem      gmean=0.81x mean=1.893x
2023-09-06T14:34:18.9733954Z dynamo_peak_mem     gmean=1.25x mean=2.091x
2023-09-06T14:34:18.9735987Z calls_captured      gmean=110.41x mean=190.000x
2023-09-06T14:34:18.9737580Z unique_graphs       gmean=2.66x mean=3.214x
2023-09-06T14:34:18.9740050Z graph_breaks        gmean=10.22x mean=10.714x
2023-09-06T14:34:18.9741845Z unique_graph_breaks gmean=7.20x mean=7.214x
2023-09-06T14:34:19.6464905Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *dynamic-true* ]]
2023-09-06T14:34:19.6466487Z + python benchmarks/dynamo/torchbench.py --performance --cold-start-latency --training --amp --backend inductor --dynamic-shapes --dynamic-batch-only --device cuda --total-partitions 4 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inductor_dynamic_torchbench_amp_training_cuda_performance.csv
2023-09-06T14:34:26.8072536Z 
2023-09-06T14:34:28.1026376Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:34:28.1026749Z loading model: 0it [00:01, ?it/s]
2023-09-06T14:34:28.1045201Z Traceback (most recent call last):
2023-09-06T14:34:28.1045890Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 495, in <module>
2023-09-06T14:34:28.1046927Z     torchbench_main()
2023-09-06T14:34:28.1047342Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 491, in torchbench_main
2023-09-06T14:34:28.1047773Z     main(TorchBenchmarkRunner(), original_dir)
2023-09-06T14:34:28.1048258Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3031, in main
2023-09-06T14:34:28.1051974Z     process_entry(0, runner, original_dir, args)
2023-09-06T14:34:28.1052399Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2988, in process_entry
2023-09-06T14:34:28.1056913Z     return maybe_fresh_cache(
2023-09-06T14:34:28.1057289Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1659, in inner
2023-09-06T14:34:28.1059158Z     return fn(*args, **kwargs)
2023-09-06T14:34:28.1059531Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3477, in run
2023-09-06T14:34:28.1065432Z     assert marked, f"nothing in example_inputs had a dim with {batch_size}"
2023-09-06T14:34:28.1065968Z AssertionError: nothing in example_inputs had a dim with 1024
2023-09-06T14:34:29.0647009Z ERROR
2023-09-06T14:34:32.6194337Z 
2023-09-06T14:34:35.9221547Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:34:35.9221907Z loading model: 0it [00:03, ?it/s]
2023-09-06T14:34:35.9222203Z cuda train BERT_pytorch                       
2023-09-06T14:36:12.6150657Z 
2023-09-06T14:36:12.7265258Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:36:12.8765347Z running benchmark:   3% 1/30 [00:00<00:03,  8.95it/s]
2023-09-06T14:36:13.0248245Z running benchmark:  10% 3/30 [00:00<00:02, 11.83it/s]
2023-09-06T14:36:13.1723013Z running benchmark:  17% 5/30 [00:00<00:01, 12.63it/s]
2023-09-06T14:36:13.3235579Z running benchmark:  23% 7/30 [00:00<00:01, 13.00it/s]
2023-09-06T14:36:13.4738379Z running benchmark:  30% 9/30 [00:00<00:01, 13.09it/s]
2023-09-06T14:36:13.6375720Z running benchmark:  37% 11/30 [00:00<00:01, 13.16it/s]
2023-09-06T14:36:13.7877809Z running benchmark:  43% 13/30 [00:01<00:01, 12.83it/s]
2023-09-06T14:36:13.9367193Z running benchmark:  50% 15/30 [00:01<00:01, 12.98it/s]
2023-09-06T14:36:14.0859285Z running benchmark:  57% 17/30 [00:01<00:00, 13.12it/s]
2023-09-06T14:36:14.2357338Z running benchmark:  63% 19/30 [00:01<00:00, 13.21it/s]
2023-09-06T14:36:14.3870158Z running benchmark:  70% 21/30 [00:01<00:00, 13.25it/s]
2023-09-06T14:36:14.5377817Z running benchmark:  77% 23/30 [00:01<00:00, 13.24it/s]
2023-09-06T14:36:14.7010075Z running benchmark:  83% 25/30 [00:01<00:00, 13.25it/s]
2023-09-06T14:36:14.8519406Z running benchmark:  90% 27/30 [00:02<00:00, 12.94it/s]
2023-09-06T14:36:14.9289343Z running benchmark:  97% 29/30 [00:02<00:00, 13.03it/s]
2023-09-06T14:36:14.9289775Z running benchmark: 100% 30/30 [00:02<00:00, 12.96it/s]
2023-09-06T14:36:14.9314730Z 3.480x
2023-09-06T14:36:20.7340202Z 
2023-09-06T14:36:26.5076909Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:36:26.5077279Z loading model: 0it [00:05, ?it/s]
2023-09-06T14:36:26.5077591Z cuda train Background_Matting                 
2023-09-06T14:37:04.2508112Z ERROR:common:Backend dynamo failed in warmup()
2023-09-06T14:37:04.2508503Z Traceback (most recent call last):
2023-09-06T14:37:04.2508910Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2288, in warmup
2023-09-06T14:37:04.2509464Z     fn(model, example_inputs)
2023-09-06T14:37:04.2510925Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T14:37:04.2511298Z     return fn(*args, **kwargs)
2023-09-06T14:37:04.2511711Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 475, in forward_and_backward_pass
2023-09-06T14:37:04.2512121Z     cloned_inputs = clone_inputs(inputs)
2023-09-06T14:37:04.2512602Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 476, in <resume in forward_and_backward_pass>
2023-09-06T14:37:04.2513103Z     self.optimizer_zero_grad(mod)
2023-09-06T14:37:04.2514145Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T14:37:04.2514621Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T14:37:04.2515247Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 634, in _convert_frame
2023-09-06T14:37:04.2515715Z     result = inner_convert(frame, cache_entry, hooks, frame_state)
2023-09-06T14:37:04.2516283Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T14:37:04.2516675Z     return fn(*args, **kwargs)
2023-09-06T14:37:04.2517255Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T14:37:04.2517651Z     return _compile(
2023-09-06T14:37:04.2518178Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T14:37:04.2518628Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T14:37:04.2519234Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T14:37:04.2519615Z     r = func(*args, **kwargs)
2023-09-06T14:37:04.2520160Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 542, in compile_inner
2023-09-06T14:37:04.2520563Z     check_fn = CheckFunctionManager(
2023-09-06T14:37:04.2521112Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 958, in __init__
2023-09-06T14:37:04.2521531Z     guard.create(local_builder, global_builder)
2023-09-06T14:37:04.2522067Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_guards.py", line 243, in create
2023-09-06T14:37:04.2522536Z     return self.create_fn(self.source.select(local_builder, global_builder), self)
2023-09-06T14:37:04.2523185Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 621, in SHAPE_ENV
2023-09-06T14:37:04.2523832Z     guards = output_graph.shape_env.produce_guards(
2023-09-06T14:37:04.2524468Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 3022, in produce_guards
2023-09-06T14:37:04.2524977Z     raise ConstraintViolationError(f"Constraints violated!\n{err}")
2023-09-06T14:37:04.2525462Z torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated!
2023-09-06T14:37:04.2526457Z   1. Not all values of RelaxedUnspecConstraint(L['cloned_inputs'][3].size()[1]) are valid because L['cloned_inputs'][3].size()[1] was inferred to be a constant (4). For more information about why it is constant, run with TORCH_LOGS=dynamic.
2023-09-06T14:37:04.2527024Z 
2023-09-06T14:37:04.2527029Z 
2023-09-06T14:37:04.2527226Z You can suppress this exception and fall back to eager by setting:
2023-09-06T14:37:04.2527566Z     import torch._dynamo
2023-09-06T14:37:04.2527889Z     torch._dynamo.config.suppress_errors = True
2023-09-06T14:37:04.2528112Z 
2023-09-06T14:37:06.6352039Z ERROR
2023-09-06T14:37:10.3497550Z 
2023-09-06T14:37:21.8901024Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:37:21.8901712Z loading model: 0it [00:11, ?it/s]
2023-09-06T14:37:21.8901990Z Eager model failed to run
2023-09-06T14:37:21.8908832Z Traceback (most recent call last):
2023-09-06T14:37:21.8909669Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T14:37:21.8910264Z     self.model_iter_fn(model, example_inputs)
2023-09-06T14:37:21.8910931Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 480, in forward_and_backward_pass
2023-09-06T14:37:21.8911485Z     self.grad_scaler.scale(loss).backward()
2023-09-06T14:37:21.8914742Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward
2023-09-06T14:37:21.8915396Z     torch.autograd.backward(
2023-09-06T14:37:21.8917310Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 251, in backward
2023-09-06T14:37:21.8917878Z     Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2023-09-06T14:37:21.8918360Z RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
2023-09-06T14:37:21.8918687Z 
2023-09-06T14:37:21.8918912Z The above exception was the direct cause of the following exception:
2023-09-06T14:37:21.8919162Z 
2023-09-06T14:37:21.8919297Z Traceback (most recent call last):
2023-09-06T14:37:21.8919679Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T14:37:21.8920026Z     ) = runner.load_model(
2023-09-06T14:37:21.8920391Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T14:37:21.8920803Z     self.validate_model(model, example_inputs)
2023-09-06T14:37:21.8921216Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T14:37:21.8921676Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T14:37:21.8922047Z NotImplementedError: Eager model failed to run
2023-09-06T14:37:21.8922254Z 
2023-09-06T14:37:21.8922391Z WARNING:root:DALLE2_pytorch failed to load
2023-09-06T14:37:26.7701771Z 
2023-09-06T14:37:31.5714828Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:37:31.5715188Z loading model: 0it [00:04, ?it/s]
2023-09-06T14:37:31.5715500Z cuda train LearningToPaint                    
2023-09-06T14:38:17.9686627Z 
2023-09-06T14:38:18.0803275Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:38:18.1832436Z running benchmark:  13% 4/30 [00:00<00:00, 35.74it/s]
2023-09-06T14:38:18.2856535Z running benchmark:  27% 8/30 [00:00<00:00, 37.49it/s]
2023-09-06T14:38:18.3893163Z running benchmark:  40% 12/30 [00:00<00:00, 38.17it/s]
2023-09-06T14:38:18.4931732Z running benchmark:  53% 16/30 [00:00<00:00, 38.36it/s]
2023-09-06T14:38:18.6078615Z running benchmark:  67% 20/30 [00:00<00:00, 38.41it/s]
2023-09-06T14:38:18.7078885Z running benchmark:  80% 24/30 [00:00<00:00, 37.13it/s]
2023-09-06T14:38:18.7579285Z running benchmark:  93% 28/30 [00:00<00:00, 38.01it/s]
2023-09-06T14:38:18.7579669Z running benchmark: 100% 30/30 [00:00<00:00, 37.99it/s]
2023-09-06T14:38:18.7601101Z 1.452x
2023-09-06T14:38:24.4609275Z 
2023-09-06T14:38:27.9022274Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:38:27.9022975Z loading model: 0it [00:03, ?it/s]
2023-09-06T14:38:27.9023284Z cuda train Super_SloMo                        
2023-09-06T14:40:15.8529472Z 
2023-09-06T14:40:16.1562690Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:40:16.2991495Z running benchmark:   3% 1/30 [00:00<00:08,  3.30it/s]
2023-09-06T14:40:16.4419122Z running benchmark:   7% 2/30 [00:00<00:05,  4.79it/s]
2023-09-06T14:40:16.5849659Z running benchmark:  10% 3/30 [00:00<00:04,  5.59it/s]
2023-09-06T14:40:16.7278957Z running benchmark:  13% 4/30 [00:00<00:04,  6.07it/s]
2023-09-06T14:40:16.8709513Z running benchmark:  17% 5/30 [00:00<00:03,  6.38it/s]
2023-09-06T14:40:17.0138234Z running benchmark:  20% 6/30 [00:01<00:03,  6.57it/s]
2023-09-06T14:40:17.1566641Z running benchmark:  23% 7/30 [00:01<00:03,  6.70it/s]
2023-09-06T14:40:17.3087600Z running benchmark:  27% 8/30 [00:01<00:03,  6.80it/s]
2023-09-06T14:40:17.4512048Z running benchmark:  30% 9/30 [00:01<00:03,  6.86it/s]
2023-09-06T14:40:17.5935175Z running benchmark:  33% 10/30 [00:01<00:02,  6.77it/s]
2023-09-06T14:40:17.7360708Z running benchmark:  37% 11/30 [00:01<00:02,  6.85it/s]
2023-09-06T14:40:17.8786851Z running benchmark:  40% 12/30 [00:01<00:02,  6.90it/s]
2023-09-06T14:40:18.0214143Z running benchmark:  43% 13/30 [00:02<00:02,  6.93it/s]
2023-09-06T14:40:18.1641571Z running benchmark:  47% 14/30 [00:02<00:02,  6.95it/s]
2023-09-06T14:40:18.3069224Z running benchmark:  50% 15/30 [00:02<00:02,  6.97it/s]
2023-09-06T14:40:18.4549586Z running benchmark:  53% 16/30 [00:02<00:02,  6.98it/s]
2023-09-06T14:40:18.5973118Z running benchmark:  57% 17/30 [00:02<00:01,  6.91it/s]
2023-09-06T14:40:18.7400766Z running benchmark:  60% 18/30 [00:02<00:01,  6.94it/s]
2023-09-06T14:40:18.8825385Z running benchmark:  63% 19/30 [00:02<00:01,  6.96it/s]
2023-09-06T14:40:19.0253374Z running benchmark:  67% 20/30 [00:03<00:01,  6.98it/s]
2023-09-06T14:40:19.1678446Z running benchmark:  70% 21/30 [00:03<00:01,  6.99it/s]
2023-09-06T14:40:19.3101523Z running benchmark:  73% 22/30 [00:03<00:01,  6.99it/s]
2023-09-06T14:40:19.4527814Z running benchmark:  77% 23/30 [00:03<00:00,  7.00it/s]
2023-09-06T14:40:19.5951397Z running benchmark:  80% 24/30 [00:03<00:00,  7.01it/s]
2023-09-06T14:40:19.7377185Z running benchmark:  83% 25/30 [00:03<00:00,  7.01it/s]
2023-09-06T14:40:19.8801332Z running benchmark:  87% 26/30 [00:03<00:00,  7.01it/s]
2023-09-06T14:40:20.0227969Z running benchmark:  90% 27/30 [00:04<00:00,  7.01it/s]
2023-09-06T14:40:20.1654472Z running benchmark:  93% 28/30 [00:04<00:00,  7.01it/s]
2023-09-06T14:40:20.3082236Z running benchmark:  97% 29/30 [00:04<00:00,  7.01it/s]
2023-09-06T14:40:20.3082810Z running benchmark: 100% 30/30 [00:04<00:00,  7.01it/s]
2023-09-06T14:40:20.3083133Z running benchmark: 100% 30/30 [00:04<00:00,  6.73it/s]
2023-09-06T14:40:20.3106700Z 1.283x
2023-09-06T14:40:26.1622011Z 
2023-09-06T14:40:28.4835308Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:40:28.4835725Z loading model: 0it [00:02, ?it/s]
2023-09-06T14:40:28.4836036Z cuda train alexnet                            
2023-09-06T14:40:55.7314230Z 
2023-09-06T14:40:55.8314807Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:40:55.9506399Z running benchmark:  13% 4/30 [00:00<00:00, 39.72it/s]
2023-09-06T14:40:56.0693237Z running benchmark:  33% 10/30 [00:00<00:00, 46.38it/s]
2023-09-06T14:40:56.1695198Z running benchmark:  53% 16/30 [00:00<00:00, 48.33it/s]
2023-09-06T14:40:56.2822538Z running benchmark:  70% 21/30 [00:00<00:00, 48.90it/s]
2023-09-06T14:40:56.3610566Z running benchmark:  87% 26/30 [00:00<00:00, 47.26it/s]
2023-09-06T14:40:56.3610948Z running benchmark: 100% 30/30 [00:00<00:00, 47.58it/s]
2023-09-06T14:40:56.3632711Z 1.126x
2023-09-06T14:41:01.6802357Z 
2023-09-06T14:41:09.4253650Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:41:09.4254335Z loading model: 0it [00:07, ?it/s]
2023-09-06T14:41:09.4254842Z cuda train attention_is_all_you_need_pytorch  
2023-09-06T14:43:01.8381357Z 
2023-09-06T14:43:02.0316334Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:43:02.2336100Z running benchmark:   3% 1/30 [00:00<00:05,  5.16it/s]
2023-09-06T14:43:02.4254544Z running benchmark:  10% 3/30 [00:00<00:03,  7.99it/s]
2023-09-06T14:43:02.6148664Z running benchmark:  17% 5/30 [00:00<00:02,  9.08it/s]
2023-09-06T14:43:02.8027562Z running benchmark:  23% 7/30 [00:00<00:02,  9.65it/s]
2023-09-06T14:43:02.9888708Z running benchmark:  30% 9/30 [00:00<00:02, 10.00it/s]
2023-09-06T14:43:03.1740651Z running benchmark:  37% 11/30 [00:01<00:01, 10.25it/s]
2023-09-06T14:43:03.3814856Z running benchmark:  43% 13/30 [00:01<00:01, 10.43it/s]
2023-09-06T14:43:03.5641078Z running benchmark:  50% 15/30 [00:01<00:01, 10.16it/s]
2023-09-06T14:43:03.7481291Z running benchmark:  57% 17/30 [00:01<00:01, 10.40it/s]
2023-09-06T14:43:03.9368976Z running benchmark:  63% 19/30 [00:01<00:01, 10.54it/s]
2023-09-06T14:43:04.1307395Z running benchmark:  70% 21/30 [00:02<00:00, 10.56it/s]
2023-09-06T14:43:04.3206034Z running benchmark:  77% 23/30 [00:02<00:00, 10.48it/s]
2023-09-06T14:43:04.5129690Z running benchmark:  83% 25/30 [00:02<00:00, 10.50it/s]
2023-09-06T14:43:04.7026591Z running benchmark:  90% 27/30 [00:02<00:00, 10.47it/s]
2023-09-06T14:43:04.7977760Z running benchmark:  97% 29/30 [00:02<00:00, 10.49it/s]
2023-09-06T14:43:04.7978210Z running benchmark: 100% 30/30 [00:02<00:00, 10.13it/s]
2023-09-06T14:43:04.8004753Z 1.722x
2023-09-06T14:43:11.0031560Z 
2023-09-06T14:43:13.7544598Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:43:13.7544960Z loading model: 0it [00:02, ?it/s]
2023-09-06T14:43:13.7547959Z cuda train basic_gnn_edgecnn                  
2023-09-06T14:43:14.5007671Z [2023-09-06 14:43:14,499] [3/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T14:43:33.4432724Z 
2023-09-06T14:43:33.5462697Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:43:33.6600336Z running benchmark:  23% 7/30 [00:00<00:00, 67.75it/s]
2023-09-06T14:43:33.7708660Z running benchmark:  50% 15/30 [00:00<00:00, 69.23it/s]
2023-09-06T14:43:33.8686669Z running benchmark:  77% 23/30 [00:00<00:00, 70.61it/s]
2023-09-06T14:43:33.8691356Z running benchmark: 100% 30/30 [00:00<00:00, 70.43it/s]
2023-09-06T14:43:33.8706663Z 1.395x
2023-09-06T14:43:38.8864628Z 
2023-09-06T14:43:41.7832982Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:43:41.7833568Z loading model: 0it [00:02, ?it/s]
2023-09-06T14:43:41.7833875Z cuda train basic_gnn_gcn                      
2023-09-06T14:43:52.4709685Z [2023-09-06 14:43:52,469] [13/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T14:44:02.1708458Z 
2023-09-06T14:44:02.2812981Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:44:02.3926821Z running benchmark:  33% 10/30 [00:00<00:00, 90.71it/s]
2023-09-06T14:44:02.5042493Z running benchmark:  67% 20/30 [00:00<00:00, 89.98it/s]
2023-09-06T14:44:02.5043070Z running benchmark: 100% 30/30 [00:00<00:00, 89.86it/s]
2023-09-06T14:44:02.5043634Z running benchmark: 100% 30/30 [00:00<00:00, 89.90it/s]
2023-09-06T14:44:02.5081546Z 1.035x
2023-09-06T14:44:07.5328979Z 
2023-09-06T14:44:10.4966792Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:44:10.4967319Z loading model: 0it [00:02, ?it/s]
2023-09-06T14:44:10.4967606Z cuda train basic_gnn_gin                      
2023-09-06T14:44:11.2503573Z [2023-09-06 14:44:11,249] [3/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T14:44:22.8954097Z 
2023-09-06T14:44:22.9979967Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:44:23.1083617Z running benchmark:  50% 15/30 [00:00<00:00, 146.10it/s]
2023-09-06T14:44:23.1084199Z running benchmark: 100% 30/30 [00:00<00:00, 139.85it/s]
2023-09-06T14:44:23.1084526Z running benchmark: 100% 30/30 [00:00<00:00, 140.62it/s]
2023-09-06T14:44:23.1101306Z 2.001x
2023-09-06T14:44:28.1001483Z 
2023-09-06T14:44:31.0364589Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:44:31.0365000Z loading model: 0it [00:02, ?it/s]
2023-09-06T14:44:31.0365303Z cuda train basic_gnn_sage                     
2023-09-06T14:44:31.8080951Z [2023-09-06 14:44:31,806] [3/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T14:44:43.3504494Z 
2023-09-06T14:44:43.4506755Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:44:43.5675555Z running benchmark:  43% 13/30 [00:00<00:00, 129.82it/s]
2023-09-06T14:44:43.5899531Z running benchmark:  90% 27/30 [00:00<00:00, 123.60it/s]
2023-09-06T14:44:43.5900206Z running benchmark: 100% 30/30 [00:00<00:00, 125.16it/s]
2023-09-06T14:44:43.5925299Z 1.794x
2023-09-06T14:44:48.5674548Z 
2023-09-06T14:44:49.0958338Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:44:49.0959114Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:44:49.0959420Z Test train is not implemented.
2023-09-06T14:44:49.0964058Z Traceback (most recent call last):
2023-09-06T14:44:49.0964477Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T14:44:49.0965364Z     ) = runner.load_model(
2023-09-06T14:44:49.0965757Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 368, in load_model
2023-09-06T14:44:49.0969691Z     benchmark = benchmark_cls(
2023-09-06T14:44:49.0970130Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 25, in __call__
2023-09-06T14:44:49.0970525Z     obj = type.__call__(cls, *args, **kwargs)
2023-09-06T14:44:49.0970977Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate/__init__.py", line 12, in __init__
2023-09-06T14:44:49.0971485Z     super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args)
2023-09-06T14:44:49.0971975Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 117, in __init__
2023-09-06T14:44:49.0972380Z     self.determine_batch_size(batch_size)
2023-09-06T14:44:49.0972813Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 175, in determine_batch_size
2023-09-06T14:44:49.0973319Z     raise NotImplementedError(f"Test {self.test} is not implemented.")
2023-09-06T14:44:49.0973740Z NotImplementedError: Test train is not implemented.
2023-09-06T14:44:49.0974021Z 
2023-09-06T14:44:49.0974167Z WARNING:root:cm3leon_generate failed to load
2023-09-06T14:44:53.5604344Z 
2023-09-06T14:44:54.6141567Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:44:54.6142008Z loading model: 0it [00:01, ?it/s]
2023-09-06T14:44:54.6142307Z cuda train dcgan                              
2023-09-06T14:45:11.4653649Z 
2023-09-06T14:45:11.5660396Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:45:11.6464343Z running benchmark:  53% 16/30 [00:00<00:00, 158.43it/s]
2023-09-06T14:45:11.6464739Z running benchmark: 100% 30/30 [00:00<00:00, 165.18it/s]
2023-09-06T14:45:11.6485236Z 1.406x
2023-09-06T14:45:16.8535792Z 
2023-09-06T14:45:21.4179491Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:45:21.4180377Z loading model: 0it [00:04, ?it/s]
2023-09-06T14:45:21.4181980Z cuda train demucs                             
2023-09-06T14:45:31.0118104Z ERROR:common:Backend dynamo failed in warmup()
2023-09-06T14:45:31.0118483Z Traceback (most recent call last):
2023-09-06T14:45:31.0118912Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2288, in warmup
2023-09-06T14:45:31.0119289Z     fn(model, example_inputs)
2023-09-06T14:45:31.0120607Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T14:45:31.0124174Z     return fn(*args, **kwargs)
2023-09-06T14:45:31.0124618Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 475, in forward_and_backward_pass
2023-09-06T14:45:31.0125058Z     cloned_inputs = clone_inputs(inputs)
2023-09-06T14:45:31.0125507Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 476, in <resume in forward_and_backward_pass>
2023-09-06T14:45:31.0125922Z     self.optimizer_zero_grad(mod)
2023-09-06T14:45:31.0126388Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 478, in <resume in forward_and_backward_pass>
2023-09-06T14:45:31.0126801Z     pred = mod(*cloned_inputs)
2023-09-06T14:45:31.0128767Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T14:45:31.0129373Z     return self._call_impl(*args, **kwargs)
2023-09-06T14:45:31.0130060Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T14:45:31.0130709Z     return forward_call(*args, **kwargs)
2023-09-06T14:45:31.0131513Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T14:45:31.0132422Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T14:45:31.0133236Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 634, in _convert_frame
2023-09-06T14:45:31.0134163Z     result = inner_convert(frame, cache_entry, hooks, frame_state)
2023-09-06T14:45:31.0134780Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T14:45:31.0135198Z     return fn(*args, **kwargs)
2023-09-06T14:45:31.0135791Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T14:45:31.0136181Z     return _compile(
2023-09-06T14:45:31.0136741Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T14:45:31.0163344Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T14:45:31.0164503Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T14:45:31.0165078Z     r = func(*args, **kwargs)
2023-09-06T14:45:31.0166182Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 542, in compile_inner
2023-09-06T14:45:31.0166903Z     check_fn = CheckFunctionManager(
2023-09-06T14:45:31.0167850Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 958, in __init__
2023-09-06T14:45:31.0168288Z     guard.create(local_builder, global_builder)
2023-09-06T14:45:31.0168841Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_guards.py", line 243, in create
2023-09-06T14:45:31.0169302Z     return self.create_fn(self.source.select(local_builder, global_builder), self)
2023-09-06T14:45:31.0169919Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 621, in SHAPE_ENV
2023-09-06T14:45:31.0170355Z     guards = output_graph.shape_env.produce_guards(
2023-09-06T14:45:31.0170992Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 3022, in produce_guards
2023-09-06T14:45:31.0171511Z     raise ConstraintViolationError(f"Constraints violated!\n{err}")
2023-09-06T14:45:31.0172460Z torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated!
2023-09-06T14:45:31.0173429Z   1. Not all values of RelaxedUnspecConstraint(L['streams'].size()[0]) are valid because L['streams'].size()[0] was inferred to be a constant (4). For more information about why it is constant, run with TORCH_LOGS=dynamic.
2023-09-06T14:45:31.0173882Z 
2023-09-06T14:45:31.0173888Z 
2023-09-06T14:45:31.0174086Z You can suppress this exception and fall back to eager by setting:
2023-09-06T14:45:31.0174450Z     import torch._dynamo
2023-09-06T14:45:31.0174764Z     torch._dynamo.config.suppress_errors = True
2023-09-06T14:45:31.0174956Z 
2023-09-06T14:45:32.3415771Z ERROR
2023-09-06T14:45:36.0255240Z 
2023-09-06T14:45:40.4595848Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:45:40.4596198Z loading model: 0it [00:04, ?it/s]
2023-09-06T14:45:40.4596502Z cuda train densenet121                        
2023-09-06T14:49:42.0941736Z 
2023-09-06T14:49:42.2147373Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T14:49:42.3219646Z running benchmark:   3% 1/30 [00:00<00:03,  8.28it/s]
2023-09-06T14:49:42.4357996Z running benchmark:   7% 2/30 [00:00<00:03,  8.86it/s]
2023-09-06T14:49:42.5569576Z running benchmark:  10% 3/30 [00:00<00:03,  8.82it/s]
2023-09-06T14:49:42.6661777Z running benchmark:  13% 4/30 [00:00<00:03,  8.59it/s]
2023-09-06T14:49:42.7707750Z running benchmark:  17% 5/30 [00:00<00:02,  8.78it/s]
2023-09-06T14:49:42.8778321Z running benchmark:  20% 6/30 [00:00<00:02,  9.03it/s]
2023-09-06T14:49:42.9865780Z running benchmark:  23% 7/30 [00:00<00:02,  9.13it/s]
2023-09-06T14:49:43.0895861Z running benchmark:  27% 8/30 [00:00<00:02,  9.15it/s]
2023-09-06T14:49:43.1946666Z running benchmark:  30% 9/30 [00:00<00:02,  9.32it/s]
2023-09-06T14:49:43.3019941Z running benchmark:  33% 10/30 [00:01<00:02,  9.38it/s]
2023-09-06T14:49:43.4089370Z running benchmark:  37% 11/30 [00:01<00:02,  9.36it/s]
2023-09-06T14:49:43.5173708Z running benchmark:  40% 12/30 [00:01<00:01,  9.36it/s]
2023-09-06T14:49:43.6323928Z running benchmark:  43% 13/30 [00:01<00:01,  9.32it/s]
2023-09-06T14:49:43.7400065Z running benchmark:  47% 14/30 [00:01<00:01,  9.12it/s]
2023-09-06T14:49:43.8479229Z running benchmark:  50% 15/30 [00:01<00:01,  9.17it/s]
2023-09-06T14:49:43.9532426Z running benchmark:  53% 16/30 [00:01<00:01,  9.20it/s]
2023-09-06T14:49:44.0586252Z running benchmark:  57% 17/30 [00:01<00:01,  9.28it/s]
2023-09-06T14:49:44.1672534Z running benchmark:  60% 18/30 [00:01<00:01,  9.35it/s]
2023-09-06T14:49:44.2727055Z running benchmark:  63% 19/30 [00:02<00:01,  9.31it/s]
2023-09-06T14:49:44.3799809Z running benchmark:  67% 20/30 [00:02<00:01,  9.36it/s]
2023-09-06T14:49:44.4900584Z running benchmark:  70% 21/30 [00:02<00:00,  9.35it/s]
2023-09-06T14:49:44.6007689Z running benchmark:  73% 22/30 [00:02<00:00,  9.26it/s]
2023-09-06T14:49:44.7057022Z running benchmark:  77% 23/30 [00:02<00:00,  9.20it/s]
2023-09-06T14:49:44.8138449Z running benchmark:  80% 24/30 [00:02<00:00,  9.29it/s]
2023-09-06T14:49:44.9220351Z running benchmark:  83% 25/30 [00:02<00:00,  9.28it/s]
2023-09-06T14:49:45.0291830Z running benchmark:  87% 26/30 [00:02<00:00,  9.27it/s]
2023-09-06T14:49:45.1371060Z running benchmark:  90% 27/30 [00:02<00:00,  9.29it/s]
2023-09-06T14:49:45.2453870Z running benchmark:  93% 28/30 [00:03<00:00,  9.28it/s]
2023-09-06T14:49:45.3534366Z running benchmark:  97% 29/30 [00:03<00:00,  9.26it/s]
2023-09-06T14:49:45.3535036Z running benchmark: 100% 30/30 [00:03<00:00,  9.26it/s]
2023-09-06T14:49:45.3537949Z running benchmark: 100% 30/30 [00:03<00:00,  9.20it/s]
2023-09-06T14:49:45.3562361Z 2.309x
2023-09-06T14:49:52.6261204Z 
2023-09-06T14:49:54.1671241Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:49:54.1671604Z loading model: 0it [00:01, ?it/s]
2023-09-06T14:49:54.1672291Z FCOS train is not supported by upstream detectron2. See GH Issue: https://github.com/facebookresearch/detectron2/issues/4369.
2023-09-06T14:49:54.1676777Z Traceback (most recent call last):
2023-09-06T14:49:54.1678127Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T14:49:54.1678662Z     ) = runner.load_model(
2023-09-06T14:49:54.1681179Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 368, in load_model
2023-09-06T14:49:54.1682086Z     benchmark = benchmark_cls(
2023-09-06T14:49:54.1682901Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 25, in __call__
2023-09-06T14:49:54.1683667Z     obj = type.__call__(cls, *args, **kwargs)
2023-09-06T14:49:54.1684205Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn/__init__.py", line 15, in __init__
2023-09-06T14:49:54.1685360Z     super().__init__(variant="COCO-Detection/fcos_R_50_FPN_1x.py", test=test, device=device,
2023-09-06T14:49:54.1685929Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/detectron2/model_factory.py", line 105, in __init__
2023-09-06T14:49:54.1686507Z     raise NotImplementedError("FCOS train is not supported by upstream detectron2. " \
2023-09-06T14:49:54.1687106Z NotImplementedError: FCOS train is not supported by upstream detectron2. See GH Issue: https://github.com/facebookresearch/detectron2/issues/4369.
2023-09-06T14:49:54.1687477Z 
2023-09-06T14:49:54.1687697Z WARNING:root:detectron2_fcos_r_50_fpn failed to load
2023-09-06T14:49:55.1582974Z speedup             gmean=0.00x mean=1.357x
2023-09-06T14:49:55.1583659Z abs_latency         gmean=0.00x mean=13.029x
2023-09-06T14:49:55.1584187Z compilation_latency mean=50.721 seconds
2023-09-06T14:49:55.1584682Z compression_ratio   mean=0.556x
2023-09-06T14:49:55.1584975Z eager_peak_mem      gmean=0.00x mean=0.954x
2023-09-06T14:49:55.1586644Z dynamo_peak_mem     gmean=0.00x mean=1.028x
2023-09-06T14:49:55.1588427Z calls_captured      gmean=0.00x mean=163.786x
2023-09-06T14:49:55.1590807Z unique_graphs       gmean=0.00x mean=2.429x
2023-09-06T14:49:55.1592125Z graph_breaks        gmean=0.00x mean=8.571x
2023-09-06T14:49:55.1594936Z unique_graph_breaks gmean=0.00x mean=5.643x
2023-09-06T14:49:55.9116713Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *cppwrapper-true* ]]
2023-09-06T14:49:55.9118493Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *freezing_cudagraphs-true* ]]
2023-09-06T14:49:55.9119579Z + [[ training == \i\n\f\e\r\e\n\c\e ]]
2023-09-06T14:49:55.9120851Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *aotinductor-true* ]]
2023-09-06T14:49:55.9121879Z + [[ training == \i\n\f\e\r\e\n\c\e ]]
2023-09-06T14:49:55.9122748Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *maxautotune-true* ]]
2023-09-06T14:49:55.9123508Z + for mode in "${modes[@]}"
2023-09-06T14:49:55.9123794Z + [[ inference == \i\n\f\e\r\e\n\c\e ]]
2023-09-06T14:49:55.9124291Z + dtype=bfloat16
2023-09-06T14:49:55.9124554Z + for target in "${targets[@]}"
2023-09-06T14:49:55.9124889Z + target_flag=("--${target}")
2023-09-06T14:49:55.9125163Z + local target_flag
2023-09-06T14:49:55.9125430Z + [[ accuracy == \p\e\r\f\o\r\m\a\n\c\e ]]
2023-09-06T14:49:55.9125726Z + [[ accuracy == \a\c\c\u\r\a\c\y ]]
2023-09-06T14:49:55.9126122Z + target_flag+=(--no-translation-validation)
2023-09-06T14:49:55.9126991Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *default-true* ]]
2023-09-06T14:49:55.9128443Z + python benchmarks/dynamo/torchbench.py --accuracy --no-translation-validation --inference --bfloat16 --backend inductor --disable-cudagraphs --device cuda --total-partitions 4 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inductor_no_cudagraphs_torchbench_bfloat16_inference_cuda_accuracy.csv
2023-09-06T14:50:03.1767356Z 
2023-09-06T14:50:04.0803037Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:50:04.0804184Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:50:04.0804452Z Eager model failed to run
2023-09-06T14:50:04.0825447Z Traceback (most recent call last):
2023-09-06T14:50:04.0826255Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T14:50:04.0826974Z     self.model_iter_fn(model, example_inputs)
2023-09-06T14:50:04.0827768Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T14:50:04.0828460Z     return mod(*inputs)
2023-09-06T14:50:04.0831315Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T14:50:04.0832156Z     return self._call_impl(*args, **kwargs)
2023-09-06T14:50:04.0833222Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T14:50:04.0833899Z     return forward_call(*args, **kwargs)
2023-09-06T14:50:04.0834967Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 893, in forward
2023-09-06T14:50:04.0835818Z     logits = self.model(batch.dense_features, batch.sparse_features)
2023-09-06T14:50:04.0836935Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T14:50:04.0837783Z     return self._call_impl(*args, **kwargs)
2023-09-06T14:50:04.0838894Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T14:50:04.0839403Z     return forward_call(*args, **kwargs)
2023-09-06T14:50:04.0839951Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 570, in forward
2023-09-06T14:50:04.0840389Z     embedded_dense = self.dense_arch(dense_features)
2023-09-06T14:50:04.0840966Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T14:50:04.0841770Z     return self._call_impl(*args, **kwargs)
2023-09-06T14:50:04.0842358Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T14:50:04.0842784Z     return forward_call(*args, **kwargs)
2023-09-06T14:50:04.0843333Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 149, in forward
2023-09-06T14:50:04.0843707Z     return self.model(features)
2023-09-06T14:50:04.0844262Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T14:50:04.0844692Z     return self._call_impl(*args, **kwargs)
2023-09-06T14:50:04.0845292Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T14:50:04.0845679Z     return forward_call(*args, **kwargs)
2023-09-06T14:50:04.0846227Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/mlp.py", line 172, in forward
2023-09-06T14:50:04.0846628Z     return self._mlp(input)
2023-09-06T14:50:04.0847217Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T14:50:04.0847702Z     return self._call_impl(*args, **kwargs)
2023-09-06T14:50:04.0848238Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T14:50:04.0848640Z     return forward_call(*args, **kwargs)
2023-09-06T14:50:04.0849194Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/container.py", line 215, in forward
2023-09-06T14:50:04.0849579Z     input = module(input)
2023-09-06T14:50:04.0850112Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T14:50:04.0850526Z     return self._call_impl(*args, **kwargs)
2023-09-06T14:50:04.0851089Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T14:50:04.0851681Z     return forward_call(*args, **kwargs)
2023-09-06T14:50:04.0852227Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/mlp.py", line 73, in forward
2023-09-06T14:50:04.0852639Z     return self._activation_fn(self._linear(input))
2023-09-06T14:50:04.0853282Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T14:50:04.0853707Z     return self._call_impl(*args, **kwargs)
2023-09-06T14:50:04.0854260Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T14:50:04.0854646Z     return forward_call(*args, **kwargs)
2023-09-06T14:50:04.0855186Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
2023-09-06T14:50:04.0855628Z     return F.linear(input, self.weight, self.bias)
2023-09-06T14:50:04.0856054Z RuntimeError: mat1 and mat2 must have the same dtype, but got Float and BFloat16
2023-09-06T14:50:04.0856316Z 
2023-09-06T14:50:04.0856512Z The above exception was the direct cause of the following exception:
2023-09-06T14:50:04.0856753Z 
2023-09-06T14:50:04.0856866Z Traceback (most recent call last):
2023-09-06T14:50:04.0857245Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T14:50:04.0857588Z     ) = runner.load_model(
2023-09-06T14:50:04.0858010Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T14:50:04.0858394Z     self.validate_model(model, example_inputs)
2023-09-06T14:50:04.0858802Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T14:50:04.0859248Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T14:50:04.0859631Z NotImplementedError: Eager model failed to run
2023-09-06T14:50:04.0859847Z 
2023-09-06T14:50:04.0860112Z WARNING:root:torchrec_dlrm failed to load
2023-09-06T14:50:08.5471936Z 
2023-09-06T14:50:11.7581329Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:50:11.7581858Z loading model: 0it [00:03, ?it/s]
2023-09-06T14:50:11.7643138Z cuda eval  BERT_pytorch                       
2023-09-06T14:50:37.7479150Z pass
2023-09-06T14:50:42.9359390Z 
2023-09-06T14:50:46.6393016Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:50:46.6393542Z loading model: 0it [00:03, ?it/s]
2023-09-06T14:50:46.6444828Z cuda eval  Background_Matting                 
2023-09-06T14:50:46.6449602Z pass_due_to_skip
2023-09-06T14:50:51.3473226Z 
2023-09-06T14:51:00.6739386Z loading model: 0it [00:00, ?it/s]WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T14:51:02.4131416Z 
2023-09-06T14:51:02.4132123Z loading model: 0it [00:11, ?it/s]
2023-09-06T14:51:02.4132854Z WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T14:51:02.4433365Z cuda eval  DALLE2_pytorch                     
2023-09-06T14:51:02.6816958Z WARNING:common:fp64 golden ref were not generated for DALLE2_pytorch. Setting accuracy check to cosine
2023-09-06T14:51:03.0544785Z WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T14:51:16.5067721Z [2023-09-06 14:51:16,505] [3/1] torch._dynamo.output_graph: [WARNING] nn.Module forward/_pre hooks are only partially supported, and were detected in your model. In particular, if you do not change/remove hooks after calling .compile(), you can disregard this warning, and otherwise you may need to set torch._dynamo.config.skip_nnmodule_hook_guards=False to ensure recompiling after changing hooks.See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations. 
2023-09-06T14:51:31.7522086Z [2023-09-06 14:51:31,750] [5/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T14:51:32.0701736Z [2023-09-06 14:51:32,069] [5/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T14:51:32.1786460Z [2023-09-06 14:51:32,177] [5/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T14:51:32.2865448Z [2023-09-06 14:51:32,285] [5/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T14:51:32.3963442Z [2023-09-06 14:51:32,395] [5/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T14:51:32.5048656Z [2023-09-06 14:51:32,504] [5/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T14:51:32.6152915Z [2023-09-06 14:51:32,614] [5/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T14:51:32.7253325Z [2023-09-06 14:51:32,724] [5/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T14:51:32.8324575Z [2023-09-06 14:51:32,831] [5/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T14:51:32.9409005Z [2023-09-06 14:51:32,939] [5/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T14:51:33.0469291Z [2023-09-06 14:51:33,046] [5/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T14:51:33.1594307Z [2023-09-06 14:51:33,158] [5/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T14:52:26.6806314Z [2023-09-06 14:52:26,678] [29/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T14:52:27.0143211Z [2023-09-06 14:52:27,013] [29/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T14:52:27.1360221Z [2023-09-06 14:52:27,135] [29/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T14:52:27.2524481Z [2023-09-06 14:52:27,251] [29/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T14:52:27.3686505Z [2023-09-06 14:52:27,367] [29/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T14:52:27.4983984Z [2023-09-06 14:52:27,497] [29/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T14:52:27.6190639Z [2023-09-06 14:52:27,618] [29/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T14:52:27.7373820Z [2023-09-06 14:52:27,736] [29/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T14:52:27.8523842Z [2023-09-06 14:52:27,851] [29/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T14:52:27.9690239Z [2023-09-06 14:52:27,968] [29/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T14:52:28.0849777Z [2023-09-06 14:52:28,084] [29/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T14:52:28.2003965Z [2023-09-06 14:52:28,199] [29/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T14:52:50.7626581Z ERROR:common:Failed running call_function <function rearrange at 0x7f4e8f039fc0>(*(FakeTensor(..., device='cuda:0', size=(4, 128, s0, s1)), 'b c ... -> b ... c'), **{}):
2023-09-06T14:52:50.7627665Z unhashable type: 'SymInt'
2023-09-06T14:52:50.7627903Z 
2023-09-06T14:52:50.7628032Z from user code:
2023-09-06T14:52:50.7628908Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 1670, in forward
2023-09-06T14:52:50.7629911Z     h = rearrange(h, 'b c ... -> b ... c')
2023-09-06T14:52:50.7630296Z 
2023-09-06T14:52:50.7630605Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T14:52:50.7630939Z 
2023-09-06T14:52:50.7630946Z 
2023-09-06T14:52:50.7631224Z You can suppress this exception and fall back to eager by setting:
2023-09-06T14:52:50.7631772Z     import torch._dynamo
2023-09-06T14:52:50.7632344Z     torch._dynamo.config.suppress_errors = True
2023-09-06T14:52:50.7632830Z Traceback (most recent call last):
2023-09-06T14:52:50.7633485Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2135, in check_accuracy
2023-09-06T14:52:50.7634270Z     new_result = optimized_model_iter_fn(model_copy, example_inputs)
2023-09-06T14:52:50.7635231Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T14:52:50.7636339Z     return fn(*args, **kwargs)
2023-09-06T14:52:50.7636914Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1899, in run_n_iterations
2023-09-06T14:52:50.7637559Z     self.model_iter_fn(mod, inputs, collect_outputs=False)
2023-09-06T14:52:50.7638240Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T14:52:50.7638841Z     return mod(*inputs)
2023-09-06T14:52:50.7639823Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T14:52:50.7640611Z     return self._call_impl(*args, **kwargs)
2023-09-06T14:52:50.7641542Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T14:52:50.7642153Z     return forward_call(*args, **kwargs)
2023-09-06T14:52:50.7643005Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T14:52:50.7643711Z     return func(*args, **kwargs)
2023-09-06T14:52:50.7644492Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 95, in inner
2023-09-06T14:52:50.7645035Z     model.eval()
2023-09-06T14:52:50.7645823Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 96, in <resume in inner>
2023-09-06T14:52:50.7646426Z     out = fn(model, *args, **kwargs)
2023-09-06T14:52:50.7647236Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3329, in forward
2023-09-06T14:52:50.7648173Z     image_embed = self.prior.sample(text, num_samples_per_batch = self.prior_num_samples, cond_scale = prior_cond_scale)
2023-09-06T14:52:50.7649250Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3332, in <resume in forward>
2023-09-06T14:52:50.7650368Z     images = self.decoder.sample(image_embed = image_embed, text = text_cond, cond_scale = cond_scale)
2023-09-06T14:52:50.7651376Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T14:52:50.7651961Z     return func(*args, **kwargs)
2023-09-06T14:52:50.7652757Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 95, in inner
2023-09-06T14:52:50.7653366Z     model.eval()
2023-09-06T14:52:50.7654123Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 96, in <resume in inner>
2023-09-06T14:52:50.7654846Z     out = fn(model, *args, **kwargs)
2023-09-06T14:52:50.7655628Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3199, in sample
2023-09-06T14:52:50.7656168Z     img = self.p_sample_loop(
2023-09-06T14:52:50.7656952Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T14:52:50.7657568Z     return func(*args, **kwargs)
2023-09-06T14:52:50.7658374Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3028, in p_sample_loop
2023-09-06T14:52:50.7659063Z     return self.p_sample_loop_ddpm(*args, noise_scheduler = noise_scheduler, **kwargs)
2023-09-06T14:52:50.7659993Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T14:52:50.7660573Z     return func(*args, **kwargs)
2023-09-06T14:52:50.7661393Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2885, in p_sample_loop_ddpm
2023-09-06T14:52:50.7661995Z     img, x_start = self.p_sample(
2023-09-06T14:52:50.7663101Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T14:52:50.7663752Z     return func(*args, **kwargs)
2023-09-06T14:52:50.7664861Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2822, in p_sample
2023-09-06T14:52:50.7666294Z     model_mean, _, model_log_variance, x_start = self.p_mean_variance(unet, x = x, t = t, image_embed = image_embed, text_encodings = text_encodings, cond_scale = cond_scale, lowres_cond_img = lowres_cond_img, self_cond = self_cond, clip_denoised = clip_denoised, predict_x_start = predict_x_start, predict_v = predict_v, noise_scheduler = noise_scheduler, learned_variance = learned_variance, lowres_noise_level = lowres_noise_level)
2023-09-06T14:52:50.7667967Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2787, in p_mean_variance
2023-09-06T14:52:50.7670681Z     model_output = default(model_output, lambda: unet.forward_with_cond_scale(x, t, image_embed = image_embed, text_encodings = text_encodings, cond_scale = cond_scale, lowres_cond_img = lowres_cond_img, self_cond = self_cond, lowres_noise_level = lowres_noise_level))
2023-09-06T14:52:50.7672175Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 65, in default
2023-09-06T14:52:50.7672750Z     return d() if callable(d) else d
2023-09-06T14:52:50.7673653Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2787, in <lambda>
2023-09-06T14:52:50.7674711Z     model_output = default(model_output, lambda: unet.forward_with_cond_scale(x, t, image_embed = image_embed, text_encodings = text_encodings, cond_scale = cond_scale, lowres_cond_img = lowres_cond_img, self_cond = self_cond, lowres_noise_level = lowres_noise_level))
2023-09-06T14:52:50.7675995Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2159, in forward_with_cond_scale
2023-09-06T14:52:50.7676603Z     logits = self.forward(*args, **kwargs)
2023-09-06T14:52:50.7677695Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2340, in forward
2023-09-06T14:52:50.7678272Z     x = resnet_block(x, t, c)
2023-09-06T14:52:50.7679092Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T14:52:50.7679716Z     return self._call_impl(*args, **kwargs)
2023-09-06T14:52:50.7680559Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T14:52:50.7681143Z     return forward_call(*args, **kwargs)
2023-09-06T14:52:50.7681950Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T14:52:50.7682580Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T14:52:50.7683517Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 634, in _convert_frame
2023-09-06T14:52:50.7684187Z     result = inner_convert(frame, cache_entry, hooks, frame_state)
2023-09-06T14:52:50.7685104Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T14:52:50.7685661Z     return fn(*args, **kwargs)
2023-09-06T14:52:50.7686554Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T14:52:50.7687167Z     return _compile(
2023-09-06T14:52:50.7687975Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T14:52:50.7688686Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T14:52:50.7689501Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T14:52:50.7690066Z     r = func(*args, **kwargs)
2023-09-06T14:52:50.7690919Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T14:52:50.7691791Z     out_code = transform_code_object(code, transform)
2023-09-06T14:52:50.7692688Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T14:52:50.7693406Z     transformations(instructions, code_options)
2023-09-06T14:52:50.7694212Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T14:52:50.7694721Z     tracer.run()
2023-09-06T14:52:50.7695482Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T14:52:50.7696001Z     super().run()
2023-09-06T14:52:50.7696742Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T14:52:50.7697274Z     and self.step()
2023-09-06T14:52:50.7698016Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T14:52:50.7698603Z     getattr(self, inst.opname)(inst)
2023-09-06T14:52:50.7699440Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T14:52:50.7700043Z     return inner_fn(self, inst)
2023-09-06T14:52:50.7700943Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1119, in CALL_FUNCTION
2023-09-06T14:52:50.7701590Z     self.call_function(fn, args, {})
2023-09-06T14:52:50.7702549Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 565, in call_function
2023-09-06T14:52:50.7703312Z     self.push(fn.call_function(self, args, kwargs))
2023-09-06T14:52:50.7704253Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/torch.py", line 731, in call_function
2023-09-06T14:52:50.7704815Z     tensor_variable = wrap_fx_proxy(
2023-09-06T14:52:50.7705821Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1216, in wrap_fx_proxy
2023-09-06T14:52:50.7706418Z     return wrap_fx_proxy_cls(
2023-09-06T14:52:50.7707217Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1303, in wrap_fx_proxy_cls
2023-09-06T14:52:50.7707828Z     example_value = get_fake_value(proxy.node, tx)
2023-09-06T14:52:50.7708592Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1381, in get_fake_value
2023-09-06T14:52:50.7709411Z     raise TorchRuntimeError(str(e)).with_traceback(e.__traceback__) from None
2023-09-06T14:52:50.7710333Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1342, in get_fake_value
2023-09-06T14:52:50.7710938Z     return wrap_fake_exception(
2023-09-06T14:52:50.7711806Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 917, in wrap_fake_exception
2023-09-06T14:52:50.7712367Z     return fn()
2023-09-06T14:52:50.7713308Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1343, in <lambda>
2023-09-06T14:52:50.7714025Z     lambda: run_node(tx.output, node, args, kwargs, nnmodule)
2023-09-06T14:52:50.7715012Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1415, in run_node
2023-09-06T14:52:50.7715873Z     raise RuntimeError(fn_str + str(e)).with_traceback(e.__traceback__) from e
2023-09-06T14:52:50.7716973Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1402, in run_node
2023-09-06T14:52:50.7717534Z     return node.target(*args, **kwargs)
2023-09-06T14:52:50.7718265Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/einops/einops.py", line 483, in rearrange
2023-09-06T14:52:50.7719053Z     return reduce(cast(Tensor, tensor), pattern, reduction='rearrange', **axes_lengths)
2023-09-06T14:52:50.7719892Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/einops/einops.py", line 412, in reduce
2023-09-06T14:52:50.7720816Z     return _apply_recipe(recipe, tensor, reduction_type=reduction)
2023-09-06T14:52:50.7721631Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/einops/einops.py", line 235, in _apply_recipe
2023-09-06T14:52:50.7722328Z     _reconstruct_from_shape(recipe, backend.shape(tensor))
2023-09-06T14:52:50.7723585Z torch._dynamo.exc.TorchRuntimeError: Failed running call_function <function rearrange at 0x7f4e8f039fc0>(*(FakeTensor(..., device='cuda:0', size=(4, 128, s0, s1)), 'b c ... -> b ... c'), **{}):
2023-09-06T14:52:50.7724438Z unhashable type: 'SymInt'
2023-09-06T14:52:50.7724718Z 
2023-09-06T14:52:50.7724870Z from user code:
2023-09-06T14:52:50.7725756Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 1670, in forward
2023-09-06T14:52:50.7726593Z     h = rearrange(h, 'b c ... -> b ... c')
2023-09-06T14:52:50.7726915Z 
2023-09-06T14:52:50.7727248Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T14:52:50.7727666Z 
2023-09-06T14:52:50.7727675Z 
2023-09-06T14:52:50.7728003Z You can suppress this exception and fall back to eager by setting:
2023-09-06T14:52:50.7728551Z     import torch._dynamo
2023-09-06T14:52:50.7729051Z     torch._dynamo.config.suppress_errors = True
2023-09-06T14:52:50.7729326Z 
2023-09-06T14:52:50.7729600Z TorchDynamo optimized model failed to run because of following error
2023-09-06T14:52:50.7730045Z fail_to_run
2023-09-06T14:52:57.1001862Z 
2023-09-06T14:53:00.9638195Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:53:00.9638714Z loading model: 0it [00:03, ?it/s]
2023-09-06T14:53:00.9663295Z cuda eval  LearningToPaint                    
2023-09-06T14:53:13.7106429Z pass
2023-09-06T14:53:19.0062218Z 
2023-09-06T14:53:21.4165095Z loading model: 0it [00:00, ?it/s]WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T14:53:21.9351167Z 
2023-09-06T14:53:21.9351674Z loading model: 0it [00:02, ?it/s]
2023-09-06T14:53:21.9352711Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T14:53:21.9356618Z cuda eval  Super_SloMo                        
2023-09-06T14:53:22.0385649Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T14:54:10.3361860Z pass
2023-09-06T14:54:16.0826885Z 
2023-09-06T14:54:17.6325328Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:54:17.6326382Z loading model: 0it [00:01, ?it/s]
2023-09-06T14:54:17.6326697Z cuda eval  alexnet                            
2023-09-06T14:54:27.2929893Z pass
2023-09-06T14:54:32.2473960Z 
2023-09-06T14:54:39.2156712Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:54:39.2213022Z loading model: 0it [00:06, ?it/s]
2023-09-06T14:54:39.2213454Z cuda eval  attention_is_all_you_need_pytorch  
2023-09-06T14:55:06.9971826Z pass
2023-09-06T14:55:12.4998161Z 
2023-09-06T14:55:15.3239147Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:55:15.3239630Z loading model: 0it [00:02, ?it/s]
2023-09-06T14:55:15.3245820Z cuda eval  basic_gnn_edgecnn                  
2023-09-06T14:55:15.9024524Z [2023-09-06 14:55:15,901] [1/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T14:55:22.3642143Z pass
2023-09-06T14:55:27.1988776Z 
2023-09-06T14:55:30.0549372Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:55:30.0550655Z loading model: 0it [00:02, ?it/s]
2023-09-06T14:55:30.0554799Z cuda eval  basic_gnn_gcn                      
2023-09-06T14:55:38.6415268Z [2023-09-06 14:55:38,639] [11/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T14:55:41.0178412Z pass
2023-09-06T14:55:45.8618879Z 
2023-09-06T14:55:48.8350877Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:55:48.8351231Z loading model: 0it [00:02, ?it/s]
2023-09-06T14:55:48.8359454Z cuda eval  basic_gnn_gin                      
2023-09-06T14:55:49.5819990Z [2023-09-06 14:55:49,580] [1/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T14:55:56.0378967Z pass
2023-09-06T14:56:00.8551277Z 
2023-09-06T14:56:03.8281200Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:56:03.8281775Z loading model: 0it [00:02, ?it/s]
2023-09-06T14:56:03.8289148Z cuda eval  basic_gnn_sage                     
2023-09-06T14:56:04.5514011Z [2023-09-06 14:56:04,550] [1/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T14:56:10.8196169Z pass
2023-09-06T14:56:15.6102447Z 
2023-09-06T14:56:21.0135328Z loading model: 0it [00:00, ?it/s]
2023-09-06T14:56:21.0136110Z loading model: 0it [00:05, ?it/s]
2023-09-06T14:56:21.0249480Z cuda eval  cm3leon_generate                   
2023-09-06T15:00:06.5457637Z pass
2023-09-06T15:00:13.2773155Z 
2023-09-06T15:00:14.1550793Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:00:14.1551160Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:00:14.1556250Z cuda eval  dcgan                              
2023-09-06T15:00:21.7643211Z pass
2023-09-06T15:00:26.6607908Z 
2023-09-06T15:00:29.9600035Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:00:29.9600557Z loading model: 0it [00:03, ?it/s]
2023-09-06T15:00:29.9601012Z Eager model failed to run
2023-09-06T15:00:29.9613560Z Traceback (most recent call last):
2023-09-06T15:00:29.9615377Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:00:29.9615873Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:00:29.9616314Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:00:29.9616692Z     return mod(*inputs)
2023-09-06T15:00:29.9623312Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:00:29.9623912Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:00:29.9624870Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:00:29.9625660Z     return forward_call(*args, **kwargs)
2023-09-06T15:00:29.9626462Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/__init__.py", line 31, in forward
2023-09-06T15:00:29.9627221Z     return sources, self.model(mix)
2023-09-06T15:00:29.9628041Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:00:29.9628467Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:00:29.9629051Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:00:29.9629864Z     return forward_call(*args, **kwargs)
2023-09-06T15:00:29.9630306Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/demucs/model.py", line 209, in forward
2023-09-06T15:00:29.9630695Z     x = self.lstm(x)
2023-09-06T15:00:29.9631259Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:00:29.9631679Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:00:29.9632219Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:00:29.9633017Z     return forward_call(*args, **kwargs)
2023-09-06T15:00:29.9633452Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/demucs/model.py", line 27, in forward
2023-09-06T15:00:29.9633909Z     x = self.lstm(x)[0]
2023-09-06T15:00:29.9634459Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:00:29.9634891Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:00:29.9635444Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:00:29.9635844Z     return forward_call(*args, **kwargs)
2023-09-06T15:00:29.9636383Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/rnn.py", line 879, in forward
2023-09-06T15:00:29.9636844Z     result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
2023-09-06T15:00:29.9637378Z RuntimeError: "_thnn_fused_lstm_cell_cuda" not implemented for 'BFloat16'
2023-09-06T15:00:29.9637637Z 
2023-09-06T15:00:29.9637834Z The above exception was the direct cause of the following exception:
2023-09-06T15:00:29.9638072Z 
2023-09-06T15:00:29.9638198Z Traceback (most recent call last):
2023-09-06T15:00:29.9638562Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:00:29.9638911Z     ) = runner.load_model(
2023-09-06T15:00:29.9639299Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:00:29.9639706Z     self.validate_model(model, example_inputs)
2023-09-06T15:00:29.9640117Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:00:29.9640543Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:00:29.9640925Z NotImplementedError: Eager model failed to run
2023-09-06T15:00:29.9641137Z 
2023-09-06T15:00:29.9641262Z WARNING:root:demucs failed to load
2023-09-06T15:00:34.4695659Z 
2023-09-06T15:00:36.3589569Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:00:36.3590356Z loading model: 0it [00:01, ?it/s]
2023-09-06T15:00:36.3737071Z cuda eval  densenet121                        
2023-09-06T15:01:19.9284446Z pass
2023-09-06T15:01:25.8139871Z 
2023-09-06T15:01:30.7417869Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:01:30.7418232Z loading model: 0it [00:04, ?it/s]
2023-09-06T15:01:30.7418510Z Eager model failed to run
2023-09-06T15:01:30.7434923Z Traceback (most recent call last):
2023-09-06T15:01:30.7435378Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:01:30.7435793Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:01:30.7441500Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:01:30.7441888Z     return mod(*inputs)
2023-09-06T15:01:30.7443248Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:01:30.7443893Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:01:30.7444469Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:01:30.7444860Z     return forward_call(*args, **kwargs)
2023-09-06T15:01:30.7445439Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:01:30.7445873Z     return self.inference(batched_inputs)
2023-09-06T15:01:30.7446458Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:01:30.7446937Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:01:30.7447550Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:01:30.7447972Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:01:30.7448533Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:01:30.7449514Z     return forward_call(*args, **kwargs)
2023-09-06T15:01:30.7450118Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2023-09-06T15:01:30.7450538Z     box_features = self._shared_roi_transform(
2023-09-06T15:01:30.7451175Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2023-09-06T15:01:30.7451621Z     x = self.pooler(features, boxes)
2023-09-06T15:01:30.7452197Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:01:30.7452694Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:01:30.7453495Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:01:30.7453927Z     return forward_call(*args, **kwargs)
2023-09-06T15:01:30.7454490Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T15:01:30.7455037Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T15:01:30.7456148Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:01:30.7456599Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:01:30.7457164Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:01:30.7457568Z     return forward_call(*args, **kwargs)
2023-09-06T15:01:30.7458143Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:01:30.7458503Z     return roi_align(
2023-09-06T15:01:30.7459386Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:01:30.7459927Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:01:30.7460593Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:01:30.7461112Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:01:30.7461773Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:01:30.7462192Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:01:30.7462888Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:01:30.7463275Z     return input[
2023-09-06T15:01:30.7464248Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 26902.82 GiB. GPU 0 has a total capacty of 39.39 GiB of which 35.54 GiB is free. Process 2181336 has 3.84 GiB memory in use. Of the allocated memory 2.99 GiB is allocated by PyTorch, and 335.94 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:01:30.7465161Z 
2023-09-06T15:01:30.7465359Z The above exception was the direct cause of the following exception:
2023-09-06T15:01:30.7465605Z 
2023-09-06T15:01:30.7465733Z Traceback (most recent call last):
2023-09-06T15:01:30.7466100Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:01:30.7466449Z     ) = runner.load_model(
2023-09-06T15:01:30.7466830Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:01:30.7467233Z     self.validate_model(model, example_inputs)
2023-09-06T15:01:30.7467632Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:01:30.7468238Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:01:30.7468671Z NotImplementedError: Eager model failed to run
2023-09-06T15:01:30.7468881Z 
2023-09-06T15:01:30.7469047Z WARNING:root:detectron2_fasterrcnn_r_101_c4 failed to load
2023-09-06T15:01:35.4984059Z 
2023-09-06T15:01:42.6139634Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:01:42.6140162Z loading model: 0it [00:07, ?it/s]
2023-09-06T15:01:42.6140443Z Eager model failed to run
2023-09-06T15:01:42.6155165Z Traceback (most recent call last):
2023-09-06T15:01:42.6156692Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:01:42.6157507Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:01:42.6158334Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:01:42.6158773Z     return mod(*inputs)
2023-09-06T15:01:42.6160216Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:01:42.6160710Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:01:42.6161290Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:01:42.6161697Z     return forward_call(*args, **kwargs)
2023-09-06T15:01:42.6162271Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:01:42.6162699Z     return self.inference(batched_inputs)
2023-09-06T15:01:42.6163269Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:01:42.6163827Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:01:42.6164454Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:01:42.6165382Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:01:42.6165967Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:01:42.6166357Z     return forward_call(*args, **kwargs)
2023-09-06T15:01:42.6166944Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T15:01:42.6167420Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T15:01:42.6168059Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T15:01:42.6168559Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T15:01:42.6169211Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:01:42.6169635Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:01:42.6170201Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:01:42.6170601Z     return forward_call(*args, **kwargs)
2023-09-06T15:01:42.6171145Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T15:01:42.6171582Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T15:01:42.6172183Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:01:42.6172608Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:01:42.6173166Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:01:42.6173626Z     return forward_call(*args, **kwargs)
2023-09-06T15:01:42.6174186Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:01:42.6174765Z     return roi_align(
2023-09-06T15:01:42.6175303Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:01:42.6175809Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:01:42.6176464Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:01:42.6176984Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:01:42.6177660Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:01:42.6178087Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:01:42.6178626Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:01:42.6179012Z     return input[
2023-09-06T15:01:42.6179997Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 13299.95 GiB. GPU 0 has a total capacty of 39.39 GiB of which 34.73 GiB is free. Process 2181443 has 4.65 GiB memory in use. Of the allocated memory 3.82 GiB is allocated by PyTorch, and 283.52 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:01:42.6181155Z 
2023-09-06T15:01:42.6181370Z The above exception was the direct cause of the following exception:
2023-09-06T15:01:42.6181619Z 
2023-09-06T15:01:42.6181750Z Traceback (most recent call last):
2023-09-06T15:01:42.6182131Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:01:42.6182471Z     ) = runner.load_model(
2023-09-06T15:01:42.6182853Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:01:42.6183430Z     self.validate_model(model, example_inputs)
2023-09-06T15:01:42.6183855Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:01:42.6184295Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:01:42.6184678Z NotImplementedError: Eager model failed to run
2023-09-06T15:01:42.6184888Z 
2023-09-06T15:01:42.6185059Z WARNING:root:detectron2_fasterrcnn_r_101_dc5 failed to load
2023-09-06T15:01:47.2919324Z 
2023-09-06T15:01:52.0280466Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:01:52.0280822Z loading model: 0it [00:04, ?it/s]
2023-09-06T15:01:52.0281103Z Eager model failed to run
2023-09-06T15:01:52.0296354Z Traceback (most recent call last):
2023-09-06T15:01:52.0296942Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:01:52.0297652Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:01:52.0298476Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:01:52.0299142Z     return mod(*inputs)
2023-09-06T15:01:52.0300644Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:01:52.0301507Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:01:52.0302518Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:01:52.0303269Z     return forward_call(*args, **kwargs)
2023-09-06T15:01:52.0304351Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:01:52.0304769Z     return self.inference(batched_inputs)
2023-09-06T15:01:52.0305355Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:01:52.0305835Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:01:52.0307001Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:01:52.0307422Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:01:52.0307969Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:01:52.0308455Z     return forward_call(*args, **kwargs)
2023-09-06T15:01:52.0309250Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T15:01:52.0309721Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T15:01:52.0310367Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T15:01:52.0310873Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T15:01:52.0311525Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:01:52.0311956Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:01:52.0312514Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:01:52.0312918Z     return forward_call(*args, **kwargs)
2023-09-06T15:01:52.0313451Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2023-09-06T15:01:52.0313911Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2023-09-06T15:01:52.0314530Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:01:52.0314953Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:01:52.0315486Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:01:52.0315887Z     return forward_call(*args, **kwargs)
2023-09-06T15:01:52.0316630Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:01:52.0317012Z     return roi_align(
2023-09-06T15:01:52.0317547Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:01:52.0318055Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:01:52.0318759Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:01:52.0319278Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:01:52.0319956Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:01:52.0320361Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:01:52.0320922Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:01:52.0321312Z     return input[
2023-09-06T15:01:52.0322278Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 19183.01 GiB. GPU 0 has a total capacty of 39.39 GiB of which 34.81 GiB is free. Process 2181535 has 4.57 GiB memory in use. Of the allocated memory 3.79 GiB is allocated by PyTorch, and 257.07 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:01:52.0323109Z 
2023-09-06T15:01:52.0323308Z The above exception was the direct cause of the following exception:
2023-09-06T15:01:52.0323551Z 
2023-09-06T15:01:52.0323677Z Traceback (most recent call last):
2023-09-06T15:01:52.0324040Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:01:52.0324391Z     ) = runner.load_model(
2023-09-06T15:01:52.0324918Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:01:52.0325328Z     self.validate_model(model, example_inputs)
2023-09-06T15:01:52.0325725Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:01:52.0326172Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:01:52.0326557Z NotImplementedError: Eager model failed to run
2023-09-06T15:01:52.0326767Z 
2023-09-06T15:01:52.0326935Z WARNING:root:detectron2_fasterrcnn_r_101_fpn failed to load
2023-09-06T15:01:56.6523096Z 
2023-09-06T15:02:00.9348849Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:02:00.9349418Z loading model: 0it [00:04, ?it/s]
2023-09-06T15:02:00.9349904Z Eager model failed to run
2023-09-06T15:02:00.9364142Z Traceback (most recent call last):
2023-09-06T15:02:00.9364909Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:02:00.9366037Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:02:00.9366514Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:02:00.9367057Z     return mod(*inputs)
2023-09-06T15:02:00.9370661Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:02:00.9371510Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:02:00.9372501Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:02:00.9373182Z     return forward_call(*args, **kwargs)
2023-09-06T15:02:00.9374217Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:02:00.9374968Z     return self.inference(batched_inputs)
2023-09-06T15:02:00.9376626Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:02:00.9377608Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:02:00.9378711Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:02:00.9379510Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:02:00.9380491Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:02:00.9381186Z     return forward_call(*args, **kwargs)
2023-09-06T15:02:00.9382222Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2023-09-06T15:02:00.9382966Z     box_features = self._shared_roi_transform(
2023-09-06T15:02:00.9384099Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2023-09-06T15:02:00.9384856Z     x = self.pooler(features, boxes)
2023-09-06T15:02:00.9385862Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:02:00.9386576Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:02:00.9387556Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:02:00.9388257Z     return forward_call(*args, **kwargs)
2023-09-06T15:02:00.9389643Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T15:02:00.9390421Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T15:02:00.9391518Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:02:00.9392214Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:02:00.9393220Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:02:00.9394311Z     return forward_call(*args, **kwargs)
2023-09-06T15:02:00.9395289Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:02:00.9395977Z     return roi_align(
2023-09-06T15:02:00.9396916Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:02:00.9397826Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:02:00.9398990Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:02:00.9399883Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:02:00.9401088Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:02:00.9401805Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:02:00.9402746Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:02:00.9403396Z     return input[
2023-09-06T15:02:00.9405062Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 27492.29 GiB. GPU 0 has a total capacty of 39.39 GiB of which 36.78 GiB is free. Process 2181632 has 2.61 GiB memory in use. Of the allocated memory 1.86 GiB is allocated by PyTorch, and 229.54 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:02:00.9406361Z 
2023-09-06T15:02:00.9406747Z The above exception was the direct cause of the following exception:
2023-09-06T15:02:00.9407049Z 
2023-09-06T15:02:00.9407179Z Traceback (most recent call last):
2023-09-06T15:02:00.9407916Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:02:00.9408505Z     ) = runner.load_model(
2023-09-06T15:02:00.9408937Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:02:00.9409345Z     self.validate_model(model, example_inputs)
2023-09-06T15:02:00.9409774Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:02:00.9410364Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:02:00.9410753Z NotImplementedError: Eager model failed to run
2023-09-06T15:02:00.9410968Z 
2023-09-06T15:02:00.9411133Z WARNING:root:detectron2_fasterrcnn_r_50_c4 failed to load
2023-09-06T15:02:05.5682028Z 
2023-09-06T15:02:12.0463269Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:02:12.0463860Z loading model: 0it [00:06, ?it/s]
2023-09-06T15:02:12.0464139Z Eager model failed to run
2023-09-06T15:02:12.0477723Z Traceback (most recent call last):
2023-09-06T15:02:12.0478560Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:02:12.0479204Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:02:12.0479981Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:02:12.0480617Z     return mod(*inputs)
2023-09-06T15:02:12.0481678Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:02:12.0482286Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:02:12.0483255Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:02:12.0483969Z     return forward_call(*args, **kwargs)
2023-09-06T15:02:12.0485019Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:02:12.0485745Z     return self.inference(batched_inputs)
2023-09-06T15:02:12.0487382Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:02:12.0488301Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:02:12.0489431Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:02:12.0490169Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:02:12.0491176Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:02:12.0491892Z     return forward_call(*args, **kwargs)
2023-09-06T15:02:12.0492931Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T15:02:12.0493819Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T15:02:12.0494978Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T15:02:12.0495888Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T15:02:12.0497046Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:02:12.0497821Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:02:12.0498816Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:02:12.0499563Z     return forward_call(*args, **kwargs)
2023-09-06T15:02:12.0500360Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T15:02:12.0501008Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T15:02:12.0501767Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:02:12.0502431Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:02:12.0503257Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:02:12.0503678Z     return forward_call(*args, **kwargs)
2023-09-06T15:02:12.0504432Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:02:12.0504802Z     return roi_align(
2023-09-06T15:02:12.0505346Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:02:12.0505877Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:02:12.0506546Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:02:12.0507076Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:02:12.0507825Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:02:12.0508405Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:02:12.0508988Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:02:12.0509736Z     return input[
2023-09-06T15:02:12.0510730Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 13521.00 GiB. GPU 0 has a total capacty of 39.39 GiB of which 35.88 GiB is free. Process 2181725 has 3.50 GiB memory in use. Of the allocated memory 2.68 GiB is allocated by PyTorch, and 270.78 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:02:12.0511685Z 
2023-09-06T15:02:12.0511888Z The above exception was the direct cause of the following exception:
2023-09-06T15:02:12.0512385Z 
2023-09-06T15:02:12.0512555Z Traceback (most recent call last):
2023-09-06T15:02:12.0512926Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:02:12.0513282Z     ) = runner.load_model(
2023-09-06T15:02:12.0513670Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:02:12.0514076Z     self.validate_model(model, example_inputs)
2023-09-06T15:02:12.0514482Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:02:12.0515012Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:02:12.0515509Z NotImplementedError: Eager model failed to run
2023-09-06T15:02:12.0515723Z 
2023-09-06T15:02:12.0515888Z WARNING:root:detectron2_fasterrcnn_r_50_dc5 failed to load
2023-09-06T15:02:16.7029467Z 
2023-09-06T15:02:21.0067227Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:02:21.0067581Z loading model: 0it [00:04, ?it/s]
2023-09-06T15:02:21.0067905Z Eager model failed to run
2023-09-06T15:02:21.0083288Z Traceback (most recent call last):
2023-09-06T15:02:21.0083925Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:02:21.0084578Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:02:21.0085414Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:02:21.0085823Z     return mod(*inputs)
2023-09-06T15:02:21.0086618Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:02:21.0087051Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:02:21.0087652Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:02:21.0088065Z     return forward_call(*args, **kwargs)
2023-09-06T15:02:21.0089150Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:02:21.0089609Z     return self.inference(batched_inputs)
2023-09-06T15:02:21.0090202Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:02:21.0090683Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:02:21.0091304Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:02:21.0091714Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:02:21.0092323Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:02:21.0092733Z     return forward_call(*args, **kwargs)
2023-09-06T15:02:21.0093329Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T15:02:21.0093852Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T15:02:21.0094487Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T15:02:21.0095008Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T15:02:21.0095665Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:02:21.0096096Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:02:21.0096643Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:02:21.0097041Z     return forward_call(*args, **kwargs)
2023-09-06T15:02:21.0097601Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2023-09-06T15:02:21.0098064Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2023-09-06T15:02:21.0098904Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:02:21.0099317Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:02:21.0099875Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:02:21.0100278Z     return forward_call(*args, **kwargs)
2023-09-06T15:02:21.0100830Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:02:21.0101195Z     return roi_align(
2023-09-06T15:02:21.0101727Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:02:21.0102289Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:02:21.0102942Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:02:21.0103473Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:02:21.0104177Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:02:21.0104594Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:02:21.0105152Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:02:21.0105535Z     return input[
2023-09-06T15:02:21.0106515Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 21352.77 GiB. GPU 0 has a total capacty of 39.39 GiB of which 36.12 GiB is free. Process 2181817 has 3.27 GiB memory in use. Of the allocated memory 2.65 GiB is allocated by PyTorch, and 99.25 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:02:21.0107476Z 
2023-09-06T15:02:21.0107688Z The above exception was the direct cause of the following exception:
2023-09-06T15:02:21.0107921Z 
2023-09-06T15:02:21.0108054Z Traceback (most recent call last):
2023-09-06T15:02:21.0108439Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:02:21.0108795Z     ) = runner.load_model(
2023-09-06T15:02:21.0109360Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:02:21.0109763Z     self.validate_model(model, example_inputs)
2023-09-06T15:02:21.0110188Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:02:21.0110634Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:02:21.0111026Z NotImplementedError: Eager model failed to run
2023-09-06T15:02:21.0111238Z 
2023-09-06T15:02:21.0111407Z WARNING:root:detectron2_fasterrcnn_r_50_fpn failed to load
2023-09-06T15:02:25.6684284Z 
2023-09-06T15:02:29.7332495Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:02:29.7332909Z loading model: 0it [00:04, ?it/s]
2023-09-06T15:02:29.7441761Z cuda eval  detectron2_fcos_r_50_fpn           
2023-09-06T15:03:17.3762895Z [2023-09-06 15:03:17,374] torch._dynamo.convert_frame: [WARNING] torch._dynamo hit config.cache_size_limit (8)
2023-09-06T15:03:17.3763843Z [2023-09-06 15:03:17,374] torch._dynamo.convert_frame: [WARNING]    function: 'forward' (/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/batch_norm.py:318)
2023-09-06T15:03:17.3765228Z [2023-09-06 15:03:17,374] torch._dynamo.convert_frame: [WARNING] to diagnose recompilation issues, set env variable TORCHDYNAMO_REPORT_GUARD_FAILURES=1 and also see https://pytorch.org/docs/master/compile/troubleshooting.html.
2023-09-06T15:04:17.8547019Z [2023-09-06 15:04:17,852] torch._dynamo.utils: [ERROR] Accuracy failed: allclose not within tol=0.01
2023-09-06T15:04:17.8547701Z [2023-09-06 15:04:17,853] torch._dynamo.utils: [ERROR] Accuracy failed for key name pred_boxes
2023-09-06T15:04:17.8548719Z [2023-09-06 15:04:17,853] torch._dynamo.utils: [ERROR] Accuracy failed for key name instances
2023-09-06T15:04:17.8695705Z fail_accuracy
2023-09-06T15:04:20.2956946Z accuracy            pass_rate=80.00%
2023-09-06T15:04:20.2957546Z calls_captured      gmean=0.00x mean=584.267x
2023-09-06T15:04:20.2958706Z unique_graphs       gmean=0.00x mean=8.200x
2023-09-06T15:04:20.2961142Z graph_breaks        gmean=0.00x mean=5.467x
2023-09-06T15:04:20.2962956Z unique_graph_breaks gmean=0.00x mean=1.067x
2023-09-06T15:04:21.0337491Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *cudagraphs-true* ]]
2023-09-06T15:04:21.0339953Z + python benchmarks/dynamo/torchbench.py --accuracy --no-translation-validation --inference --bfloat16 --backend inductor --device cuda --total-partitions 4 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inductor_with_cudagraphs_torchbench_bfloat16_inference_cuda_accuracy.csv
2023-09-06T15:04:28.3595112Z 
2023-09-06T15:04:29.2653627Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:04:29.2654217Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:04:29.2654492Z Eager model failed to run
2023-09-06T15:04:29.2671256Z Traceback (most recent call last):
2023-09-06T15:04:29.2672016Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:04:29.2674597Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:04:29.2675370Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:04:29.2675927Z     return mod(*inputs)
2023-09-06T15:04:29.2676742Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:04:29.2677174Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:04:29.2678234Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:04:29.2678676Z     return forward_call(*args, **kwargs)
2023-09-06T15:04:29.2680316Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 893, in forward
2023-09-06T15:04:29.2680998Z     logits = self.model(batch.dense_features, batch.sparse_features)
2023-09-06T15:04:29.2682025Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:04:29.2682498Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:04:29.2683080Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:04:29.2683488Z     return forward_call(*args, **kwargs)
2023-09-06T15:04:29.2684028Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 570, in forward
2023-09-06T15:04:29.2684459Z     embedded_dense = self.dense_arch(dense_features)
2023-09-06T15:04:29.2685119Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:04:29.2685674Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:04:29.2686246Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:04:29.2686637Z     return forward_call(*args, **kwargs)
2023-09-06T15:04:29.2687191Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 149, in forward
2023-09-06T15:04:29.2687581Z     return self.model(features)
2023-09-06T15:04:29.2688162Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:04:29.2688625Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:04:29.2689186Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:04:29.2690036Z     return forward_call(*args, **kwargs)
2023-09-06T15:04:29.2690597Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/mlp.py", line 172, in forward
2023-09-06T15:04:29.2690964Z     return self._mlp(input)
2023-09-06T15:04:29.2691532Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:04:29.2691957Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:04:29.2692527Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:04:29.2692934Z     return forward_call(*args, **kwargs)
2023-09-06T15:04:29.2693486Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/container.py", line 215, in forward
2023-09-06T15:04:29.2693868Z     input = module(input)
2023-09-06T15:04:29.2694489Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:04:29.2694923Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:04:29.2695472Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:04:29.2695870Z     return forward_call(*args, **kwargs)
2023-09-06T15:04:29.2696412Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/mlp.py", line 73, in forward
2023-09-06T15:04:29.2696833Z     return self._activation_fn(self._linear(input))
2023-09-06T15:04:29.2697425Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:04:29.2697840Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:04:29.2698397Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:04:29.2698845Z     return forward_call(*args, **kwargs)
2023-09-06T15:04:29.2699639Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
2023-09-06T15:04:29.2700083Z     return F.linear(input, self.weight, self.bias)
2023-09-06T15:04:29.2700499Z RuntimeError: mat1 and mat2 must have the same dtype, but got Float and BFloat16
2023-09-06T15:04:29.2700760Z 
2023-09-06T15:04:29.2700958Z The above exception was the direct cause of the following exception:
2023-09-06T15:04:29.2701208Z 
2023-09-06T15:04:29.2701339Z Traceback (most recent call last):
2023-09-06T15:04:29.2701725Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:04:29.2702061Z     ) = runner.load_model(
2023-09-06T15:04:29.2702446Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:04:29.2702855Z     self.validate_model(model, example_inputs)
2023-09-06T15:04:29.2703269Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:04:29.2703706Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:04:29.2704096Z NotImplementedError: Eager model failed to run
2023-09-06T15:04:29.2704307Z 
2023-09-06T15:04:29.2704442Z WARNING:root:torchrec_dlrm failed to load
2023-09-06T15:04:33.7858544Z 
2023-09-06T15:04:36.8739825Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:04:36.8740200Z loading model: 0it [00:03, ?it/s]
2023-09-06T15:04:36.8807043Z cuda eval  BERT_pytorch                       
2023-09-06T15:05:02.8956821Z pass
2023-09-06T15:05:08.0281109Z 
2023-09-06T15:05:11.6936559Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:05:11.6936916Z loading model: 0it [00:03, ?it/s]
2023-09-06T15:05:11.6991177Z cuda eval  Background_Matting                 
2023-09-06T15:05:11.6994330Z pass_due_to_skip
2023-09-06T15:05:16.2691721Z 
2023-09-06T15:05:25.4259343Z loading model: 0it [00:00, ?it/s]WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T15:05:27.1375900Z 
2023-09-06T15:05:27.1377021Z loading model: 0it [00:10, ?it/s]
2023-09-06T15:05:27.1378184Z WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T15:05:27.1381357Z cuda eval  DALLE2_pytorch                     
2023-09-06T15:05:27.3817792Z WARNING:common:fp64 golden ref were not generated for DALLE2_pytorch. Setting accuracy check to cosine
2023-09-06T15:05:27.7809683Z WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T15:05:41.1215075Z [2023-09-06 15:05:41,119] [3/1] torch._dynamo.output_graph: [WARNING] nn.Module forward/_pre hooks are only partially supported, and were detected in your model. In particular, if you do not change/remove hooks after calling .compile(), you can disregard this warning, and otherwise you may need to set torch._dynamo.config.skip_nnmodule_hook_guards=False to ensure recompiling after changing hooks.See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations. 
2023-09-06T15:05:56.3238815Z [2023-09-06 15:05:56,322] [5/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T15:05:56.6456255Z [2023-09-06 15:05:56,644] [5/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T15:05:56.7557615Z [2023-09-06 15:05:56,754] [5/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T15:05:56.8642584Z [2023-09-06 15:05:56,863] [5/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T15:05:56.9715660Z [2023-09-06 15:05:56,970] [5/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T15:05:57.0791727Z [2023-09-06 15:05:57,078] [5/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T15:05:57.1897398Z [2023-09-06 15:05:57,188] [5/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T15:05:57.3010113Z [2023-09-06 15:05:57,300] [5/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T15:05:57.4121692Z [2023-09-06 15:05:57,411] [5/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T15:05:57.5230291Z [2023-09-06 15:05:57,522] [5/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T15:05:57.6313782Z [2023-09-06 15:05:57,630] [5/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T15:05:57.7388845Z [2023-09-06 15:05:57,738] [5/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T15:06:06.4872904Z skipping cudagraphs due to multiple devices
2023-09-06T15:06:47.4983901Z [2023-09-06 15:06:47,496] [29/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T15:06:47.8391592Z [2023-09-06 15:06:47,838] [29/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T15:06:47.9544785Z [2023-09-06 15:06:47,953] [29/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T15:06:48.0710445Z [2023-09-06 15:06:48,070] [29/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T15:06:48.1879854Z [2023-09-06 15:06:48,187] [29/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T15:06:48.3037658Z [2023-09-06 15:06:48,303] [29/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T15:06:48.4188298Z [2023-09-06 15:06:48,417] [29/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T15:06:48.5362063Z [2023-09-06 15:06:48,535] [29/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T15:06:48.6567673Z [2023-09-06 15:06:48,655] [29/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T15:06:48.7709007Z [2023-09-06 15:06:48,770] [29/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T15:06:48.8868129Z [2023-09-06 15:06:48,885] [29/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T15:06:49.0040648Z [2023-09-06 15:06:49,003] [29/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T15:06:51.8470631Z skipping cudagraphs due to multiple devices
2023-09-06T15:07:08.0793252Z ERROR:common:Failed running call_function <function rearrange at 0x7f8b79a39f30>(*(FakeTensor(..., device='cuda:0', size=(4, 128, s0, s1)), 'b c ... -> b ... c'), **{}):
2023-09-06T15:07:08.0793878Z unhashable type: 'SymInt'
2023-09-06T15:07:08.0794041Z 
2023-09-06T15:07:08.0794176Z from user code:
2023-09-06T15:07:08.0795001Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 1670, in forward
2023-09-06T15:07:08.0795773Z     h = rearrange(h, 'b c ... -> b ... c')
2023-09-06T15:07:08.0799435Z 
2023-09-06T15:07:08.0800036Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T15:07:08.0800548Z 
2023-09-06T15:07:08.0800651Z 
2023-09-06T15:07:08.0801006Z You can suppress this exception and fall back to eager by setting:
2023-09-06T15:07:08.0801531Z     import torch._dynamo
2023-09-06T15:07:08.0802004Z     torch._dynamo.config.suppress_errors = True
2023-09-06T15:07:08.0802648Z Traceback (most recent call last):
2023-09-06T15:07:08.0803246Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2135, in check_accuracy
2023-09-06T15:07:08.0803681Z     new_result = optimized_model_iter_fn(model_copy, example_inputs)
2023-09-06T15:07:08.0804425Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T15:07:08.0804811Z     return fn(*args, **kwargs)
2023-09-06T15:07:08.0805197Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1899, in run_n_iterations
2023-09-06T15:07:08.0805608Z     self.model_iter_fn(mod, inputs, collect_outputs=False)
2023-09-06T15:07:08.0806065Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:07:08.0806699Z     return mod(*inputs)
2023-09-06T15:07:08.0807741Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:07:08.0808412Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:07:08.0809816Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:07:08.0810433Z     return forward_call(*args, **kwargs)
2023-09-06T15:07:08.0811251Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T15:07:08.0811826Z     return func(*args, **kwargs)
2023-09-06T15:07:08.0812573Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 95, in inner
2023-09-06T15:07:08.0813104Z     model.eval()
2023-09-06T15:07:08.0813902Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 96, in <resume in inner>
2023-09-06T15:07:08.0814501Z     out = fn(model, *args, **kwargs)
2023-09-06T15:07:08.0815263Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3329, in forward
2023-09-06T15:07:08.0816073Z     image_embed = self.prior.sample(text, num_samples_per_batch = self.prior_num_samples, cond_scale = prior_cond_scale)
2023-09-06T15:07:08.0817121Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3332, in <resume in forward>
2023-09-06T15:07:08.0817896Z     images = self.decoder.sample(image_embed = image_embed, text = text_cond, cond_scale = cond_scale)
2023-09-06T15:07:08.0818908Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T15:07:08.0819472Z     return func(*args, **kwargs)
2023-09-06T15:07:08.0820277Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 95, in inner
2023-09-06T15:07:08.0820819Z     model.eval()
2023-09-06T15:07:08.0821629Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 96, in <resume in inner>
2023-09-06T15:07:08.0822236Z     out = fn(model, *args, **kwargs)
2023-09-06T15:07:08.0823389Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3199, in sample
2023-09-06T15:07:08.0823957Z     img = self.p_sample_loop(
2023-09-06T15:07:08.0824734Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T15:07:08.0825312Z     return func(*args, **kwargs)
2023-09-06T15:07:08.0826078Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3028, in p_sample_loop
2023-09-06T15:07:08.0826764Z     return self.p_sample_loop_ddpm(*args, noise_scheduler = noise_scheduler, **kwargs)
2023-09-06T15:07:08.0827633Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T15:07:08.0828164Z     return func(*args, **kwargs)
2023-09-06T15:07:08.0829028Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2885, in p_sample_loop_ddpm
2023-09-06T15:07:08.0829904Z     img, x_start = self.p_sample(
2023-09-06T15:07:08.0830731Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T15:07:08.0831315Z     return func(*args, **kwargs)
2023-09-06T15:07:08.0832115Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2822, in p_sample
2023-09-06T15:07:08.0833646Z     model_mean, _, model_log_variance, x_start = self.p_mean_variance(unet, x = x, t = t, image_embed = image_embed, text_encodings = text_encodings, cond_scale = cond_scale, lowres_cond_img = lowres_cond_img, self_cond = self_cond, clip_denoised = clip_denoised, predict_x_start = predict_x_start, predict_v = predict_v, noise_scheduler = noise_scheduler, learned_variance = learned_variance, lowres_noise_level = lowres_noise_level)
2023-09-06T15:07:08.0835672Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2787, in p_mean_variance
2023-09-06T15:07:08.0837089Z     model_output = default(model_output, lambda: unet.forward_with_cond_scale(x, t, image_embed = image_embed, text_encodings = text_encodings, cond_scale = cond_scale, lowres_cond_img = lowres_cond_img, self_cond = self_cond, lowres_noise_level = lowres_noise_level))
2023-09-06T15:07:08.0838381Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 65, in default
2023-09-06T15:07:08.0839001Z     return d() if callable(d) else d
2023-09-06T15:07:08.0839824Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2787, in <lambda>
2023-09-06T15:07:08.0840940Z     model_output = default(model_output, lambda: unet.forward_with_cond_scale(x, t, image_embed = image_embed, text_encodings = text_encodings, cond_scale = cond_scale, lowres_cond_img = lowres_cond_img, self_cond = self_cond, lowres_noise_level = lowres_noise_level))
2023-09-06T15:07:08.0842223Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2159, in forward_with_cond_scale
2023-09-06T15:07:08.0842828Z     logits = self.forward(*args, **kwargs)
2023-09-06T15:07:08.0843584Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2340, in forward
2023-09-06T15:07:08.0844109Z     x = resnet_block(x, t, c)
2023-09-06T15:07:08.0844914Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:07:08.0845496Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:07:08.0846274Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:07:08.0846804Z     return forward_call(*args, **kwargs)
2023-09-06T15:07:08.0847573Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T15:07:08.0848199Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T15:07:08.0849365Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 634, in _convert_frame
2023-09-06T15:07:08.0850022Z     result = inner_convert(frame, cache_entry, hooks, frame_state)
2023-09-06T15:07:08.0850860Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T15:07:08.0851413Z     return fn(*args, **kwargs)
2023-09-06T15:07:08.0852261Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T15:07:08.0852839Z     return _compile(
2023-09-06T15:07:08.0853589Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T15:07:08.0854263Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T15:07:08.0855149Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T15:07:08.0855710Z     r = func(*args, **kwargs)
2023-09-06T15:07:08.0856514Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T15:07:08.0857141Z     out_code = transform_code_object(code, transform)
2023-09-06T15:07:08.0858088Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T15:07:08.0858818Z     transformations(instructions, code_options)
2023-09-06T15:07:08.0859658Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T15:07:08.0860179Z     tracer.run()
2023-09-06T15:07:08.0860923Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T15:07:08.0861449Z     super().run()
2023-09-06T15:07:08.0862413Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T15:07:08.0863169Z     and self.step()
2023-09-06T15:07:08.0863945Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T15:07:08.0864494Z     getattr(self, inst.opname)(inst)
2023-09-06T15:07:08.0865261Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T15:07:08.0865854Z     return inner_fn(self, inst)
2023-09-06T15:07:08.0866630Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1119, in CALL_FUNCTION
2023-09-06T15:07:08.0867198Z     self.call_function(fn, args, {})
2023-09-06T15:07:08.0868005Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 565, in call_function
2023-09-06T15:07:08.0868611Z     self.push(fn.call_function(self, args, kwargs))
2023-09-06T15:07:08.0869668Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/torch.py", line 731, in call_function
2023-09-06T15:07:08.0870261Z     tensor_variable = wrap_fx_proxy(
2023-09-06T15:07:08.0871111Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1216, in wrap_fx_proxy
2023-09-06T15:07:08.0871694Z     return wrap_fx_proxy_cls(
2023-09-06T15:07:08.0872520Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1303, in wrap_fx_proxy_cls
2023-09-06T15:07:08.0873130Z     example_value = get_fake_value(proxy.node, tx)
2023-09-06T15:07:08.0873973Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1381, in get_fake_value
2023-09-06T15:07:08.0874644Z     raise TorchRuntimeError(str(e)).with_traceback(e.__traceback__) from None
2023-09-06T15:07:08.0875601Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1342, in get_fake_value
2023-09-06T15:07:08.0876527Z     return wrap_fake_exception(
2023-09-06T15:07:08.0877352Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 917, in wrap_fake_exception
2023-09-06T15:07:08.0877893Z     return fn()
2023-09-06T15:07:08.0878698Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1343, in <lambda>
2023-09-06T15:07:08.0879353Z     lambda: run_node(tx.output, node, args, kwargs, nnmodule)
2023-09-06T15:07:08.0880197Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1415, in run_node
2023-09-06T15:07:08.0880889Z     raise RuntimeError(fn_str + str(e)).with_traceback(e.__traceback__) from e
2023-09-06T15:07:08.0881764Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1402, in run_node
2023-09-06T15:07:08.0882305Z     return node.target(*args, **kwargs)
2023-09-06T15:07:08.0883032Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/einops/einops.py", line 483, in rearrange
2023-09-06T15:07:08.0883866Z     return reduce(cast(Tensor, tensor), pattern, reduction='rearrange', **axes_lengths)
2023-09-06T15:07:08.0884694Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/einops/einops.py", line 412, in reduce
2023-09-06T15:07:08.0885282Z     return _apply_recipe(recipe, tensor, reduction_type=reduction)
2023-09-06T15:07:08.0886090Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/einops/einops.py", line 235, in _apply_recipe
2023-09-06T15:07:08.0886657Z     _reconstruct_from_shape(recipe, backend.shape(tensor))
2023-09-06T15:07:08.0887729Z torch._dynamo.exc.TorchRuntimeError: Failed running call_function <function rearrange at 0x7f8b79a39f30>(*(FakeTensor(..., device='cuda:0', size=(4, 128, s0, s1)), 'b c ... -> b ... c'), **{}):
2023-09-06T15:07:08.0888475Z unhashable type: 'SymInt'
2023-09-06T15:07:08.0888790Z 
2023-09-06T15:07:08.0888931Z from user code:
2023-09-06T15:07:08.0889989Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 1670, in forward
2023-09-06T15:07:08.0890706Z     h = rearrange(h, 'b c ... -> b ... c')
2023-09-06T15:07:08.0890983Z 
2023-09-06T15:07:08.0891276Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T15:07:08.0891623Z 
2023-09-06T15:07:08.0891632Z 
2023-09-06T15:07:08.0891917Z You can suppress this exception and fall back to eager by setting:
2023-09-06T15:07:08.0892414Z     import torch._dynamo
2023-09-06T15:07:08.0892854Z     torch._dynamo.config.suppress_errors = True
2023-09-06T15:07:08.0893153Z 
2023-09-06T15:07:08.0893445Z TorchDynamo optimized model failed to run because of following error
2023-09-06T15:07:08.0893923Z fail_to_run
2023-09-06T15:07:14.1801340Z 
2023-09-06T15:07:17.9345246Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:07:17.9345605Z loading model: 0it [00:03, ?it/s]
2023-09-06T15:07:17.9375406Z cuda eval  LearningToPaint                    
2023-09-06T15:07:30.7267184Z pass
2023-09-06T15:07:35.9424601Z 
2023-09-06T15:07:38.3325864Z loading model: 0it [00:00, ?it/s]WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T15:07:38.8716573Z 
2023-09-06T15:07:38.8717062Z loading model: 0it [00:02, ?it/s]
2023-09-06T15:07:38.8717517Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T15:07:38.8717925Z cuda eval  Super_SloMo                        
2023-09-06T15:07:38.9757212Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T15:08:18.4098961Z pass
2023-09-06T15:08:24.1185815Z 
2023-09-06T15:08:25.6588750Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:08:25.6589775Z loading model: 0it [00:01, ?it/s]
2023-09-06T15:08:25.6593431Z cuda eval  alexnet                            
2023-09-06T15:08:34.5368097Z pass
2023-09-06T15:08:39.4326282Z 
2023-09-06T15:08:46.3174901Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:08:46.3178382Z loading model: 0it [00:06, ?it/s]
2023-09-06T15:08:46.3232833Z cuda eval  attention_is_all_you_need_pytorch  
2023-09-06T15:09:13.9574680Z pass
2023-09-06T15:09:19.4066361Z 
2023-09-06T15:09:22.1411845Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:09:22.1415326Z loading model: 0it [00:02, ?it/s]
2023-09-06T15:09:22.1420682Z cuda eval  basic_gnn_edgecnn                  
2023-09-06T15:09:22.7146049Z [2023-09-06 15:09:22,713] [1/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T15:09:28.5234617Z skipping cudagraphs for unknown reason
2023-09-06T15:09:28.8308658Z pass
2023-09-06T15:09:33.6596726Z 
2023-09-06T15:09:36.5068002Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:09:36.5068375Z loading model: 0it [00:02, ?it/s]
2023-09-06T15:09:36.5070864Z cuda eval  basic_gnn_gcn                      
2023-09-06T15:09:44.4935355Z skipping cudagraphs for unknown reason
2023-09-06T15:09:44.5113872Z [2023-09-06 15:09:44,510] [11/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T15:09:45.7213929Z skipping cudagraphs for unknown reason
2023-09-06T15:09:46.3893204Z skipping cudagraphs for unknown reason
2023-09-06T15:09:46.7178373Z skipping cudagraphs for unknown reason
2023-09-06T15:09:46.8393850Z pass
2023-09-06T15:09:51.6768571Z 
2023-09-06T15:09:54.7093883Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:09:54.7094299Z loading model: 0it [00:03, ?it/s]
2023-09-06T15:09:54.7101769Z cuda eval  basic_gnn_gin                      
2023-09-06T15:09:55.4232763Z [2023-09-06 15:09:55,422] [1/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T15:10:01.1787170Z skipping cudagraphs for unknown reason
2023-09-06T15:10:01.5153438Z pass
2023-09-06T15:10:06.3290845Z 
2023-09-06T15:10:09.2444071Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:10:09.2444596Z loading model: 0it [00:02, ?it/s]
2023-09-06T15:10:09.2447015Z cuda eval  basic_gnn_sage                     
2023-09-06T15:10:09.9605883Z [2023-09-06 15:10:09,959] [1/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T15:10:15.6801284Z skipping cudagraphs for unknown reason
2023-09-06T15:10:16.0235808Z pass
2023-09-06T15:10:20.8203046Z 
2023-09-06T15:10:26.1244455Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:10:26.1244805Z loading model: 0it [00:05, ?it/s]
2023-09-06T15:10:26.1347134Z cuda eval  cm3leon_generate                   
2023-09-06T15:14:06.9282538Z pass
2023-09-06T15:14:13.5741165Z 
2023-09-06T15:14:14.4513138Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:14:14.4513811Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:14:14.4515459Z cuda eval  dcgan                              
2023-09-06T15:14:22.1918450Z pass
2023-09-06T15:14:27.0112978Z 
2023-09-06T15:14:30.2873111Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:14:30.2873998Z loading model: 0it [00:03, ?it/s]
2023-09-06T15:14:30.2874423Z Eager model failed to run
2023-09-06T15:14:30.2881668Z Traceback (most recent call last):
2023-09-06T15:14:30.2882636Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:14:30.2883349Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:14:30.2884213Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:14:30.2895525Z     return mod(*inputs)
2023-09-06T15:14:30.2897107Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:14:30.2897533Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:14:30.2898102Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:14:30.2898563Z     return forward_call(*args, **kwargs)
2023-09-06T15:14:30.2898995Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/__init__.py", line 31, in forward
2023-09-06T15:14:30.2899404Z     return sources, self.model(mix)
2023-09-06T15:14:30.2899961Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:14:30.2900398Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:14:30.2901120Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:14:30.2901528Z     return forward_call(*args, **kwargs)
2023-09-06T15:14:30.2901960Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/demucs/model.py", line 209, in forward
2023-09-06T15:14:30.2902332Z     x = self.lstm(x)
2023-09-06T15:14:30.2902872Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:14:30.2903366Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:14:30.2903925Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:14:30.2904311Z     return forward_call(*args, **kwargs)
2023-09-06T15:14:30.2904742Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/demucs/model.py", line 27, in forward
2023-09-06T15:14:30.2905131Z     x = self.lstm(x)[0]
2023-09-06T15:14:30.2905942Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:14:30.2906369Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:14:30.2906911Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:14:30.2907312Z     return forward_call(*args, **kwargs)
2023-09-06T15:14:30.2907847Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/rnn.py", line 879, in forward
2023-09-06T15:14:30.2908319Z     result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
2023-09-06T15:14:30.2908834Z RuntimeError: "_thnn_fused_lstm_cell_cuda" not implemented for 'BFloat16'
2023-09-06T15:14:30.2909321Z 
2023-09-06T15:14:30.2909576Z The above exception was the direct cause of the following exception:
2023-09-06T15:14:30.2909819Z 
2023-09-06T15:14:30.2909947Z Traceback (most recent call last):
2023-09-06T15:14:30.2910339Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:14:30.2910686Z     ) = runner.load_model(
2023-09-06T15:14:30.2911048Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:14:30.2911682Z     self.validate_model(model, example_inputs)
2023-09-06T15:14:30.2912097Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:14:30.2912523Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:14:30.2912903Z NotImplementedError: Eager model failed to run
2023-09-06T15:14:30.2913110Z 
2023-09-06T15:14:30.2913280Z WARNING:root:demucs failed to load
2023-09-06T15:14:34.7657272Z 
2023-09-06T15:14:36.6466447Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:14:36.6466867Z loading model: 0it [00:01, ?it/s]
2023-09-06T15:14:36.6612396Z cuda eval  densenet121                        
2023-09-06T15:15:19.4618811Z pass
2023-09-06T15:15:25.2188661Z 
2023-09-06T15:15:30.0111022Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:15:30.0111452Z loading model: 0it [00:04, ?it/s]
2023-09-06T15:15:30.0111733Z Eager model failed to run
2023-09-06T15:15:30.0127913Z Traceback (most recent call last):
2023-09-06T15:15:30.0128643Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:15:30.0129311Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:15:30.0130000Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:15:30.0130683Z     return mod(*inputs)
2023-09-06T15:15:30.0132115Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:15:30.0132533Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:15:30.0135160Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:15:30.0139261Z     return forward_call(*args, **kwargs)
2023-09-06T15:15:30.0140463Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:15:30.0141220Z     return self.inference(batched_inputs)
2023-09-06T15:15:30.0142279Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:15:30.0143093Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:15:30.0144149Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:15:30.0144858Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:15:30.0145821Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:15:30.0146625Z     return forward_call(*args, **kwargs)
2023-09-06T15:15:30.0148436Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2023-09-06T15:15:30.0149061Z     box_features = self._shared_roi_transform(
2023-09-06T15:15:30.0150283Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2023-09-06T15:15:30.0150738Z     x = self.pooler(features, boxes)
2023-09-06T15:15:30.0151322Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:15:30.0151747Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:15:30.0152296Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:15:30.0152699Z     return forward_call(*args, **kwargs)
2023-09-06T15:15:30.0153253Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T15:15:30.0153702Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T15:15:30.0154304Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:15:30.0154727Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:15:30.0155285Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:15:30.0155685Z     return forward_call(*args, **kwargs)
2023-09-06T15:15:30.0156242Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:15:30.0156632Z     return roi_align(
2023-09-06T15:15:30.0157162Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:15:30.0157694Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:15:30.0158413Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:15:30.0159175Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:15:30.0159851Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:15:30.0160275Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:15:30.0160829Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:15:30.0161212Z     return input[
2023-09-06T15:15:30.0162191Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 26902.82 GiB. GPU 0 has a total capacty of 39.39 GiB of which 35.54 GiB is free. Process 2184289 has 3.84 GiB memory in use. Of the allocated memory 2.99 GiB is allocated by PyTorch, and 335.94 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:15:30.0164433Z 
2023-09-06T15:15:30.0164636Z The above exception was the direct cause of the following exception:
2023-09-06T15:15:30.0164886Z 
2023-09-06T15:15:30.0165013Z Traceback (most recent call last):
2023-09-06T15:15:30.0165383Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:15:30.0165740Z     ) = runner.load_model(
2023-09-06T15:15:30.0166119Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:15:30.0166527Z     self.validate_model(model, example_inputs)
2023-09-06T15:15:30.0166944Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:15:30.0167375Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:15:30.0167757Z NotImplementedError: Eager model failed to run
2023-09-06T15:15:30.0167970Z 
2023-09-06T15:15:30.0168351Z WARNING:root:detectron2_fasterrcnn_r_101_c4 failed to load
2023-09-06T15:15:34.6482548Z 
2023-09-06T15:15:41.5936019Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:15:41.5936639Z loading model: 0it [00:06, ?it/s]
2023-09-06T15:15:41.5936923Z Eager model failed to run
2023-09-06T15:15:41.5951701Z Traceback (most recent call last):
2023-09-06T15:15:41.5952411Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:15:41.5953014Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:15:41.5953582Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:15:41.5953955Z     return mod(*inputs)
2023-09-06T15:15:41.5955548Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:15:41.5956173Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:15:41.5957310Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:15:41.5958192Z     return forward_call(*args, **kwargs)
2023-09-06T15:15:41.5959271Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:15:41.5960089Z     return self.inference(batched_inputs)
2023-09-06T15:15:41.5961184Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:15:41.5961739Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:15:41.5962376Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:15:41.5962802Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:15:41.5963364Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:15:41.5963814Z     return forward_call(*args, **kwargs)
2023-09-06T15:15:41.5964953Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T15:15:41.5965426Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T15:15:41.5966067Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T15:15:41.5966595Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T15:15:41.5967229Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:15:41.5967653Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:15:41.5968286Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:15:41.5968701Z     return forward_call(*args, **kwargs)
2023-09-06T15:15:41.5969276Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T15:15:41.5969706Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T15:15:41.5970307Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:15:41.5970729Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:15:41.5971285Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:15:41.5971691Z     return forward_call(*args, **kwargs)
2023-09-06T15:15:41.5972227Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:15:41.5972605Z     return roi_align(
2023-09-06T15:15:41.5973140Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:15:41.5973844Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:15:41.5974508Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:15:41.5975028Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:15:41.5975709Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:15:41.5976133Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:15:41.5976695Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:15:41.5977064Z     return input[
2023-09-06T15:15:41.5978088Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 13299.95 GiB. GPU 0 has a total capacty of 39.39 GiB of which 34.73 GiB is free. Process 2184396 has 4.65 GiB memory in use. Of the allocated memory 3.82 GiB is allocated by PyTorch, and 283.52 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:15:41.5978942Z 
2023-09-06T15:15:41.5979147Z The above exception was the direct cause of the following exception:
2023-09-06T15:15:41.5979394Z 
2023-09-06T15:15:41.5979522Z Traceback (most recent call last):
2023-09-06T15:15:41.5979901Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:15:41.5980257Z     ) = runner.load_model(
2023-09-06T15:15:41.5980624Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:15:41.5981027Z     self.validate_model(model, example_inputs)
2023-09-06T15:15:41.5981445Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:15:41.5981890Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:15:41.5982402Z NotImplementedError: Eager model failed to run
2023-09-06T15:15:41.5982616Z 
2023-09-06T15:15:41.5982792Z WARNING:root:detectron2_fasterrcnn_r_101_dc5 failed to load
2023-09-06T15:15:46.3002994Z 
2023-09-06T15:15:50.8913029Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:15:50.8913395Z loading model: 0it [00:04, ?it/s]
2023-09-06T15:15:50.8913673Z Eager model failed to run
2023-09-06T15:15:50.8928851Z Traceback (most recent call last):
2023-09-06T15:15:50.8929648Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:15:50.8930417Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:15:50.8931042Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:15:50.8931414Z     return mod(*inputs)
2023-09-06T15:15:50.8932213Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:15:50.8932893Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:15:50.8933463Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:15:50.8933870Z     return forward_call(*args, **kwargs)
2023-09-06T15:15:50.8934441Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:15:50.8934851Z     return self.inference(batched_inputs)
2023-09-06T15:15:50.8935439Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:15:50.8935912Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:15:50.8936525Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:15:50.8936940Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:15:50.8937912Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:15:50.8938338Z     return forward_call(*args, **kwargs)
2023-09-06T15:15:50.8939017Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T15:15:50.8939481Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T15:15:50.8940115Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T15:15:50.8940620Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T15:15:50.8941261Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:15:50.8941684Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:15:50.8942237Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:15:50.8942633Z     return forward_call(*args, **kwargs)
2023-09-06T15:15:50.8943192Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2023-09-06T15:15:50.8943657Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2023-09-06T15:15:50.8944274Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:15:50.8944689Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:15:50.8945228Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:15:50.8945624Z     return forward_call(*args, **kwargs)
2023-09-06T15:15:50.8946168Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:15:50.8946540Z     return roi_align(
2023-09-06T15:15:50.8947057Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:15:50.8947786Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:15:50.8948506Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:15:50.8949025Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:15:50.8950209Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:15:50.8950633Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:15:50.8951176Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:15:50.8951555Z     return input[
2023-09-06T15:15:50.8952531Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 19183.01 GiB. GPU 0 has a total capacty of 39.39 GiB of which 34.81 GiB is free. Process 2184490 has 4.57 GiB memory in use. Of the allocated memory 3.79 GiB is allocated by PyTorch, and 257.07 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:15:50.8953361Z 
2023-09-06T15:15:50.8953560Z The above exception was the direct cause of the following exception:
2023-09-06T15:15:50.8953806Z 
2023-09-06T15:15:50.8953935Z Traceback (most recent call last):
2023-09-06T15:15:50.8954298Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:15:50.8954648Z     ) = runner.load_model(
2023-09-06T15:15:50.8955032Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:15:50.8955430Z     self.validate_model(model, example_inputs)
2023-09-06T15:15:50.8956019Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:15:50.8956466Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:15:50.8956849Z NotImplementedError: Eager model failed to run
2023-09-06T15:15:50.8957061Z 
2023-09-06T15:15:50.8957229Z WARNING:root:detectron2_fasterrcnn_r_101_fpn failed to load
2023-09-06T15:15:55.5361416Z 
2023-09-06T15:15:59.8685134Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:15:59.8685832Z loading model: 0it [00:04, ?it/s]
2023-09-06T15:15:59.8686113Z Eager model failed to run
2023-09-06T15:15:59.8698929Z Traceback (most recent call last):
2023-09-06T15:15:59.8699737Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:15:59.8700404Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:15:59.8701099Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:15:59.8701730Z     return mod(*inputs)
2023-09-06T15:15:59.8703857Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:15:59.8704352Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:15:59.8705259Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:15:59.8706062Z     return forward_call(*args, **kwargs)
2023-09-06T15:15:59.8707050Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:15:59.8707830Z     return self.inference(batched_inputs)
2023-09-06T15:15:59.8708820Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:15:59.8709912Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:15:59.8711036Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:15:59.8712527Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:15:59.8713935Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:15:59.8714625Z     return forward_call(*args, **kwargs)
2023-09-06T15:15:59.8715703Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2023-09-06T15:15:59.8716432Z     box_features = self._shared_roi_transform(
2023-09-06T15:15:59.8717558Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2023-09-06T15:15:59.8718343Z     x = self.pooler(features, boxes)
2023-09-06T15:15:59.8719347Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:15:59.8720075Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:15:59.8721070Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:15:59.8721992Z     return forward_call(*args, **kwargs)
2023-09-06T15:15:59.8722968Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T15:15:59.8723791Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T15:15:59.8724985Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:15:59.8725726Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:15:59.8726697Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:15:59.8727436Z     return forward_call(*args, **kwargs)
2023-09-06T15:15:59.8728378Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:15:59.8729597Z     return roi_align(
2023-09-06T15:15:59.8730554Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:15:59.8731700Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:15:59.8732825Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:15:59.8733766Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:15:59.8735233Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:15:59.8735983Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:15:59.8736961Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:15:59.8737600Z     return input[
2023-09-06T15:15:59.8739464Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 27492.29 GiB. GPU 0 has a total capacty of 39.39 GiB of which 36.78 GiB is free. Process 2184586 has 2.61 GiB memory in use. Of the allocated memory 1.86 GiB is allocated by PyTorch, and 229.54 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:15:59.8741957Z 
2023-09-06T15:15:59.8742299Z The above exception was the direct cause of the following exception:
2023-09-06T15:15:59.8742774Z 
2023-09-06T15:15:59.8742921Z Traceback (most recent call last):
2023-09-06T15:15:59.8743295Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:15:59.8743839Z     ) = runner.load_model(
2023-09-06T15:15:59.8744456Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:15:59.8745200Z     self.validate_model(model, example_inputs)
2023-09-06T15:15:59.8745599Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:15:59.8746181Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:15:59.8746567Z NotImplementedError: Eager model failed to run
2023-09-06T15:15:59.8746777Z 
2023-09-06T15:15:59.8746944Z WARNING:root:detectron2_fasterrcnn_r_50_c4 failed to load
2023-09-06T15:16:04.5045900Z 
2023-09-06T15:16:10.9832785Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:16:10.9833599Z loading model: 0it [00:06, ?it/s]
2023-09-06T15:16:10.9834042Z Eager model failed to run
2023-09-06T15:16:10.9848230Z Traceback (most recent call last):
2023-09-06T15:16:10.9848919Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:16:10.9849608Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:16:10.9850366Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:16:10.9850970Z     return mod(*inputs)
2023-09-06T15:16:10.9852158Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:16:10.9852571Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:16:10.9853190Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:16:10.9853597Z     return forward_call(*args, **kwargs)
2023-09-06T15:16:10.9854162Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:16:10.9854588Z     return self.inference(batched_inputs)
2023-09-06T15:16:10.9855155Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:16:10.9855640Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:16:10.9856974Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:16:10.9857564Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:16:10.9858371Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:16:10.9858879Z     return forward_call(*args, **kwargs)
2023-09-06T15:16:10.9859840Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T15:16:10.9860648Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T15:16:10.9861767Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T15:16:10.9862461Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T15:16:10.9863331Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:16:10.9863758Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:16:10.9864310Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:16:10.9864695Z     return forward_call(*args, **kwargs)
2023-09-06T15:16:10.9865246Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T15:16:10.9865685Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T15:16:10.9866278Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:16:10.9866701Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:16:10.9867236Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:16:10.9867635Z     return forward_call(*args, **kwargs)
2023-09-06T15:16:10.9868452Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:16:10.9868831Z     return roi_align(
2023-09-06T15:16:10.9869667Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:16:10.9870193Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:16:10.9870851Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:16:10.9871368Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:16:10.9872046Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:16:10.9872449Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:16:10.9873057Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:16:10.9873445Z     return input[
2023-09-06T15:16:10.9874415Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 13521.00 GiB. GPU 0 has a total capacty of 39.39 GiB of which 35.88 GiB is free. Process 2184678 has 3.50 GiB memory in use. Of the allocated memory 2.68 GiB is allocated by PyTorch, and 270.78 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:16:10.9875249Z 
2023-09-06T15:16:10.9875447Z The above exception was the direct cause of the following exception:
2023-09-06T15:16:10.9875692Z 
2023-09-06T15:16:10.9875819Z Traceback (most recent call last):
2023-09-06T15:16:10.9876183Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:16:10.9876531Z     ) = runner.load_model(
2023-09-06T15:16:10.9877115Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:16:10.9877520Z     self.validate_model(model, example_inputs)
2023-09-06T15:16:10.9877927Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:16:10.9878354Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:16:10.9878743Z NotImplementedError: Eager model failed to run
2023-09-06T15:16:10.9878955Z 
2023-09-06T15:16:10.9879144Z WARNING:root:detectron2_fasterrcnn_r_50_dc5 failed to load
2023-09-06T15:16:15.5953846Z 
2023-09-06T15:16:19.8911094Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:16:19.8911461Z loading model: 0it [00:04, ?it/s]
2023-09-06T15:16:19.8911734Z Eager model failed to run
2023-09-06T15:16:19.8927850Z Traceback (most recent call last):
2023-09-06T15:16:19.8928587Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:16:19.8929537Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:16:19.8930134Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:16:19.8930499Z     return mod(*inputs)
2023-09-06T15:16:19.8931298Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:16:19.8931734Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:16:19.8932300Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:16:19.8932707Z     return forward_call(*args, **kwargs)
2023-09-06T15:16:19.8933282Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:16:19.8933691Z     return self.inference(batched_inputs)
2023-09-06T15:16:19.8934273Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:16:19.8935275Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:16:19.8935907Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:16:19.8936319Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:16:19.8936871Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:16:19.8937279Z     return forward_call(*args, **kwargs)
2023-09-06T15:16:19.8937857Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T15:16:19.8938324Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T15:16:19.8939008Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T15:16:19.8939532Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T15:16:19.8940186Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:16:19.8940605Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:16:19.8941144Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:16:19.8941542Z     return forward_call(*args, **kwargs)
2023-09-06T15:16:19.8942092Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2023-09-06T15:16:19.8942584Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2023-09-06T15:16:19.8943201Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:16:19.8943601Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:16:19.8944340Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:16:19.8944754Z     return forward_call(*args, **kwargs)
2023-09-06T15:16:19.8945316Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:16:19.8945676Z     return roi_align(
2023-09-06T15:16:19.8946225Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:16:19.8946754Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:16:19.8947399Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:16:19.8947912Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:16:19.8948568Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:16:19.8949038Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:16:19.8949767Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:16:19.8950152Z     return input[
2023-09-06T15:16:19.8951118Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 21352.77 GiB. GPU 0 has a total capacty of 39.39 GiB of which 36.12 GiB is free. Process 2184770 has 3.27 GiB memory in use. Of the allocated memory 2.65 GiB is allocated by PyTorch, and 99.25 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:16:19.8952107Z 
2023-09-06T15:16:19.8952308Z The above exception was the direct cause of the following exception:
2023-09-06T15:16:19.8952553Z 
2023-09-06T15:16:19.8952680Z Traceback (most recent call last):
2023-09-06T15:16:19.8953328Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:16:19.8953678Z     ) = runner.load_model(
2023-09-06T15:16:19.8954058Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:16:19.8954463Z     self.validate_model(model, example_inputs)
2023-09-06T15:16:19.8954862Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:16:19.8955302Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:16:19.8955686Z NotImplementedError: Eager model failed to run
2023-09-06T15:16:19.8955894Z 
2023-09-06T15:16:19.8956095Z WARNING:root:detectron2_fasterrcnn_r_50_fpn failed to load
2023-09-06T15:16:24.5822149Z 
2023-09-06T15:16:28.6152892Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:16:28.6153503Z loading model: 0it [00:04, ?it/s]
2023-09-06T15:16:28.6252605Z cuda eval  detectron2_fcos_r_50_fpn           
2023-09-06T15:16:42.9150997Z skipping cudagraphs for unknown reason
2023-09-06T15:16:44.5543543Z skipping cudagraphs due to input mutation
2023-09-06T15:17:12.1631582Z [2023-09-06 15:17:12,161] torch._dynamo.convert_frame: [WARNING] torch._dynamo hit config.cache_size_limit (8)
2023-09-06T15:17:12.1632643Z [2023-09-06 15:17:12,161] torch._dynamo.convert_frame: [WARNING]    function: 'forward' (/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/batch_norm.py:318)
2023-09-06T15:17:12.1634081Z [2023-09-06 15:17:12,161] torch._dynamo.convert_frame: [WARNING] to diagnose recompilation issues, set env variable TORCHDYNAMO_REPORT_GUARD_FAILURES=1 and also see https://pytorch.org/docs/master/compile/troubleshooting.html.
2023-09-06T15:17:23.7784393Z skipping cudagraphs due to input mutation
2023-09-06T15:17:23.9270151Z skipping cudagraphs for unknown reason
2023-09-06T15:17:24.7591273Z skipping cudagraphs due to input mutation
2023-09-06T15:17:24.8570110Z skipping cudagraphs for unknown reason
2023-09-06T15:17:25.6375876Z skipping cudagraphs due to input mutation
2023-09-06T15:17:26.4263664Z skipping cudagraphs due to input mutation
2023-09-06T15:18:07.7263331Z skipping cudagraphs due to input mutation
2023-09-06T15:18:07.7517441Z [2023-09-06 15:18:07,750] torch._dynamo.utils: [ERROR] Accuracy failed: allclose not within tol=0.01
2023-09-06T15:18:07.7518105Z [2023-09-06 15:18:07,751] torch._dynamo.utils: [ERROR] Accuracy failed for key name pred_boxes
2023-09-06T15:18:07.7518702Z [2023-09-06 15:18:07,751] torch._dynamo.utils: [ERROR] Accuracy failed for key name instances
2023-09-06T15:18:07.7669320Z fail_accuracy
2023-09-06T15:18:10.1948800Z accuracy            pass_rate=80.00%
2023-09-06T15:18:10.1950352Z calls_captured      gmean=0.00x mean=584.267x
2023-09-06T15:18:10.1950713Z unique_graphs       gmean=0.00x mean=8.200x
2023-09-06T15:18:10.1951039Z graph_breaks        gmean=0.00x mean=5.467x
2023-09-06T15:18:10.1953400Z unique_graph_breaks gmean=0.00x mean=1.067x
2023-09-06T15:18:10.9494464Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *dynamic-true* ]]
2023-09-06T15:18:10.9496191Z + python benchmarks/dynamo/torchbench.py --accuracy --no-translation-validation --inference --bfloat16 --backend inductor --dynamic-shapes --dynamic-batch-only --device cuda --total-partitions 4 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inductor_dynamic_torchbench_bfloat16_inference_cuda_accuracy.csv
2023-09-06T15:18:18.1632984Z 
2023-09-06T15:18:19.0795802Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:18:19.0796399Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:18:19.0796685Z Eager model failed to run
2023-09-06T15:18:19.0812619Z Traceback (most recent call last):
2023-09-06T15:18:19.0813459Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:18:19.0814060Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:18:19.0814819Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:18:19.0815777Z     return mod(*inputs)
2023-09-06T15:18:19.0817152Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:18:19.0817576Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:18:19.0822021Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:18:19.0822563Z     return forward_call(*args, **kwargs)
2023-09-06T15:18:19.0823145Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 893, in forward
2023-09-06T15:18:19.0823670Z     logits = self.model(batch.dense_features, batch.sparse_features)
2023-09-06T15:18:19.0824318Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:18:19.0824752Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:18:19.0825331Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:18:19.0825736Z     return forward_call(*args, **kwargs)
2023-09-06T15:18:19.0826274Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 570, in forward
2023-09-06T15:18:19.0826703Z     embedded_dense = self.dense_arch(dense_features)
2023-09-06T15:18:19.0827285Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:18:19.0827706Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:18:19.0828260Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:18:19.0828669Z     return forward_call(*args, **kwargs)
2023-09-06T15:18:19.0829500Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 149, in forward
2023-09-06T15:18:19.0829986Z     return self.model(features)
2023-09-06T15:18:19.0830924Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:18:19.0831354Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:18:19.0831920Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:18:19.0832308Z     return forward_call(*args, **kwargs)
2023-09-06T15:18:19.0832846Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/mlp.py", line 172, in forward
2023-09-06T15:18:19.0833229Z     return self._mlp(input)
2023-09-06T15:18:19.0833782Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:18:19.0834193Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:18:19.0834727Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:18:19.0835137Z     return forward_call(*args, **kwargs)
2023-09-06T15:18:19.0835689Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/container.py", line 215, in forward
2023-09-06T15:18:19.0836068Z     input = module(input)
2023-09-06T15:18:19.0836604Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:18:19.0837016Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:18:19.0837564Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:18:19.0837962Z     return forward_call(*args, **kwargs)
2023-09-06T15:18:19.0838499Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/mlp.py", line 73, in forward
2023-09-06T15:18:19.0838958Z     return self._activation_fn(self._linear(input))
2023-09-06T15:18:19.0839565Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:18:19.0840201Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:18:19.0840769Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:18:19.0841164Z     return forward_call(*args, **kwargs)
2023-09-06T15:18:19.0841701Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
2023-09-06T15:18:19.0842135Z     return F.linear(input, self.weight, self.bias)
2023-09-06T15:18:19.0842542Z RuntimeError: mat1 and mat2 must have the same dtype, but got Float and BFloat16
2023-09-06T15:18:19.0842808Z 
2023-09-06T15:18:19.0843010Z The above exception was the direct cause of the following exception:
2023-09-06T15:18:19.0843239Z 
2023-09-06T15:18:19.0843364Z Traceback (most recent call last):
2023-09-06T15:18:19.0843751Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:18:19.0844102Z     ) = runner.load_model(
2023-09-06T15:18:19.0844486Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:18:19.0844880Z     self.validate_model(model, example_inputs)
2023-09-06T15:18:19.0845289Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:18:19.0845736Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:18:19.0846120Z NotImplementedError: Eager model failed to run
2023-09-06T15:18:19.0846327Z 
2023-09-06T15:18:19.0846456Z WARNING:root:torchrec_dlrm failed to load
2023-09-06T15:18:23.5535565Z 
2023-09-06T15:18:26.6188509Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:18:26.6188936Z loading model: 0it [00:03, ?it/s]
2023-09-06T15:18:26.6255477Z cuda eval  BERT_pytorch                       
2023-09-06T15:19:03.1785545Z pass
2023-09-06T15:19:08.4475399Z 
2023-09-06T15:19:12.1304615Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:19:12.1304980Z loading model: 0it [00:03, ?it/s]
2023-09-06T15:19:12.1378270Z cuda eval  Background_Matting                 
2023-09-06T15:19:12.1380730Z pass_due_to_skip
2023-09-06T15:19:16.8297195Z 
2023-09-06T15:19:26.0411557Z loading model: 0it [00:00, ?it/s]WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T15:19:27.8092864Z 
2023-09-06T15:19:27.8093464Z loading model: 0it [00:10, ?it/s]
2023-09-06T15:19:27.8094129Z WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T15:19:27.8094571Z cuda eval  DALLE2_pytorch                     
2023-09-06T15:19:28.0516504Z WARNING:common:fp64 golden ref were not generated for DALLE2_pytorch. Setting accuracy check to cosine
2023-09-06T15:19:28.4591370Z WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T15:19:39.7043115Z ERROR:common:name 'Eq' is not defined
2023-09-06T15:19:39.7043535Z Traceback (most recent call last):
2023-09-06T15:19:39.7044074Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2135, in check_accuracy
2023-09-06T15:19:39.7044535Z     new_result = optimized_model_iter_fn(model_copy, example_inputs)
2023-09-06T15:19:39.7046289Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T15:19:39.7046824Z     return fn(*args, **kwargs)
2023-09-06T15:19:39.7047208Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1899, in run_n_iterations
2023-09-06T15:19:39.7047636Z     self.model_iter_fn(mod, inputs, collect_outputs=False)
2023-09-06T15:19:39.7048255Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:19:39.7049001Z     return mod(*inputs)
2023-09-06T15:19:39.7049614Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:19:39.7050043Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:19:39.7051250Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:19:39.7051985Z     return forward_call(*args, **kwargs)
2023-09-06T15:19:39.7052774Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T15:19:39.7053172Z     return func(*args, **kwargs)
2023-09-06T15:19:39.7053778Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 95, in inner
2023-09-06T15:19:39.7054378Z     model.eval()
2023-09-06T15:19:39.7055253Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 96, in <resume in inner>
2023-09-06T15:19:39.7055954Z     out = fn(model, *args, **kwargs)
2023-09-06T15:19:39.7056993Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3313, in forward
2023-09-06T15:19:39.7057711Z     @torch.no_grad()
2023-09-06T15:19:39.7058817Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T15:19:39.7059280Z     return fn(*args, **kwargs)
2023-09-06T15:19:39.7059827Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/external_utils.py", line 17, in inner
2023-09-06T15:19:39.7060220Z     return fn(*args, **kwargs)
2023-09-06T15:19:39.7060771Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3905, in forward
2023-09-06T15:19:39.7061171Z     return compiled_fn(full_args)
2023-09-06T15:19:39.7061696Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1482, in g
2023-09-06T15:19:39.7062070Z     return f(*args)
2023-09-06T15:19:39.7062784Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2533, in runtime_wrapper
2023-09-06T15:19:39.7063202Z     all_outs = call_func_with_args(
2023-09-06T15:19:39.7064185Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1506, in call_func_with_args
2023-09-06T15:19:39.7064598Z     out = normalize_as_list(f(args))
2023-09-06T15:19:39.7065237Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1594, in rng_functionalization_wrapper
2023-09-06T15:19:39.7065673Z     return compiled_fw(args)
2023-09-06T15:19:39.7066215Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 384, in __call__
2023-09-06T15:19:39.7066620Z     return self.get_current_callable()(inputs)
2023-09-06T15:19:39.7067174Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 689, in run
2023-09-06T15:19:39.7067564Z     return compiled_fn(new_inputs)
2023-09-06T15:19:39.7068160Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 378, in deferred_cudagraphify
2023-09-06T15:19:39.7068741Z     fn, out = cudagraphify(model, inputs, new_static_input_idxs, *args, **kwargs)
2023-09-06T15:19:39.7069707Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 405, in cudagraphify
2023-09-06T15:19:39.7070130Z     return manager.add_function(
2023-09-06T15:19:39.7070710Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 1921, in add_function
2023-09-06T15:19:39.7071103Z     return fn, fn(inputs)
2023-09-06T15:19:39.7071628Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 1741, in run
2023-09-06T15:19:39.7072046Z     out = self._run(new_inputs, function_id)
2023-09-06T15:19:39.7072612Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 1782, in _run
2023-09-06T15:19:39.7073044Z     return self.run_eager(new_inputs, function_id)
2023-09-06T15:19:39.7073666Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 1897, in run_eager
2023-09-06T15:19:39.7074258Z     return node.run(new_inputs)
2023-09-06T15:19:39.7074807Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/cudagraph_trees.py", line 603, in run
2023-09-06T15:19:39.7075237Z     out = self.wrapped_function.model(new_inputs)
2023-09-06T15:19:39.7075827Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 411, in _run_from_cache
2023-09-06T15:19:39.7076252Z     return compiled_graph.compiled_artifact(inputs)
2023-09-06T15:19:39.7076768Z   File "/tmp/torchinductor_jenkins/rw/crws42g3jmmnbmezsagvvyimf775tzd4rq4yay4b333hj5tbdfyo.py", line 30, in call
2023-09-06T15:19:39.7077215Z     return (Eq(s0, 1), )
2023-09-06T15:19:39.7077557Z NameError: name 'Eq' is not defined
2023-09-06T15:19:39.7077930Z TorchDynamo optimized model failed to run because of following error
2023-09-06T15:19:39.7381701Z fail_to_run
2023-09-06T15:19:45.2075139Z 
2023-09-06T15:19:49.0054929Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:19:49.0082795Z loading model: 0it [00:03, ?it/s]
2023-09-06T15:19:49.0083143Z cuda eval  LearningToPaint                    
2023-09-06T15:20:01.9640435Z pass
2023-09-06T15:20:07.1987413Z 
2023-09-06T15:20:09.6091060Z loading model: 0it [00:00, ?it/s]WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T15:20:10.1076057Z 
2023-09-06T15:20:10.1076619Z loading model: 0it [00:02, ?it/s]
2023-09-06T15:20:10.1077157Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T15:20:10.1077565Z cuda eval  Super_SloMo                        
2023-09-06T15:20:10.2108962Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T15:21:12.6118610Z pass
2023-09-06T15:21:18.4418073Z 
2023-09-06T15:21:19.9785880Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:21:19.9786288Z loading model: 0it [00:01, ?it/s]
2023-09-06T15:21:19.9793719Z cuda eval  alexnet                            
2023-09-06T15:21:28.8420505Z pass
2023-09-06T15:21:33.8226791Z 
2023-09-06T15:21:40.6405931Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:21:40.6406276Z loading model: 0it [00:06, ?it/s]
2023-09-06T15:21:40.6460389Z cuda eval  attention_is_all_you_need_pytorch  
2023-09-06T15:22:24.1858137Z pass
2023-09-06T15:22:29.8112231Z 
2023-09-06T15:22:32.4506057Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:22:32.4506437Z loading model: 0it [00:02, ?it/s]
2023-09-06T15:22:32.4512828Z cuda eval  basic_gnn_edgecnn                  
2023-09-06T15:22:33.0115961Z [2023-09-06 15:22:33,010] [1/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T15:22:38.8444014Z skipping cudagraphs for unknown reason
2023-09-06T15:22:39.1719935Z pass
2023-09-06T15:22:43.8738965Z 
2023-09-06T15:22:46.9113243Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:22:46.9113784Z loading model: 0it [00:03, ?it/s]
2023-09-06T15:22:46.9116524Z cuda eval  basic_gnn_gcn                      
2023-09-06T15:22:54.9859244Z skipping cudagraphs for unknown reason
2023-09-06T15:22:55.0044997Z [2023-09-06 15:22:55,003] [11/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T15:22:56.2165912Z skipping cudagraphs for unknown reason
2023-09-06T15:22:56.9055729Z skipping cudagraphs for unknown reason
2023-09-06T15:22:57.2256588Z skipping cudagraphs for unknown reason
2023-09-06T15:22:57.3486119Z pass
2023-09-06T15:23:02.0760028Z 
2023-09-06T15:23:05.0051682Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:23:05.0052045Z loading model: 0it [00:02, ?it/s]
2023-09-06T15:23:05.0059651Z cuda eval  basic_gnn_gin                      
2023-09-06T15:23:05.7248198Z [2023-09-06 15:23:05,723] [1/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T15:23:11.4646872Z skipping cudagraphs for unknown reason
2023-09-06T15:23:11.8012460Z pass
2023-09-06T15:23:16.5812512Z 
2023-09-06T15:23:19.4844136Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:23:19.4844506Z loading model: 0it [00:02, ?it/s]
2023-09-06T15:23:19.4848464Z cuda eval  basic_gnn_sage                     
2023-09-06T15:23:20.2150711Z [2023-09-06 15:23:20,213] [1/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T15:23:25.9628411Z skipping cudagraphs for unknown reason
2023-09-06T15:23:26.3089665Z pass
2023-09-06T15:23:31.1150060Z 
2023-09-06T15:23:36.4202441Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:23:36.4202801Z loading model: 0it [00:05, ?it/s]
2023-09-06T15:23:36.4312727Z cuda eval  cm3leon_generate                   
2023-09-06T15:28:22.1808137Z pass
2023-09-06T15:28:28.8444667Z 
2023-09-06T15:28:29.6861691Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:28:29.6862240Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:28:29.6867779Z cuda eval  dcgan                              
2023-09-06T15:28:37.4967153Z pass
2023-09-06T15:28:42.3115855Z 
2023-09-06T15:28:45.6228903Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:28:45.6229493Z loading model: 0it [00:03, ?it/s]
2023-09-06T15:28:45.6229775Z Eager model failed to run
2023-09-06T15:28:45.6243487Z Traceback (most recent call last):
2023-09-06T15:28:45.6244470Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:28:45.6244901Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:28:45.6245323Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:28:45.6247065Z     return mod(*inputs)
2023-09-06T15:28:45.6248587Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:28:45.6249026Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:28:45.6249600Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:28:45.6250032Z     return forward_call(*args, **kwargs)
2023-09-06T15:28:45.6250475Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/__init__.py", line 31, in forward
2023-09-06T15:28:45.6250867Z     return sources, self.model(mix)
2023-09-06T15:28:45.6251456Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:28:45.6251897Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:28:45.6252651Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:28:45.6253215Z     return forward_call(*args, **kwargs)
2023-09-06T15:28:45.6253893Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/demucs/model.py", line 209, in forward
2023-09-06T15:28:45.6254339Z     x = self.lstm(x)
2023-09-06T15:28:45.6255095Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:28:45.6255856Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:28:45.6256765Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:28:45.6257456Z     return forward_call(*args, **kwargs)
2023-09-06T15:28:45.6258849Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/demucs/model.py", line 27, in forward
2023-09-06T15:28:45.6259593Z     x = self.lstm(x)[0]
2023-09-06T15:28:45.6260555Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:28:45.6261336Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:28:45.6262293Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:28:45.6263160Z     return forward_call(*args, **kwargs)
2023-09-06T15:28:45.6264127Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/rnn.py", line 879, in forward
2023-09-06T15:28:45.6264956Z     result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
2023-09-06T15:28:45.6265879Z RuntimeError: "_thnn_fused_lstm_cell_cuda" not implemented for 'BFloat16'
2023-09-06T15:28:45.6266336Z 
2023-09-06T15:28:45.6266706Z The above exception was the direct cause of the following exception:
2023-09-06T15:28:45.6266951Z 
2023-09-06T15:28:45.6267159Z Traceback (most recent call last):
2023-09-06T15:28:45.6267840Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:28:45.6268228Z     ) = runner.load_model(
2023-09-06T15:28:45.6268728Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:28:45.6269712Z     self.validate_model(model, example_inputs)
2023-09-06T15:28:45.6270288Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:28:45.6270719Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:28:45.6271407Z NotImplementedError: Eager model failed to run
2023-09-06T15:28:45.6271714Z 
2023-09-06T15:28:45.6271840Z WARNING:root:demucs failed to load
2023-09-06T15:28:50.0605915Z 
2023-09-06T15:28:51.9558604Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:28:51.9559681Z loading model: 0it [00:01, ?it/s]
2023-09-06T15:28:51.9710002Z cuda eval  densenet121                        
2023-09-06T15:29:34.8338335Z pass
2023-09-06T15:29:40.5902491Z 
2023-09-06T15:29:45.3605103Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:29:45.3605670Z loading model: 0it [00:04, ?it/s]
2023-09-06T15:29:45.3606118Z Eager model failed to run
2023-09-06T15:29:45.3623229Z Traceback (most recent call last):
2023-09-06T15:29:45.3623942Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:29:45.3624539Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:29:45.3625147Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:29:45.3625725Z     return mod(*inputs)
2023-09-06T15:29:45.3629842Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:29:45.3630633Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:29:45.3631419Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:29:45.3631833Z     return forward_call(*args, **kwargs)
2023-09-06T15:29:45.3632660Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:29:45.3633306Z     return self.inference(batched_inputs)
2023-09-06T15:29:45.3633920Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:29:45.3634402Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:29:45.3635030Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:29:45.3635469Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:29:45.3636020Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:29:45.3637099Z     return forward_call(*args, **kwargs)
2023-09-06T15:29:45.3638352Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2023-09-06T15:29:45.3639206Z     box_features = self._shared_roi_transform(
2023-09-06T15:29:45.3639977Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2023-09-06T15:29:45.3640593Z     x = self.pooler(features, boxes)
2023-09-06T15:29:45.3641431Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:29:45.3641999Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:29:45.3642895Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:29:45.3643604Z     return forward_call(*args, **kwargs)
2023-09-06T15:29:45.3644582Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T15:29:45.3645362Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T15:29:45.3646406Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:29:45.3647140Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:29:45.3648178Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:29:45.3648893Z     return forward_call(*args, **kwargs)
2023-09-06T15:29:45.3649834Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:29:45.3650525Z     return roi_align(
2023-09-06T15:29:45.3651470Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:29:45.3652711Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:29:45.3653916Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:29:45.3654809Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:29:45.3656012Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:29:45.3656779Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:29:45.3657735Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:29:45.3658475Z     return input[
2023-09-06T15:29:45.3660133Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 26902.82 GiB. GPU 0 has a total capacty of 39.39 GiB of which 35.54 GiB is free. Process 2187428 has 3.84 GiB memory in use. Of the allocated memory 2.99 GiB is allocated by PyTorch, and 335.94 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:29:45.3661608Z 
2023-09-06T15:29:45.3661994Z The above exception was the direct cause of the following exception:
2023-09-06T15:29:45.3662380Z 
2023-09-06T15:29:45.3662758Z Traceback (most recent call last):
2023-09-06T15:29:45.3663401Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:29:45.3664000Z     ) = runner.load_model(
2023-09-06T15:29:45.3664702Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:29:45.3665382Z     self.validate_model(model, example_inputs)
2023-09-06T15:29:45.3666083Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:29:45.3666944Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:29:45.3667644Z NotImplementedError: Eager model failed to run
2023-09-06T15:29:45.3667936Z 
2023-09-06T15:29:45.3668105Z WARNING:root:detectron2_fasterrcnn_r_101_c4 failed to load
2023-09-06T15:29:49.8843670Z 
2023-09-06T15:29:56.9178231Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:29:56.9178599Z loading model: 0it [00:07, ?it/s]
2023-09-06T15:29:56.9178951Z Eager model failed to run
2023-09-06T15:29:56.9195604Z Traceback (most recent call last):
2023-09-06T15:29:56.9196226Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:29:56.9196958Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:29:56.9197696Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:29:56.9198299Z     return mod(*inputs)
2023-09-06T15:29:56.9199972Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:29:56.9200414Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:29:56.9200993Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:29:56.9201408Z     return forward_call(*args, **kwargs)
2023-09-06T15:29:56.9201985Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:29:56.9202399Z     return self.inference(batched_inputs)
2023-09-06T15:29:56.9202989Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:29:56.9203471Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:29:56.9204092Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:29:56.9204515Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:29:56.9205562Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:29:56.9205978Z     return forward_call(*args, **kwargs)
2023-09-06T15:29:56.9206577Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T15:29:56.9207044Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T15:29:56.9207665Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T15:29:56.9208184Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T15:29:56.9208923Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:29:56.9209349Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:29:56.9209909Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:29:56.9210306Z     return forward_call(*args, **kwargs)
2023-09-06T15:29:56.9210861Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T15:29:56.9211299Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T15:29:56.9211895Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:29:56.9212302Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:29:56.9212853Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:29:56.9213353Z     return forward_call(*args, **kwargs)
2023-09-06T15:29:56.9214038Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:29:56.9214416Z     return roi_align(
2023-09-06T15:29:56.9215215Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:29:56.9215739Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:29:56.9216399Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:29:56.9216917Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:29:56.9217591Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:29:56.9217997Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:29:56.9218557Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:29:56.9218995Z     return input[
2023-09-06T15:29:56.9219975Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 13299.95 GiB. GPU 0 has a total capacty of 39.39 GiB of which 34.73 GiB is free. Process 2187531 has 4.65 GiB memory in use. Of the allocated memory 3.82 GiB is allocated by PyTorch, and 283.52 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:29:56.9220824Z 
2023-09-06T15:29:56.9221026Z The above exception was the direct cause of the following exception:
2023-09-06T15:29:56.9221271Z 
2023-09-06T15:29:56.9221401Z Traceback (most recent call last):
2023-09-06T15:29:56.9221765Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:29:56.9222150Z     ) = runner.load_model(
2023-09-06T15:29:56.9222535Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:29:56.9222940Z     self.validate_model(model, example_inputs)
2023-09-06T15:29:56.9223466Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:29:56.9223923Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:29:56.9224305Z NotImplementedError: Eager model failed to run
2023-09-06T15:29:56.9224516Z 
2023-09-06T15:29:56.9224685Z WARNING:root:detectron2_fasterrcnn_r_101_dc5 failed to load
2023-09-06T15:30:01.5903032Z 
2023-09-06T15:30:06.1720276Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:30:06.1720860Z loading model: 0it [00:04, ?it/s]
2023-09-06T15:30:06.1721358Z Eager model failed to run
2023-09-06T15:30:06.1738422Z Traceback (most recent call last):
2023-09-06T15:30:06.1739227Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:30:06.1739972Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:30:06.1740718Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:30:06.1741339Z     return mod(*inputs)
2023-09-06T15:30:06.1744662Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:30:06.1745111Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:30:06.1745729Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:30:06.1746141Z     return forward_call(*args, **kwargs)
2023-09-06T15:30:06.1746721Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:30:06.1747149Z     return self.inference(batched_inputs)
2023-09-06T15:30:06.1747720Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:30:06.1748200Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:30:06.1748900Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:30:06.1750212Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:30:06.1750785Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:30:06.1751189Z     return forward_call(*args, **kwargs)
2023-09-06T15:30:06.1751796Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T15:30:06.1752263Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T15:30:06.1752895Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T15:30:06.1753394Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T15:30:06.1754048Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:30:06.1754485Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:30:06.1755043Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:30:06.1755427Z     return forward_call(*args, **kwargs)
2023-09-06T15:30:06.1755987Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2023-09-06T15:30:06.1756448Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2023-09-06T15:30:06.1757070Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:30:06.1757494Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:30:06.1758036Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:30:06.1758436Z     return forward_call(*args, **kwargs)
2023-09-06T15:30:06.1759244Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:30:06.1759626Z     return roi_align(
2023-09-06T15:30:06.1760153Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:30:06.1760673Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:30:06.1761330Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:30:06.1761848Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:30:06.1762524Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:30:06.1762929Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:30:06.1763488Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:30:06.1763878Z     return input[
2023-09-06T15:30:06.1764849Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 19183.01 GiB. GPU 0 has a total capacty of 39.39 GiB of which 34.81 GiB is free. Process 2187629 has 4.57 GiB memory in use. Of the allocated memory 3.79 GiB is allocated by PyTorch, and 257.07 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:30:06.1766011Z 
2023-09-06T15:30:06.1766213Z The above exception was the direct cause of the following exception:
2023-09-06T15:30:06.1766462Z 
2023-09-06T15:30:06.1766590Z Traceback (most recent call last):
2023-09-06T15:30:06.1766974Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:30:06.1767306Z     ) = runner.load_model(
2023-09-06T15:30:06.1767829Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:30:06.1768236Z     self.validate_model(model, example_inputs)
2023-09-06T15:30:06.1768688Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:30:06.1769120Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:30:06.1769498Z NotImplementedError: Eager model failed to run
2023-09-06T15:30:06.1769714Z 
2023-09-06T15:30:06.1769884Z WARNING:root:detectron2_fasterrcnn_r_101_fpn failed to load
2023-09-06T15:30:10.7955935Z 
2023-09-06T15:30:15.0897491Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:30:15.0898055Z loading model: 0it [00:04, ?it/s]
2023-09-06T15:30:15.0898334Z Eager model failed to run
2023-09-06T15:30:15.0913757Z Traceback (most recent call last):
2023-09-06T15:30:15.0914458Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:30:15.0915314Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:30:15.0916036Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:30:15.0916641Z     return mod(*inputs)
2023-09-06T15:30:15.0917888Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:30:15.0918304Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:30:15.0918939Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:30:15.0919674Z     return forward_call(*args, **kwargs)
2023-09-06T15:30:15.0920761Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:30:15.0921568Z     return self.inference(batched_inputs)
2023-09-06T15:30:15.0922308Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:30:15.0923357Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:30:15.0924010Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:30:15.0924432Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:30:15.0924974Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:30:15.0925381Z     return forward_call(*args, **kwargs)
2023-09-06T15:30:15.0925963Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2023-09-06T15:30:15.0926448Z     box_features = self._shared_roi_transform(
2023-09-06T15:30:15.0927086Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2023-09-06T15:30:15.0927563Z     x = self.pooler(features, boxes)
2023-09-06T15:30:15.0928189Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:30:15.0928612Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:30:15.0929164Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:30:15.0929566Z     return forward_call(*args, **kwargs)
2023-09-06T15:30:15.0930104Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T15:30:15.0930538Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T15:30:15.0931129Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:30:15.0931554Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:30:15.0932089Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:30:15.0932680Z     return forward_call(*args, **kwargs)
2023-09-06T15:30:15.0933280Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:30:15.0933653Z     return roi_align(
2023-09-06T15:30:15.0934164Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:30:15.0934683Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:30:15.0935334Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:30:15.0935847Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:30:15.0936517Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:30:15.0936922Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:30:15.0937486Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:30:15.0937870Z     return input[
2023-09-06T15:30:15.0938833Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 27492.29 GiB. GPU 0 has a total capacty of 39.39 GiB of which 36.78 GiB is free. Process 2187725 has 2.61 GiB memory in use. Of the allocated memory 1.86 GiB is allocated by PyTorch, and 229.54 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:30:15.0939738Z 
2023-09-06T15:30:15.0939939Z The above exception was the direct cause of the following exception:
2023-09-06T15:30:15.0940182Z 
2023-09-06T15:30:15.0940312Z Traceback (most recent call last):
2023-09-06T15:30:15.0940690Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:30:15.0941138Z     ) = runner.load_model(
2023-09-06T15:30:15.0941519Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:30:15.0941923Z     self.validate_model(model, example_inputs)
2023-09-06T15:30:15.0942335Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:30:15.0942767Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:30:15.0943194Z NotImplementedError: Eager model failed to run
2023-09-06T15:30:15.0943405Z 
2023-09-06T15:30:15.0943570Z WARNING:root:detectron2_fasterrcnn_r_50_c4 failed to load
2023-09-06T15:30:19.7506107Z 
2023-09-06T15:30:26.1816230Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:30:26.1816749Z loading model: 0it [00:06, ?it/s]
2023-09-06T15:30:26.1817161Z Eager model failed to run
2023-09-06T15:30:26.1833532Z Traceback (most recent call last):
2023-09-06T15:30:26.1834295Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:30:26.1834930Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:30:26.1835583Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:30:26.1836132Z     return mod(*inputs)
2023-09-06T15:30:26.1837671Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:30:26.1838405Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:30:26.1839290Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:30:26.1839915Z     return forward_call(*args, **kwargs)
2023-09-06T15:30:26.1840783Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:30:26.1841389Z     return self.inference(batched_inputs)
2023-09-06T15:30:26.1842324Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:30:26.1843727Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:30:26.1844766Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:30:26.1845417Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:30:26.1846303Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:30:26.1846927Z     return forward_call(*args, **kwargs)
2023-09-06T15:30:26.1847862Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T15:30:26.1848659Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T15:30:26.1849640Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T15:30:26.1850488Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T15:30:26.1851537Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:30:26.1852203Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:30:26.1853077Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:30:26.1853669Z     return forward_call(*args, **kwargs)
2023-09-06T15:30:26.1854555Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T15:30:26.1855233Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T15:30:26.1856156Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:30:26.1856806Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:30:26.1858230Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:30:26.1858875Z     return forward_call(*args, **kwargs)
2023-09-06T15:30:26.1859746Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:30:26.1860312Z     return roi_align(
2023-09-06T15:30:26.1861112Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:30:26.1861886Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:30:26.1863081Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:30:26.1863863Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:30:26.1864948Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:30:26.1865660Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:30:26.1866647Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:30:26.1867230Z     return input[
2023-09-06T15:30:26.1868821Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 13521.00 GiB. GPU 0 has a total capacty of 39.39 GiB of which 35.88 GiB is free. Process 2187818 has 3.50 GiB memory in use. Of the allocated memory 2.68 GiB is allocated by PyTorch, and 270.78 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:30:26.1870554Z 
2023-09-06T15:30:26.1870901Z The above exception was the direct cause of the following exception:
2023-09-06T15:30:26.1871264Z 
2023-09-06T15:30:26.1871887Z Traceback (most recent call last):
2023-09-06T15:30:26.1872457Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:30:26.1872994Z     ) = runner.load_model(
2023-09-06T15:30:26.1873438Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:30:26.1874062Z     self.validate_model(model, example_inputs)
2023-09-06T15:30:26.1874670Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:30:26.1875359Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:30:26.1875987Z NotImplementedError: Eager model failed to run
2023-09-06T15:30:26.1876335Z 
2023-09-06T15:30:26.1876635Z WARNING:root:detectron2_fasterrcnn_r_50_dc5 failed to load
2023-09-06T15:30:30.8421157Z 
2023-09-06T15:30:35.1173122Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:30:35.1173838Z loading model: 0it [00:04, ?it/s]
2023-09-06T15:30:35.1174100Z Eager model failed to run
2023-09-06T15:30:35.1186498Z Traceback (most recent call last):
2023-09-06T15:30:35.1187176Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:30:35.1188063Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:30:35.1188755Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:30:35.1189508Z     return mod(*inputs)
2023-09-06T15:30:35.1191807Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:30:35.1192242Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:30:35.1192805Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:30:35.1193198Z     return forward_call(*args, **kwargs)
2023-09-06T15:30:35.1193773Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:30:35.1194663Z     return self.inference(batched_inputs)
2023-09-06T15:30:35.1195269Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:30:35.1195759Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:30:35.1196368Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:30:35.1196799Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:30:35.1197356Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:30:35.1197881Z     return forward_call(*args, **kwargs)
2023-09-06T15:30:35.1198464Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T15:30:35.1198916Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T15:30:35.1199572Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T15:30:35.1200146Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T15:30:35.1200794Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:30:35.1201215Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:30:35.1201755Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:30:35.1202154Z     return forward_call(*args, **kwargs)
2023-09-06T15:30:35.1202703Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2023-09-06T15:30:35.1203172Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2023-09-06T15:30:35.1203784Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:30:35.1204408Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:30:35.1204974Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:30:35.1205377Z     return forward_call(*args, **kwargs)
2023-09-06T15:30:35.1205926Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:30:35.1206286Z     return roi_align(
2023-09-06T15:30:35.1206818Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:30:35.1207338Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:30:35.1208051Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:30:35.1208558Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:30:35.1209237Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:30:35.1209659Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:30:35.1210208Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:30:35.1210590Z     return input[
2023-09-06T15:30:35.1211535Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 21352.77 GiB. GPU 0 has a total capacty of 39.39 GiB of which 36.12 GiB is free. Process 2187912 has 3.27 GiB memory in use. Of the allocated memory 2.65 GiB is allocated by PyTorch, and 99.25 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:30:35.1212602Z 
2023-09-06T15:30:35.1212905Z The above exception was the direct cause of the following exception:
2023-09-06T15:30:35.1213155Z 
2023-09-06T15:30:35.1213285Z Traceback (most recent call last):
2023-09-06T15:30:35.1213665Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:30:35.1214015Z     ) = runner.load_model(
2023-09-06T15:30:35.1214397Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:30:35.1214784Z     self.validate_model(model, example_inputs)
2023-09-06T15:30:35.1215198Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:30:35.1215639Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:30:35.1216023Z NotImplementedError: Eager model failed to run
2023-09-06T15:30:35.1216234Z 
2023-09-06T15:30:35.1216387Z WARNING:root:detectron2_fasterrcnn_r_50_fpn failed to load
2023-09-06T15:30:39.7098487Z 
2023-09-06T15:30:43.8581605Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:30:43.8582492Z loading model: 0it [00:04, ?it/s]
2023-09-06T15:30:43.8583010Z Traceback (most recent call last):
2023-09-06T15:30:43.8583691Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 495, in <module>
2023-09-06T15:30:43.8585518Z     torchbench_main()
2023-09-06T15:30:43.8586229Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 491, in torchbench_main
2023-09-06T15:30:43.8589930Z     main(TorchBenchmarkRunner(), original_dir)
2023-09-06T15:30:43.8590730Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3031, in main
2023-09-06T15:30:43.8596875Z     process_entry(0, runner, original_dir, args)
2023-09-06T15:30:43.8597686Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2988, in process_entry
2023-09-06T15:30:43.8602478Z     return maybe_fresh_cache(
2023-09-06T15:30:43.8603267Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1659, in inner
2023-09-06T15:30:43.8605262Z     return fn(*args, **kwargs)
2023-09-06T15:30:43.8605961Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3477, in run
2023-09-06T15:30:43.8613262Z     assert marked, f"nothing in example_inputs had a dim with {batch_size}"
2023-09-06T15:30:43.8613796Z AssertionError: nothing in example_inputs had a dim with 4
2023-09-06T15:30:44.9375164Z ERROR
2023-09-06T15:30:44.9417652Z accuracy            pass_rate=80.00%
2023-09-06T15:30:44.9419524Z calls_captured      gmean=0.00x mean=534.000x
2023-09-06T15:30:44.9422955Z unique_graphs       gmean=0.00x mean=1.867x
2023-09-06T15:30:44.9423563Z graph_breaks        gmean=0.00x mean=1.067x
2023-09-06T15:30:44.9427238Z unique_graph_breaks gmean=0.00x mean=0.267x
2023-09-06T15:30:45.7220827Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *cppwrapper-true* ]]
2023-09-06T15:30:45.7222237Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *freezing_cudagraphs-true* ]]
2023-09-06T15:30:45.7222986Z + [[ inference == \i\n\f\e\r\e\n\c\e ]]
2023-09-06T15:30:45.7224082Z + python benchmarks/dynamo/torchbench.py --accuracy --no-translation-validation --inference --bfloat16 --backend inductor --device cuda --total-partitions 4 --partition-id 0 --freezing --output /var/lib/jenkins/workspace/test/test-reports/inductor_with_cudagraphs_freezing_torchbench_bfloat16_inference_cuda_accuracy.csv
2023-09-06T15:30:52.9486575Z 
2023-09-06T15:30:53.8561050Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:30:53.8561576Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:30:53.8562009Z Eager model failed to run
2023-09-06T15:30:53.8578435Z Traceback (most recent call last):
2023-09-06T15:30:53.8579183Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:30:53.8579876Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:30:53.8581731Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:30:53.8582318Z     return mod(*inputs)
2023-09-06T15:30:53.8583725Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:30:53.8584419Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:30:53.8585269Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:30:53.8585939Z     return forward_call(*args, **kwargs)
2023-09-06T15:30:53.8586785Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 893, in forward
2023-09-06T15:30:53.8587460Z     logits = self.model(batch.dense_features, batch.sparse_features)
2023-09-06T15:30:53.8588632Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:30:53.8589646Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:30:53.8590606Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:30:53.8591211Z     return forward_call(*args, **kwargs)
2023-09-06T15:30:53.8592072Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 570, in forward
2023-09-06T15:30:53.8592716Z     embedded_dense = self.dense_arch(dense_features)
2023-09-06T15:30:53.8593672Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:30:53.8594327Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:30:53.8595220Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:30:53.8595865Z     return forward_call(*args, **kwargs)
2023-09-06T15:30:53.8596783Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 149, in forward
2023-09-06T15:30:53.8597950Z     return self.model(features)
2023-09-06T15:30:53.8598914Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:30:53.8599564Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:30:53.8600568Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:30:53.8601246Z     return forward_call(*args, **kwargs)
2023-09-06T15:30:53.8602170Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/mlp.py", line 172, in forward
2023-09-06T15:30:53.8602796Z     return self._mlp(input)
2023-09-06T15:30:53.8603764Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:30:53.8604478Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:30:53.8605493Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:30:53.8606137Z     return forward_call(*args, **kwargs)
2023-09-06T15:30:53.8607133Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/container.py", line 215, in forward
2023-09-06T15:30:53.8607816Z     input = module(input)
2023-09-06T15:30:53.8608909Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:30:53.8609663Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:30:53.8610680Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:30:53.8611385Z     return forward_call(*args, **kwargs)
2023-09-06T15:30:53.8612164Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/mlp.py", line 73, in forward
2023-09-06T15:30:53.8612842Z     return self._activation_fn(self._linear(input))
2023-09-06T15:30:53.8614390Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:30:53.8615206Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:30:53.8616228Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:30:53.8616867Z     return forward_call(*args, **kwargs)
2023-09-06T15:30:53.8617809Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
2023-09-06T15:30:53.8618648Z     return F.linear(input, self.weight, self.bias)
2023-09-06T15:30:53.8619326Z RuntimeError: mat1 and mat2 must have the same dtype, but got Float and BFloat16
2023-09-06T15:30:53.8619779Z 
2023-09-06T15:30:53.8620129Z The above exception was the direct cause of the following exception:
2023-09-06T15:30:53.8620540Z 
2023-09-06T15:30:53.8620761Z Traceback (most recent call last):
2023-09-06T15:30:53.8621412Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:30:53.8622031Z     ) = runner.load_model(
2023-09-06T15:30:53.8622632Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:30:53.8623269Z     self.validate_model(model, example_inputs)
2023-09-06T15:30:53.8623919Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:30:53.8624709Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:30:53.8625381Z NotImplementedError: Eager model failed to run
2023-09-06T15:30:53.8625762Z 
2023-09-06T15:30:53.8626008Z WARNING:root:torchrec_dlrm failed to load
2023-09-06T15:30:58.2937121Z 
2023-09-06T15:31:01.4060605Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:31:01.4061526Z loading model: 0it [00:03, ?it/s]
2023-09-06T15:31:01.4131716Z cuda eval  BERT_pytorch                       
2023-09-06T15:31:36.6846986Z pass
2023-09-06T15:31:41.8715045Z 
2023-09-06T15:31:45.5778460Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:31:45.5779689Z loading model: 0it [00:03, ?it/s]
2023-09-06T15:31:45.5833230Z cuda eval  Background_Matting                 
2023-09-06T15:31:45.5836208Z pass_due_to_skip
2023-09-06T15:31:50.2118262Z 
2023-09-06T15:31:59.5228819Z loading model: 0it [00:00, ?it/s]WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T15:32:01.2582301Z 
2023-09-06T15:32:01.2583213Z loading model: 0it [00:11, ?it/s]
2023-09-06T15:32:01.2583847Z WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T15:32:01.2584394Z cuda eval  DALLE2_pytorch                     
2023-09-06T15:32:01.4971878Z WARNING:common:fp64 golden ref were not generated for DALLE2_pytorch. Setting accuracy check to cosine
2023-09-06T15:32:01.8708346Z WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T15:32:15.1469554Z [2023-09-06 15:32:15,145] [3/1] torch._dynamo.output_graph: [WARNING] nn.Module forward/_pre hooks are only partially supported, and were detected in your model. In particular, if you do not change/remove hooks after calling .compile(), you can disregard this warning, and otherwise you may need to set torch._dynamo.config.skip_nnmodule_hook_guards=False to ensure recompiling after changing hooks.See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations. 
2023-09-06T15:32:48.9742816Z skipping cudagraphs due to multiple devices
2023-09-06T15:33:50.7006975Z skipping cudagraphs due to multiple devices
2023-09-06T15:34:11.3728394Z ERROR:common:Failed running call_function <function rearrange at 0x7fa0b57d9fc0>(*(FakeTensor(..., device='cuda:0', size=(4, 128, s0, s1)), 'b c ... -> b ... c'), **{}):
2023-09-06T15:34:11.3729555Z unhashable type: 'SymInt'
2023-09-06T15:34:11.3729843Z 
2023-09-06T15:34:11.3732422Z from user code:
2023-09-06T15:34:11.3733584Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 1670, in forward
2023-09-06T15:34:11.3735197Z     h = rearrange(h, 'b c ... -> b ... c')
2023-09-06T15:34:11.3735636Z 
2023-09-06T15:34:11.3735930Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T15:34:11.3736315Z 
2023-09-06T15:34:11.3741887Z 
2023-09-06T15:34:11.3742491Z You can suppress this exception and fall back to eager by setting:
2023-09-06T15:34:11.3743279Z     import torch._dynamo
2023-09-06T15:34:11.3743942Z     torch._dynamo.config.suppress_errors = True
2023-09-06T15:34:11.3744547Z Traceback (most recent call last):
2023-09-06T15:34:11.3745319Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2135, in check_accuracy
2023-09-06T15:34:11.3746156Z     new_result = optimized_model_iter_fn(model_copy, example_inputs)
2023-09-06T15:34:11.3747147Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T15:34:11.3747531Z     return fn(*args, **kwargs)
2023-09-06T15:34:11.3747938Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1899, in run_n_iterations
2023-09-06T15:34:11.3748375Z     self.model_iter_fn(mod, inputs, collect_outputs=False)
2023-09-06T15:34:11.3748790Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:34:11.3749441Z     return mod(*inputs)
2023-09-06T15:34:11.3750276Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:34:11.3750886Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:34:11.3751723Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:34:11.3752320Z     return forward_call(*args, **kwargs)
2023-09-06T15:34:11.3753302Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T15:34:11.3753957Z     return func(*args, **kwargs)
2023-09-06T15:34:11.3755331Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 95, in inner
2023-09-06T15:34:11.3755852Z     model.eval()
2023-09-06T15:34:11.3756679Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 96, in <resume in inner>
2023-09-06T15:34:11.3757318Z     out = fn(model, *args, **kwargs)
2023-09-06T15:34:11.3758241Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3329, in forward
2023-09-06T15:34:11.3759019Z     image_embed = self.prior.sample(text, num_samples_per_batch = self.prior_num_samples, cond_scale = prior_cond_scale)
2023-09-06T15:34:11.3759755Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3332, in <resume in forward>
2023-09-06T15:34:11.3760302Z     images = self.decoder.sample(image_embed = image_embed, text = text_cond, cond_scale = cond_scale)
2023-09-06T15:34:11.3760984Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T15:34:11.3761393Z     return func(*args, **kwargs)
2023-09-06T15:34:11.3761937Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 95, in inner
2023-09-06T15:34:11.3762296Z     model.eval()
2023-09-06T15:34:11.3762843Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 96, in <resume in inner>
2023-09-06T15:34:11.3763320Z     out = fn(model, *args, **kwargs)
2023-09-06T15:34:11.3763873Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3199, in sample
2023-09-06T15:34:11.3764257Z     img = self.p_sample_loop(
2023-09-06T15:34:11.3764804Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T15:34:11.3765202Z     return func(*args, **kwargs)
2023-09-06T15:34:11.3766035Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3028, in p_sample_loop
2023-09-06T15:34:11.3766545Z     return self.p_sample_loop_ddpm(*args, noise_scheduler = noise_scheduler, **kwargs)
2023-09-06T15:34:11.3767184Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T15:34:11.3767573Z     return func(*args, **kwargs)
2023-09-06T15:34:11.3768155Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2885, in p_sample_loop_ddpm
2023-09-06T15:34:11.3768569Z     img, x_start = self.p_sample(
2023-09-06T15:34:11.3769104Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T15:34:11.3769500Z     return func(*args, **kwargs)
2023-09-06T15:34:11.3770047Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2822, in p_sample
2023-09-06T15:34:11.3771076Z     model_mean, _, model_log_variance, x_start = self.p_mean_variance(unet, x = x, t = t, image_embed = image_embed, text_encodings = text_encodings, cond_scale = cond_scale, lowres_cond_img = lowres_cond_img, self_cond = self_cond, clip_denoised = clip_denoised, predict_x_start = predict_x_start, predict_v = predict_v, noise_scheduler = noise_scheduler, learned_variance = learned_variance, lowres_noise_level = lowres_noise_level)
2023-09-06T15:34:11.3772534Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2787, in p_mean_variance
2023-09-06T15:34:11.3773347Z     model_output = default(model_output, lambda: unet.forward_with_cond_scale(x, t, image_embed = image_embed, text_encodings = text_encodings, cond_scale = cond_scale, lowres_cond_img = lowres_cond_img, self_cond = self_cond, lowres_noise_level = lowres_noise_level))
2023-09-06T15:34:11.3774227Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 65, in default
2023-09-06T15:34:11.3774765Z     return d() if callable(d) else d
2023-09-06T15:34:11.3775334Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2787, in <lambda>
2023-09-06T15:34:11.3776099Z     model_output = default(model_output, lambda: unet.forward_with_cond_scale(x, t, image_embed = image_embed, text_encodings = text_encodings, cond_scale = cond_scale, lowres_cond_img = lowres_cond_img, self_cond = self_cond, lowres_noise_level = lowres_noise_level))
2023-09-06T15:34:11.3777024Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2159, in forward_with_cond_scale
2023-09-06T15:34:11.3777458Z     logits = self.forward(*args, **kwargs)
2023-09-06T15:34:11.3778032Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2340, in forward
2023-09-06T15:34:11.3778409Z     x = resnet_block(x, t, c)
2023-09-06T15:34:11.3778971Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:34:11.3779397Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:34:11.3779955Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:34:11.3780344Z     return forward_call(*args, **kwargs)
2023-09-06T15:34:11.3780899Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T15:34:11.3781346Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T15:34:11.3781946Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 634, in _convert_frame
2023-09-06T15:34:11.3782407Z     result = inner_convert(frame, cache_entry, hooks, frame_state)
2023-09-06T15:34:11.3783021Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T15:34:11.3783516Z     return fn(*args, **kwargs)
2023-09-06T15:34:11.3784097Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T15:34:11.3784490Z     return _compile(
2023-09-06T15:34:11.3785006Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T15:34:11.3785464Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T15:34:11.3786063Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T15:34:11.3786445Z     r = func(*args, **kwargs)
2023-09-06T15:34:11.3786983Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T15:34:11.3787422Z     out_code = transform_code_object(code, transform)
2023-09-06T15:34:11.3788078Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T15:34:11.3788557Z     transformations(instructions, code_options)
2023-09-06T15:34:11.3789349Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T15:34:11.3789855Z     tracer.run()
2023-09-06T15:34:11.3790394Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T15:34:11.3790768Z     super().run()
2023-09-06T15:34:11.3791282Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T15:34:11.3791641Z     and self.step()
2023-09-06T15:34:11.3792174Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T15:34:11.3792572Z     getattr(self, inst.opname)(inst)
2023-09-06T15:34:11.3793202Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T15:34:11.3793773Z     return inner_fn(self, inst)
2023-09-06T15:34:11.3794335Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1119, in CALL_FUNCTION
2023-09-06T15:34:11.3794751Z     self.call_function(fn, args, {})
2023-09-06T15:34:11.3795324Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 565, in call_function
2023-09-06T15:34:11.3795770Z     self.push(fn.call_function(self, args, kwargs))
2023-09-06T15:34:11.3796360Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/torch.py", line 731, in call_function
2023-09-06T15:34:11.3796791Z     tensor_variable = wrap_fx_proxy(
2023-09-06T15:34:11.3797378Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1216, in wrap_fx_proxy
2023-09-06T15:34:11.3797787Z     return wrap_fx_proxy_cls(
2023-09-06T15:34:11.3798394Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1303, in wrap_fx_proxy_cls
2023-09-06T15:34:11.3798837Z     example_value = get_fake_value(proxy.node, tx)
2023-09-06T15:34:11.3799421Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1381, in get_fake_value
2023-09-06T15:34:11.3799898Z     raise TorchRuntimeError(str(e)).with_traceback(e.__traceback__) from None
2023-09-06T15:34:11.3800512Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1342, in get_fake_value
2023-09-06T15:34:11.3800887Z     return wrap_fake_exception(
2023-09-06T15:34:11.3801445Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 917, in wrap_fake_exception
2023-09-06T15:34:11.3801820Z     return fn()
2023-09-06T15:34:11.3802326Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1343, in <lambda>
2023-09-06T15:34:11.3802967Z     lambda: run_node(tx.output, node, args, kwargs, nnmodule)
2023-09-06T15:34:11.3803548Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1415, in run_node
2023-09-06T15:34:11.3804015Z     raise RuntimeError(fn_str + str(e)).with_traceback(e.__traceback__) from e
2023-09-06T15:34:11.3804616Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1402, in run_node
2023-09-06T15:34:11.3805011Z     return node.target(*args, **kwargs)
2023-09-06T15:34:11.3805532Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/einops/einops.py", line 483, in rearrange
2023-09-06T15:34:11.3806137Z     return reduce(cast(Tensor, tensor), pattern, reduction='rearrange', **axes_lengths)
2023-09-06T15:34:11.3806734Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/einops/einops.py", line 412, in reduce
2023-09-06T15:34:11.3807176Z     return _apply_recipe(recipe, tensor, reduction_type=reduction)
2023-09-06T15:34:11.3807779Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/einops/einops.py", line 235, in _apply_recipe
2023-09-06T15:34:11.3808206Z     _reconstruct_from_shape(recipe, backend.shape(tensor))
2023-09-06T15:34:11.3809010Z torch._dynamo.exc.TorchRuntimeError: Failed running call_function <function rearrange at 0x7fa0b57d9fc0>(*(FakeTensor(..., device='cuda:0', size=(4, 128, s0, s1)), 'b c ... -> b ... c'), **{}):
2023-09-06T15:34:11.3809571Z unhashable type: 'SymInt'
2023-09-06T15:34:11.3809747Z 
2023-09-06T15:34:11.3809843Z from user code:
2023-09-06T15:34:11.3810369Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 1670, in forward
2023-09-06T15:34:11.3810860Z     h = rearrange(h, 'b c ... -> b ... c')
2023-09-06T15:34:11.3811047Z 
2023-09-06T15:34:11.3811239Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T15:34:11.3811484Z 
2023-09-06T15:34:11.3811490Z 
2023-09-06T15:34:11.3811693Z You can suppress this exception and fall back to eager by setting:
2023-09-06T15:34:11.3812177Z     import torch._dynamo
2023-09-06T15:34:11.3812482Z     torch._dynamo.config.suppress_errors = True
2023-09-06T15:34:11.3812687Z 
2023-09-06T15:34:11.3812933Z TorchDynamo optimized model failed to run because of following error
2023-09-06T15:34:11.3813261Z fail_to_run
2023-09-06T15:34:17.6223696Z 
2023-09-06T15:34:21.4637713Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:34:21.4638630Z loading model: 0it [00:03, ?it/s]
2023-09-06T15:34:21.4664588Z cuda eval  LearningToPaint                    
2023-09-06T15:34:37.8172342Z pass
2023-09-06T15:34:43.1248170Z 
2023-09-06T15:34:45.5412592Z loading model: 0it [00:00, ?it/s]WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T15:34:46.0638031Z 
2023-09-06T15:34:46.0643044Z loading model: 0it [00:02, ?it/s]
2023-09-06T15:34:46.0643516Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T15:34:46.0643970Z cuda eval  Super_SloMo                        
2023-09-06T15:34:46.1673203Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T15:35:37.6661836Z pass
2023-09-06T15:35:43.3047087Z 
2023-09-06T15:35:44.8175717Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:35:44.8176136Z loading model: 0it [00:01, ?it/s]
2023-09-06T15:35:44.8177998Z cuda eval  alexnet                            
2023-09-06T15:35:55.6124156Z pass
2023-09-06T15:36:00.5494799Z 
2023-09-06T15:36:07.3949295Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:36:07.3949657Z loading model: 0it [00:06, ?it/s]
2023-09-06T15:36:07.4005491Z cuda eval  attention_is_all_you_need_pytorch  
2023-09-06T15:36:44.3589794Z pass
2023-09-06T15:36:49.7816821Z 
2023-09-06T15:36:52.4786708Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:36:52.4787278Z loading model: 0it [00:02, ?it/s]
2023-09-06T15:36:52.4792314Z cuda eval  basic_gnn_edgecnn                  
2023-09-06T15:36:53.0444191Z [2023-09-06 15:36:53,043] [1/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T15:36:59.4301388Z skipping cudagraphs for unknown reason
2023-09-06T15:36:59.7062795Z pass
2023-09-06T15:37:04.5130586Z 
2023-09-06T15:37:07.4151324Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:37:07.4151892Z loading model: 0it [00:02, ?it/s]
2023-09-06T15:37:07.4154636Z cuda eval  basic_gnn_gcn                      
2023-09-06T15:37:15.5797503Z skipping cudagraphs for unknown reason
2023-09-06T15:37:15.5983762Z [2023-09-06 15:37:15,597] [11/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T15:37:16.9195583Z skipping cudagraphs for unknown reason
2023-09-06T15:37:18.6262081Z skipping cudagraphs for unknown reason
2023-09-06T15:37:19.0651388Z skipping cudagraphs for unknown reason
2023-09-06T15:37:19.1696043Z pass
2023-09-06T15:37:23.9729061Z 
2023-09-06T15:37:27.0122030Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:37:27.0122727Z loading model: 0it [00:03, ?it/s]
2023-09-06T15:37:27.0129778Z cuda eval  basic_gnn_gin                      
2023-09-06T15:37:27.7374405Z [2023-09-06 15:37:27,736] [1/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T15:37:34.1017977Z skipping cudagraphs for unknown reason
2023-09-06T15:37:34.4303418Z pass
2023-09-06T15:37:39.3050199Z 
2023-09-06T15:37:42.3117630Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:37:42.3119526Z loading model: 0it [00:03, ?it/s]
2023-09-06T15:37:42.3136486Z cuda eval  basic_gnn_sage                     
2023-09-06T15:37:43.0148955Z [2023-09-06 15:37:43,013] [1/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T15:37:49.2557934Z skipping cudagraphs for unknown reason
2023-09-06T15:37:49.5912136Z pass
2023-09-06T15:37:54.3593916Z 
2023-09-06T15:37:59.6294403Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:37:59.6295085Z loading model: 0it [00:05, ?it/s]
2023-09-06T15:37:59.6406988Z cuda eval  cm3leon_generate                   
2023-09-06T15:42:53.7919553Z pass
2023-09-06T15:43:00.2252284Z 
2023-09-06T15:43:01.0947448Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:43:01.0947806Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:43:01.0953548Z cuda eval  dcgan                              
2023-09-06T15:43:10.1461804Z pass
2023-09-06T15:43:15.0196185Z 
2023-09-06T15:43:18.3378100Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:43:18.3378858Z loading model: 0it [00:03, ?it/s]
2023-09-06T15:43:18.3379140Z Eager model failed to run
2023-09-06T15:43:18.3388790Z Traceback (most recent call last):
2023-09-06T15:43:18.3389808Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:43:18.3390465Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:43:18.3391180Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:43:18.3391862Z     return mod(*inputs)
2023-09-06T15:43:18.3393291Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:43:18.3393734Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:43:18.3394969Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:43:18.3395434Z     return forward_call(*args, **kwargs)
2023-09-06T15:43:18.3395887Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/__init__.py", line 31, in forward
2023-09-06T15:43:18.3396297Z     return sources, self.model(mix)
2023-09-06T15:43:18.3396879Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:43:18.3397301Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:43:18.3397858Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:43:18.3398262Z     return forward_call(*args, **kwargs)
2023-09-06T15:43:18.3398682Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/demucs/model.py", line 209, in forward
2023-09-06T15:43:18.3399071Z     x = self.lstm(x)
2023-09-06T15:43:18.3399622Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:43:18.3400055Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:43:18.3400593Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:43:18.3400996Z     return forward_call(*args, **kwargs)
2023-09-06T15:43:18.3401426Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/demucs/model.py", line 27, in forward
2023-09-06T15:43:18.3401823Z     x = self.lstm(x)[0]
2023-09-06T15:43:18.3402368Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:43:18.3402769Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:43:18.3403370Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:43:18.3403773Z     return forward_call(*args, **kwargs)
2023-09-06T15:43:18.3404602Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/rnn.py", line 879, in forward
2023-09-06T15:43:18.3405061Z     result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
2023-09-06T15:43:18.3405599Z RuntimeError: "_thnn_fused_lstm_cell_cuda" not implemented for 'BFloat16'
2023-09-06T15:43:18.3405846Z 
2023-09-06T15:43:18.3406047Z The above exception was the direct cause of the following exception:
2023-09-06T15:43:18.3406288Z 
2023-09-06T15:43:18.3406415Z Traceback (most recent call last):
2023-09-06T15:43:18.3406798Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:43:18.3407132Z     ) = runner.load_model(
2023-09-06T15:43:18.3407510Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:43:18.3407916Z     self.validate_model(model, example_inputs)
2023-09-06T15:43:18.3408331Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:43:18.3408772Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:43:18.3409159Z NotImplementedError: Eager model failed to run
2023-09-06T15:43:18.3409368Z 
2023-09-06T15:43:18.3409494Z WARNING:root:demucs failed to load
2023-09-06T15:43:22.7970784Z 
2023-09-06T15:43:24.6769120Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:43:24.6769489Z loading model: 0it [00:01, ?it/s]
2023-09-06T15:43:24.6921754Z cuda eval  densenet121                        
2023-09-06T15:44:47.2119591Z [2023-09-06 15:44:47,210] torch._dynamo.utils: [ERROR] RMSE (res-fp64): 0.01925, (ref-fp64): 0.00878 and shape=torch.Size([4, 1000])
2023-09-06T15:44:47.2283614Z fail_accuracy
2023-09-06T15:44:52.9984407Z 
2023-09-06T15:44:57.7860106Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:44:57.7860609Z loading model: 0it [00:04, ?it/s]
2023-09-06T15:44:57.7861028Z Eager model failed to run
2023-09-06T15:44:57.7876854Z Traceback (most recent call last):
2023-09-06T15:44:57.7878116Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:44:57.7879048Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:44:57.7879782Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:44:57.7880385Z     return mod(*inputs)
2023-09-06T15:44:57.7881149Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:44:57.7881582Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:44:57.7882157Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:44:57.7882645Z     return forward_call(*args, **kwargs)
2023-09-06T15:44:57.7883540Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:44:57.7884121Z     return self.inference(batched_inputs)
2023-09-06T15:44:57.7885712Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:44:57.7886231Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:44:57.7886893Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:44:57.7887336Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:44:57.7887908Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:44:57.7888312Z     return forward_call(*args, **kwargs)
2023-09-06T15:44:57.7888941Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2023-09-06T15:44:57.7889384Z     box_features = self._shared_roi_transform(
2023-09-06T15:44:57.7917348Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2023-09-06T15:44:57.7919018Z     x = self.pooler(features, boxes)
2023-09-06T15:44:57.7920036Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:44:57.7920471Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:44:57.7921023Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:44:57.7921615Z     return forward_call(*args, **kwargs)
2023-09-06T15:44:57.7922482Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T15:44:57.7923091Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T15:44:57.7923701Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:44:57.7924140Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:44:57.7924722Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:44:57.7925129Z     return forward_call(*args, **kwargs)
2023-09-06T15:44:57.7925674Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:44:57.7926048Z     return roi_align(
2023-09-06T15:44:57.7926564Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:44:57.7927082Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:44:57.7927725Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:44:57.7928236Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:44:57.7929252Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:44:57.7929684Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:44:57.7930236Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:44:57.7930620Z     return input[
2023-09-06T15:44:57.7931582Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 26902.82 GiB. GPU 0 has a total capacty of 39.39 GiB of which 35.54 GiB is free. Process 2190647 has 3.84 GiB memory in use. Of the allocated memory 2.99 GiB is allocated by PyTorch, and 335.94 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:44:57.7932523Z 
2023-09-06T15:44:57.7932724Z The above exception was the direct cause of the following exception:
2023-09-06T15:44:57.7932969Z 
2023-09-06T15:44:57.7933109Z Traceback (most recent call last):
2023-09-06T15:44:57.7933477Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:44:57.7933823Z     ) = runner.load_model(
2023-09-06T15:44:57.7934199Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:44:57.7934602Z     self.validate_model(model, example_inputs)
2023-09-06T15:44:57.7935000Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:44:57.7935442Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:44:57.7935822Z NotImplementedError: Eager model failed to run
2023-09-06T15:44:57.7936030Z 
2023-09-06T15:44:57.7936194Z WARNING:root:detectron2_fasterrcnn_r_101_c4 failed to load
2023-09-06T15:45:02.4978826Z 
2023-09-06T15:45:09.4974589Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:45:09.4974952Z loading model: 0it [00:06, ?it/s]
2023-09-06T15:45:09.4975407Z Eager model failed to run
2023-09-06T15:45:09.4988927Z Traceback (most recent call last):
2023-09-06T15:45:09.4989761Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:45:09.4990518Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:45:09.4991391Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:45:09.4991983Z     return mod(*inputs)
2023-09-06T15:45:09.4993465Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:45:09.4993976Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:45:09.4994552Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:45:09.4994964Z     return forward_call(*args, **kwargs)
2023-09-06T15:45:09.4995535Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:45:09.4995979Z     return self.inference(batched_inputs)
2023-09-06T15:45:09.4996561Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:45:09.4997041Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:45:09.4997658Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:45:09.4998069Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:45:09.4998629Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:45:09.4999034Z     return forward_call(*args, **kwargs)
2023-09-06T15:45:09.4999640Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T15:45:09.5000148Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T15:45:09.5001183Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T15:45:09.5001697Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T15:45:09.5002352Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:45:09.5002774Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:45:09.5003329Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:45:09.5003718Z     return forward_call(*args, **kwargs)
2023-09-06T15:45:09.5004375Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T15:45:09.5004811Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T15:45:09.5005412Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:45:09.5005875Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:45:09.5006437Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:45:09.5006836Z     return forward_call(*args, **kwargs)
2023-09-06T15:45:09.5007389Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:45:09.5007761Z     return roi_align(
2023-09-06T15:45:09.5008272Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:45:09.5008794Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:45:09.5009445Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:45:09.5009966Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:45:09.5010927Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:45:09.5011332Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:45:09.5011881Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:45:09.5012264Z     return input[
2023-09-06T15:45:09.5013236Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 13299.95 GiB. GPU 0 has a total capacty of 39.39 GiB of which 34.73 GiB is free. Process 2190754 has 4.65 GiB memory in use. Of the allocated memory 3.82 GiB is allocated by PyTorch, and 283.52 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:45:09.5014337Z 
2023-09-06T15:45:09.5014550Z The above exception was the direct cause of the following exception:
2023-09-06T15:45:09.5014796Z 
2023-09-06T15:45:09.5014924Z Traceback (most recent call last):
2023-09-06T15:45:09.5015308Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:45:09.5015661Z     ) = runner.load_model(
2023-09-06T15:45:09.5016040Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:45:09.5016475Z     self.validate_model(model, example_inputs)
2023-09-06T15:45:09.5016889Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:45:09.5017318Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:45:09.5017704Z NotImplementedError: Eager model failed to run
2023-09-06T15:45:09.5017911Z 
2023-09-06T15:45:09.5018081Z WARNING:root:detectron2_fasterrcnn_r_101_dc5 failed to load
2023-09-06T15:45:14.1549301Z 
2023-09-06T15:45:18.9109046Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:45:18.9110291Z loading model: 0it [00:04, ?it/s]
2023-09-06T15:45:18.9110586Z Eager model failed to run
2023-09-06T15:45:18.9127592Z Traceback (most recent call last):
2023-09-06T15:45:18.9128433Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:45:18.9129153Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:45:18.9130025Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:45:18.9130642Z     return mod(*inputs)
2023-09-06T15:45:18.9133131Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:45:18.9133578Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:45:18.9134154Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:45:18.9134584Z     return forward_call(*args, **kwargs)
2023-09-06T15:45:18.9135226Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:45:18.9135658Z     return self.inference(batched_inputs)
2023-09-06T15:45:18.9136256Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:45:18.9136750Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:45:18.9137380Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:45:18.9137808Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:45:18.9138355Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:45:18.9138771Z     return forward_call(*args, **kwargs)
2023-09-06T15:45:18.9139361Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T15:45:18.9140271Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T15:45:18.9140908Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T15:45:18.9141435Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T15:45:18.9142097Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:45:18.9142521Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:45:18.9143139Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:45:18.9143540Z     return forward_call(*args, **kwargs)
2023-09-06T15:45:18.9144101Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2023-09-06T15:45:18.9144582Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2023-09-06T15:45:18.9145205Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:45:18.9145653Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:45:18.9146218Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:45:18.9146645Z     return forward_call(*args, **kwargs)
2023-09-06T15:45:18.9147212Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:45:18.9147606Z     return roi_align(
2023-09-06T15:45:18.9148123Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:45:18.9148659Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:45:18.9149740Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:45:18.9150278Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:45:18.9150975Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:45:18.9151384Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:45:18.9151943Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:45:18.9152336Z     return input[
2023-09-06T15:45:18.9153364Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 19183.01 GiB. GPU 0 has a total capacty of 39.39 GiB of which 34.81 GiB is free. Process 2190847 has 4.57 GiB memory in use. Of the allocated memory 3.79 GiB is allocated by PyTorch, and 257.07 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:45:18.9154858Z 
2023-09-06T15:45:18.9155066Z The above exception was the direct cause of the following exception:
2023-09-06T15:45:18.9155317Z 
2023-09-06T15:45:18.9155444Z Traceback (most recent call last):
2023-09-06T15:45:18.9155811Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:45:18.9156165Z     ) = runner.load_model(
2023-09-06T15:45:18.9156547Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:45:18.9156952Z     self.validate_model(model, example_inputs)
2023-09-06T15:45:18.9157369Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:45:18.9157807Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:45:18.9158194Z NotImplementedError: Eager model failed to run
2023-09-06T15:45:18.9158583Z 
2023-09-06T15:45:18.9158786Z WARNING:root:detectron2_fasterrcnn_r_101_fpn failed to load
2023-09-06T15:45:23.5732842Z 
2023-09-06T15:45:27.8769606Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:45:27.8770195Z loading model: 0it [00:04, ?it/s]
2023-09-06T15:45:27.8770633Z Eager model failed to run
2023-09-06T15:45:27.8782198Z Traceback (most recent call last):
2023-09-06T15:45:27.8782680Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:45:27.8783146Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:45:27.8783936Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:45:27.8784539Z     return mod(*inputs)
2023-09-06T15:45:27.8785783Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:45:27.8786545Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:45:27.8787569Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:45:27.8788322Z     return forward_call(*args, **kwargs)
2023-09-06T15:45:27.8789479Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:45:27.8790268Z     return self.inference(batched_inputs)
2023-09-06T15:45:27.8791408Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:45:27.8791898Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:45:27.8792531Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:45:27.8792939Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:45:27.8793548Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:45:27.8794426Z     return forward_call(*args, **kwargs)
2023-09-06T15:45:27.8795042Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2023-09-06T15:45:27.8795479Z     box_features = self._shared_roi_transform(
2023-09-06T15:45:27.8796098Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2023-09-06T15:45:27.8796540Z     x = self.pooler(features, boxes)
2023-09-06T15:45:27.8797111Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:45:27.8797530Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:45:27.8798068Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:45:27.8798468Z     return forward_call(*args, **kwargs)
2023-09-06T15:45:27.8799023Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T15:45:27.8799467Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T15:45:27.8800057Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:45:27.8800493Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:45:27.8801049Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:45:27.8801456Z     return forward_call(*args, **kwargs)
2023-09-06T15:45:27.8802001Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:45:27.8802358Z     return roi_align(
2023-09-06T15:45:27.8802884Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:45:27.8803461Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:45:27.8804338Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:45:27.8804852Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:45:27.8805509Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:45:27.8805927Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:45:27.8806474Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:45:27.8806854Z     return input[
2023-09-06T15:45:27.8807823Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 27492.29 GiB. GPU 0 has a total capacty of 39.39 GiB of which 36.78 GiB is free. Process 2190939 has 2.61 GiB memory in use. Of the allocated memory 1.86 GiB is allocated by PyTorch, and 229.54 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:45:27.8808981Z 
2023-09-06T15:45:27.8809181Z The above exception was the direct cause of the following exception:
2023-09-06T15:45:27.8809424Z 
2023-09-06T15:45:27.8809550Z Traceback (most recent call last):
2023-09-06T15:45:27.8809915Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:45:27.8810263Z     ) = runner.load_model(
2023-09-06T15:45:27.8810645Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:45:27.8811048Z     self.validate_model(model, example_inputs)
2023-09-06T15:45:27.8811442Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:45:27.8811887Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:45:27.8812389Z NotImplementedError: Eager model failed to run
2023-09-06T15:45:27.8812611Z 
2023-09-06T15:45:27.8812776Z WARNING:root:detectron2_fasterrcnn_r_50_c4 failed to load
2023-09-06T15:45:32.5539458Z 
2023-09-06T15:45:39.1030666Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:45:39.1031371Z loading model: 0it [00:06, ?it/s]
2023-09-06T15:45:39.1031649Z Eager model failed to run
2023-09-06T15:45:39.1044691Z Traceback (most recent call last):
2023-09-06T15:45:39.1045436Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:45:39.1046126Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:45:39.1046767Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:45:39.1047386Z     return mod(*inputs)
2023-09-06T15:45:39.1048553Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:45:39.1049223Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:45:39.1049798Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:45:39.1050204Z     return forward_call(*args, **kwargs)
2023-09-06T15:45:39.1050758Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:45:39.1051192Z     return self.inference(batched_inputs)
2023-09-06T15:45:39.1051780Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:45:39.1052255Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:45:39.1052873Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:45:39.1053280Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:45:39.1053842Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:45:39.1054745Z     return forward_call(*args, **kwargs)
2023-09-06T15:45:39.1055343Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T15:45:39.1055797Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T15:45:39.1056437Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T15:45:39.1056960Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T15:45:39.1057606Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:45:39.1058131Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:45:39.1058695Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:45:39.1059116Z     return forward_call(*args, **kwargs)
2023-09-06T15:45:39.1059679Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T15:45:39.1060120Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T15:45:39.1060715Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:45:39.1061125Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:45:39.1061681Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:45:39.1062083Z     return forward_call(*args, **kwargs)
2023-09-06T15:45:39.1062784Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:45:39.1063150Z     return roi_align(
2023-09-06T15:45:39.1063881Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:45:39.1064416Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:45:39.1065080Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:45:39.1065602Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:45:39.1066266Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:45:39.1066682Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:45:39.1067236Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:45:39.1067621Z     return input[
2023-09-06T15:45:39.1068633Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 13521.00 GiB. GPU 0 has a total capacty of 39.39 GiB of which 35.88 GiB is free. Process 2191037 has 3.50 GiB memory in use. Of the allocated memory 2.68 GiB is allocated by PyTorch, and 270.78 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:45:39.1069907Z 
2023-09-06T15:45:39.1070112Z The above exception was the direct cause of the following exception:
2023-09-06T15:45:39.1070347Z 
2023-09-06T15:45:39.1070479Z Traceback (most recent call last):
2023-09-06T15:45:39.1070864Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:45:39.1071247Z     ) = runner.load_model(
2023-09-06T15:45:39.1071627Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:45:39.1072017Z     self.validate_model(model, example_inputs)
2023-09-06T15:45:39.1072444Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:45:39.1073078Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:45:39.1073468Z NotImplementedError: Eager model failed to run
2023-09-06T15:45:39.1073679Z 
2023-09-06T15:45:39.1073845Z WARNING:root:detectron2_fasterrcnn_r_50_dc5 failed to load
2023-09-06T15:45:43.7477571Z 
2023-09-06T15:45:47.8803128Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:45:47.8803496Z loading model: 0it [00:04, ?it/s]
2023-09-06T15:45:47.8803778Z Eager model failed to run
2023-09-06T15:45:47.8821701Z Traceback (most recent call last):
2023-09-06T15:45:47.8822280Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:45:47.8822987Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:45:47.8823823Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:45:47.8824459Z     return mod(*inputs)
2023-09-06T15:45:47.8826793Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:45:47.8827263Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:45:47.8827832Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:45:47.8828299Z     return forward_call(*args, **kwargs)
2023-09-06T15:45:47.8828878Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:45:47.8829516Z     return self.inference(batched_inputs)
2023-09-06T15:45:47.8830096Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:45:47.8830575Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:45:47.8831191Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:45:47.8832075Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:45:47.8832629Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:45:47.8833036Z     return forward_call(*args, **kwargs)
2023-09-06T15:45:47.8833622Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T15:45:47.8834092Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T15:45:47.8834732Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T15:45:47.8835232Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T15:45:47.8835882Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:45:47.8836302Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:45:47.8836874Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:45:47.8837260Z     return forward_call(*args, **kwargs)
2023-09-06T15:45:47.8837817Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2023-09-06T15:45:47.8838343Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2023-09-06T15:45:47.8838968Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:45:47.8839390Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:45:47.8839924Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:45:47.8840324Z     return forward_call(*args, **kwargs)
2023-09-06T15:45:47.8840873Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:45:47.8841474Z     return roi_align(
2023-09-06T15:45:47.8841997Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:45:47.8842514Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:45:47.8843168Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:45:47.8843685Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:45:47.8844358Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:45:47.8844762Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:45:47.8845317Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:45:47.8845742Z     return input[
2023-09-06T15:45:47.8846720Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 21352.77 GiB. GPU 0 has a total capacty of 39.39 GiB of which 36.12 GiB is free. Process 2191129 has 3.27 GiB memory in use. Of the allocated memory 2.65 GiB is allocated by PyTorch, and 99.25 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:45:47.8847556Z 
2023-09-06T15:45:47.8847754Z The above exception was the direct cause of the following exception:
2023-09-06T15:45:47.8847998Z 
2023-09-06T15:45:47.8848126Z Traceback (most recent call last):
2023-09-06T15:45:47.8848543Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:45:47.8848876Z     ) = runner.load_model(
2023-09-06T15:45:47.8849258Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:45:47.8849789Z     self.validate_model(model, example_inputs)
2023-09-06T15:45:47.8850217Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:45:47.8850641Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:45:47.8851026Z NotImplementedError: Eager model failed to run
2023-09-06T15:45:47.8851235Z 
2023-09-06T15:45:47.8851404Z WARNING:root:detectron2_fasterrcnn_r_50_fpn failed to load
2023-09-06T15:45:52.5029963Z 
2023-09-06T15:45:56.7313596Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:45:56.7314006Z loading model: 0it [00:04, ?it/s]
2023-09-06T15:45:56.7402649Z cuda eval  detectron2_fcos_r_50_fpn           
2023-09-06T15:46:11.2848342Z skipping cudagraphs for unknown reason
2023-09-06T15:46:13.0789264Z skipping cudagraphs due to input mutation
2023-09-06T15:46:58.0796529Z [2023-09-06 15:46:58,077] torch._dynamo.convert_frame: [WARNING] torch._dynamo hit config.cache_size_limit (8)
2023-09-06T15:46:58.0797470Z [2023-09-06 15:46:58,077] torch._dynamo.convert_frame: [WARNING]    function: 'forward' (/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/batch_norm.py:318)
2023-09-06T15:46:58.0798952Z [2023-09-06 15:46:58,077] torch._dynamo.convert_frame: [WARNING] to diagnose recompilation issues, set env variable TORCHDYNAMO_REPORT_GUARD_FAILURES=1 and also see https://pytorch.org/docs/master/compile/troubleshooting.html.
2023-09-06T15:47:13.3502477Z skipping cudagraphs due to input mutation
2023-09-06T15:47:13.5422746Z skipping cudagraphs for unknown reason
2023-09-06T15:47:14.6348137Z skipping cudagraphs due to input mutation
2023-09-06T15:47:14.7599089Z skipping cudagraphs for unknown reason
2023-09-06T15:47:15.7983688Z skipping cudagraphs due to input mutation
2023-09-06T15:47:16.8376765Z skipping cudagraphs due to input mutation
2023-09-06T15:48:02.1699364Z skipping cudagraphs due to input mutation
2023-09-06T15:48:02.2082692Z pass
2023-09-06T15:48:04.5999697Z accuracy            pass_rate=80.00%
2023-09-06T15:48:04.6001084Z calls_captured      gmean=0.00x mean=569.800x
2023-09-06T15:48:04.6003286Z unique_graphs       gmean=0.00x mean=7.800x
2023-09-06T15:48:04.6004857Z graph_breaks        gmean=0.00x mean=5.067x
2023-09-06T15:48:04.6006991Z unique_graph_breaks gmean=0.00x mean=1.067x
2023-09-06T15:48:05.3594044Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *aotinductor-true* ]]
2023-09-06T15:48:05.3595053Z + [[ inference == \i\n\f\e\r\e\n\c\e ]]
2023-09-06T15:48:05.3596698Z + python benchmarks/dynamo/torchbench.py --accuracy --no-translation-validation --inference --bfloat16 --export-aot-inductor --disable-cudagraphs --device cuda --total-partitions 4 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inductor_aot_inductor_torchbench_bfloat16_inference_cuda_accuracy.csv
2023-09-06T15:48:12.6141759Z 
2023-09-06T15:48:13.5346627Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:48:13.5347209Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:48:13.5347506Z Eager model failed to run
2023-09-06T15:48:13.5362366Z Traceback (most recent call last):
2023-09-06T15:48:13.5362962Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:48:13.5363523Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:48:13.5364238Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:48:13.5364714Z     return mod(*inputs)
2023-09-06T15:48:13.5366612Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:48:13.5370744Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:48:13.5371854Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:48:13.5372278Z     return forward_call(*args, **kwargs)
2023-09-06T15:48:13.5373351Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 893, in forward
2023-09-06T15:48:13.5373851Z     logits = self.model(batch.dense_features, batch.sparse_features)
2023-09-06T15:48:13.5374493Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:48:13.5374906Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:48:13.5375460Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:48:13.5375867Z     return forward_call(*args, **kwargs)
2023-09-06T15:48:13.5376403Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 570, in forward
2023-09-06T15:48:13.5376822Z     embedded_dense = self.dense_arch(dense_features)
2023-09-06T15:48:13.5377417Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:48:13.5377841Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:48:13.5378420Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:48:13.5378819Z     return forward_call(*args, **kwargs)
2023-09-06T15:48:13.5379344Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 149, in forward
2023-09-06T15:48:13.5379730Z     return self.model(features)
2023-09-06T15:48:13.5380283Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:48:13.5380704Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:48:13.5381238Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:48:13.5381636Z     return forward_call(*args, **kwargs)
2023-09-06T15:48:13.5382168Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/mlp.py", line 172, in forward
2023-09-06T15:48:13.5382821Z     return self._mlp(input)
2023-09-06T15:48:13.5383381Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:48:13.5383781Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:48:13.5384339Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:48:13.5384744Z     return forward_call(*args, **kwargs)
2023-09-06T15:48:13.5385292Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/container.py", line 215, in forward
2023-09-06T15:48:13.5385660Z     input = module(input)
2023-09-06T15:48:13.5386204Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:48:13.5386622Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:48:13.5387181Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:48:13.5387570Z     return forward_call(*args, **kwargs)
2023-09-06T15:48:13.5388101Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/mlp.py", line 73, in forward
2023-09-06T15:48:13.5388520Z     return self._activation_fn(self._linear(input))
2023-09-06T15:48:13.5389323Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:48:13.5389747Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:48:13.5390285Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:48:13.5390684Z     return forward_call(*args, **kwargs)
2023-09-06T15:48:13.5391225Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
2023-09-06T15:48:13.5391659Z     return F.linear(input, self.weight, self.bias)
2023-09-06T15:48:13.5392197Z RuntimeError: mat1 and mat2 must have the same dtype, but got Float and BFloat16
2023-09-06T15:48:13.5392479Z 
2023-09-06T15:48:13.5392722Z The above exception was the direct cause of the following exception:
2023-09-06T15:48:13.5392966Z 
2023-09-06T15:48:13.5393092Z Traceback (most recent call last):
2023-09-06T15:48:13.5393469Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:48:13.5393846Z     ) = runner.load_model(
2023-09-06T15:48:13.5394207Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:48:13.5394623Z     self.validate_model(model, example_inputs)
2023-09-06T15:48:13.5395032Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:48:13.5395475Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:48:13.5395845Z NotImplementedError: Eager model failed to run
2023-09-06T15:48:13.5396057Z 
2023-09-06T15:48:13.5396192Z WARNING:root:torchrec_dlrm failed to load
2023-09-06T15:48:18.0133566Z 
2023-09-06T15:48:21.1079893Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:48:21.1080808Z loading model: 0it [00:03, ?it/s]
2023-09-06T15:48:21.1142959Z cuda eval  BERT_pytorch                       
2023-09-06T15:48:23.1491502Z ERROR:common:Mutating module attribute mask during export.
2023-09-06T15:48:23.1491975Z 
2023-09-06T15:48:23.1492128Z from user code:
2023-09-06T15:48:23.1492825Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch/bert_pytorch/model/bert.py", line 47, in forward
2023-09-06T15:48:23.1493577Z     x = transformer.forward(x, mask)
2023-09-06T15:48:23.1494292Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch/bert_pytorch/model/transformer.py", line 46, in forward
2023-09-06T15:48:23.1494951Z     self.lambda_module.set_mask(mask)
2023-09-06T15:48:23.1495660Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch/bert_pytorch/model/transformer.py", line 17, in set_mask
2023-09-06T15:48:23.1500836Z     self.mask = mask
2023-09-06T15:48:23.1501044Z 
2023-09-06T15:48:23.1501348Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T15:48:23.1501988Z Traceback (most recent call last):
2023-09-06T15:48:23.1502485Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2135, in check_accuracy
2023-09-06T15:48:23.1503368Z     new_result = optimized_model_iter_fn(model_copy, example_inputs)
2023-09-06T15:48:23.1504143Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1190, in opt_aot_inductor
2023-09-06T15:48:23.1505055Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-09-06T15:48:23.1505990Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1134, in load
2023-09-06T15:48:23.1506748Z     so_path, exported = torch._export.aot_compile(
2023-09-06T15:48:23.1507768Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 635, in aot_compile
2023-09-06T15:48:23.1508201Z     ep = export(f, args, kwargs, constraints)
2023-09-06T15:48:23.1508765Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 307, in export
2023-09-06T15:48:23.1509610Z     gm_torch_level, _ = torch._dynamo.export(
2023-09-06T15:48:23.1510194Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1150, in inner
2023-09-06T15:48:23.1510582Z     result_traced = opt_f(*args, **kwargs)
2023-09-06T15:48:23.1511171Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:48:23.1511600Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:48:23.1512163Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:48:23.1512567Z     return forward_call(*args, **kwargs)
2023-09-06T15:48:23.1513583Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T15:48:23.1513993Z     return fn(*args, **kwargs)
2023-09-06T15:48:23.1514619Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:48:23.1515046Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:48:23.1515606Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:48:23.1516013Z     return forward_call(*args, **kwargs)
2023-09-06T15:48:23.1516556Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T15:48:23.1517005Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T15:48:23.1517580Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T15:48:23.1517964Z     return fn(*args, **kwargs)
2023-09-06T15:48:23.1518539Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T15:48:23.1518943Z     return _compile(
2023-09-06T15:48:23.1519476Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T15:48:23.1519946Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T15:48:23.1520526Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T15:48:23.1520923Z     r = func(*args, **kwargs)
2023-09-06T15:48:23.1521484Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T15:48:23.1521930Z     out_code = transform_code_object(code, transform)
2023-09-06T15:48:23.1522581Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T15:48:23.1523284Z     transformations(instructions, code_options)
2023-09-06T15:48:23.1523870Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T15:48:23.1524251Z     tracer.run()
2023-09-06T15:48:23.1524768Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T15:48:23.1525125Z     super().run()
2023-09-06T15:48:23.1525643Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T15:48:23.1526013Z     and self.step()
2023-09-06T15:48:23.1526529Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T15:48:23.1526930Z     getattr(self, inst.opname)(inst)
2023-09-06T15:48:23.1527473Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T15:48:23.1527882Z     return inner_fn(self, inst)
2023-09-06T15:48:23.1528453Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1119, in CALL_FUNCTION
2023-09-06T15:48:23.1528860Z     self.call_function(fn, args, {})
2023-09-06T15:48:23.1529414Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 565, in call_function
2023-09-06T15:48:23.1529859Z     self.push(fn.call_function(self, args, kwargs))
2023-09-06T15:48:23.1530459Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-09-06T15:48:23.1530898Z     return super().call_function(tx, args, kwargs)
2023-09-06T15:48:23.1531493Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-09-06T15:48:23.1531921Z     return super().call_function(tx, args, kwargs)
2023-09-06T15:48:23.1532632Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-09-06T15:48:23.1533117Z     return tx.inline_user_function_return(
2023-09-06T15:48:23.1533744Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 601, in inline_user_function_return
2023-09-06T15:48:23.1534284Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-09-06T15:48:23.1534954Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2183, in inline_call
2023-09-06T15:48:23.1535401Z     return cls.inline_call_(parent, func, args, kwargs)
2023-09-06T15:48:23.1536011Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2290, in inline_call_
2023-09-06T15:48:23.1536399Z     tracer.run()
2023-09-06T15:48:23.1536911Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T15:48:23.1537301Z     and self.step()
2023-09-06T15:48:23.1537829Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T15:48:23.1538249Z     getattr(self, inst.opname)(inst)
2023-09-06T15:48:23.1538792Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T15:48:23.1539191Z     return inner_fn(self, inst)
2023-09-06T15:48:23.1539752Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1119, in CALL_FUNCTION
2023-09-06T15:48:23.1540168Z     self.call_function(fn, args, {})
2023-09-06T15:48:23.1540737Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 565, in call_function
2023-09-06T15:48:23.1541159Z     self.push(fn.call_function(self, args, kwargs))
2023-09-06T15:48:23.1541766Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-09-06T15:48:23.1542400Z     return super().call_function(tx, args, kwargs)
2023-09-06T15:48:23.1543046Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-09-06T15:48:23.1543476Z     return super().call_function(tx, args, kwargs)
2023-09-06T15:48:23.1544072Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-09-06T15:48:23.1544504Z     return tx.inline_user_function_return(
2023-09-06T15:48:23.1545128Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 601, in inline_user_function_return
2023-09-06T15:48:23.1545669Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-09-06T15:48:23.1546320Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2183, in inline_call
2023-09-06T15:48:23.1546793Z     return cls.inline_call_(parent, func, args, kwargs)
2023-09-06T15:48:23.1547406Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2290, in inline_call_
2023-09-06T15:48:23.1547796Z     tracer.run()
2023-09-06T15:48:23.1548300Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T15:48:23.1548677Z     and self.step()
2023-09-06T15:48:23.1549382Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T15:48:23.1549793Z     getattr(self, inst.opname)(inst)
2023-09-06T15:48:23.1550374Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1207, in STORE_ATTR
2023-09-06T15:48:23.1550730Z     assert (
2023-09-06T15:48:23.1551088Z AssertionError: Mutating module attribute mask during export.
2023-09-06T15:48:23.1551453Z 
2023-09-06T15:48:23.1551885Z from user code:
2023-09-06T15:48:23.1552760Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch/bert_pytorch/model/bert.py", line 47, in forward
2023-09-06T15:48:23.1553284Z     x = transformer.forward(x, mask)
2023-09-06T15:48:23.1553767Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch/bert_pytorch/model/transformer.py", line 46, in forward
2023-09-06T15:48:23.1554211Z     self.lambda_module.set_mask(mask)
2023-09-06T15:48:23.1554689Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch/bert_pytorch/model/transformer.py", line 17, in set_mask
2023-09-06T15:48:23.1555113Z     self.mask = mask
2023-09-06T15:48:23.1555254Z 
2023-09-06T15:48:23.1555448Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T15:48:23.1555687Z 
2023-09-06T15:48:23.1555887Z TorchDynamo optimized model failed to run because of following error
2023-09-06T15:48:23.1589264Z fail_to_run
2023-09-06T15:48:27.8168611Z 
2023-09-06T15:48:31.5698205Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:48:31.5698763Z loading model: 0it [00:03, ?it/s]
2023-09-06T15:48:31.5753294Z cuda eval  Background_Matting                 
2023-09-06T15:48:31.5757685Z pass_due_to_skip
2023-09-06T15:48:36.1889896Z 
2023-09-06T15:48:45.4576982Z loading model: 0it [00:00, ?it/s]WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T15:48:47.1947947Z 
2023-09-06T15:48:47.1949286Z loading model: 0it [00:11, ?it/s]
2023-09-06T15:48:47.1949792Z WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T15:48:47.1950198Z cuda eval  DALLE2_pytorch                     
2023-09-06T15:48:47.4415463Z WARNING:common:fp64 golden ref were not generated for DALLE2_pytorch. Setting accuracy check to cosine
2023-09-06T15:48:47.8489164Z WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T15:48:52.1996341Z ERROR:common:call_method NNModuleVariable() eval [] {}
2023-09-06T15:48:52.1996773Z 
2023-09-06T15:48:52.1996927Z from user code:
2023-09-06T15:48:52.1999982Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/external_utils.py", line 17, in inner
2023-09-06T15:48:52.2000658Z     return fn(*args, **kwargs)
2023-09-06T15:48:52.2001597Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:48:52.2002345Z     return forward_call(*args, **kwargs)
2023-09-06T15:48:52.2006750Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T15:48:52.2007333Z     return func(*args, **kwargs)
2023-09-06T15:48:52.2008464Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 95, in inner
2023-09-06T15:48:52.2009207Z     model.eval()
2023-09-06T15:48:52.2009454Z 
2023-09-06T15:48:52.2009671Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T15:48:52.2010047Z Traceback (most recent call last):
2023-09-06T15:48:52.2010433Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2135, in check_accuracy
2023-09-06T15:48:52.2010873Z     new_result = optimized_model_iter_fn(model_copy, example_inputs)
2023-09-06T15:48:52.2011315Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1190, in opt_aot_inductor
2023-09-06T15:48:52.2011799Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-09-06T15:48:52.2012262Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1134, in load
2023-09-06T15:48:52.2012638Z     so_path, exported = torch._export.aot_compile(
2023-09-06T15:48:52.2013220Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 635, in aot_compile
2023-09-06T15:48:52.2013635Z     ep = export(f, args, kwargs, constraints)
2023-09-06T15:48:52.2014537Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 307, in export
2023-09-06T15:48:52.2014952Z     gm_torch_level, _ = torch._dynamo.export(
2023-09-06T15:48:52.2015506Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1150, in inner
2023-09-06T15:48:52.2015911Z     result_traced = opt_f(*args, **kwargs)
2023-09-06T15:48:52.2016484Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:48:52.2016902Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:48:52.2017438Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:48:52.2017840Z     return forward_call(*args, **kwargs)
2023-09-06T15:48:52.2018434Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T15:48:52.2018811Z     return fn(*args, **kwargs)
2023-09-06T15:48:52.2019350Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T15:48:52.2019795Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T15:48:52.2020366Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T15:48:52.2020741Z     return fn(*args, **kwargs)
2023-09-06T15:48:52.2021309Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T15:48:52.2021690Z     return _compile(
2023-09-06T15:48:52.2022212Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T15:48:52.2022671Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T15:48:52.2023260Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T15:48:52.2023825Z     r = func(*args, **kwargs)
2023-09-06T15:48:52.2024382Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T15:48:52.2024819Z     out_code = transform_code_object(code, transform)
2023-09-06T15:48:52.2025463Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T15:48:52.2025936Z     transformations(instructions, code_options)
2023-09-06T15:48:52.2026497Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T15:48:52.2026867Z     tracer.run()
2023-09-06T15:48:52.2027387Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T15:48:52.2027752Z     super().run()
2023-09-06T15:48:52.2028293Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T15:48:52.2028670Z     and self.step()
2023-09-06T15:48:52.2029389Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T15:48:52.2029790Z     getattr(self, inst.opname)(inst)
2023-09-06T15:48:52.2030336Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T15:48:52.2030734Z     return inner_fn(self, inst)
2023-09-06T15:48:52.2031312Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1159, in CALL_FUNCTION_EX
2023-09-06T15:48:52.2031787Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-09-06T15:48:52.2032402Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 565, in call_function
2023-09-06T15:48:52.2032828Z     self.push(fn.call_function(self, args, kwargs))
2023-09-06T15:48:52.2033571Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 336, in call_function
2023-09-06T15:48:52.2034003Z     return tx.inline_user_function_return(
2023-09-06T15:48:52.2034625Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 601, in inline_user_function_return
2023-09-06T15:48:52.2035153Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-09-06T15:48:52.2035812Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2183, in inline_call
2023-09-06T15:48:52.2036259Z     return cls.inline_call_(parent, func, args, kwargs)
2023-09-06T15:48:52.2036857Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2290, in inline_call_
2023-09-06T15:48:52.2037232Z     tracer.run()
2023-09-06T15:48:52.2037732Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T15:48:52.2038109Z     and self.step()
2023-09-06T15:48:52.2038678Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T15:48:52.2039075Z     getattr(self, inst.opname)(inst)
2023-09-06T15:48:52.2039614Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T15:48:52.2040012Z     return inner_fn(self, inst)
2023-09-06T15:48:52.2040580Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1159, in CALL_FUNCTION_EX
2023-09-06T15:48:52.2041059Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-09-06T15:48:52.2041667Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 565, in call_function
2023-09-06T15:48:52.2042093Z     self.push(fn.call_function(self, args, kwargs))
2023-09-06T15:48:52.2042692Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-09-06T15:48:52.2043306Z     return super().call_function(tx, args, kwargs)
2023-09-06T15:48:52.2043905Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-09-06T15:48:52.2044340Z     return super().call_function(tx, args, kwargs)
2023-09-06T15:48:52.2044916Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-09-06T15:48:52.2045339Z     return tx.inline_user_function_return(
2023-09-06T15:48:52.2045948Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 601, in inline_user_function_return
2023-09-06T15:48:52.2046483Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-09-06T15:48:52.2047137Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2183, in inline_call
2023-09-06T15:48:52.2047587Z     return cls.inline_call_(parent, func, args, kwargs)
2023-09-06T15:48:52.2048184Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2290, in inline_call_
2023-09-06T15:48:52.2048607Z     tracer.run()
2023-09-06T15:48:52.2049128Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T15:48:52.2049485Z     and self.step()
2023-09-06T15:48:52.2050006Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T15:48:52.2050410Z     getattr(self, inst.opname)(inst)
2023-09-06T15:48:52.2050970Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T15:48:52.2051349Z     return inner_fn(self, inst)
2023-09-06T15:48:52.2052037Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1159, in CALL_FUNCTION_EX
2023-09-06T15:48:52.2052522Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-09-06T15:48:52.2053141Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 565, in call_function
2023-09-06T15:48:52.2053580Z     self.push(fn.call_function(self, args, kwargs))
2023-09-06T15:48:52.2054162Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-09-06T15:48:52.2054601Z     return super().call_function(tx, args, kwargs)
2023-09-06T15:48:52.2055199Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-09-06T15:48:52.2055672Z     return tx.inline_user_function_return(
2023-09-06T15:48:52.2056280Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 601, in inline_user_function_return
2023-09-06T15:48:52.2056811Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-09-06T15:48:52.2057475Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2183, in inline_call
2023-09-06T15:48:52.2057924Z     return cls.inline_call_(parent, func, args, kwargs)
2023-09-06T15:48:52.2058565Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2290, in inline_call_
2023-09-06T15:48:52.2058942Z     tracer.run()
2023-09-06T15:48:52.2059442Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T15:48:52.2059821Z     and self.step()
2023-09-06T15:48:52.2060348Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T15:48:52.2060747Z     getattr(self, inst.opname)(inst)
2023-09-06T15:48:52.2061294Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T15:48:52.2061795Z     return inner_fn(self, inst)
2023-09-06T15:48:52.2062365Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1119, in CALL_FUNCTION
2023-09-06T15:48:52.2062985Z     self.call_function(fn, args, {})
2023-09-06T15:48:52.2063544Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 565, in call_function
2023-09-06T15:48:52.2064017Z     self.push(fn.call_function(self, args, kwargs))
2023-09-06T15:48:52.2064624Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 304, in call_function
2023-09-06T15:48:52.2065038Z     return self.obj.call_method(
2023-09-06T15:48:52.2065599Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 601, in call_method
2023-09-06T15:48:52.2066026Z     return super().call_method(tx, name, args, kwargs)
2023-09-06T15:48:52.2066625Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/base.py", line 329, in call_method
2023-09-06T15:48:52.2067090Z     raise unimplemented(f"call_method {self} {name} {args} {kwargs}")
2023-09-06T15:48:52.2067684Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 176, in unimplemented
2023-09-06T15:48:52.2068058Z     raise Unsupported(msg)
2023-09-06T15:48:52.2068518Z torch._dynamo.exc.Unsupported: call_method NNModuleVariable() eval [] {}
2023-09-06T15:48:52.2068777Z 
2023-09-06T15:48:52.2068875Z from user code:
2023-09-06T15:48:52.2069632Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/external_utils.py", line 17, in inner
2023-09-06T15:48:52.2070032Z     return fn(*args, **kwargs)
2023-09-06T15:48:52.2070556Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:48:52.2071141Z     return forward_call(*args, **kwargs)
2023-09-06T15:48:52.2071722Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T15:48:52.2072120Z     return func(*args, **kwargs)
2023-09-06T15:48:52.2072648Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 95, in inner
2023-09-06T15:48:52.2073021Z     model.eval()
2023-09-06T15:48:52.2073172Z 
2023-09-06T15:48:52.2073370Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T15:48:52.2073608Z 
2023-09-06T15:48:52.2073805Z TorchDynamo optimized model failed to run because of following error
2023-09-06T15:48:52.2327346Z fail_to_run
2023-09-06T15:48:57.3426259Z 
2023-09-06T15:49:01.0970960Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:49:01.0971305Z loading model: 0it [00:03, ?it/s]
2023-09-06T15:49:01.0992573Z cuda eval  LearningToPaint                    
2023-09-06T15:49:54.6251489Z pass
2023-09-06T15:49:59.8904981Z 
2023-09-06T15:50:02.3179911Z loading model: 0it [00:00, ?it/s]WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T15:50:02.8422385Z 
2023-09-06T15:50:02.8423029Z loading model: 0it [00:02, ?it/s]
2023-09-06T15:50:02.8427159Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T15:50:02.8427944Z cuda eval  Super_SloMo                        
2023-09-06T15:50:02.9447641Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T15:51:26.0528080Z pass
2023-09-06T15:51:31.9795585Z 
2023-09-06T15:51:33.5078765Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:51:33.5079299Z loading model: 0it [00:01, ?it/s]
2023-09-06T15:51:33.5082580Z cuda eval  alexnet                            
2023-09-06T15:51:56.8778787Z pass
2023-09-06T15:52:01.8244708Z 
2023-09-06T15:52:08.6779232Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:52:08.6779780Z loading model: 0it [00:06, ?it/s]
2023-09-06T15:52:08.6842337Z cuda eval  attention_is_all_you_need_pytorch  
2023-09-06T15:53:14.5047114Z pass
2023-09-06T15:53:20.0123142Z 
2023-09-06T15:53:22.8024098Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:53:22.8024800Z loading model: 0it [00:02, ?it/s]
2023-09-06T15:53:22.8028256Z cuda eval  basic_gnn_edgecnn                  
2023-09-06T15:53:23.3609070Z [2023-09-06 15:53:23,359] [0/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T15:53:43.2849278Z pass
2023-09-06T15:53:48.2361500Z 
2023-09-06T15:53:51.1410117Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:53:51.1410644Z loading model: 0it [00:02, ?it/s]
2023-09-06T15:53:51.1413690Z cuda eval  basic_gnn_gcn                      
2023-09-06T15:53:52.0258009Z [2023-09-06 15:53:52,024] [0/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T15:53:54.3965782Z ERROR:common:aten.nonzero.default
2023-09-06T15:53:54.3966694Z 
2023-09-06T15:53:54.3967047Z While executing %index : [num_users=2] = call_function[target=torch.ops.aten.index.Tensor](args = (%slice_1, [None, %ne]), kwargs = {})
2023-09-06T15:53:54.3970021Z Original traceback:
2023-09-06T15:53:54.3973241Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/models/basic_gnn.py", line 222, in forward
2023-09-06T15:53:54.3973755Z     x = self.convs[i](x, edge_index, edge_weight=edge_weight)
2023-09-06T15:53:54.3974601Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:53:54.3975022Z     return forward_call(*args, **kwargs)
2023-09-06T15:53:54.3975793Z   File "/tmp/jenkins_pyg/tmpjvozk9n_.py", line 221, in forward
2023-09-06T15:53:54.3976174Z     edge_index, edge_weight = gcn_norm(  # yapf: disable
2023-09-06T15:53:54.3976798Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/gcn_conv.py", line 91, in gcn_norm
2023-09-06T15:53:54.3977245Z     edge_index, edge_weight = add_remaining_self_loops(
2023-09-06T15:53:54.3978650Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/loop.py", line 370, in add_remaining_self_loops
2023-09-06T15:53:54.3979222Z     edge_index = torch.cat([edge_index[:, mask], loop_index], dim=1)
2023-09-06T15:53:54.3979580Z Traceback (most recent call last):
2023-09-06T15:53:54.3980125Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2135, in check_accuracy
2023-09-06T15:53:54.3980899Z     new_result = optimized_model_iter_fn(model_copy, example_inputs)
2023-09-06T15:53:54.3981781Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1190, in opt_aot_inductor
2023-09-06T15:53:54.3982300Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-09-06T15:53:54.3982785Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1134, in load
2023-09-06T15:53:54.3983239Z     so_path, exported = torch._export.aot_compile(
2023-09-06T15:53:54.3983890Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 645, in aot_compile
2023-09-06T15:53:54.3984368Z     so_path = torch._inductor.aot_compile(ep.graph_module, list(all_args), options)
2023-09-06T15:53:54.3984993Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/__init__.py", line 48, in aot_compile
2023-09-06T15:53:54.3985430Z     result = compile_fx_aot(
2023-09-06T15:53:54.3986299Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 874, in compile_fx_aot
2023-09-06T15:53:54.3986939Z     return compile_fx(
2023-09-06T15:53:54.3988370Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 975, in compile_fx
2023-09-06T15:53:54.3989001Z     return compile_fx(
2023-09-06T15:53:54.3990224Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 995, in compile_fx
2023-09-06T15:53:54.3990800Z     return compile_fx(
2023-09-06T15:53:54.3992302Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1167, in compile_fx
2023-09-06T15:53:54.3993514Z     return aot_autograd(
2023-09-06T15:53:54.3995050Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 55, in compiler_fn
2023-09-06T15:53:54.3996318Z     cg = aot_module_simplified(gm, example_inputs, **kwargs)
2023-09-06T15:53:54.3998058Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3891, in aot_module_simplified
2023-09-06T15:53:54.3999280Z     compiled_fn = create_aot_dispatcher_function(
2023-09-06T15:53:54.4000895Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T15:53:54.4001942Z     r = func(*args, **kwargs)
2023-09-06T15:53:54.4003577Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3379, in create_aot_dispatcher_function
2023-09-06T15:53:54.4004356Z     fw_metadata = run_functionalized_fw_and_collect_metadata(
2023-09-06T15:53:54.4005471Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 757, in inner
2023-09-06T15:53:54.4006243Z     flat_f_outs = f(*flat_f_args)
2023-09-06T15:53:54.4007330Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3496, in functional_call
2023-09-06T15:53:54.4007970Z     out = Interpreter(mod).run(*args[params_len:], **kwargs)
2023-09-06T15:53:54.4008873Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 138, in run
2023-09-06T15:53:54.4009283Z     self.env[node] = self.run_node(node)
2023-09-06T15:53:54.4009836Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 195, in run_node
2023-09-06T15:53:54.4010273Z     return getattr(self, n.op)(n.target, args, kwargs)
2023-09-06T15:53:54.4010946Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 267, in call_function
2023-09-06T15:53:54.4011674Z     return target(*args, **kwargs)
2023-09-06T15:53:54.4012571Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 448, in __call__
2023-09-06T15:53:54.4013238Z     return self._op(*args, **kwargs or {})
2023-09-06T15:53:54.4014049Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_stats.py", line 20, in wrapper
2023-09-06T15:53:54.4014564Z     return fn(*args, **kwargs)
2023-09-06T15:53:54.4015409Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1290, in __torch_dispatch__
2023-09-06T15:53:54.4016072Z     return self.dispatch(func, types, args, kwargs)
2023-09-06T15:53:54.4016924Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1532, in dispatch
2023-09-06T15:53:54.4017544Z     op_impl_out = op_impl(self, func, *args, **kwargs)
2023-09-06T15:53:54.4018400Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 634, in index_tensor
2023-09-06T15:53:54.4019016Z     out = meta_index_Tensor(*args, **kwargs)
2023-09-06T15:53:54.4019850Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_meta_registrations.py", line 2668, in meta_index_Tensor
2023-09-06T15:53:54.4020434Z     nonzero = index.nonzero()
2023-09-06T15:53:54.4021174Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_stats.py", line 20, in wrapper
2023-09-06T15:53:54.4021709Z     return fn(*args, **kwargs)
2023-09-06T15:53:54.4023056Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1290, in __torch_dispatch__
2023-09-06T15:53:54.4023700Z     return self.dispatch(func, types, args, kwargs)
2023-09-06T15:53:54.4024547Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1532, in dispatch
2023-09-06T15:53:54.4025163Z     op_impl_out = op_impl(self, func, *args, **kwargs)
2023-09-06T15:53:54.4026052Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 556, in nonzero
2023-09-06T15:53:54.4026689Z     raise DynamicOutputShapeException(func)
2023-09-06T15:53:54.4027339Z torch._subclasses.fake_tensor.DynamicOutputShapeException: aten.nonzero.default
2023-09-06T15:53:54.4027756Z 
2023-09-06T15:53:54.4028169Z While executing %index : [num_users=2] = call_function[target=torch.ops.aten.index.Tensor](args = (%slice_1, [None, %ne]), kwargs = {})
2023-09-06T15:53:54.4028757Z Original traceback:
2023-09-06T15:53:54.4029769Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/models/basic_gnn.py", line 222, in forward
2023-09-06T15:53:54.4030410Z     x = self.convs[i](x, edge_index, edge_weight=edge_weight)
2023-09-06T15:53:54.4031213Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:53:54.4031769Z     return forward_call(*args, **kwargs)
2023-09-06T15:53:54.4032249Z   File "/tmp/jenkins_pyg/tmpjvozk9n_.py", line 221, in forward
2023-09-06T15:53:54.4032772Z     edge_index, edge_weight = gcn_norm(  # yapf: disable
2023-09-06T15:53:54.4033682Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/gcn_conv.py", line 91, in gcn_norm
2023-09-06T15:53:54.4034282Z     edge_index, edge_weight = add_remaining_self_loops(
2023-09-06T15:53:54.4035164Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/loop.py", line 370, in add_remaining_self_loops
2023-09-06T15:53:54.4036136Z     edge_index = torch.cat([edge_index[:, mask], loop_index], dim=1)
2023-09-06T15:53:54.4036472Z 
2023-09-06T15:53:54.4036757Z TorchDynamo optimized model failed to run because of following error
2023-09-06T15:53:54.4037216Z fail_to_run
2023-09-06T15:53:59.2278089Z 
2023-09-06T15:54:02.2498736Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:54:02.2499102Z loading model: 0it [00:03, ?it/s]
2023-09-06T15:54:02.2508890Z cuda eval  basic_gnn_gin                      
2023-09-06T15:54:02.9940504Z [2023-09-06 15:54:02,993] [0/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T15:54:22.8313614Z pass
2023-09-06T15:54:27.7958545Z 
2023-09-06T15:54:30.7584072Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:54:30.7584476Z loading model: 0it [00:02, ?it/s]
2023-09-06T15:54:30.7587437Z cuda eval  basic_gnn_sage                     
2023-09-06T15:54:31.4949614Z [2023-09-06 15:54:31,493] [0/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T15:54:51.6116397Z pass
2023-09-06T15:54:56.5944322Z 
2023-09-06T15:55:01.9994780Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:55:01.9995138Z loading model: 0it [00:05, ?it/s]
2023-09-06T15:55:02.0102597Z cuda eval  cm3leon_generate                   
2023-09-06T15:55:23.1853776Z ERROR:common:inline in skipfiles: _SpecialForm.__getitem__  | inner /opt/conda/envs/py_3.10/lib/python3.10/typing.py
2023-09-06T15:55:23.1854190Z 
2023-09-06T15:55:23.1854340Z from user code:
2023-09-06T15:55:23.1857396Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate/model.py", line 1116, in forward
2023-09-06T15:55:23.1858352Z     Dict[str, Dict[str, Optional[Tensor]]], {}
2023-09-06T15:55:23.1858558Z 
2023-09-06T15:55:23.1858759Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T15:55:23.1859132Z Traceback (most recent call last):
2023-09-06T15:55:23.1859627Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2135, in check_accuracy
2023-09-06T15:55:23.1860076Z     new_result = optimized_model_iter_fn(model_copy, example_inputs)
2023-09-06T15:55:23.1860513Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1190, in opt_aot_inductor
2023-09-06T15:55:23.1861002Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-09-06T15:55:23.1864918Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1134, in load
2023-09-06T15:55:23.1865514Z     so_path, exported = torch._export.aot_compile(
2023-09-06T15:55:23.1868218Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 635, in aot_compile
2023-09-06T15:55:23.1869043Z     ep = export(f, args, kwargs, constraints)
2023-09-06T15:55:23.1870369Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 307, in export
2023-09-06T15:55:23.1870980Z     gm_torch_level, _ = torch._dynamo.export(
2023-09-06T15:55:23.1871535Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1150, in inner
2023-09-06T15:55:23.1871948Z     result_traced = opt_f(*args, **kwargs)
2023-09-06T15:55:23.1872527Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:55:23.1872953Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:55:23.1873491Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:55:23.1873895Z     return forward_call(*args, **kwargs)
2023-09-06T15:55:23.1874806Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T15:55:23.1875199Z     return fn(*args, **kwargs)
2023-09-06T15:55:23.1875753Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:55:23.1876177Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:55:23.1876728Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:55:23.1877136Z     return forward_call(*args, **kwargs)
2023-09-06T15:55:23.1877772Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T15:55:23.1878204Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T15:55:23.1878784Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T15:55:23.1879224Z     return fn(*args, **kwargs)
2023-09-06T15:55:23.1879798Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T15:55:23.1880207Z     return _compile(
2023-09-06T15:55:23.1880721Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T15:55:23.1881202Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T15:55:23.1881802Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T15:55:23.1882184Z     r = func(*args, **kwargs)
2023-09-06T15:55:23.1882712Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T15:55:23.1883156Z     out_code = transform_code_object(code, transform)
2023-09-06T15:55:23.1883807Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T15:55:23.1884563Z     transformations(instructions, code_options)
2023-09-06T15:55:23.1885145Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T15:55:23.1885533Z     tracer.run()
2023-09-06T15:55:23.1886048Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T15:55:23.1886420Z     super().run()
2023-09-06T15:55:23.1886937Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T15:55:23.1887296Z     and self.step()
2023-09-06T15:55:23.1887870Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T15:55:23.1888267Z     getattr(self, inst.opname)(inst)
2023-09-06T15:55:23.1888830Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T15:55:23.1889231Z     return inner_fn(self, inst)
2023-09-06T15:55:23.1889756Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 168, in impl
2023-09-06T15:55:23.1890206Z     self.push(fn_var.call_function(self, self.popn(nargs), {}))
2023-09-06T15:55:23.1890826Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 618, in call_function
2023-09-06T15:55:23.1891255Z     result = handler(tx, *args, **kwargs)
2023-09-06T15:55:23.1891817Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 950, in call_getitem
2023-09-06T15:55:23.1892278Z     return args[0].call_method(tx, "__getitem__", args[1:], kwargs)
2023-09-06T15:55:23.1892887Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 302, in call_method
2023-09-06T15:55:23.1893473Z     ).call_function(tx, args, kwargs)
2023-09-06T15:55:23.1894062Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-09-06T15:55:23.1894487Z     return super().call_function(tx, args, kwargs)
2023-09-06T15:55:23.1895083Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-09-06T15:55:23.1895528Z     return super().call_function(tx, args, kwargs)
2023-09-06T15:55:23.1896132Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-09-06T15:55:23.1896547Z     return tx.inline_user_function_return(
2023-09-06T15:55:23.1897160Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 601, in inline_user_function_return
2023-09-06T15:55:23.1897738Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-09-06T15:55:23.1898413Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2183, in inline_call
2023-09-06T15:55:23.1898865Z     return cls.inline_call_(parent, func, args, kwargs)
2023-09-06T15:55:23.1899445Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2236, in inline_call_
2023-09-06T15:55:23.1899933Z     InliningInstructionTranslator.check_inlineable(func)
2023-09-06T15:55:23.1900587Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2217, in check_inlineable
2023-09-06T15:55:23.1900976Z     unimplemented(
2023-09-06T15:55:23.1901494Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 176, in unimplemented
2023-09-06T15:55:23.1901866Z     raise Unsupported(msg)
2023-09-06T15:55:23.1902329Z torch._dynamo.exc.Unsupported: inline in skipfiles: _SpecialForm.__getitem__  | inner /opt/conda/envs/py_3.10/lib/python3.10/typing.py
2023-09-06T15:55:23.1902773Z 
2023-09-06T15:55:23.1902877Z from user code:
2023-09-06T15:55:23.1903290Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate/model.py", line 1116, in forward
2023-09-06T15:55:23.1903713Z     Dict[str, Dict[str, Optional[Tensor]]], {}
2023-09-06T15:55:23.1903906Z 
2023-09-06T15:55:23.1904105Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T15:55:23.1904344Z 
2023-09-06T15:55:23.1904545Z TorchDynamo optimized model failed to run because of following error
2023-09-06T15:55:23.1918984Z fail_to_run
2023-09-06T15:55:27.9473295Z 
2023-09-06T15:55:28.7971817Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:55:28.7972411Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:55:28.7975786Z cuda eval  dcgan                              
2023-09-06T15:55:49.6359793Z pass
2023-09-06T15:55:54.4559417Z 
2023-09-06T15:55:57.7816469Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:55:57.7816857Z loading model: 0it [00:03, ?it/s]
2023-09-06T15:55:57.7817185Z Eager model failed to run
2023-09-06T15:55:57.7831710Z Traceback (most recent call last):
2023-09-06T15:55:57.7832470Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:55:57.7832889Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:55:57.7833316Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:55:57.7833742Z     return mod(*inputs)
2023-09-06T15:55:57.7836856Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:55:57.7837908Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:55:57.7839216Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:55:57.7840298Z     return forward_call(*args, **kwargs)
2023-09-06T15:55:57.7841269Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/__init__.py", line 31, in forward
2023-09-06T15:55:57.7841703Z     return sources, self.model(mix)
2023-09-06T15:55:57.7842489Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:55:57.7843009Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:55:57.7843575Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:55:57.7843983Z     return forward_call(*args, **kwargs)
2023-09-06T15:55:57.7844405Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/demucs/model.py", line 209, in forward
2023-09-06T15:55:57.7844790Z     x = self.lstm(x)
2023-09-06T15:55:57.7845334Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:55:57.7845756Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:55:57.7846304Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:55:57.7846706Z     return forward_call(*args, **kwargs)
2023-09-06T15:55:57.7847137Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/demucs/model.py", line 27, in forward
2023-09-06T15:55:57.7847574Z     x = self.lstm(x)[0]
2023-09-06T15:55:57.7848164Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:55:57.7848583Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:55:57.7849139Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:55:57.7849538Z     return forward_call(*args, **kwargs)
2023-09-06T15:55:57.7850071Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/rnn.py", line 879, in forward
2023-09-06T15:55:57.7850530Z     result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
2023-09-06T15:55:57.7851280Z RuntimeError: "_thnn_fused_lstm_cell_cuda" not implemented for 'BFloat16'
2023-09-06T15:55:57.7851527Z 
2023-09-06T15:55:57.7851723Z The above exception was the direct cause of the following exception:
2023-09-06T15:55:57.7851965Z 
2023-09-06T15:55:57.7852092Z Traceback (most recent call last):
2023-09-06T15:55:57.7852468Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:55:57.7852803Z     ) = runner.load_model(
2023-09-06T15:55:57.7853233Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:55:57.7853635Z     self.validate_model(model, example_inputs)
2023-09-06T15:55:57.7854080Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:55:57.7854510Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:55:57.7854890Z NotImplementedError: Eager model failed to run
2023-09-06T15:55:57.7855105Z 
2023-09-06T15:55:57.7855230Z WARNING:root:demucs failed to load
2023-09-06T15:56:02.2692060Z 
2023-09-06T15:56:04.1807901Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:56:04.1808268Z loading model: 0it [00:01, ?it/s]
2023-09-06T15:56:04.1954128Z cuda eval  densenet121                        
2023-09-06T15:58:29.9260235Z pass
2023-09-06T15:58:35.9461155Z 
2023-09-06T15:58:40.7426894Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:58:40.7427229Z loading model: 0it [00:04, ?it/s]
2023-09-06T15:58:40.7427506Z Eager model failed to run
2023-09-06T15:58:40.7439762Z Traceback (most recent call last):
2023-09-06T15:58:40.7440452Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:58:40.7440979Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:58:40.7441681Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:58:40.7444635Z     return mod(*inputs)
2023-09-06T15:58:40.7447142Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:58:40.7447624Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:58:40.7448314Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:58:40.7448875Z     return forward_call(*args, **kwargs)
2023-09-06T15:58:40.7449459Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:58:40.7449897Z     return self.inference(batched_inputs)
2023-09-06T15:58:40.7450489Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:58:40.7451108Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:58:40.7452196Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:58:40.7452629Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:58:40.7453200Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:58:40.7453600Z     return forward_call(*args, **kwargs)
2023-09-06T15:58:40.7454265Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2023-09-06T15:58:40.7454710Z     box_features = self._shared_roi_transform(
2023-09-06T15:58:40.7455326Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2023-09-06T15:58:40.7455841Z     x = self.pooler(features, boxes)
2023-09-06T15:58:40.7456771Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:58:40.7457440Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:58:40.7458487Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:58:40.7459233Z     return forward_call(*args, **kwargs)
2023-09-06T15:58:40.7459809Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T15:58:40.7460252Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T15:58:40.7460849Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:58:40.7461258Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:58:40.7461845Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:58:40.7462252Z     return forward_call(*args, **kwargs)
2023-09-06T15:58:40.7463004Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:58:40.7463386Z     return roi_align(
2023-09-06T15:58:40.7463975Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:58:40.7464507Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:58:40.7465155Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:58:40.7465673Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:58:40.7466343Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:58:40.7466745Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:58:40.7467298Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:58:40.7467685Z     return input[
2023-09-06T15:58:40.7468807Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 26902.82 GiB. GPU 0 has a total capacty of 39.39 GiB of which 35.54 GiB is free. Process 2197344 has 3.84 GiB memory in use. Of the allocated memory 2.99 GiB is allocated by PyTorch, and 335.94 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:58:40.7470139Z 
2023-09-06T15:58:40.7470351Z The above exception was the direct cause of the following exception:
2023-09-06T15:58:40.7470597Z 
2023-09-06T15:58:40.7470724Z Traceback (most recent call last):
2023-09-06T15:58:40.7471090Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:58:40.7471445Z     ) = runner.load_model(
2023-09-06T15:58:40.7471826Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:58:40.7472229Z     self.validate_model(model, example_inputs)
2023-09-06T15:58:40.7472662Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:58:40.7473091Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:58:40.7473474Z NotImplementedError: Eager model failed to run
2023-09-06T15:58:40.7473684Z 
2023-09-06T15:58:40.7473897Z WARNING:root:detectron2_fasterrcnn_r_101_c4 failed to load
2023-09-06T15:58:45.4244627Z 
2023-09-06T15:58:52.3659505Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:58:52.3660197Z loading model: 0it [00:06, ?it/s]
2023-09-06T15:58:52.3660478Z Eager model failed to run
2023-09-06T15:58:52.3673373Z Traceback (most recent call last):
2023-09-06T15:58:52.3674125Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:58:52.3674856Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:58:52.3675588Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:58:52.3676975Z     return mod(*inputs)
2023-09-06T15:58:52.3679539Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:58:52.3679992Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:58:52.3680561Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:58:52.3680970Z     return forward_call(*args, **kwargs)
2023-09-06T15:58:52.3681551Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:58:52.3681965Z     return self.inference(batched_inputs)
2023-09-06T15:58:52.3682553Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:58:52.3683029Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:58:52.3683741Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:58:52.3684175Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:58:52.3684714Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:58:52.3685120Z     return forward_call(*args, **kwargs)
2023-09-06T15:58:52.3685700Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T15:58:52.3686205Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T15:58:52.3686846Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T15:58:52.3687347Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T15:58:52.3687992Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:58:52.3688696Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:58:52.3689259Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:58:52.3689660Z     return forward_call(*args, **kwargs)
2023-09-06T15:58:52.3690195Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T15:58:52.3690630Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T15:58:52.3691224Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:58:52.3691648Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:58:52.3692188Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:58:52.3692586Z     return forward_call(*args, **kwargs)
2023-09-06T15:58:52.3693182Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:58:52.3693566Z     return roi_align(
2023-09-06T15:58:52.3694081Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:58:52.3694601Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:58:52.3695251Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:58:52.3695766Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:58:52.3696444Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:58:52.3696854Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:58:52.3697416Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:58:52.3697941Z     return input[
2023-09-06T15:58:52.3698905Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 13299.95 GiB. GPU 0 has a total capacty of 39.39 GiB of which 34.73 GiB is free. Process 2197451 has 4.65 GiB memory in use. Of the allocated memory 3.82 GiB is allocated by PyTorch, and 283.52 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:58:52.3699738Z 
2023-09-06T15:58:52.3699939Z The above exception was the direct cause of the following exception:
2023-09-06T15:58:52.3700182Z 
2023-09-06T15:58:52.3700312Z Traceback (most recent call last):
2023-09-06T15:58:52.3700696Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:58:52.3701032Z     ) = runner.load_model(
2023-09-06T15:58:52.3701422Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:58:52.3701826Z     self.validate_model(model, example_inputs)
2023-09-06T15:58:52.3702246Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:58:52.3702673Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:58:52.3703057Z NotImplementedError: Eager model failed to run
2023-09-06T15:58:52.3703301Z 
2023-09-06T15:58:52.3703472Z WARNING:root:detectron2_fasterrcnn_r_101_dc5 failed to load
2023-09-06T15:58:57.0627696Z 
2023-09-06T15:59:01.7634934Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:59:01.7635642Z loading model: 0it [00:04, ?it/s]
2023-09-06T15:59:01.7635901Z Eager model failed to run
2023-09-06T15:59:01.7651800Z Traceback (most recent call last):
2023-09-06T15:59:01.7652470Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:59:01.7653440Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:59:01.7654585Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:59:01.7654964Z     return mod(*inputs)
2023-09-06T15:59:01.7655780Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:59:01.7656213Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:59:01.7656777Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:59:01.7657168Z     return forward_call(*args, **kwargs)
2023-09-06T15:59:01.7657744Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:59:01.7658165Z     return self.inference(batched_inputs)
2023-09-06T15:59:01.7658748Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:59:01.7659224Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:59:01.7659840Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:59:01.7660265Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:59:01.7660822Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:59:01.7661222Z     return forward_call(*args, **kwargs)
2023-09-06T15:59:01.7661793Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T15:59:01.7662257Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T15:59:01.7663086Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T15:59:01.7663631Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T15:59:01.7664531Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:59:01.7664942Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:59:01.7665504Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:59:01.7665909Z     return forward_call(*args, **kwargs)
2023-09-06T15:59:01.7666461Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2023-09-06T15:59:01.7666910Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2023-09-06T15:59:01.7667532Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:59:01.7667947Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:59:01.7668503Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:59:01.7668902Z     return forward_call(*args, **kwargs)
2023-09-06T15:59:01.7669616Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:59:01.7669991Z     return roi_align(
2023-09-06T15:59:01.7670525Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:59:01.7671056Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:59:01.7671699Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:59:01.7672221Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:59:01.7672899Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:59:01.7673518Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:59:01.7674082Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:59:01.7674451Z     return input[
2023-09-06T15:59:01.7675414Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 19183.01 GiB. GPU 0 has a total capacty of 39.39 GiB of which 34.81 GiB is free. Process 2197547 has 4.57 GiB memory in use. Of the allocated memory 3.79 GiB is allocated by PyTorch, and 257.07 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:59:01.7676254Z 
2023-09-06T15:59:01.7676455Z The above exception was the direct cause of the following exception:
2023-09-06T15:59:01.7676699Z 
2023-09-06T15:59:01.7676830Z Traceback (most recent call last):
2023-09-06T15:59:01.7677224Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:59:01.7677584Z     ) = runner.load_model(
2023-09-06T15:59:01.7677951Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:59:01.7678353Z     self.validate_model(model, example_inputs)
2023-09-06T15:59:01.7678768Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:59:01.7679207Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:59:01.7679571Z NotImplementedError: Eager model failed to run
2023-09-06T15:59:01.7679779Z 
2023-09-06T15:59:01.7679965Z WARNING:root:detectron2_fasterrcnn_r_101_fpn failed to load
2023-09-06T15:59:06.4440843Z 
2023-09-06T15:59:10.7320958Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:59:10.7321331Z loading model: 0it [00:04, ?it/s]
2023-09-06T15:59:10.7321613Z Eager model failed to run
2023-09-06T15:59:10.7334453Z Traceback (most recent call last):
2023-09-06T15:59:10.7336578Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:59:10.7337209Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:59:10.7337632Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:59:10.7337997Z     return mod(*inputs)
2023-09-06T15:59:10.7338994Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:59:10.7339407Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:59:10.7339973Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:59:10.7340377Z     return forward_call(*args, **kwargs)
2023-09-06T15:59:10.7340947Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:59:10.7341369Z     return self.inference(batched_inputs)
2023-09-06T15:59:10.7341956Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:59:10.7342429Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:59:10.7343048Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:59:10.7343553Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:59:10.7344094Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:59:10.7344496Z     return forward_call(*args, **kwargs)
2023-09-06T15:59:10.7345080Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2023-09-06T15:59:10.7345515Z     box_features = self._shared_roi_transform(
2023-09-06T15:59:10.7346402Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2023-09-06T15:59:10.7346844Z     x = self.pooler(features, boxes)
2023-09-06T15:59:10.7347425Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:59:10.7347845Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:59:10.7348398Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:59:10.7348782Z     return forward_call(*args, **kwargs)
2023-09-06T15:59:10.7349634Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T15:59:10.7350278Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T15:59:10.7350895Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:59:10.7351315Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:59:10.7351874Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:59:10.7352270Z     return forward_call(*args, **kwargs)
2023-09-06T15:59:10.7352815Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:59:10.7353240Z     return roi_align(
2023-09-06T15:59:10.7353759Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:59:10.7354275Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:59:10.7354924Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:59:10.7355441Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:59:10.7356138Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:59:10.7356737Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:59:10.7357287Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:59:10.7357669Z     return input[
2023-09-06T15:59:10.7358637Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 27492.29 GiB. GPU 0 has a total capacty of 39.39 GiB of which 36.78 GiB is free. Process 2197639 has 2.61 GiB memory in use. Of the allocated memory 1.86 GiB is allocated by PyTorch, and 229.54 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:59:10.7359480Z 
2023-09-06T15:59:10.7359683Z The above exception was the direct cause of the following exception:
2023-09-06T15:59:10.7359927Z 
2023-09-06T15:59:10.7360070Z Traceback (most recent call last):
2023-09-06T15:59:10.7360433Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:59:10.7360780Z     ) = runner.load_model(
2023-09-06T15:59:10.7361156Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:59:10.7361557Z     self.validate_model(model, example_inputs)
2023-09-06T15:59:10.7361953Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:59:10.7362394Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:59:10.7362790Z NotImplementedError: Eager model failed to run
2023-09-06T15:59:10.7362998Z 
2023-09-06T15:59:10.7363225Z WARNING:root:detectron2_fasterrcnn_r_50_c4 failed to load
2023-09-06T15:59:15.3782381Z 
2023-09-06T15:59:21.9158571Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:59:21.9159274Z loading model: 0it [00:06, ?it/s]
2023-09-06T15:59:21.9159538Z Eager model failed to run
2023-09-06T15:59:21.9171070Z Traceback (most recent call last):
2023-09-06T15:59:21.9172026Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:59:21.9172651Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:59:21.9173160Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:59:21.9173635Z     return mod(*inputs)
2023-09-06T15:59:21.9177155Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:59:21.9177753Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:59:21.9178732Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:59:21.9179551Z     return forward_call(*args, **kwargs)
2023-09-06T15:59:21.9180456Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:59:21.9180910Z     return self.inference(batched_inputs)
2023-09-06T15:59:21.9181497Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:59:21.9181976Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:59:21.9182577Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:59:21.9183058Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:59:21.9183624Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:59:21.9184030Z     return forward_call(*args, **kwargs)
2023-09-06T15:59:21.9184605Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T15:59:21.9185060Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T15:59:21.9186192Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T15:59:21.9186702Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T15:59:21.9187404Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:59:21.9187826Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:59:21.9188365Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:59:21.9188771Z     return forward_call(*args, **kwargs)
2023-09-06T15:59:21.9189650Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T15:59:21.9190101Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T15:59:21.9190724Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:59:21.9191136Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:59:21.9191689Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:59:21.9192086Z     return forward_call(*args, **kwargs)
2023-09-06T15:59:21.9192627Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:59:21.9193044Z     return roi_align(
2023-09-06T15:59:21.9193575Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:59:21.9194125Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:59:21.9194779Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:59:21.9195480Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:59:21.9196160Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:59:21.9196581Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:59:21.9197134Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:59:21.9197517Z     return input[
2023-09-06T15:59:21.9198485Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 13521.00 GiB. GPU 0 has a total capacty of 39.39 GiB of which 35.88 GiB is free. Process 2197731 has 3.50 GiB memory in use. Of the allocated memory 2.68 GiB is allocated by PyTorch, and 270.78 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:59:21.9199622Z 
2023-09-06T15:59:21.9199829Z The above exception was the direct cause of the following exception:
2023-09-06T15:59:21.9200074Z 
2023-09-06T15:59:21.9200200Z Traceback (most recent call last):
2023-09-06T15:59:21.9200572Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:59:21.9200921Z     ) = runner.load_model(
2023-09-06T15:59:21.9201299Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:59:21.9201697Z     self.validate_model(model, example_inputs)
2023-09-06T15:59:21.9202093Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:59:21.9202535Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:59:21.9202950Z NotImplementedError: Eager model failed to run
2023-09-06T15:59:21.9203170Z 
2023-09-06T15:59:21.9203426Z WARNING:root:detectron2_fasterrcnn_r_50_dc5 failed to load
2023-09-06T15:59:26.5661073Z 
2023-09-06T15:59:30.8572800Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:59:30.8573571Z loading model: 0it [00:04, ?it/s]
2023-09-06T15:59:30.8573851Z Eager model failed to run
2023-09-06T15:59:30.8588445Z Traceback (most recent call last):
2023-09-06T15:59:30.8588912Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:59:30.8589560Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:59:30.8590315Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:59:30.8590991Z     return mod(*inputs)
2023-09-06T15:59:30.8592220Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:59:30.8593017Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:59:30.8593759Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:59:30.8594164Z     return forward_call(*args, **kwargs)
2023-09-06T15:59:30.8594748Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T15:59:30.8595174Z     return self.inference(batched_inputs)
2023-09-06T15:59:30.8595760Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T15:59:30.8596237Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T15:59:30.8596858Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:59:30.8597264Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:59:30.8597814Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:59:30.8598220Z     return forward_call(*args, **kwargs)
2023-09-06T15:59:30.8599309Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T15:59:30.8599801Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T15:59:30.8600423Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T15:59:30.8600931Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T15:59:30.8601576Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:59:30.8601990Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:59:30.8602528Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:59:30.8602929Z     return forward_call(*args, **kwargs)
2023-09-06T15:59:30.8603545Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2023-09-06T15:59:30.8604018Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2023-09-06T15:59:30.8604638Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:59:30.8605048Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:59:30.8605597Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:59:30.8605993Z     return forward_call(*args, **kwargs)
2023-09-06T15:59:30.8606540Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T15:59:30.8606900Z     return roi_align(
2023-09-06T15:59:30.8607427Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2023-09-06T15:59:30.8607952Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2023-09-06T15:59:30.8608830Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2023-09-06T15:59:30.8609351Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2023-09-06T15:59:30.8610008Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2023-09-06T15:59:30.8610427Z     v1 = masked_index(y_low, x_low)
2023-09-06T15:59:30.8610976Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2023-09-06T15:59:30.8611355Z     return input[
2023-09-06T15:59:30.8612327Z torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 21352.77 GiB. GPU 0 has a total capacty of 39.39 GiB of which 36.12 GiB is free. Process 2197829 has 3.27 GiB memory in use. Of the allocated memory 2.65 GiB is allocated by PyTorch, and 99.25 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
2023-09-06T15:59:30.8613221Z 
2023-09-06T15:59:30.8613424Z The above exception was the direct cause of the following exception:
2023-09-06T15:59:30.8613652Z 
2023-09-06T15:59:30.8613777Z Traceback (most recent call last):
2023-09-06T15:59:30.8614177Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:59:30.8614529Z     ) = runner.load_model(
2023-09-06T15:59:30.8614908Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:59:30.8615300Z     self.validate_model(model, example_inputs)
2023-09-06T15:59:30.8615713Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:59:30.8616156Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:59:30.8616537Z NotImplementedError: Eager model failed to run
2023-09-06T15:59:30.8616754Z 
2023-09-06T15:59:30.8617036Z WARNING:root:detectron2_fasterrcnn_r_50_fpn failed to load
2023-09-06T15:59:35.5163574Z 
2023-09-06T15:59:39.6771009Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:59:39.6771624Z loading model: 0it [00:04, ?it/s]
2023-09-06T15:59:39.6860330Z cuda eval  detectron2_fcos_r_50_fpn           
2023-09-06T15:59:44.1742587Z ERROR:common:Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time.You will need to explicitly give hint to the compiler. Please take a look at constrain_as_value OR constrain_as_size APIs
2023-09-06T15:59:44.1743852Z 
2023-09-06T15:59:44.1744026Z from user code:
2023-09-06T15:59:44.1745157Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/dense_detector.py", line 95, in forward
2023-09-06T15:59:44.1746060Z     images = self.preprocess_image(batched_inputs)
2023-09-06T15:59:44.1747352Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/dense_detector.py", line 129, in preprocess_image
2023-09-06T15:59:44.1747855Z     images = ImageList.from_tensors(
2023-09-06T15:59:44.1748769Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/structures/image_list.py", line 122, in from_tensors
2023-09-06T15:59:44.1749396Z     batched_imgs = tensors[0].new_full(batch_shape, pad_value, device=device)
2023-09-06T15:59:44.1749648Z 
2023-09-06T15:59:44.1749850Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T15:59:44.1750209Z Traceback (most recent call last):
2023-09-06T15:59:44.1750757Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1404, in run_node
2023-09-06T15:59:44.1751196Z     return getattr(args[0], node.target)(*args[1:], **kwargs)
2023-09-06T15:59:44.1751738Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_stats.py", line 20, in wrapper
2023-09-06T15:59:44.1752612Z     return fn(*args, **kwargs)
2023-09-06T15:59:44.1753250Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1290, in __torch_dispatch__
2023-09-06T15:59:44.1753716Z     return self.dispatch(func, types, args, kwargs)
2023-09-06T15:59:44.1754297Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1495, in dispatch
2023-09-06T15:59:44.1755173Z     return decomposition_table[func](*args, **kwargs)
2023-09-06T15:59:44.1755734Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_refs/__init__.py", line 4688, in new_full
2023-09-06T15:59:44.1756087Z     return torch.full(
2023-09-06T15:59:44.1756650Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_stats.py", line 20, in wrapper
2023-09-06T15:59:44.1757024Z     return fn(*args, **kwargs)
2023-09-06T15:59:44.1757605Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1290, in __torch_dispatch__
2023-09-06T15:59:44.1758045Z     return self.dispatch(func, types, args, kwargs)
2023-09-06T15:59:44.1758632Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1532, in dispatch
2023-09-06T15:59:44.1759065Z     op_impl_out = op_impl(self, func, *args, **kwargs)
2023-09-06T15:59:44.1759666Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 447, in constructors
2023-09-06T15:59:44.1760052Z     r = func(*args, **new_kwargs)
2023-09-06T15:59:44.1760546Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 448, in __call__
2023-09-06T15:59:44.1760927Z     return self._op(*args, **kwargs or {})
2023-09-06T15:59:44.1761474Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_meta_registrations.py", line 4433, in full
2023-09-06T15:59:44.1761868Z     return torch.empty(size, *args, **kwargs)
2023-09-06T15:59:44.1762692Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 985, in guard_bool
2023-09-06T15:59:44.1763260Z     r = self.shape_env.evaluate_expr(self.expr, self.hint, fx_node=self.fx_node)
2023-09-06T15:59:44.1763941Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 3491, in evaluate_expr
2023-09-06T15:59:44.1764384Z     concrete_val = self.size_hint(orig_expr)
2023-09-06T15:59:44.1764956Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 3292, in size_hint
2023-09-06T15:59:44.1765424Z     raise self._make_data_dependent_error(result_expr, expr)
2023-09-06T15:59:44.1766725Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: It appears that you're trying to get a value out of symbolic int/float whose value is data-dependent (and thus we do not know the true value.)  The expression we were trying to evaluate is Eq(12*i0*i1, 0) (unhinted: Eq(12*i0*i1, 0)).  Scroll up to see where each of these data-dependent accesses originally occurred.
2023-09-06T15:59:44.1767467Z 
2023-09-06T15:59:44.1767665Z The above exception was the direct cause of the following exception:
2023-09-06T15:59:44.1767906Z 
2023-09-06T15:59:44.1768032Z Traceback (most recent call last):
2023-09-06T15:59:44.1768589Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1342, in get_fake_value
2023-09-06T15:59:44.1768972Z     return wrap_fake_exception(
2023-09-06T15:59:44.1769683Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 917, in wrap_fake_exception
2023-09-06T15:59:44.1770229Z     return fn()
2023-09-06T15:59:44.1771026Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1343, in <lambda>
2023-09-06T15:59:44.1771683Z     lambda: run_node(tx.output, node, args, kwargs, nnmodule)
2023-09-06T15:59:44.1772604Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1415, in run_node
2023-09-06T15:59:44.1773766Z     raise RuntimeError(fn_str + str(e)).with_traceback(e.__traceback__) from e
2023-09-06T15:59:44.1774689Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1404, in run_node
2023-09-06T15:59:44.1775365Z     return getattr(args[0], node.target)(*args[1:], **kwargs)
2023-09-06T15:59:44.1776200Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_stats.py", line 20, in wrapper
2023-09-06T15:59:44.1776760Z     return fn(*args, **kwargs)
2023-09-06T15:59:44.1777649Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1290, in __torch_dispatch__
2023-09-06T15:59:44.1778343Z     return self.dispatch(func, types, args, kwargs)
2023-09-06T15:59:44.1779238Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1495, in dispatch
2023-09-06T15:59:44.1779929Z     return decomposition_table[func](*args, **kwargs)
2023-09-06T15:59:44.1780945Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_refs/__init__.py", line 4688, in new_full
2023-09-06T15:59:44.1781337Z     return torch.full(
2023-09-06T15:59:44.1782185Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_stats.py", line 20, in wrapper
2023-09-06T15:59:44.1782870Z     return fn(*args, **kwargs)
2023-09-06T15:59:44.1783982Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1290, in __torch_dispatch__
2023-09-06T15:59:44.1784660Z     return self.dispatch(func, types, args, kwargs)
2023-09-06T15:59:44.1785516Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1532, in dispatch
2023-09-06T15:59:44.1786119Z     op_impl_out = op_impl(self, func, *args, **kwargs)
2023-09-06T15:59:44.1786950Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 447, in constructors
2023-09-06T15:59:44.1787896Z     r = func(*args, **new_kwargs)
2023-09-06T15:59:44.1788654Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 448, in __call__
2023-09-06T15:59:44.1789539Z     return self._op(*args, **kwargs or {})
2023-09-06T15:59:44.1790366Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_meta_registrations.py", line 4433, in full
2023-09-06T15:59:44.1790958Z     return torch.empty(size, *args, **kwargs)
2023-09-06T15:59:44.1791801Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 985, in guard_bool
2023-09-06T15:59:44.1792507Z     r = self.shape_env.evaluate_expr(self.expr, self.hint, fx_node=self.fx_node)
2023-09-06T15:59:44.1793487Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 3491, in evaluate_expr
2023-09-06T15:59:44.1794113Z     concrete_val = self.size_hint(orig_expr)
2023-09-06T15:59:44.1794978Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 3292, in size_hint
2023-09-06T15:59:44.1795629Z     raise self._make_data_dependent_error(result_expr, expr)
2023-09-06T15:59:44.1796905Z RuntimeError: Failed running call_method new_full(*(FakeTensor(..., device='cuda:0', size=(3, 800, 1199), dtype=torch.bfloat16), [4, 3, FakeTensor(..., size=(), dtype=torch.int64), FakeTensor(..., size=(), dtype=torch.int64)], 0.0), **{'device': None}):
2023-09-06T15:59:44.1798776Z It appears that you're trying to get a value out of symbolic int/float whose value is data-dependent (and thus we do not know the true value.)  The expression we were trying to evaluate is Eq(12*i0*i1, 0) (unhinted: Eq(12*i0*i1, 0)).  Scroll up to see where each of these data-dependent accesses originally occurred.
2023-09-06T15:59:44.1799618Z 
2023-09-06T15:59:44.1799868Z During handling of the above exception, another exception occurred:
2023-09-06T15:59:44.1800252Z 
2023-09-06T15:59:44.1800765Z Traceback (most recent call last):
2023-09-06T15:59:44.1801300Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2135, in check_accuracy
2023-09-06T15:59:44.1801898Z     new_result = optimized_model_iter_fn(model_copy, example_inputs)
2023-09-06T15:59:44.1802498Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1190, in opt_aot_inductor
2023-09-06T15:59:44.1803119Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-09-06T15:59:44.1803819Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1134, in load
2023-09-06T15:59:44.1804364Z     so_path, exported = torch._export.aot_compile(
2023-09-06T15:59:44.1805231Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 635, in aot_compile
2023-09-06T15:59:44.1805799Z     ep = export(f, args, kwargs, constraints)
2023-09-06T15:59:44.1806577Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 307, in export
2023-09-06T15:59:44.1807178Z     gm_torch_level, _ = torch._dynamo.export(
2023-09-06T15:59:44.1807962Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1150, in inner
2023-09-06T15:59:44.1808531Z     result_traced = opt_f(*args, **kwargs)
2023-09-06T15:59:44.1809315Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:59:44.1809900Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:59:44.1810681Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:59:44.1811242Z     return forward_call(*args, **kwargs)
2023-09-06T15:59:44.1811969Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T15:59:44.1812499Z     return fn(*args, **kwargs)
2023-09-06T15:59:44.1813618Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:59:44.1814232Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:59:44.1815005Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:59:44.1815564Z     return forward_call(*args, **kwargs)
2023-09-06T15:59:44.1816363Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T15:59:44.1817004Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T15:59:44.1817817Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T15:59:44.1818338Z     return fn(*args, **kwargs)
2023-09-06T15:59:44.1819109Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T15:59:44.1819645Z     return _compile(
2023-09-06T15:59:44.1820369Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T15:59:44.1821031Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T15:59:44.1821847Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T15:59:44.1822371Z     r = func(*args, **kwargs)
2023-09-06T15:59:44.1823104Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T15:59:44.1823780Z     out_code = transform_code_object(code, transform)
2023-09-06T15:59:44.1824682Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T15:59:44.1825343Z     transformations(instructions, code_options)
2023-09-06T15:59:44.1826172Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T15:59:44.1827050Z     tracer.run()
2023-09-06T15:59:44.1827775Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T15:59:44.1828306Z     super().run()
2023-09-06T15:59:44.1829043Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T15:59:44.1829789Z     and self.step()
2023-09-06T15:59:44.1830551Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T15:59:44.1831100Z     getattr(self, inst.opname)(inst)
2023-09-06T15:59:44.1831905Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T15:59:44.1832465Z     return inner_fn(self, inst)
2023-09-06T15:59:44.1833326Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1119, in CALL_FUNCTION
2023-09-06T15:59:44.1833917Z     self.call_function(fn, args, {})
2023-09-06T15:59:44.1834723Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 565, in call_function
2023-09-06T15:59:44.1835348Z     self.push(fn.call_function(self, args, kwargs))
2023-09-06T15:59:44.1836208Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-09-06T15:59:44.1836819Z     return super().call_function(tx, args, kwargs)
2023-09-06T15:59:44.1837614Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-09-06T15:59:44.1838220Z     return super().call_function(tx, args, kwargs)
2023-09-06T15:59:44.1839019Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-09-06T15:59:44.1839600Z     return tx.inline_user_function_return(
2023-09-06T15:59:44.1840762Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 601, in inline_user_function_return
2023-09-06T15:59:44.1841506Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-09-06T15:59:44.1842431Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2183, in inline_call
2023-09-06T15:59:44.1843032Z     return cls.inline_call_(parent, func, args, kwargs)
2023-09-06T15:59:44.1843914Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2290, in inline_call_
2023-09-06T15:59:44.1844421Z     tracer.run()
2023-09-06T15:59:44.1845162Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T15:59:44.1845693Z     and self.step()
2023-09-06T15:59:44.1846438Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T15:59:44.1847006Z     getattr(self, inst.opname)(inst)
2023-09-06T15:59:44.1847801Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T15:59:44.1848372Z     return inner_fn(self, inst)
2023-09-06T15:59:44.1849207Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1171, in CALL_FUNCTION_KW
2023-09-06T15:59:44.1849821Z     self.call_function(fn, args, kwargs)
2023-09-06T15:59:44.1850635Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 565, in call_function
2023-09-06T15:59:44.1851264Z     self.push(fn.call_function(self, args, kwargs))
2023-09-06T15:59:44.1852124Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-09-06T15:59:44.1852753Z     return super().call_function(tx, args, kwargs)
2023-09-06T15:59:44.1853654Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-09-06T15:59:44.1854528Z     return tx.inline_user_function_return(
2023-09-06T15:59:44.1855408Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 601, in inline_user_function_return
2023-09-06T15:59:44.1856175Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-09-06T15:59:44.1857093Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2183, in inline_call
2023-09-06T15:59:44.1857675Z     return cls.inline_call_(parent, func, args, kwargs)
2023-09-06T15:59:44.1858491Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2290, in inline_call_
2023-09-06T15:59:44.1859026Z     tracer.run()
2023-09-06T15:59:44.1859740Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T15:59:44.1860233Z     and self.step()
2023-09-06T15:59:44.1860946Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T15:59:44.1861490Z     getattr(self, inst.opname)(inst)
2023-09-06T15:59:44.1862294Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T15:59:44.1863086Z     return inner_fn(self, inst)
2023-09-06T15:59:44.1863961Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1171, in CALL_FUNCTION_KW
2023-09-06T15:59:44.1864567Z     self.call_function(fn, args, kwargs)
2023-09-06T15:59:44.1865391Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 565, in call_function
2023-09-06T15:59:44.1866023Z     self.push(fn.call_function(self, args, kwargs))
2023-09-06T15:59:44.1867114Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 594, in call_function
2023-09-06T15:59:44.1867841Z     return self.obj.call_method(tx, self.name, args, kwargs).add_options(self)
2023-09-06T15:59:44.1868765Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 652, in call_method
2023-09-06T15:59:44.1869501Z     return wrap_fx_proxy(
2023-09-06T15:59:44.1870306Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1216, in wrap_fx_proxy
2023-09-06T15:59:44.1870865Z     return wrap_fx_proxy_cls(
2023-09-06T15:59:44.1871697Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1303, in wrap_fx_proxy_cls
2023-09-06T15:59:44.1872340Z     example_value = get_fake_value(proxy.node, tx)
2023-09-06T15:59:44.1873223Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1371, in get_fake_value
2023-09-06T15:59:44.1873730Z     raise UserError(
2023-09-06T15:59:44.1875197Z torch._dynamo.exc.UserError: Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time.You will need to explicitly give hint to the compiler. Please take a look at constrain_as_value OR constrain_as_size APIs
2023-09-06T15:59:44.1876059Z 
2023-09-06T15:59:44.1876184Z from user code:
2023-09-06T15:59:44.1877018Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/dense_detector.py", line 95, in forward
2023-09-06T15:59:44.1877630Z     images = self.preprocess_image(batched_inputs)
2023-09-06T15:59:44.1878500Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/dense_detector.py", line 129, in preprocess_image
2023-09-06T15:59:44.1879101Z     images = ImageList.from_tensors(
2023-09-06T15:59:44.1879925Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/structures/image_list.py", line 122, in from_tensors
2023-09-06T15:59:44.1880927Z     batched_imgs = tensors[0].new_full(batch_shape, pad_value, device=device)
2023-09-06T15:59:44.1881274Z 
2023-09-06T15:59:44.1881551Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T15:59:44.1881887Z 
2023-09-06T15:59:44.1882151Z TorchDynamo optimized model failed to run because of following error
2023-09-06T15:59:44.1882619Z fail_to_run
2023-09-06T15:59:45.5502917Z accuracy            pass_rate=60.00%
2023-09-06T15:59:45.5503653Z calls_captured      gmean=0.00x mean=116.333x
2023-09-06T15:59:45.5506900Z unique_graphs       gmean=0.00x mean=0.667x
2023-09-06T15:59:45.5507987Z graph_breaks        gmean=0.00x mean=0.000x
2023-09-06T15:59:45.5510932Z unique_graph_breaks gmean=0.00x mean=0.000x
2023-09-06T15:59:46.3014272Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *maxautotune-true* ]]
2023-09-06T15:59:46.3015104Z + for target in "${targets[@]}"
2023-09-06T15:59:46.3015810Z + target_flag=("--${target}")
2023-09-06T15:59:46.3016102Z + local target_flag
2023-09-06T15:59:46.3016403Z + [[ performance == \p\e\r\f\o\r\m\a\n\c\e ]]
2023-09-06T15:59:46.3016770Z + target_flag+=(--cold-start-latency)
2023-09-06T15:59:46.3017628Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *default-true* ]]
2023-09-06T15:59:46.3019178Z + python benchmarks/dynamo/torchbench.py --performance --cold-start-latency --inference --bfloat16 --backend inductor --disable-cudagraphs --device cuda --total-partitions 4 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inductor_no_cudagraphs_torchbench_bfloat16_inference_cuda_performance.csv
2023-09-06T15:59:53.5435511Z 
2023-09-06T15:59:54.4570869Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:59:54.4571404Z loading model: 0it [00:00, ?it/s]
2023-09-06T15:59:54.4571668Z Eager model failed to run
2023-09-06T15:59:54.4593145Z Traceback (most recent call last):
2023-09-06T15:59:54.4594661Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T15:59:54.4595291Z     self.model_iter_fn(model, example_inputs)
2023-09-06T15:59:54.4595867Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T15:59:54.4596412Z     return mod(*inputs)
2023-09-06T15:59:54.4599329Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:59:54.4600171Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:59:54.4600930Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:59:54.4601328Z     return forward_call(*args, **kwargs)
2023-09-06T15:59:54.4601883Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 893, in forward
2023-09-06T15:59:54.4602354Z     logits = self.model(batch.dense_features, batch.sparse_features)
2023-09-06T15:59:54.4603005Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:59:54.4603438Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:59:54.4604042Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:59:54.4604465Z     return forward_call(*args, **kwargs)
2023-09-06T15:59:54.4605010Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 570, in forward
2023-09-06T15:59:54.4605452Z     embedded_dense = self.dense_arch(dense_features)
2023-09-06T15:59:54.4606049Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:59:54.4606522Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:59:54.4607090Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:59:54.4607860Z     return forward_call(*args, **kwargs)
2023-09-06T15:59:54.4608416Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 149, in forward
2023-09-06T15:59:54.4608791Z     return self.model(features)
2023-09-06T15:59:54.4609353Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:59:54.4609772Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:59:54.4610333Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:59:54.4610740Z     return forward_call(*args, **kwargs)
2023-09-06T15:59:54.4611297Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/mlp.py", line 172, in forward
2023-09-06T15:59:54.4611686Z     return self._mlp(input)
2023-09-06T15:59:54.4612247Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:59:54.4612732Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:59:54.4613275Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:59:54.4613720Z     return forward_call(*args, **kwargs)
2023-09-06T15:59:54.4614274Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/container.py", line 215, in forward
2023-09-06T15:59:54.4614656Z     input = module(input)
2023-09-06T15:59:54.4615214Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:59:54.4615619Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:59:54.4616180Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:59:54.4616590Z     return forward_call(*args, **kwargs)
2023-09-06T15:59:54.4617322Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/mlp.py", line 73, in forward
2023-09-06T15:59:54.4617738Z     return self._activation_fn(self._linear(input))
2023-09-06T15:59:54.4618334Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T15:59:54.4618753Z     return self._call_impl(*args, **kwargs)
2023-09-06T15:59:54.4619305Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T15:59:54.4619713Z     return forward_call(*args, **kwargs)
2023-09-06T15:59:54.4620236Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
2023-09-06T15:59:54.4620684Z     return F.linear(input, self.weight, self.bias)
2023-09-06T15:59:54.4621105Z RuntimeError: mat1 and mat2 must have the same dtype, but got Float and BFloat16
2023-09-06T15:59:54.4621368Z 
2023-09-06T15:59:54.4621574Z The above exception was the direct cause of the following exception:
2023-09-06T15:59:54.4621831Z 
2023-09-06T15:59:54.4621943Z Traceback (most recent call last):
2023-09-06T15:59:54.4622326Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T15:59:54.4622690Z     ) = runner.load_model(
2023-09-06T15:59:54.4623078Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T15:59:54.4623472Z     self.validate_model(model, example_inputs)
2023-09-06T15:59:54.4623954Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T15:59:54.4624406Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T15:59:54.4624787Z NotImplementedError: Eager model failed to run
2023-09-06T15:59:54.4624998Z 
2023-09-06T15:59:54.4625134Z WARNING:root:torchrec_dlrm failed to load
2023-09-06T15:59:58.9772701Z 
2023-09-06T16:00:02.2963671Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:00:02.2965012Z loading model: 0it [00:03, ?it/s]
2023-09-06T16:00:02.3015655Z cuda eval  BERT_pytorch                       
2023-09-06T16:00:30.9889993Z 
2023-09-06T16:00:31.1066725Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:00:31.2429872Z running benchmark:  20% 6/30 [00:00<00:00, 50.81it/s]
2023-09-06T16:00:31.3557093Z running benchmark:  40% 12/30 [00:00<00:00, 46.57it/s]
2023-09-06T16:00:31.4746440Z running benchmark:  60% 18/30 [00:00<00:00, 49.34it/s]
2023-09-06T16:00:31.5878647Z running benchmark:  80% 24/30 [00:00<00:00, 49.79it/s]
2023-09-06T16:00:31.5882062Z running benchmark: 100% 30/30 [00:00<00:00, 50.90it/s]
2023-09-06T16:00:31.5882469Z running benchmark: 100% 30/30 [00:00<00:00, 50.02it/s]
2023-09-06T16:00:31.5895386Z 1.967x
2023-09-06T16:00:36.7960826Z 
2023-09-06T16:00:40.4920118Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:00:40.4920640Z loading model: 0it [00:03, ?it/s]
2023-09-06T16:00:40.4975313Z cuda eval  Background_Matting                 
2023-09-06T16:01:04.4187863Z 
2023-09-06T16:01:04.5421368Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:01:04.6464685Z running benchmark:  13% 4/30 [00:00<00:00, 32.37it/s]
2023-09-06T16:01:04.7509612Z running benchmark:  27% 8/30 [00:00<00:00, 35.61it/s]
2023-09-06T16:01:04.8556585Z running benchmark:  40% 12/30 [00:00<00:00, 36.77it/s]
2023-09-06T16:01:04.9769404Z running benchmark:  53% 16/30 [00:00<00:00, 37.32it/s]
2023-09-06T16:01:05.0801959Z running benchmark:  67% 20/30 [00:00<00:00, 35.65it/s]
2023-09-06T16:01:05.1832794Z running benchmark:  80% 24/30 [00:00<00:00, 36.62it/s]
2023-09-06T16:01:05.2353315Z running benchmark:  93% 28/30 [00:00<00:00, 37.30it/s]
2023-09-06T16:01:05.2353711Z running benchmark: 100% 30/30 [00:00<00:00, 36.72it/s]
2023-09-06T16:01:05.2374870Z 1.606x
2023-09-06T16:01:10.6155676Z 
2023-09-06T16:01:20.3521872Z loading model: 0it [00:00, ?it/s]WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T16:01:21.9111050Z 
2023-09-06T16:01:21.9111651Z loading model: 0it [00:11, ?it/s]
2023-09-06T16:01:21.9118003Z WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T16:01:21.9118659Z cuda eval  DALLE2_pytorch                     
2023-09-06T16:01:22.0827636Z WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T16:01:35.1569618Z [2023-09-06 16:01:35,155] [2/1] torch._dynamo.output_graph: [WARNING] nn.Module forward/_pre hooks are only partially supported, and were detected in your model. In particular, if you do not change/remove hooks after calling .compile(), you can disregard this warning, and otherwise you may need to set torch._dynamo.config.skip_nnmodule_hook_guards=False to ensure recompiling after changing hooks.See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations. 
2023-09-06T16:01:50.5578819Z [2023-09-06 16:01:50,556] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:01:50.8818636Z [2023-09-06 16:01:50,881] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:01:51.0006642Z [2023-09-06 16:01:50,990] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:01:51.1145277Z [2023-09-06 16:01:51,113] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:01:51.2243245Z [2023-09-06 16:01:51,223] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:01:51.3330451Z [2023-09-06 16:01:51,332] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:01:51.4440599Z [2023-09-06 16:01:51,443] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:01:51.5559339Z [2023-09-06 16:01:51,554] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:01:51.6681874Z [2023-09-06 16:01:51,667] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:01:51.7773820Z [2023-09-06 16:01:51,776] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:01:51.8845099Z [2023-09-06 16:01:51,883] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:01:51.9923377Z [2023-09-06 16:01:51,991] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:02:53.8683953Z [2023-09-06 16:02:53,866] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:02:53.9674831Z [2023-09-06 16:02:53,966] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:02:54.0622491Z [2023-09-06 16:02:54,061] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:02:54.1564296Z [2023-09-06 16:02:54,155] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:02:54.2551359Z [2023-09-06 16:02:54,254] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:02:54.3496096Z [2023-09-06 16:02:54,348] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:02:54.4438108Z [2023-09-06 16:02:54,443] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:02:54.5542460Z [2023-09-06 16:02:54,553] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:02:54.6517569Z [2023-09-06 16:02:54,650] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:02:54.7463725Z [2023-09-06 16:02:54,745] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:02:54.8412126Z [2023-09-06 16:02:54,840] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:02:54.9376526Z [2023-09-06 16:02:54,936] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:03:25.1010429Z ERROR:common:Backend dynamo failed in warmup()
2023-09-06T16:03:25.1010806Z Traceback (most recent call last):
2023-09-06T16:03:25.1011213Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2288, in warmup
2023-09-06T16:03:25.1011615Z     fn(model, example_inputs)
2023-09-06T16:03:25.1014894Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T16:03:25.1015322Z     return fn(*args, **kwargs)
2023-09-06T16:03:25.1016840Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:03:25.1017558Z     return mod(*inputs)
2023-09-06T16:03:25.1018593Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:03:25.1019273Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:03:25.1020282Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:03:25.1020697Z     return forward_call(*args, **kwargs)
2023-09-06T16:03:25.1021279Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T16:03:25.1021724Z     return func(*args, **kwargs)
2023-09-06T16:03:25.1022302Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 95, in inner
2023-09-06T16:03:25.1022661Z     model.eval()
2023-09-06T16:03:25.1023274Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 96, in <resume in inner>
2023-09-06T16:03:25.1023688Z     out = fn(model, *args, **kwargs)
2023-09-06T16:03:25.1024246Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3329, in forward
2023-09-06T16:03:25.1024794Z     image_embed = self.prior.sample(text, num_samples_per_batch = self.prior_num_samples, cond_scale = prior_cond_scale)
2023-09-06T16:03:25.1025520Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3332, in <resume in forward>
2023-09-06T16:03:25.1026065Z     images = self.decoder.sample(image_embed = image_embed, text = text_cond, cond_scale = cond_scale)
2023-09-06T16:03:25.1027068Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T16:03:25.1027465Z     return func(*args, **kwargs)
2023-09-06T16:03:25.1027993Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 95, in inner
2023-09-06T16:03:25.1028367Z     model.eval()
2023-09-06T16:03:25.1028914Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 96, in <resume in inner>
2023-09-06T16:03:25.1029678Z     out = fn(model, *args, **kwargs)
2023-09-06T16:03:25.1030235Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3199, in sample
2023-09-06T16:03:25.1030629Z     img = self.p_sample_loop(
2023-09-06T16:03:25.1031181Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T16:03:25.1031614Z     return func(*args, **kwargs)
2023-09-06T16:03:25.1032288Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3028, in p_sample_loop
2023-09-06T16:03:25.1032816Z     return self.p_sample_loop_ddpm(*args, noise_scheduler = noise_scheduler, **kwargs)
2023-09-06T16:03:25.1033455Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T16:03:25.1033855Z     return func(*args, **kwargs)
2023-09-06T16:03:25.1034432Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2885, in p_sample_loop_ddpm
2023-09-06T16:03:25.1034833Z     img, x_start = self.p_sample(
2023-09-06T16:03:25.1035390Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T16:03:25.1035785Z     return func(*args, **kwargs)
2023-09-06T16:03:25.1036521Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2822, in p_sample
2023-09-06T16:03:25.1037551Z     model_mean, _, model_log_variance, x_start = self.p_mean_variance(unet, x = x, t = t, image_embed = image_embed, text_encodings = text_encodings, cond_scale = cond_scale, lowres_cond_img = lowres_cond_img, self_cond = self_cond, clip_denoised = clip_denoised, predict_x_start = predict_x_start, predict_v = predict_v, noise_scheduler = noise_scheduler, learned_variance = learned_variance, lowres_noise_level = lowres_noise_level)
2023-09-06T16:03:25.1038721Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2787, in p_mean_variance
2023-09-06T16:03:25.1039561Z     model_output = default(model_output, lambda: unet.forward_with_cond_scale(x, t, image_embed = image_embed, text_encodings = text_encodings, cond_scale = cond_scale, lowres_cond_img = lowres_cond_img, self_cond = self_cond, lowres_noise_level = lowres_noise_level))
2023-09-06T16:03:25.1040455Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 65, in default
2023-09-06T16:03:25.1040852Z     return d() if callable(d) else d
2023-09-06T16:03:25.1041414Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2787, in <lambda>
2023-09-06T16:03:25.1042177Z     model_output = default(model_output, lambda: unet.forward_with_cond_scale(x, t, image_embed = image_embed, text_encodings = text_encodings, cond_scale = cond_scale, lowres_cond_img = lowres_cond_img, self_cond = self_cond, lowres_noise_level = lowres_noise_level))
2023-09-06T16:03:25.1043126Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2159, in forward_with_cond_scale
2023-09-06T16:03:25.1043560Z     logits = self.forward(*args, **kwargs)
2023-09-06T16:03:25.1044114Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2340, in forward
2023-09-06T16:03:25.1044503Z     x = resnet_block(x, t, c)
2023-09-06T16:03:25.1045230Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:03:25.1045652Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:03:25.1046213Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:03:25.1046603Z     return forward_call(*args, **kwargs)
2023-09-06T16:03:25.1047163Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T16:03:25.1047615Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T16:03:25.1048215Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 634, in _convert_frame
2023-09-06T16:03:25.1048684Z     result = inner_convert(frame, cache_entry, hooks, frame_state)
2023-09-06T16:03:25.1049257Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T16:03:25.1049646Z     return fn(*args, **kwargs)
2023-09-06T16:03:25.1050208Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T16:03:25.1050610Z     return _compile(
2023-09-06T16:03:25.1051123Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T16:03:25.1051587Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T16:03:25.1052188Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:03:25.1052564Z     r = func(*args, **kwargs)
2023-09-06T16:03:25.1053139Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T16:03:25.1053591Z     out_code = transform_code_object(code, transform)
2023-09-06T16:03:25.1054405Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T16:03:25.1054884Z     transformations(instructions, code_options)
2023-09-06T16:03:25.1055467Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T16:03:25.1055829Z     tracer.run()
2023-09-06T16:03:25.1056347Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T16:03:25.1056716Z     super().run()
2023-09-06T16:03:25.1057230Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T16:03:25.1057584Z     and self.step()
2023-09-06T16:03:25.1058111Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T16:03:25.1058505Z     getattr(self, inst.opname)(inst)
2023-09-06T16:03:25.1059070Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T16:03:25.1059470Z     return inner_fn(self, inst)
2023-09-06T16:03:25.1060018Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1119, in CALL_FUNCTION
2023-09-06T16:03:25.1060427Z     self.call_function(fn, args, {})
2023-09-06T16:03:25.1060991Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 565, in call_function
2023-09-06T16:03:25.1061427Z     self.push(fn.call_function(self, args, kwargs))
2023-09-06T16:03:25.1061996Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/torch.py", line 731, in call_function
2023-09-06T16:03:25.1062408Z     tensor_variable = wrap_fx_proxy(
2023-09-06T16:03:25.1063201Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1216, in wrap_fx_proxy
2023-09-06T16:03:25.1063815Z     return wrap_fx_proxy_cls(
2023-09-06T16:03:25.1064398Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1303, in wrap_fx_proxy_cls
2023-09-06T16:03:25.1064831Z     example_value = get_fake_value(proxy.node, tx)
2023-09-06T16:03:25.1065411Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1381, in get_fake_value
2023-09-06T16:03:25.1065882Z     raise TorchRuntimeError(str(e)).with_traceback(e.__traceback__) from None
2023-09-06T16:03:25.1066491Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1342, in get_fake_value
2023-09-06T16:03:25.1066858Z     return wrap_fake_exception(
2023-09-06T16:03:25.1067406Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 917, in wrap_fake_exception
2023-09-06T16:03:25.1067769Z     return fn()
2023-09-06T16:03:25.1068270Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1343, in <lambda>
2023-09-06T16:03:25.1068716Z     lambda: run_node(tx.output, node, args, kwargs, nnmodule)
2023-09-06T16:03:25.1069605Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1415, in run_node
2023-09-06T16:03:25.1070077Z     raise RuntimeError(fn_str + str(e)).with_traceback(e.__traceback__) from e
2023-09-06T16:03:25.1070678Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1402, in run_node
2023-09-06T16:03:25.1071071Z     return node.target(*args, **kwargs)
2023-09-06T16:03:25.1071582Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/einops/einops.py", line 483, in rearrange
2023-09-06T16:03:25.1072165Z     return reduce(cast(Tensor, tensor), pattern, reduction='rearrange', **axes_lengths)
2023-09-06T16:03:25.1072802Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/einops/einops.py", line 412, in reduce
2023-09-06T16:03:25.1073436Z     return _apply_recipe(recipe, tensor, reduction_type=reduction)
2023-09-06T16:03:25.1074043Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/einops/einops.py", line 235, in _apply_recipe
2023-09-06T16:03:25.1074463Z     _reconstruct_from_shape(recipe, backend.shape(tensor))
2023-09-06T16:03:25.1075252Z torch._dynamo.exc.TorchRuntimeError: Failed running call_function <function rearrange at 0x7fc950ae1fc0>(*(FakeTensor(..., device='cuda:0', size=(1, 128, s0, s1)), 'b c ... -> b ... c'), **{}):
2023-09-06T16:03:25.1075799Z unhashable type: 'SymInt'
2023-09-06T16:03:25.1075970Z 
2023-09-06T16:03:25.1076062Z from user code:
2023-09-06T16:03:25.1076580Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 1670, in forward
2023-09-06T16:03:25.1077056Z     h = rearrange(h, 'b c ... -> b ... c')
2023-09-06T16:03:25.1077235Z 
2023-09-06T16:03:25.1077426Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T16:03:25.1077666Z 
2023-09-06T16:03:25.1077678Z 
2023-09-06T16:03:25.1077876Z You can suppress this exception and fall back to eager by setting:
2023-09-06T16:03:25.1078210Z     import torch._dynamo
2023-09-06T16:03:25.1078510Z     torch._dynamo.config.suppress_errors = True
2023-09-06T16:03:25.1078715Z 
2023-09-06T16:03:27.7179113Z ERROR
2023-09-06T16:03:31.3267182Z 
2023-09-06T16:03:35.8382407Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:03:35.8383018Z loading model: 0it [00:04, ?it/s]
2023-09-06T16:03:35.8404594Z cuda eval  LearningToPaint                    
2023-09-06T16:03:51.2955194Z 
2023-09-06T16:03:51.4007386Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:03:51.5045226Z running benchmark:  27% 8/30 [00:00<00:00, 76.05it/s]
2023-09-06T16:03:51.6078506Z running benchmark:  60% 18/30 [00:00<00:00, 87.83it/s]
2023-09-06T16:03:51.6288155Z running benchmark:  93% 28/30 [00:00<00:00, 91.85it/s]
2023-09-06T16:03:51.6288728Z running benchmark: 100% 30/30 [00:00<00:00, 89.92it/s]
2023-09-06T16:03:51.6309268Z 1.802x
2023-09-06T16:03:56.9986797Z 
2023-09-06T16:03:59.4462972Z loading model: 0it [00:00, ?it/s]WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T16:03:59.9276448Z 
2023-09-06T16:03:59.9276931Z loading model: 0it [00:02, ?it/s]
2023-09-06T16:03:59.9285048Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T16:03:59.9285492Z cuda eval  Super_SloMo                        
2023-09-06T16:03:59.9390477Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T16:04:48.8951541Z 
2023-09-06T16:04:49.0263337Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:04:49.1792856Z running benchmark:   3% 1/30 [00:00<00:03,  7.60it/s]
2023-09-06T16:04:49.3262849Z running benchmark:  10% 3/30 [00:00<00:02, 11.02it/s]
2023-09-06T16:04:49.4730940Z running benchmark:  17% 5/30 [00:00<00:02, 12.20it/s]
2023-09-06T16:04:49.6203354Z running benchmark:  23% 7/30 [00:00<00:01, 12.77it/s]
2023-09-06T16:04:49.7670559Z running benchmark:  30% 9/30 [00:00<00:01, 13.07it/s]
2023-09-06T16:04:49.9140844Z running benchmark:  37% 11/30 [00:00<00:01, 13.26it/s]
2023-09-06T16:04:50.0611519Z running benchmark:  43% 13/30 [00:01<00:01, 13.37it/s]
2023-09-06T16:04:50.2079243Z running benchmark:  50% 15/30 [00:01<00:01, 13.44it/s]
2023-09-06T16:04:50.3547147Z running benchmark:  57% 17/30 [00:01<00:00, 13.50it/s]
2023-09-06T16:04:50.5017987Z running benchmark:  63% 19/30 [00:01<00:00, 13.54it/s]
2023-09-06T16:04:50.6486403Z running benchmark:  70% 21/30 [00:01<00:00, 13.56it/s]
2023-09-06T16:04:50.7956917Z running benchmark:  77% 23/30 [00:01<00:00, 13.58it/s]
2023-09-06T16:04:50.9423387Z running benchmark:  83% 25/30 [00:01<00:00, 13.58it/s]
2023-09-06T16:04:51.0891347Z running benchmark:  90% 27/30 [00:02<00:00, 13.59it/s]
2023-09-06T16:04:51.1627297Z running benchmark:  97% 29/30 [00:02<00:00, 13.61it/s]
2023-09-06T16:04:51.1627695Z running benchmark: 100% 30/30 [00:02<00:00, 13.23it/s]
2023-09-06T16:04:51.1670325Z 1.509x
2023-09-06T16:04:57.0365222Z 
2023-09-06T16:05:00.6039578Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:05:00.6039958Z loading model: 0it [00:03, ?it/s]
2023-09-06T16:05:00.6042127Z cuda eval  alexnet                            
2023-09-06T16:05:11.1611609Z 
2023-09-06T16:05:11.2693493Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:05:11.3985683Z running benchmark:   3% 1/30 [00:00<00:03,  9.24it/s]
2023-09-06T16:05:11.5276921Z running benchmark:  13% 4/30 [00:00<00:01, 18.03it/s]
2023-09-06T16:05:11.6569118Z running benchmark:  23% 7/30 [00:00<00:01, 20.49it/s]
2023-09-06T16:05:11.8022282Z running benchmark:  33% 10/30 [00:00<00:00, 21.60it/s]
2023-09-06T16:05:11.9313044Z running benchmark:  43% 13/30 [00:00<00:00, 21.23it/s]
2023-09-06T16:05:12.0603352Z running benchmark:  53% 16/30 [00:00<00:00, 21.89it/s]
2023-09-06T16:05:12.1893698Z running benchmark:  63% 19/30 [00:00<00:00, 22.33it/s]
2023-09-06T16:05:12.3184157Z running benchmark:  73% 22/30 [00:01<00:00, 22.62it/s]
2023-09-06T16:05:12.4475150Z running benchmark:  83% 25/30 [00:01<00:00, 22.81it/s]
2023-09-06T16:05:12.5336945Z running benchmark:  93% 28/30 [00:01<00:00, 22.95it/s]
2023-09-06T16:05:12.5337563Z running benchmark: 100% 30/30 [00:01<00:00, 21.85it/s]
2023-09-06T16:05:12.5353157Z 1.416x
2023-09-06T16:05:17.5342143Z 
2023-09-06T16:05:24.3524189Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:05:24.3525174Z loading model: 0it [00:06, ?it/s]
2023-09-06T16:05:24.3570743Z cuda eval  attention_is_all_you_need_pytorch  
2023-09-06T16:05:56.9737492Z 
2023-09-06T16:05:57.0821587Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:05:57.2070611Z running benchmark:  13% 4/30 [00:00<00:00, 36.86it/s]
2023-09-06T16:05:57.3084200Z running benchmark:  30% 9/30 [00:00<00:00, 38.82it/s]
2023-09-06T16:05:57.4326475Z running benchmark:  43% 13/30 [00:00<00:00, 39.09it/s]
2023-09-06T16:05:57.5328835Z running benchmark:  60% 18/30 [00:00<00:00, 39.57it/s]
2023-09-06T16:05:57.6573386Z running benchmark:  73% 22/30 [00:00<00:00, 39.68it/s]
2023-09-06T16:05:57.7335136Z running benchmark:  90% 27/30 [00:00<00:00, 39.87it/s]
2023-09-06T16:05:57.7335811Z running benchmark: 100% 30/30 [00:00<00:00, 39.47it/s]
2023-09-06T16:05:57.7359818Z 1.535x
2023-09-06T16:06:03.2970363Z 
2023-09-06T16:06:06.0624601Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:06:06.0624935Z loading model: 0it [00:02, ?it/s]
2023-09-06T16:06:06.0630126Z cuda eval  basic_gnn_edgecnn                  
2023-09-06T16:06:06.5489842Z [2023-09-06 16:06:06,547] [0/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T16:06:14.1378764Z 
2023-09-06T16:06:14.2387523Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:06:14.3008942Z running benchmark:  53% 16/30 [00:00<00:00, 158.87it/s]
2023-09-06T16:06:14.3009442Z running benchmark: 100% 30/30 [00:00<00:00, 183.91it/s]
2023-09-06T16:06:14.3023512Z 1.724x
2023-09-06T16:06:19.1209140Z 
2023-09-06T16:06:21.9090176Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:06:21.9090521Z loading model: 0it [00:02, ?it/s]
2023-09-06T16:06:21.9090838Z cuda eval  basic_gnn_gcn                      
2023-09-06T16:06:30.8978519Z ERROR:common:Backend dynamo failed in warmup()
2023-09-06T16:06:30.8979397Z Traceback (most recent call last):
2023-09-06T16:06:30.8980098Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2288, in warmup
2023-09-06T16:06:30.8980462Z     fn(model, example_inputs)
2023-09-06T16:06:30.8983783Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T16:06:30.8984219Z     return fn(*args, **kwargs)
2023-09-06T16:06:30.8984619Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:06:30.8985000Z     return mod(*inputs)
2023-09-06T16:06:30.8986002Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:06:30.8986442Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:06:30.8987052Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:06:30.8987903Z     return forward_call(*args, **kwargs)
2023-09-06T16:06:30.8988907Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/models/basic_gnn.py", line 222, in forward
2023-09-06T16:06:30.8990143Z     x = self.convs[i](x, edge_index, edge_weight=edge_weight)
2023-09-06T16:06:30.8990786Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:06:30.8991200Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:06:30.8991770Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:06:30.8992205Z     return forward_call(*args, **kwargs)
2023-09-06T16:06:30.8992558Z   File "/tmp/jenkins_pyg/tmpo5a_gllp.py", line 221, in forward
2023-09-06T16:06:30.8992925Z     edge_index, edge_weight = gcn_norm(  # yapf: disable
2023-09-06T16:06:30.8993540Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/gcn_conv.py", line 91, in gcn_norm
2023-09-06T16:06:30.8994001Z     edge_index, edge_weight = add_remaining_self_loops(
2023-09-06T16:06:30.8994608Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T16:06:30.8995074Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T16:06:30.8995671Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 634, in _convert_frame
2023-09-06T16:06:30.8996154Z     result = inner_convert(frame, cache_entry, hooks, frame_state)
2023-09-06T16:06:30.8996803Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T16:06:30.8997525Z     return fn(*args, **kwargs)
2023-09-06T16:06:30.8998170Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T16:06:30.8998582Z     return _compile(
2023-09-06T16:06:30.8999128Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T16:06:30.8999600Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T16:06:30.9000217Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:06:30.9000594Z     r = func(*args, **kwargs)
2023-09-06T16:06:30.9001149Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T16:06:30.9001604Z     out_code = transform_code_object(code, transform)
2023-09-06T16:06:30.9002281Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T16:06:30.9002744Z     transformations(instructions, code_options)
2023-09-06T16:06:30.9003342Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T16:06:30.9003721Z     tracer.run()
2023-09-06T16:06:30.9004253Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T16:06:30.9004628Z     super().run()
2023-09-06T16:06:30.9005132Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T16:06:30.9005512Z     and self.step()
2023-09-06T16:06:30.9006048Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T16:06:30.9006461Z     getattr(self, inst.opname)(inst)
2023-09-06T16:06:30.9007196Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2166, in RETURN_VALUE
2023-09-06T16:06:30.9007634Z     self.output.compile_subgraph(
2023-09-06T16:06:30.9008262Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 905, in compile_subgraph
2023-09-06T16:06:30.9008754Z     self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
2023-09-06T16:06:30.9009172Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T16:06:30.9009529Z     return func(*args, **kwds)
2023-09-06T16:06:30.9010116Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1008, in compile_and_call_fx_graph
2023-09-06T16:06:30.9010565Z     compiled_fn = self.call_user_compiler(gm)
2023-09-06T16:06:30.9011153Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:06:30.9011537Z     r = func(*args, **kwargs)
2023-09-06T16:06:30.9012112Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1075, in call_user_compiler
2023-09-06T16:06:30.9012613Z     raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
2023-09-06T16:06:30.9013257Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1060, in call_user_compiler
2023-09-06T16:06:30.9013701Z     compiled_fn = compiler_fn(gm, self.example_inputs())
2023-09-06T16:06:30.9014318Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper
2023-09-06T16:06:30.9014760Z     compiled_gm = compiler_fn(gm, example_inputs)
2023-09-06T16:06:30.9015350Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/inductor.py", line 9, in inductor
2023-09-06T16:06:30.9015845Z     return compile_fx(*args, **kwargs)
2023-09-06T16:06:30.9016413Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1167, in compile_fx
2023-09-06T16:06:30.9016922Z     return aot_autograd(
2023-09-06T16:06:30.9017476Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 55, in compiler_fn
2023-09-06T16:06:30.9017989Z     cg = aot_module_simplified(gm, example_inputs, **kwargs)
2023-09-06T16:06:30.9018625Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3891, in aot_module_simplified
2023-09-06T16:06:30.9019077Z     compiled_fn = create_aot_dispatcher_function(
2023-09-06T16:06:30.9019624Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:06:30.9020010Z     r = func(*args, **kwargs)
2023-09-06T16:06:30.9020613Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3429, in create_aot_dispatcher_function
2023-09-06T16:06:30.9021173Z     compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:06:30.9021822Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2212, in aot_wrapper_dedupe
2023-09-06T16:06:30.9022334Z     return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:06:30.9023005Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2392, in aot_wrapper_synthetic_base
2023-09-06T16:06:30.9023515Z     return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:06:30.9024159Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1573, in aot_dispatch_base
2023-09-06T16:06:30.9024603Z     compiled_fw = compiler(fw_module, flat_args)
2023-09-06T16:06:30.9025170Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:06:30.9025741Z     r = func(*args, **kwargs)
2023-09-06T16:06:30.9026333Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1109, in fw_compiler_base
2023-09-06T16:06:30.9026738Z     return inner_compile(
2023-09-06T16:06:30.9027280Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 80, in debug_wrapper
2023-09-06T16:06:30.9027776Z     inner_compiled_fn = compiler_fn(gm, example_inputs)
2023-09-06T16:06:30.9028358Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 228, in inner
2023-09-06T16:06:30.9028734Z     return fn(*args, **kwargs)
2023-09-06T16:06:30.9029275Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T16:06:30.9029765Z     return func(*args, **kwds)
2023-09-06T16:06:30.9030339Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 55, in newFunction
2023-09-06T16:06:30.9030761Z     return old_func(*args, **kwargs)
2023-09-06T16:06:30.9031313Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 348, in compile_fx_inner
2023-09-06T16:06:30.9031780Z     compiled_graph: CompiledFxGraph = fx_codegen_and_compile(
2023-09-06T16:06:30.9032412Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 574, in fx_codegen_and_compile
2023-09-06T16:06:30.9032849Z     compiled_fn = graph.compile_to_fn()
2023-09-06T16:06:30.9033412Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 991, in compile_to_fn
2023-09-06T16:06:30.9033812Z     return self.compile_to_module().call
2023-09-06T16:06:30.9034363Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:06:30.9034747Z     r = func(*args, **kwargs)
2023-09-06T16:06:30.9035300Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 946, in compile_to_module
2023-09-06T16:06:30.9035977Z     mod = PyCodeCache.load_by_key_path(key, path, linemap=linemap)
2023-09-06T16:06:30.9036612Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1161, in load_by_key_path
2023-09-06T16:06:30.9037030Z     exec(code, mod.__dict__, mod.__dict__)
2023-09-06T16:06:30.9037519Z   File "/tmp/tmp0us7gfic/ou/couprzu664fw7hjvzetesztsdbv43dudusk4sjwve6kd4tfmkt4d.py", line 131, in <module>
2023-09-06T16:06:30.9038036Z     async_compile.wait(globals())
2023-09-06T16:06:30.9038561Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1440, in wait
2023-09-06T16:06:30.9038954Z     scope[key] = result.result()
2023-09-06T16:06:30.9039510Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1299, in result
2023-09-06T16:06:30.9039896Z     self.future.result()
2023-09-06T16:06:30.9040271Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 458, in result
2023-09-06T16:06:30.9040644Z     return self.__get_result()
2023-09-06T16:06:30.9041037Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
2023-09-06T16:06:30.9041406Z     raise self._exception
2023-09-06T16:06:30.9041866Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
2023-09-06T16:06:30.9042346Z CompilationError: at 11:70:def triton_(in_ptr0, out_ptr0, xnumel, XBLOCK : tl.constexpr):
2023-09-06T16:06:30.9042711Z     xnumel = 209982
2023-09-06T16:06:30.9042992Z     xoffset = tl.program_id(0) * XBLOCK
2023-09-06T16:06:30.9043300Z     xindex = xoffset + tl.arange(0, XBLOCK)[:]
2023-09-06T16:06:30.9043595Z     xmask = xindex < xnumel
2023-09-06T16:06:30.9043879Z     x0 = xindex
2023-09-06T16:06:30.9044154Z     tmp0 = tl.load(in_ptr0 + (209982 + x0), xmask)
2023-09-06T16:06:30.9044470Z     tmp1 = tl.where(tmp0 < 0, tmp0 + 10000, tmp0)
2023-09-06T16:06:30.9045060Z     tl.device_assert(((0 <= tmp1) & (tmp1 < 10000)) | ~xmask, "index out of bounds: 0 <= tmp1 < 10000")
2023-09-06T16:06:30.9045410Z     tmp2 = 1.0
2023-09-06T16:06:30.9045744Z     tl.atomic_add(out_ptr0 + (tl.broadcast_to(tmp1, [XBLOCK])), tmp2, xmask)
2023-09-06T16:06:30.9046122Z                                                                       ^
2023-09-06T16:06:30.9046511Z ValueError('atomic_add does not support bf16')
2023-09-06T16:06:30.9046728Z 
2023-09-06T16:06:30.9046926Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T16:06:30.9047165Z 
2023-09-06T16:06:30.9047170Z 
2023-09-06T16:06:30.9047366Z You can suppress this exception and fall back to eager by setting:
2023-09-06T16:06:30.9047747Z     import torch._dynamo
2023-09-06T16:06:30.9048054Z     torch._dynamo.config.suppress_errors = True
2023-09-06T16:06:30.9048267Z 
2023-09-06T16:06:32.2899566Z ERROR
2023-09-06T16:06:35.9072893Z 
2023-09-06T16:06:38.7129602Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:06:38.7130009Z loading model: 0it [00:02, ?it/s]
2023-09-06T16:06:38.7136349Z cuda eval  basic_gnn_gin                      
2023-09-06T16:06:39.2398805Z [2023-09-06 16:06:39,238] [0/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T16:06:45.6995538Z ERROR:common:Backend dynamo failed in warmup()
2023-09-06T16:06:45.6996152Z Traceback (most recent call last):
2023-09-06T16:06:45.6996790Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2288, in warmup
2023-09-06T16:06:45.6997361Z     fn(model, example_inputs)
2023-09-06T16:06:45.6998594Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T16:06:45.6999253Z     return fn(*args, **kwargs)
2023-09-06T16:06:45.7003695Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T16:06:45.7005204Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T16:06:45.7006494Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 634, in _convert_frame
2023-09-06T16:06:45.7007743Z     result = inner_convert(frame, cache_entry, hooks, frame_state)
2023-09-06T16:06:45.7009573Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T16:06:45.7010628Z     return fn(*args, **kwargs)
2023-09-06T16:06:45.7012192Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T16:06:45.7013208Z     return _compile(
2023-09-06T16:06:45.7014351Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T16:06:45.7015156Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T16:06:45.7015769Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:06:45.7016272Z     r = func(*args, **kwargs)
2023-09-06T16:06:45.7017334Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T16:06:45.7018157Z     out_code = transform_code_object(code, transform)
2023-09-06T16:06:45.7019342Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T16:06:45.7020129Z     transformations(instructions, code_options)
2023-09-06T16:06:45.7021298Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T16:06:45.7022060Z     tracer.run()
2023-09-06T16:06:45.7023265Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T16:06:45.7024035Z     super().run()
2023-09-06T16:06:45.7025090Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T16:06:45.7025840Z     and self.step()
2023-09-06T16:06:45.7026405Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T16:06:45.7026818Z     getattr(self, inst.opname)(inst)
2023-09-06T16:06:45.7027462Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2166, in RETURN_VALUE
2023-09-06T16:06:45.7027883Z     self.output.compile_subgraph(
2023-09-06T16:06:45.7028512Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 881, in compile_subgraph
2023-09-06T16:06:45.7029009Z     self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
2023-09-06T16:06:45.7029642Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T16:06:45.7029992Z     return func(*args, **kwds)
2023-09-06T16:06:45.7030594Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1008, in compile_and_call_fx_graph
2023-09-06T16:06:45.7031046Z     compiled_fn = self.call_user_compiler(gm)
2023-09-06T16:06:45.7031624Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:06:45.7032014Z     r = func(*args, **kwargs)
2023-09-06T16:06:45.7032565Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1075, in call_user_compiler
2023-09-06T16:06:45.7033056Z     raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
2023-09-06T16:06:45.7033699Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1060, in call_user_compiler
2023-09-06T16:06:45.7034153Z     compiled_fn = compiler_fn(gm, self.example_inputs())
2023-09-06T16:06:45.7034990Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper
2023-09-06T16:06:45.7035435Z     compiled_gm = compiler_fn(gm, example_inputs)
2023-09-06T16:06:45.7036052Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/inductor.py", line 9, in inductor
2023-09-06T16:06:45.7036468Z     return compile_fx(*args, **kwargs)
2023-09-06T16:06:45.7037033Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1167, in compile_fx
2023-09-06T16:06:45.7037406Z     return aot_autograd(
2023-09-06T16:06:45.7037960Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 55, in compiler_fn
2023-09-06T16:06:45.7038458Z     cg = aot_module_simplified(gm, example_inputs, **kwargs)
2023-09-06T16:06:45.7039100Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3891, in aot_module_simplified
2023-09-06T16:06:45.7039552Z     compiled_fn = create_aot_dispatcher_function(
2023-09-06T16:06:45.7040126Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:06:45.7040512Z     r = func(*args, **kwargs)
2023-09-06T16:06:45.7041125Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3429, in create_aot_dispatcher_function
2023-09-06T16:06:45.7041660Z     compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:06:45.7042312Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2212, in aot_wrapper_dedupe
2023-09-06T16:06:45.7042822Z     return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:06:45.7043637Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2392, in aot_wrapper_synthetic_base
2023-09-06T16:06:45.7044153Z     return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:06:45.7044930Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1573, in aot_dispatch_base
2023-09-06T16:06:45.7045362Z     compiled_fw = compiler(fw_module, flat_args)
2023-09-06T16:06:45.7045927Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:06:45.7046311Z     r = func(*args, **kwargs)
2023-09-06T16:06:45.7046865Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1109, in fw_compiler_base
2023-09-06T16:06:45.7047243Z     return inner_compile(
2023-09-06T16:06:45.7047793Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 80, in debug_wrapper
2023-09-06T16:06:45.7048282Z     inner_compiled_fn = compiler_fn(gm, example_inputs)
2023-09-06T16:06:45.7048866Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 228, in inner
2023-09-06T16:06:45.7049247Z     return fn(*args, **kwargs)
2023-09-06T16:06:45.7049591Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T16:06:45.7049933Z     return func(*args, **kwds)
2023-09-06T16:06:45.7050481Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 55, in newFunction
2023-09-06T16:06:45.7050887Z     return old_func(*args, **kwargs)
2023-09-06T16:06:45.7051435Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 348, in compile_fx_inner
2023-09-06T16:06:45.7051913Z     compiled_graph: CompiledFxGraph = fx_codegen_and_compile(
2023-09-06T16:06:45.7052544Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 574, in fx_codegen_and_compile
2023-09-06T16:06:45.7052983Z     compiled_fn = graph.compile_to_fn()
2023-09-06T16:06:45.7053693Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 991, in compile_to_fn
2023-09-06T16:06:45.7054255Z     return self.compile_to_module().call
2023-09-06T16:06:45.7055135Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:06:45.7055715Z     r = func(*args, **kwargs)
2023-09-06T16:06:45.7056566Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 946, in compile_to_module
2023-09-06T16:06:45.7057259Z     mod = PyCodeCache.load_by_key_path(key, path, linemap=linemap)
2023-09-06T16:06:45.7058334Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1161, in load_by_key_path
2023-09-06T16:06:45.7058976Z     exec(code, mod.__dict__, mod.__dict__)
2023-09-06T16:06:45.7131510Z   File "/tmp/tmpilbioyik/sg/csg4dekrvn722zlwdxgeznffs6lasdssqaiaj2b6l32ses22wwne.py", line 239, in <module>
2023-09-06T16:06:45.7132762Z     async_compile.wait(globals())
2023-09-06T16:06:45.7133875Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1440, in wait
2023-09-06T16:06:45.7134469Z     scope[key] = result.result()
2023-09-06T16:06:45.7135385Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1299, in result
2023-09-06T16:06:45.7136099Z     self.future.result()
2023-09-06T16:06:45.7136734Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 451, in result
2023-09-06T16:06:45.7137231Z     return self.__get_result()
2023-09-06T16:06:45.7137853Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
2023-09-06T16:06:45.7138426Z     raise self._exception
2023-09-06T16:06:45.7139094Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
2023-09-06T16:06:45.7139721Z CompilationError: at 15:53:    xindex = xoffset + tl.arange(0, XBLOCK)[:]
2023-09-06T16:06:45.7140823Z     xmask = xindex < xnumel
2023-09-06T16:06:45.7141100Z     x1 = (xindex // 64)
2023-09-06T16:06:45.7141334Z     x0 = xindex % 64
2023-09-06T16:06:45.7141813Z     tmp0 = tl.load(in_ptr0 + (200000 + x1), None, eviction_policy='evict_last')
2023-09-06T16:06:45.7142315Z     tmp2 = tl.load(in_ptr0 + (x1), None, eviction_policy='evict_last')
2023-09-06T16:06:45.7142675Z     tmp1 = tl.where(tmp0 < 0, tmp0 + 10000, tmp0)
2023-09-06T16:06:45.7143053Z     tl.device_assert((0 <= tmp1) & (tmp1 < 10000), "index out of bounds: 0 <= tmp1 < 10000")
2023-09-06T16:06:45.7143440Z     tmp3 = tl.where(tmp2 < 0, tmp2 + 10000, tmp2)
2023-09-06T16:06:45.7143827Z     tl.device_assert((0 <= tmp3) & (tmp3 < 10000), "index out of bounds: 0 <= tmp3 < 10000")
2023-09-06T16:06:45.7144228Z     tmp4 = tl.load(in_ptr1 + (x0 + (64*tmp3)), None).to(tl.float32)
2023-09-06T16:06:45.7144575Z     tl.atomic_add(out_ptr0 + (x0 + (64*tmp1)), tmp4, None)
2023-09-06T16:06:45.7144886Z                                                      ^
2023-09-06T16:06:45.7145268Z ValueError('atomic_add does not support bf16')
2023-09-06T16:06:45.7145468Z 
2023-09-06T16:06:45.7145662Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T16:06:45.7145897Z 
2023-09-06T16:06:45.7145903Z 
2023-09-06T16:06:45.7146094Z You can suppress this exception and fall back to eager by setting:
2023-09-06T16:06:45.7146413Z     import torch._dynamo
2023-09-06T16:06:45.7146723Z     torch._dynamo.config.suppress_errors = True
2023-09-06T16:06:45.7146931Z 
2023-09-06T16:06:46.9525421Z ERROR
2023-09-06T16:06:50.5430220Z 
2023-09-06T16:06:53.4193549Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:06:53.4194594Z loading model: 0it [00:02, ?it/s]
2023-09-06T16:06:53.4196482Z cuda eval  basic_gnn_sage                     
2023-09-06T16:06:53.9329187Z [2023-09-06 16:06:53,931] [0/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T16:07:00.3711048Z ERROR:common:Backend dynamo failed in warmup()
2023-09-06T16:07:00.3711760Z Traceback (most recent call last):
2023-09-06T16:07:00.3712485Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2288, in warmup
2023-09-06T16:07:00.3713239Z     fn(model, example_inputs)
2023-09-06T16:07:00.3715487Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T16:07:00.3716217Z     return fn(*args, **kwargs)
2023-09-06T16:07:00.3717240Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T16:07:00.3718087Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T16:07:00.3719299Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 634, in _convert_frame
2023-09-06T16:07:00.3720218Z     result = inner_convert(frame, cache_entry, hooks, frame_state)
2023-09-06T16:07:00.3721307Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T16:07:00.3722023Z     return fn(*args, **kwargs)
2023-09-06T16:07:00.3723010Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T16:07:00.3723715Z     return _compile(
2023-09-06T16:07:00.3724675Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T16:07:00.3725502Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T16:07:00.3726607Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:07:00.3727261Z     r = func(*args, **kwargs)
2023-09-06T16:07:00.3728771Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T16:07:00.3729561Z     out_code = transform_code_object(code, transform)
2023-09-06T16:07:00.3730674Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T16:07:00.3731482Z     transformations(instructions, code_options)
2023-09-06T16:07:00.3732539Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T16:07:00.3733169Z     tracer.run()
2023-09-06T16:07:00.3734066Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T16:07:00.3734711Z     super().run()
2023-09-06T16:07:00.3735638Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T16:07:00.3736288Z     and self.step()
2023-09-06T16:07:00.3737244Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T16:07:00.3737940Z     getattr(self, inst.opname)(inst)
2023-09-06T16:07:00.3739102Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2166, in RETURN_VALUE
2023-09-06T16:07:00.3739841Z     self.output.compile_subgraph(
2023-09-06T16:07:00.3740906Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 881, in compile_subgraph
2023-09-06T16:07:00.3741773Z     self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
2023-09-06T16:07:00.3742517Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T16:07:00.3743119Z     return func(*args, **kwds)
2023-09-06T16:07:00.3744188Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1008, in compile_and_call_fx_graph
2023-09-06T16:07:00.3744980Z     compiled_fn = self.call_user_compiler(gm)
2023-09-06T16:07:00.3746542Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:07:00.3747230Z     r = func(*args, **kwargs)
2023-09-06T16:07:00.3748217Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1075, in call_user_compiler
2023-09-06T16:07:00.3749373Z     raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
2023-09-06T16:07:00.3750543Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1060, in call_user_compiler
2023-09-06T16:07:00.3751388Z     compiled_fn = compiler_fn(gm, self.example_inputs())
2023-09-06T16:07:00.3752457Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper
2023-09-06T16:07:00.3753231Z     compiled_gm = compiler_fn(gm, example_inputs)
2023-09-06T16:07:00.3754287Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/inductor.py", line 9, in inductor
2023-09-06T16:07:00.3755005Z     return compile_fx(*args, **kwargs)
2023-09-06T16:07:00.3756008Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1167, in compile_fx
2023-09-06T16:07:00.3756696Z     return aot_autograd(
2023-09-06T16:07:00.3757669Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 55, in compiler_fn
2023-09-06T16:07:00.3758567Z     cg = aot_module_simplified(gm, example_inputs, **kwargs)
2023-09-06T16:07:00.3759701Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3891, in aot_module_simplified
2023-09-06T16:07:00.3760603Z     compiled_fn = create_aot_dispatcher_function(
2023-09-06T16:07:00.3761611Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:07:00.3762237Z     r = func(*args, **kwargs)
2023-09-06T16:07:00.3763678Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3429, in create_aot_dispatcher_function
2023-09-06T16:07:00.3764672Z     compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:07:00.3765906Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2212, in aot_wrapper_dedupe
2023-09-06T16:07:00.3766799Z     return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:07:00.3767980Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2392, in aot_wrapper_synthetic_base
2023-09-06T16:07:00.3768815Z     return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:07:00.3769832Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1573, in aot_dispatch_base
2023-09-06T16:07:00.3770554Z     compiled_fw = compiler(fw_module, flat_args)
2023-09-06T16:07:00.3771408Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:07:00.3771991Z     r = func(*args, **kwargs)
2023-09-06T16:07:00.3772800Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1109, in fw_compiler_base
2023-09-06T16:07:00.3773357Z     return inner_compile(
2023-09-06T16:07:00.3774194Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 80, in debug_wrapper
2023-09-06T16:07:00.3774718Z     inner_compiled_fn = compiler_fn(gm, example_inputs)
2023-09-06T16:07:00.3775281Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 228, in inner
2023-09-06T16:07:00.3775667Z     return fn(*args, **kwargs)
2023-09-06T16:07:00.3776025Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T16:07:00.3776676Z     return func(*args, **kwds)
2023-09-06T16:07:00.3777212Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 55, in newFunction
2023-09-06T16:07:00.3777614Z     return old_func(*args, **kwargs)
2023-09-06T16:07:00.3778176Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 348, in compile_fx_inner
2023-09-06T16:07:00.3778707Z     compiled_graph: CompiledFxGraph = fx_codegen_and_compile(
2023-09-06T16:07:00.3779338Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 574, in fx_codegen_and_compile
2023-09-06T16:07:00.3779750Z     compiled_fn = graph.compile_to_fn()
2023-09-06T16:07:00.3780314Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 991, in compile_to_fn
2023-09-06T16:07:00.3780727Z     return self.compile_to_module().call
2023-09-06T16:07:00.3781283Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:07:00.3781653Z     r = func(*args, **kwargs)
2023-09-06T16:07:00.3782200Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 946, in compile_to_module
2023-09-06T16:07:00.3782674Z     mod = PyCodeCache.load_by_key_path(key, path, linemap=linemap)
2023-09-06T16:07:00.3783296Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1161, in load_by_key_path
2023-09-06T16:07:00.3783712Z     exec(code, mod.__dict__, mod.__dict__)
2023-09-06T16:07:00.3784188Z   File "/tmp/tmplb7wnl15/p5/cp5hsqq7dr6hpemfmcthikvswgxiwpfsyzg6i7nf3xrdger2nme4.py", line 247, in <module>
2023-09-06T16:07:00.3784661Z     async_compile.wait(globals())
2023-09-06T16:07:00.3785199Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1440, in wait
2023-09-06T16:07:00.3785592Z     scope[key] = result.result()
2023-09-06T16:07:00.3786552Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1299, in result
2023-09-06T16:07:00.3787176Z     self.future.result()
2023-09-06T16:07:00.3787783Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 451, in result
2023-09-06T16:07:00.3788393Z     return self.__get_result()
2023-09-06T16:07:00.3789264Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
2023-09-06T16:07:00.3789829Z     raise self._exception
2023-09-06T16:07:00.3790558Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
2023-09-06T16:07:00.3791081Z CompilationError: at 15:53:    xindex = xoffset + tl.arange(0, XBLOCK)[:]
2023-09-06T16:07:00.3791635Z     xmask = xindex < xnumel
2023-09-06T16:07:00.3792018Z     x1 = (xindex // 64)
2023-09-06T16:07:00.3792406Z     x0 = xindex % 64
2023-09-06T16:07:00.3793133Z     tmp0 = tl.load(in_ptr0 + (200000 + x1), None, eviction_policy='evict_last')
2023-09-06T16:07:00.3793959Z     tmp2 = tl.load(in_ptr0 + (x1), None, eviction_policy='evict_last')
2023-09-06T16:07:00.3794489Z     tmp1 = tl.where(tmp0 < 0, tmp0 + 10000, tmp0)
2023-09-06T16:07:00.3795139Z     tl.device_assert((0 <= tmp1) & (tmp1 < 10000), "index out of bounds: 0 <= tmp1 < 10000")
2023-09-06T16:07:00.3795821Z     tmp3 = tl.where(tmp2 < 0, tmp2 + 10000, tmp2)
2023-09-06T16:07:00.3796538Z     tl.device_assert((0 <= tmp3) & (tmp3 < 10000), "index out of bounds: 0 <= tmp3 < 10000")
2023-09-06T16:07:00.3797163Z     tmp4 = tl.load(in_ptr1 + (x0 + (64*tmp3)), None).to(tl.float32)
2023-09-06T16:07:00.3797690Z     tl.atomic_add(out_ptr0 + (x0 + (64*tmp1)), tmp4, None)
2023-09-06T16:07:00.3798158Z                                                      ^
2023-09-06T16:07:00.3798881Z ValueError('atomic_add does not support bf16')
2023-09-06T16:07:00.3799176Z 
2023-09-06T16:07:00.3799439Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T16:07:00.3799768Z 
2023-09-06T16:07:00.3800123Z 
2023-09-06T16:07:00.3800428Z You can suppress this exception and fall back to eager by setting:
2023-09-06T16:07:00.3800903Z     import torch._dynamo
2023-09-06T16:07:00.3801340Z     torch._dynamo.config.suppress_errors = True
2023-09-06T16:07:00.3801627Z 
2023-09-06T16:07:01.6365767Z ERROR
2023-09-06T16:07:05.1970309Z 
2023-09-06T16:07:09.4201891Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:07:09.4202430Z loading model: 0it [00:04, ?it/s]
2023-09-06T16:07:09.4287074Z cuda eval  cm3leon_generate                   
2023-09-06T16:10:16.0538272Z 
2023-09-06T16:10:18.1456149Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:10:20.2154297Z running benchmark:   3% 1/30 [00:02<01:00,  2.09s/it]
2023-09-06T16:10:22.2921045Z running benchmark:   7% 2/30 [00:04<00:58,  2.08s/it]
2023-09-06T16:10:24.3578076Z running benchmark:  10% 3/30 [00:06<00:56,  2.08s/it]
2023-09-06T16:10:26.4293947Z running benchmark:  13% 4/30 [00:08<00:53,  2.07s/it]
2023-09-06T16:10:28.5161730Z running benchmark:  17% 5/30 [00:10<00:51,  2.07s/it]
2023-09-06T16:10:30.5835575Z running benchmark:  20% 6/30 [00:12<00:49,  2.08s/it]
2023-09-06T16:10:32.6864664Z running benchmark:  23% 7/30 [00:14<00:47,  2.07s/it]
2023-09-06T16:10:34.7604800Z running benchmark:  27% 8/30 [00:16<00:45,  2.08s/it]
2023-09-06T16:10:36.8405841Z running benchmark:  30% 9/30 [00:18<00:43,  2.08s/it]
2023-09-06T16:10:38.8993796Z running benchmark:  33% 10/30 [00:20<00:41,  2.08s/it]
2023-09-06T16:10:40.9686374Z running benchmark:  37% 11/30 [00:22<00:39,  2.07s/it]
2023-09-06T16:10:43.0362971Z running benchmark:  40% 12/30 [00:24<00:37,  2.07s/it]
2023-09-06T16:10:45.1160335Z running benchmark:  43% 13/30 [00:26<00:35,  2.07s/it]
2023-09-06T16:10:47.1693155Z running benchmark:  47% 14/30 [00:29<00:33,  2.07s/it]
2023-09-06T16:10:49.2171902Z running benchmark:  50% 15/30 [00:31<00:31,  2.07s/it]
2023-09-06T16:10:51.2815216Z running benchmark:  53% 16/30 [00:33<00:28,  2.06s/it]
2023-09-06T16:10:53.3420718Z running benchmark:  57% 17/30 [00:35<00:26,  2.06s/it]
2023-09-06T16:10:55.3957055Z running benchmark:  60% 18/30 [00:37<00:24,  2.06s/it]
2023-09-06T16:10:57.4595792Z running benchmark:  63% 19/30 [00:39<00:22,  2.06s/it]
2023-09-06T16:10:59.5174152Z running benchmark:  67% 20/30 [00:41<00:20,  2.06s/it]
2023-09-06T16:11:01.5603933Z running benchmark:  70% 21/30 [00:43<00:18,  2.06s/it]
2023-09-06T16:11:03.6117170Z running benchmark:  73% 22/30 [00:45<00:16,  2.05s/it]
2023-09-06T16:11:05.6496045Z running benchmark:  77% 23/30 [00:47<00:14,  2.05s/it]
2023-09-06T16:11:07.6903931Z running benchmark:  80% 24/30 [00:49<00:12,  2.05s/it]
2023-09-06T16:11:09.7510497Z running benchmark:  83% 25/30 [00:51<00:10,  2.05s/it]
2023-09-06T16:11:11.7830394Z running benchmark:  87% 26/30 [00:53<00:08,  2.05s/it]
2023-09-06T16:11:13.8158817Z running benchmark:  90% 27/30 [00:55<00:06,  2.05s/it]
2023-09-06T16:11:15.8834006Z running benchmark:  93% 28/30 [00:57<00:04,  2.04s/it]
2023-09-06T16:11:17.9561380Z running benchmark:  97% 29/30 [00:59<00:02,  2.05s/it]
2023-09-06T16:11:17.9561791Z running benchmark: 100% 30/30 [01:01<00:00,  2.06s/it]
2023-09-06T16:11:17.9562121Z running benchmark: 100% 30/30 [01:01<00:00,  2.06s/it]
2023-09-06T16:11:17.9601540Z 1.118x
2023-09-06T16:11:24.5431273Z 
2023-09-06T16:11:25.4172397Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:11:25.4172904Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:11:25.4177073Z cuda eval  dcgan                              
2023-09-06T16:11:34.0620561Z 
2023-09-06T16:11:34.1689192Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:11:34.2750234Z running benchmark:  47% 14/30 [00:00<00:00, 131.17it/s]
2023-09-06T16:11:34.2822009Z running benchmark:  97% 29/30 [00:00<00:00, 137.12it/s]
2023-09-06T16:11:34.2822387Z running benchmark: 100% 30/30 [00:00<00:00, 136.23it/s]
2023-09-06T16:11:34.2841124Z 1.156x
2023-09-06T16:11:39.1913430Z 
2023-09-06T16:11:42.8499437Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:11:42.8500036Z loading model: 0it [00:03, ?it/s]
2023-09-06T16:11:42.8500547Z Eager model failed to run
2023-09-06T16:11:42.8517192Z Traceback (most recent call last):
2023-09-06T16:11:42.8517882Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T16:11:42.8518505Z     self.model_iter_fn(model, example_inputs)
2023-09-06T16:11:42.8519228Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:11:42.8519814Z     return mod(*inputs)
2023-09-06T16:11:42.8525516Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:11:42.8525988Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:11:42.8526568Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:11:42.8526978Z     return forward_call(*args, **kwargs)
2023-09-06T16:11:42.8527438Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/__init__.py", line 31, in forward
2023-09-06T16:11:42.8527951Z     return sources, self.model(mix)
2023-09-06T16:11:42.8528535Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:11:42.8528955Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:11:42.8529512Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:11:42.8529913Z     return forward_call(*args, **kwargs)
2023-09-06T16:11:42.8530333Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/demucs/model.py", line 209, in forward
2023-09-06T16:11:42.8530721Z     x = self.lstm(x)
2023-09-06T16:11:42.8531258Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:11:42.8531675Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:11:42.8532228Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:11:42.8533043Z     return forward_call(*args, **kwargs)
2023-09-06T16:11:42.8533487Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/demucs/model.py", line 27, in forward
2023-09-06T16:11:42.8533879Z     x = self.lstm(x)[0]
2023-09-06T16:11:42.8534434Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:11:42.8534836Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:11:42.8535388Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:11:42.8535787Z     return forward_call(*args, **kwargs)
2023-09-06T16:11:42.8536331Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/rnn.py", line 879, in forward
2023-09-06T16:11:42.8536801Z     result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
2023-09-06T16:11:42.8537333Z RuntimeError: "_thnn_fused_lstm_cell_cuda" not implemented for 'BFloat16'
2023-09-06T16:11:42.8537586Z 
2023-09-06T16:11:42.8537835Z The above exception was the direct cause of the following exception:
2023-09-06T16:11:42.8538082Z 
2023-09-06T16:11:42.8538232Z Traceback (most recent call last):
2023-09-06T16:11:42.8538615Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T16:11:42.8538950Z     ) = runner.load_model(
2023-09-06T16:11:42.8539327Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T16:11:42.8539729Z     self.validate_model(model, example_inputs)
2023-09-06T16:11:42.8540143Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T16:11:42.8540585Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T16:11:42.8540953Z NotImplementedError: Eager model failed to run
2023-09-06T16:11:42.8541165Z 
2023-09-06T16:11:42.8541286Z WARNING:root:demucs failed to load
2023-09-06T16:11:47.3745744Z 
2023-09-06T16:11:49.4864613Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:11:49.4865202Z loading model: 0it [00:02, ?it/s]
2023-09-06T16:11:49.4984755Z cuda eval  densenet121                        
2023-09-06T16:12:47.5524445Z 
2023-09-06T16:12:47.6775922Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:12:47.7947540Z running benchmark:  10% 3/30 [00:00<00:01, 23.83it/s]
2023-09-06T16:12:47.9190159Z running benchmark:  20% 6/30 [00:00<00:00, 24.81it/s]
2023-09-06T16:12:48.0388917Z running benchmark:  30% 9/30 [00:00<00:00, 24.51it/s]
2023-09-06T16:12:48.1560266Z running benchmark:  40% 12/30 [00:00<00:00, 24.70it/s]
2023-09-06T16:12:48.2729743Z running benchmark:  50% 15/30 [00:00<00:00, 25.03it/s]
2023-09-06T16:12:48.3924828Z running benchmark:  60% 18/30 [00:00<00:00, 25.24it/s]
2023-09-06T16:12:48.5197350Z running benchmark:  70% 21/30 [00:00<00:00, 25.19it/s]
2023-09-06T16:12:48.6366368Z running benchmark:  80% 24/30 [00:00<00:00, 24.66it/s]
2023-09-06T16:12:48.7543798Z running benchmark:  90% 27/30 [00:01<00:00, 24.95it/s]
2023-09-06T16:12:48.7544297Z running benchmark: 100% 30/30 [00:01<00:00, 25.11it/s]
2023-09-06T16:12:48.7547890Z running benchmark: 100% 30/30 [00:01<00:00, 24.93it/s]
2023-09-06T16:12:48.7612124Z 1.238x
2023-09-06T16:12:54.6695834Z 
2023-09-06T16:12:59.2892686Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:12:59.2893060Z loading model: 0it [00:04, ?it/s]
2023-09-06T16:12:59.2893376Z Eager model failed to run
2023-09-06T16:12:59.2909884Z Traceback (most recent call last):
2023-09-06T16:12:59.2910360Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T16:12:59.2910772Z     self.model_iter_fn(model, example_inputs)
2023-09-06T16:12:59.2911193Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:12:59.2911568Z     return mod(*inputs)
2023-09-06T16:12:59.2914486Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:12:59.2914950Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:12:59.2915541Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:12:59.2917997Z     return forward_call(*args, **kwargs)
2023-09-06T16:12:59.2919358Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T16:12:59.2919855Z     return self.inference(batched_inputs)
2023-09-06T16:12:59.2920462Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T16:12:59.2920946Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T16:12:59.2921576Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:12:59.2921984Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:12:59.2922803Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:12:59.2923480Z     return forward_call(*args, **kwargs)
2023-09-06T16:12:59.2924086Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2023-09-06T16:12:59.2924517Z     box_features = self._shared_roi_transform(
2023-09-06T16:12:59.2925147Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2023-09-06T16:12:59.2925608Z     x = self.pooler(features, boxes)
2023-09-06T16:12:59.2926179Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:12:59.2926600Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:12:59.2927152Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:12:59.2928262Z     return forward_call(*args, **kwargs)
2023-09-06T16:12:59.2928839Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T16:12:59.2929278Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T16:12:59.2929871Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:12:59.2930278Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:12:59.2930834Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:12:59.2931234Z     return forward_call(*args, **kwargs)
2023-09-06T16:12:59.2931779Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T16:12:59.2932136Z     return roi_align(
2023-09-06T16:12:59.2932672Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T16:12:59.2933116Z     return torch.ops.torchvision.roi_align(
2023-09-06T16:12:59.2933751Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T16:12:59.2934140Z     return self._op(*args, **kwargs or {})
2023-09-06T16:12:59.2934601Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T16:12:59.2934844Z 
2023-09-06T16:12:59.2935044Z The above exception was the direct cause of the following exception:
2023-09-06T16:12:59.2935289Z 
2023-09-06T16:12:59.2935415Z Traceback (most recent call last):
2023-09-06T16:12:59.2935792Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T16:12:59.2936122Z     ) = runner.load_model(
2023-09-06T16:12:59.2936500Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T16:12:59.2937067Z     self.validate_model(model, example_inputs)
2023-09-06T16:12:59.2937493Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T16:12:59.2938006Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T16:12:59.2938380Z NotImplementedError: Eager model failed to run
2023-09-06T16:12:59.2938590Z 
2023-09-06T16:12:59.2938758Z WARNING:root:detectron2_fasterrcnn_r_101_c4 failed to load
2023-09-06T16:13:04.0085177Z 
2023-09-06T16:13:11.0949707Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:13:11.0950314Z loading model: 0it [00:07, ?it/s]
2023-09-06T16:13:11.0950806Z Eager model failed to run
2023-09-06T16:13:11.0966624Z Traceback (most recent call last):
2023-09-06T16:13:11.0967368Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T16:13:11.0967988Z     self.model_iter_fn(model, example_inputs)
2023-09-06T16:13:11.0968661Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:13:11.0969624Z     return mod(*inputs)
2023-09-06T16:13:11.0970992Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:13:11.0971670Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:13:11.0972566Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:13:11.0973198Z     return forward_call(*args, **kwargs)
2023-09-06T16:13:11.0974095Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T16:13:11.0974767Z     return self.inference(batched_inputs)
2023-09-06T16:13:11.0975677Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T16:13:11.0976364Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T16:13:11.0977327Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:13:11.0978795Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:13:11.0979692Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:13:11.0980333Z     return forward_call(*args, **kwargs)
2023-09-06T16:13:11.0981237Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T16:13:11.0981972Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T16:13:11.0982990Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T16:13:11.0983799Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T16:13:11.0984878Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:13:11.0985546Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:13:11.0986460Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:13:11.0987101Z     return forward_call(*args, **kwargs)
2023-09-06T16:13:11.0987998Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T16:13:11.0988753Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T16:13:11.0990035Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:13:11.0990705Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:13:11.0991551Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:13:11.0992170Z     return forward_call(*args, **kwargs)
2023-09-06T16:13:11.0993472Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T16:13:11.0994066Z     return roi_align(
2023-09-06T16:13:11.0994910Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T16:13:11.0995582Z     return torch.ops.torchvision.roi_align(
2023-09-06T16:13:11.0996399Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T16:13:11.0997046Z     return self._op(*args, **kwargs or {})
2023-09-06T16:13:11.0997771Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T16:13:11.0998154Z 
2023-09-06T16:13:11.0998540Z The above exception was the direct cause of the following exception:
2023-09-06T16:13:11.0998889Z 
2023-09-06T16:13:11.0999085Z Traceback (most recent call last):
2023-09-06T16:13:11.0999645Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T16:13:11.1000198Z     ) = runner.load_model(
2023-09-06T16:13:11.1000788Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T16:13:11.1001388Z     self.validate_model(model, example_inputs)
2023-09-06T16:13:11.1001992Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T16:13:11.1002672Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T16:13:11.1003273Z NotImplementedError: Eager model failed to run
2023-09-06T16:13:11.1003600Z 
2023-09-06T16:13:11.1003861Z WARNING:root:detectron2_fasterrcnn_r_101_dc5 failed to load
2023-09-06T16:13:15.7320821Z 
2023-09-06T16:13:20.1254099Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:13:20.1254624Z loading model: 0it [00:04, ?it/s]
2023-09-06T16:13:20.1254911Z Eager model failed to run
2023-09-06T16:13:20.1279546Z Traceback (most recent call last):
2023-09-06T16:13:20.1280899Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T16:13:20.1282385Z     self.model_iter_fn(model, example_inputs)
2023-09-06T16:13:20.1283093Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:13:20.1283580Z     return mod(*inputs)
2023-09-06T16:13:20.1284868Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:13:20.1285592Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:13:20.1286576Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:13:20.1287300Z     return forward_call(*args, **kwargs)
2023-09-06T16:13:20.1288350Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T16:13:20.1289106Z     return self.inference(batched_inputs)
2023-09-06T16:13:20.1290172Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T16:13:20.1290991Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T16:13:20.1292112Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:13:20.1292838Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:13:20.1293926Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:13:20.1294427Z     return forward_call(*args, **kwargs)
2023-09-06T16:13:20.1295395Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T16:13:20.1295893Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T16:13:20.1296911Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T16:13:20.1297786Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T16:13:20.1298626Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:13:20.1299068Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:13:20.1299645Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:13:20.1300061Z     return forward_call(*args, **kwargs)
2023-09-06T16:13:20.1300625Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2023-09-06T16:13:20.1301079Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2023-09-06T16:13:20.1301704Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:13:20.1302141Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:13:20.1302861Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:13:20.1303256Z     return forward_call(*args, **kwargs)
2023-09-06T16:13:20.1303863Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T16:13:20.1304242Z     return roi_align(
2023-09-06T16:13:20.1304782Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T16:13:20.1305233Z     return torch.ops.torchvision.roi_align(
2023-09-06T16:13:20.1305767Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T16:13:20.1306154Z     return self._op(*args, **kwargs or {})
2023-09-06T16:13:20.1306630Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T16:13:20.1306879Z 
2023-09-06T16:13:20.1307084Z The above exception was the direct cause of the following exception:
2023-09-06T16:13:20.1307620Z 
2023-09-06T16:13:20.1307738Z Traceback (most recent call last):
2023-09-06T16:13:20.1308127Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T16:13:20.1308485Z     ) = runner.load_model(
2023-09-06T16:13:20.1308873Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T16:13:20.1309697Z     self.validate_model(model, example_inputs)
2023-09-06T16:13:20.1310112Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T16:13:20.1310562Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T16:13:20.1310955Z NotImplementedError: Eager model failed to run
2023-09-06T16:13:20.1311171Z 
2023-09-06T16:13:20.1311343Z WARNING:root:detectron2_fasterrcnn_r_101_fpn failed to load
2023-09-06T16:13:24.7894662Z 
2023-09-06T16:13:28.8705253Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:13:28.8705673Z loading model: 0it [00:04, ?it/s]
2023-09-06T16:13:28.8705954Z Eager model failed to run
2023-09-06T16:13:28.8723655Z Traceback (most recent call last):
2023-09-06T16:13:28.8725056Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T16:13:28.8725840Z     self.model_iter_fn(model, example_inputs)
2023-09-06T16:13:28.8726665Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:13:28.8727225Z     return mod(*inputs)
2023-09-06T16:13:28.8728971Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:13:28.8729406Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:13:28.8729986Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:13:28.8730402Z     return forward_call(*args, **kwargs)
2023-09-06T16:13:28.8731469Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T16:13:28.8731938Z     return self.inference(batched_inputs)
2023-09-06T16:13:28.8732542Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T16:13:28.8733031Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T16:13:28.8733651Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:13:28.8734059Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:13:28.8734616Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:13:28.8735026Z     return forward_call(*args, **kwargs)
2023-09-06T16:13:28.8735610Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2023-09-06T16:13:28.8736052Z     box_features = self._shared_roi_transform(
2023-09-06T16:13:28.8736708Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2023-09-06T16:13:28.8737165Z     x = self.pooler(features, boxes)
2023-09-06T16:13:28.8737742Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:13:28.8738167Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:13:28.8738772Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:13:28.8739184Z     return forward_call(*args, **kwargs)
2023-09-06T16:13:28.8739753Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T16:13:28.8740202Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T16:13:28.8740801Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:13:28.8741441Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:13:28.8742005Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:13:28.8742410Z     return forward_call(*args, **kwargs)
2023-09-06T16:13:28.8742962Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T16:13:28.8743324Z     return roi_align(
2023-09-06T16:13:28.8743852Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T16:13:28.8744304Z     return torch.ops.torchvision.roi_align(
2023-09-06T16:13:28.8744855Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T16:13:28.8745232Z     return self._op(*args, **kwargs or {})
2023-09-06T16:13:28.8745714Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T16:13:28.8745966Z 
2023-09-06T16:13:28.8746183Z The above exception was the direct cause of the following exception:
2023-09-06T16:13:28.8746425Z 
2023-09-06T16:13:28.8746555Z Traceback (most recent call last):
2023-09-06T16:13:28.8746935Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T16:13:28.8747272Z     ) = runner.load_model(
2023-09-06T16:13:28.8747654Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T16:13:28.8748069Z     self.validate_model(model, example_inputs)
2023-09-06T16:13:28.8748539Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T16:13:28.8748968Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T16:13:28.8749564Z NotImplementedError: Eager model failed to run
2023-09-06T16:13:28.8749780Z 
2023-09-06T16:13:28.8750156Z WARNING:root:detectron2_fasterrcnn_r_50_c4 failed to load
2023-09-06T16:13:33.5913101Z 
2023-09-06T16:13:39.9821061Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:13:39.9821594Z loading model: 0it [00:06, ?it/s]
2023-09-06T16:13:39.9821853Z Eager model failed to run
2023-09-06T16:13:39.9836504Z Traceback (most recent call last):
2023-09-06T16:13:39.9836953Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T16:13:39.9837452Z     self.model_iter_fn(model, example_inputs)
2023-09-06T16:13:39.9838194Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:13:39.9838808Z     return mod(*inputs)
2023-09-06T16:13:39.9839947Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:13:39.9840689Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:13:39.9841804Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:13:39.9842263Z     return forward_call(*args, **kwargs)
2023-09-06T16:13:39.9842845Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T16:13:39.9843274Z     return self.inference(batched_inputs)
2023-09-06T16:13:39.9843854Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T16:13:39.9844328Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T16:13:39.9844946Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:13:39.9845356Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:13:39.9845907Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:13:39.9846310Z     return forward_call(*args, **kwargs)
2023-09-06T16:13:39.9847367Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T16:13:39.9847821Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T16:13:39.9848533Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T16:13:39.9849043Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T16:13:39.9849688Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:13:39.9850101Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:13:39.9850642Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:13:39.9851042Z     return forward_call(*args, **kwargs)
2023-09-06T16:13:39.9851594Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T16:13:39.9852030Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T16:13:39.9852602Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:13:39.9853015Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:13:39.9853563Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:13:39.9853960Z     return forward_call(*args, **kwargs)
2023-09-06T16:13:39.9854512Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T16:13:39.9854868Z     return roi_align(
2023-09-06T16:13:39.9855387Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T16:13:39.9855815Z     return torch.ops.torchvision.roi_align(
2023-09-06T16:13:39.9856522Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T16:13:39.9856900Z     return self._op(*args, **kwargs or {})
2023-09-06T16:13:39.9857384Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T16:13:39.9857633Z 
2023-09-06T16:13:39.9857829Z The above exception was the direct cause of the following exception:
2023-09-06T16:13:39.9858069Z 
2023-09-06T16:13:39.9858238Z Traceback (most recent call last):
2023-09-06T16:13:39.9858616Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T16:13:39.9858944Z     ) = runner.load_model(
2023-09-06T16:13:39.9859320Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T16:13:39.9859718Z     self.validate_model(model, example_inputs)
2023-09-06T16:13:39.9860135Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T16:13:39.9860574Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T16:13:39.9860957Z NotImplementedError: Eager model failed to run
2023-09-06T16:13:39.9861163Z 
2023-09-06T16:13:39.9861328Z WARNING:root:detectron2_fasterrcnn_r_50_dc5 failed to load
2023-09-06T16:13:44.6222684Z 
2023-09-06T16:13:48.6921702Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:13:48.6922230Z loading model: 0it [00:04, ?it/s]
2023-09-06T16:13:48.6923143Z Eager model failed to run
2023-09-06T16:13:48.6939557Z Traceback (most recent call last):
2023-09-06T16:13:48.6940374Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T16:13:48.6941066Z     self.model_iter_fn(model, example_inputs)
2023-09-06T16:13:48.6941761Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:13:48.6942136Z     return mod(*inputs)
2023-09-06T16:13:48.6943199Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:13:48.6944273Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:13:48.6944872Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:13:48.6945283Z     return forward_call(*args, **kwargs)
2023-09-06T16:13:48.6945840Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T16:13:48.6946276Z     return self.inference(batched_inputs)
2023-09-06T16:13:48.6946864Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T16:13:48.6947337Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T16:13:48.6947957Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:13:48.6948366Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:13:48.6948987Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:13:48.6949601Z     return forward_call(*args, **kwargs)
2023-09-06T16:13:48.6950206Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T16:13:48.6950673Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T16:13:48.6951297Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T16:13:48.6951823Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T16:13:48.6952474Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:13:48.6952899Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:13:48.6953726Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:13:48.6954129Z     return forward_call(*args, **kwargs)
2023-09-06T16:13:48.6954693Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2023-09-06T16:13:48.6955160Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2023-09-06T16:13:48.6955784Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:13:48.6956208Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:13:48.6956777Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:13:48.6957177Z     return forward_call(*args, **kwargs)
2023-09-06T16:13:48.6957732Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T16:13:48.6958108Z     return roi_align(
2023-09-06T16:13:48.6958631Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T16:13:48.6959073Z     return torch.ops.torchvision.roi_align(
2023-09-06T16:13:48.6959617Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T16:13:48.6960006Z     return self._op(*args, **kwargs or {})
2023-09-06T16:13:48.6960461Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T16:13:48.6960710Z 
2023-09-06T16:13:48.6960908Z The above exception was the direct cause of the following exception:
2023-09-06T16:13:48.6961152Z 
2023-09-06T16:13:48.6961279Z Traceback (most recent call last):
2023-09-06T16:13:48.6961661Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T16:13:48.6962009Z     ) = runner.load_model(
2023-09-06T16:13:48.6962376Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T16:13:48.6962931Z     self.validate_model(model, example_inputs)
2023-09-06T16:13:48.6963341Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T16:13:48.6963828Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T16:13:48.6964200Z NotImplementedError: Eager model failed to run
2023-09-06T16:13:48.6964412Z 
2023-09-06T16:13:48.6964581Z WARNING:root:detectron2_fasterrcnn_r_50_fpn failed to load
2023-09-06T16:13:53.3471026Z 
2023-09-06T16:13:57.3892443Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:13:57.3892827Z loading model: 0it [00:04, ?it/s]
2023-09-06T16:13:57.3960128Z cuda eval  detectron2_fcos_r_50_fpn           
2023-09-06T16:14:58.2343354Z [2023-09-06 16:14:58,232] torch._dynamo.convert_frame: [WARNING] torch._dynamo hit config.cache_size_limit (8)
2023-09-06T16:14:58.2344258Z [2023-09-06 16:14:58,232] torch._dynamo.convert_frame: [WARNING]    function: 'forward' (/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/batch_norm.py:318)
2023-09-06T16:14:58.2345651Z [2023-09-06 16:14:58,232] torch._dynamo.convert_frame: [WARNING] to diagnose recompilation issues, set env variable TORCHDYNAMO_REPORT_GUARD_FAILURES=1 and also see https://pytorch.org/docs/master/compile/troubleshooting.html.
2023-09-06T16:16:10.5331895Z 
2023-09-06T16:16:10.6662942Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:16:10.8007965Z running benchmark:   3% 1/30 [00:00<00:03,  7.48it/s]
2023-09-06T16:16:10.9325863Z running benchmark:   7% 2/30 [00:00<00:03,  7.45it/s]
2023-09-06T16:16:11.0651288Z running benchmark:  10% 3/30 [00:00<00:03,  7.51it/s]
2023-09-06T16:16:11.2147712Z running benchmark:  13% 4/30 [00:00<00:03,  7.52it/s]
2023-09-06T16:16:11.3473580Z running benchmark:  17% 5/30 [00:00<00:03,  7.20it/s]
2023-09-06T16:16:11.4791858Z running benchmark:  20% 6/30 [00:00<00:03,  7.31it/s]
2023-09-06T16:16:11.6091299Z running benchmark:  23% 7/30 [00:00<00:03,  7.40it/s]
2023-09-06T16:16:11.7366342Z running benchmark:  27% 8/30 [00:01<00:02,  7.49it/s]
2023-09-06T16:16:11.8666769Z running benchmark:  30% 9/30 [00:01<00:02,  7.60it/s]
2023-09-06T16:16:12.0018022Z running benchmark:  33% 10/30 [00:01<00:02,  7.63it/s]
2023-09-06T16:16:12.1335296Z running benchmark:  37% 11/30 [00:01<00:02,  7.56it/s]
2023-09-06T16:16:12.2796551Z running benchmark:  40% 12/30 [00:01<00:02,  7.57it/s]
2023-09-06T16:16:12.4110471Z running benchmark:  43% 13/30 [00:01<00:02,  7.33it/s]
2023-09-06T16:16:12.5425865Z running benchmark:  47% 14/30 [00:01<00:02,  7.41it/s]
2023-09-06T16:16:12.6746996Z running benchmark:  50% 15/30 [00:02<00:02,  7.47it/s]
2023-09-06T16:16:12.8083317Z running benchmark:  53% 16/30 [00:02<00:01,  7.50it/s]
2023-09-06T16:16:12.9413303Z running benchmark:  57% 17/30 [00:02<00:01,  7.50it/s]
2023-09-06T16:16:13.0750906Z running benchmark:  60% 18/30 [00:02<00:01,  7.50it/s]
2023-09-06T16:16:13.2260697Z running benchmark:  63% 19/30 [00:02<00:01,  7.49it/s]
2023-09-06T16:16:13.3578935Z running benchmark:  67% 20/30 [00:02<00:01,  7.21it/s]
2023-09-06T16:16:13.4908759Z running benchmark:  70% 21/30 [00:02<00:01,  7.32it/s]
2023-09-06T16:16:13.6232353Z running benchmark:  73% 22/30 [00:02<00:01,  7.38it/s]
2023-09-06T16:16:13.7549402Z running benchmark:  77% 23/30 [00:03<00:00,  7.43it/s]
2023-09-06T16:16:13.8870117Z running benchmark:  80% 24/30 [00:03<00:00,  7.48it/s]
2023-09-06T16:16:14.0170187Z running benchmark:  83% 25/30 [00:03<00:00,  7.51it/s]
2023-09-06T16:16:14.1462092Z running benchmark:  87% 26/30 [00:03<00:00,  7.56it/s]
2023-09-06T16:16:14.2832581Z running benchmark:  90% 27/30 [00:03<00:00,  7.61it/s]
2023-09-06T16:16:14.4157998Z running benchmark:  93% 28/30 [00:03<00:00,  7.52it/s]
2023-09-06T16:16:14.5467853Z running benchmark:  97% 29/30 [00:03<00:00,  7.52it/s]
2023-09-06T16:16:14.5468468Z running benchmark: 100% 30/30 [00:04<00:00,  7.56it/s]
2023-09-06T16:16:14.5468792Z running benchmark: 100% 30/30 [00:04<00:00,  7.47it/s]
2023-09-06T16:16:14.5503532Z 1.319x
2023-09-06T16:16:17.0217506Z speedup             gmean=0.00x mean=1.093x
2023-09-06T16:16:17.0218093Z abs_latency         gmean=0.00x mean=75.266x
2023-09-06T16:16:17.0218638Z compilation_latency mean=36.061 seconds
2023-09-06T16:16:17.0219195Z compression_ratio   mean=0.539x
2023-09-06T16:16:17.0219747Z eager_peak_mem      gmean=0.00x mean=0.723x
2023-09-06T16:16:17.0220297Z dynamo_peak_mem     gmean=0.00x mean=0.978x
2023-09-06T16:16:17.0221650Z calls_captured      gmean=0.00x mean=433.733x
2023-09-06T16:16:17.0224366Z unique_graphs       gmean=0.00x mean=5.733x
2023-09-06T16:16:17.0226064Z graph_breaks        gmean=0.00x mean=3.667x
2023-09-06T16:16:17.0228447Z unique_graph_breaks gmean=0.00x mean=0.733x
2023-09-06T16:16:17.7803799Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *cudagraphs-true* ]]
2023-09-06T16:16:17.7805395Z + python benchmarks/dynamo/torchbench.py --performance --cold-start-latency --inference --bfloat16 --backend inductor --device cuda --total-partitions 4 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inductor_with_cudagraphs_torchbench_bfloat16_inference_cuda_performance.csv
2023-09-06T16:16:25.0834981Z 
2023-09-06T16:16:26.0227955Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:16:26.0228416Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:16:26.0228701Z Eager model failed to run
2023-09-06T16:16:26.0242439Z Traceback (most recent call last):
2023-09-06T16:16:26.0243342Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T16:16:26.0243946Z     self.model_iter_fn(model, example_inputs)
2023-09-06T16:16:26.0244375Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:16:26.0244743Z     return mod(*inputs)
2023-09-06T16:16:26.0245779Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:16:26.0246192Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:16:26.0247332Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:16:26.0247771Z     return forward_call(*args, **kwargs)
2023-09-06T16:16:26.0248812Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 893, in forward
2023-09-06T16:16:26.0249714Z     logits = self.model(batch.dense_features, batch.sparse_features)
2023-09-06T16:16:26.0250346Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:16:26.0250806Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:16:26.0251362Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:16:26.0251765Z     return forward_call(*args, **kwargs)
2023-09-06T16:16:26.0252298Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 570, in forward
2023-09-06T16:16:26.0252738Z     embedded_dense = self.dense_arch(dense_features)
2023-09-06T16:16:26.0253334Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:16:26.0253757Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:16:26.0254311Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:16:26.0254698Z     return forward_call(*args, **kwargs)
2023-09-06T16:16:26.0255233Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 149, in forward
2023-09-06T16:16:26.0255620Z     return self.model(features)
2023-09-06T16:16:26.0256183Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:16:26.0256604Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:16:26.0257160Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:16:26.0257810Z     return forward_call(*args, **kwargs)
2023-09-06T16:16:26.0258412Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/mlp.py", line 172, in forward
2023-09-06T16:16:26.0258801Z     return self._mlp(input)
2023-09-06T16:16:26.0259345Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:16:26.0259769Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:16:26.0260321Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:16:26.0260725Z     return forward_call(*args, **kwargs)
2023-09-06T16:16:26.0261278Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/container.py", line 215, in forward
2023-09-06T16:16:26.0261645Z     input = module(input)
2023-09-06T16:16:26.0262200Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:16:26.0262625Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:16:26.0263324Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:16:26.0263708Z     return forward_call(*args, **kwargs)
2023-09-06T16:16:26.0264246Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/mlp.py", line 73, in forward
2023-09-06T16:16:26.0264664Z     return self._activation_fn(self._linear(input))
2023-09-06T16:16:26.0265253Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:16:26.0265669Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:16:26.0266206Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:16:26.0266604Z     return forward_call(*args, **kwargs)
2023-09-06T16:16:26.0267272Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
2023-09-06T16:16:26.0267708Z     return F.linear(input, self.weight, self.bias)
2023-09-06T16:16:26.0268145Z RuntimeError: mat1 and mat2 must have the same dtype, but got Float and BFloat16
2023-09-06T16:16:26.0268413Z 
2023-09-06T16:16:26.0268610Z The above exception was the direct cause of the following exception:
2023-09-06T16:16:26.0268851Z 
2023-09-06T16:16:26.0268977Z Traceback (most recent call last):
2023-09-06T16:16:26.0269530Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T16:16:26.0269875Z     ) = runner.load_model(
2023-09-06T16:16:26.0270235Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T16:16:26.0270636Z     self.validate_model(model, example_inputs)
2023-09-06T16:16:26.0271049Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T16:16:26.0271499Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T16:16:26.0271866Z NotImplementedError: Eager model failed to run
2023-09-06T16:16:26.0272074Z 
2023-09-06T16:16:26.0272211Z WARNING:root:torchrec_dlrm failed to load
2023-09-06T16:16:30.5407494Z 
2023-09-06T16:16:33.7518591Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:16:33.7518970Z loading model: 0it [00:03, ?it/s]
2023-09-06T16:16:33.7572050Z cuda eval  BERT_pytorch                       
2023-09-06T16:17:03.1861531Z 
2023-09-06T16:17:03.2927822Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:17:03.4080948Z running benchmark:  17% 5/30 [00:00<00:00, 46.72it/s]
2023-09-06T16:17:03.5175638Z running benchmark:  33% 10/30 [00:00<00:00, 44.71it/s]
2023-09-06T16:17:03.6300550Z running benchmark:  53% 16/30 [00:00<00:00, 49.24it/s]
2023-09-06T16:17:03.7439831Z running benchmark:  73% 22/30 [00:00<00:00, 50.89it/s]
2023-09-06T16:17:03.7820569Z running benchmark:  93% 28/30 [00:00<00:00, 51.53it/s]
2023-09-06T16:17:03.7820951Z running benchmark: 100% 30/30 [00:00<00:00, 50.31it/s]
2023-09-06T16:17:03.7842625Z 2.125x
2023-09-06T16:17:08.9556209Z 
2023-09-06T16:17:12.7033606Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:17:12.7033966Z loading model: 0it [00:03, ?it/s]
2023-09-06T16:17:12.7088884Z cuda eval  Background_Matting                 
2023-09-06T16:17:37.6228417Z 
2023-09-06T16:17:37.7271286Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:17:37.8316406Z running benchmark:  10% 3/30 [00:00<00:00, 28.61it/s]
2023-09-06T16:17:37.9360219Z running benchmark:  23% 7/30 [00:00<00:00, 34.25it/s]
2023-09-06T16:17:38.0403393Z running benchmark:  37% 11/30 [00:00<00:00, 36.10it/s]
2023-09-06T16:17:38.1439127Z running benchmark:  50% 15/30 [00:00<00:00, 36.98it/s]
2023-09-06T16:17:38.2642338Z running benchmark:  63% 19/30 [00:00<00:00, 37.57it/s]
2023-09-06T16:17:38.3690998Z running benchmark:  77% 23/30 [00:00<00:00, 35.98it/s]
2023-09-06T16:17:38.4474815Z running benchmark:  90% 27/30 [00:00<00:00, 36.66it/s]
2023-09-06T16:17:38.4475220Z running benchmark: 100% 30/30 [00:00<00:00, 36.35it/s]
2023-09-06T16:17:38.4494992Z 1.636x
2023-09-06T16:17:43.7818643Z 
2023-09-06T16:17:53.4658871Z loading model: 0it [00:00, ?it/s]WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T16:17:55.0655113Z 
2023-09-06T16:17:55.0655716Z loading model: 0it [00:11, ?it/s]
2023-09-06T16:17:55.0656381Z WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T16:17:55.0657011Z cuda eval  DALLE2_pytorch                     
2023-09-06T16:17:55.2223063Z WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T16:18:08.3222012Z [2023-09-06 16:18:08,320] [2/1] torch._dynamo.output_graph: [WARNING] nn.Module forward/_pre hooks are only partially supported, and were detected in your model. In particular, if you do not change/remove hooks after calling .compile(), you can disregard this warning, and otherwise you may need to set torch._dynamo.config.skip_nnmodule_hook_guards=False to ensure recompiling after changing hooks.See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations. 
2023-09-06T16:18:23.6579426Z [2023-09-06 16:18:23,656] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:18:23.9773081Z [2023-09-06 16:18:23,976] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:18:24.0861661Z [2023-09-06 16:18:24,085] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:18:24.1972022Z [2023-09-06 16:18:24,196] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:18:24.3140098Z [2023-09-06 16:18:24,313] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:18:24.4232834Z [2023-09-06 16:18:24,422] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:18:24.5350117Z [2023-09-06 16:18:24,534] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:18:24.6467052Z [2023-09-06 16:18:24,645] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:18:24.7556595Z [2023-09-06 16:18:24,754] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:18:24.8647911Z [2023-09-06 16:18:24,863] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:18:24.9733056Z [2023-09-06 16:18:24,972] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:18:25.0801974Z [2023-09-06 16:18:25,079] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:18:37.4586288Z skipping cudagraphs due to multiple devices
2023-09-06T16:19:27.6152488Z [2023-09-06 16:19:27,613] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:19:27.7161356Z [2023-09-06 16:19:27,715] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:19:27.8233925Z [2023-09-06 16:19:27,822] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:19:27.9212551Z [2023-09-06 16:19:27,920] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:19:28.0175805Z [2023-09-06 16:19:28,016] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:19:28.1124995Z [2023-09-06 16:19:28,111] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:19:28.2076798Z [2023-09-06 16:19:28,206] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:19:28.3032589Z [2023-09-06 16:19:28,302] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:19:28.3981127Z [2023-09-06 16:19:28,397] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:19:28.4958801Z [2023-09-06 16:19:28,495] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:19:28.5919655Z [2023-09-06 16:19:28,591] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:19:28.6893766Z [2023-09-06 16:19:28,688] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:19:32.4255194Z skipping cudagraphs due to multiple devices
2023-09-06T16:19:59.1086147Z ERROR:common:Backend dynamo failed in warmup()
2023-09-06T16:19:59.1087140Z Traceback (most recent call last):
2023-09-06T16:19:59.1087901Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2288, in warmup
2023-09-06T16:19:59.1088581Z     fn(model, example_inputs)
2023-09-06T16:19:59.1090911Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T16:19:59.1092047Z     return fn(*args, **kwargs)
2023-09-06T16:19:59.1093152Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:19:59.1093643Z     return mod(*inputs)
2023-09-06T16:19:59.1094888Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:19:59.1095623Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:19:59.1096546Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:19:59.1097335Z     return forward_call(*args, **kwargs)
2023-09-06T16:19:59.1097975Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T16:19:59.1098446Z     return func(*args, **kwargs)
2023-09-06T16:19:59.1098999Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 95, in inner
2023-09-06T16:19:59.1099385Z     model.eval()
2023-09-06T16:19:59.1099948Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 96, in <resume in inner>
2023-09-06T16:19:59.1100380Z     out = fn(model, *args, **kwargs)
2023-09-06T16:19:59.1100998Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3329, in forward
2023-09-06T16:19:59.1101592Z     image_embed = self.prior.sample(text, num_samples_per_batch = self.prior_num_samples, cond_scale = prior_cond_scale)
2023-09-06T16:19:59.1102351Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3332, in <resume in forward>
2023-09-06T16:19:59.1102923Z     images = self.decoder.sample(image_embed = image_embed, text = text_cond, cond_scale = cond_scale)
2023-09-06T16:19:59.1103681Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T16:19:59.1104093Z     return func(*args, **kwargs)
2023-09-06T16:19:59.1104656Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 95, in inner
2023-09-06T16:19:59.1105049Z     model.eval()
2023-09-06T16:19:59.1105602Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 96, in <resume in inner>
2023-09-06T16:19:59.1106336Z     out = fn(model, *args, **kwargs)
2023-09-06T16:19:59.1106904Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3199, in sample
2023-09-06T16:19:59.1107331Z     img = self.p_sample_loop(
2023-09-06T16:19:59.1107905Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T16:19:59.1108295Z     return func(*args, **kwargs)
2023-09-06T16:19:59.1108862Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3028, in p_sample_loop
2023-09-06T16:19:59.1109649Z     return self.p_sample_loop_ddpm(*args, noise_scheduler = noise_scheduler, **kwargs)
2023-09-06T16:19:59.1110318Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T16:19:59.1110719Z     return func(*args, **kwargs)
2023-09-06T16:19:59.1111305Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2885, in p_sample_loop_ddpm
2023-09-06T16:19:59.1111726Z     img, x_start = self.p_sample(
2023-09-06T16:19:59.1112287Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T16:19:59.1112682Z     return func(*args, **kwargs)
2023-09-06T16:19:59.1113277Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2822, in p_sample
2023-09-06T16:19:59.1114456Z     model_mean, _, model_log_variance, x_start = self.p_mean_variance(unet, x = x, t = t, image_embed = image_embed, text_encodings = text_encodings, cond_scale = cond_scale, lowres_cond_img = lowres_cond_img, self_cond = self_cond, clip_denoised = clip_denoised, predict_x_start = predict_x_start, predict_v = predict_v, noise_scheduler = noise_scheduler, learned_variance = learned_variance, lowres_noise_level = lowres_noise_level)
2023-09-06T16:19:59.1115716Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2787, in p_mean_variance
2023-09-06T16:19:59.1116492Z     model_output = default(model_output, lambda: unet.forward_with_cond_scale(x, t, image_embed = image_embed, text_encodings = text_encodings, cond_scale = cond_scale, lowres_cond_img = lowres_cond_img, self_cond = self_cond, lowres_noise_level = lowres_noise_level))
2023-09-06T16:19:59.1117379Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 65, in default
2023-09-06T16:19:59.1117786Z     return d() if callable(d) else d
2023-09-06T16:19:59.1118354Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2787, in <lambda>
2023-09-06T16:19:59.1119116Z     model_output = default(model_output, lambda: unet.forward_with_cond_scale(x, t, image_embed = image_embed, text_encodings = text_encodings, cond_scale = cond_scale, lowres_cond_img = lowres_cond_img, self_cond = self_cond, lowres_noise_level = lowres_noise_level))
2023-09-06T16:19:59.1120034Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2159, in forward_with_cond_scale
2023-09-06T16:19:59.1120465Z     logits = self.forward(*args, **kwargs)
2023-09-06T16:19:59.1121034Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2340, in forward
2023-09-06T16:19:59.1121425Z     x = resnet_block(x, t, c)
2023-09-06T16:19:59.1121967Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:19:59.1122388Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:19:59.1122954Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:19:59.1123408Z     return forward_call(*args, **kwargs)
2023-09-06T16:19:59.1124139Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T16:19:59.1124573Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T16:19:59.1125180Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 634, in _convert_frame
2023-09-06T16:19:59.1125648Z     result = inner_convert(frame, cache_entry, hooks, frame_state)
2023-09-06T16:19:59.1126236Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T16:19:59.1126609Z     return fn(*args, **kwargs)
2023-09-06T16:19:59.1127185Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T16:19:59.1127582Z     return _compile(
2023-09-06T16:19:59.1128109Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T16:19:59.1128580Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T16:19:59.1129159Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:19:59.1129537Z     r = func(*args, **kwargs)
2023-09-06T16:19:59.1130085Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T16:19:59.1130532Z     out_code = transform_code_object(code, transform)
2023-09-06T16:19:59.1131175Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T16:19:59.1131643Z     transformations(instructions, code_options)
2023-09-06T16:19:59.1132221Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T16:19:59.1132593Z     tracer.run()
2023-09-06T16:19:59.1133284Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T16:19:59.1133651Z     super().run()
2023-09-06T16:19:59.1134169Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T16:19:59.1134540Z     and self.step()
2023-09-06T16:19:59.1135061Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T16:19:59.1135446Z     getattr(self, inst.opname)(inst)
2023-09-06T16:19:59.1136003Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T16:19:59.1136396Z     return inner_fn(self, inst)
2023-09-06T16:19:59.1136964Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1119, in CALL_FUNCTION
2023-09-06T16:19:59.1137363Z     self.call_function(fn, args, {})
2023-09-06T16:19:59.1137943Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 565, in call_function
2023-09-06T16:19:59.1138395Z     self.push(fn.call_function(self, args, kwargs))
2023-09-06T16:19:59.1138981Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/torch.py", line 731, in call_function
2023-09-06T16:19:59.1139386Z     tensor_variable = wrap_fx_proxy(
2023-09-06T16:19:59.1139945Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1216, in wrap_fx_proxy
2023-09-06T16:19:59.1140349Z     return wrap_fx_proxy_cls(
2023-09-06T16:19:59.1140927Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1303, in wrap_fx_proxy_cls
2023-09-06T16:19:59.1141376Z     example_value = get_fake_value(proxy.node, tx)
2023-09-06T16:19:59.1141930Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1381, in get_fake_value
2023-09-06T16:19:59.1142541Z     raise TorchRuntimeError(str(e)).with_traceback(e.__traceback__) from None
2023-09-06T16:19:59.1143153Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1342, in get_fake_value
2023-09-06T16:19:59.1143596Z     return wrap_fake_exception(
2023-09-06T16:19:59.1144157Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 917, in wrap_fake_exception
2023-09-06T16:19:59.1144514Z     return fn()
2023-09-06T16:19:59.1145021Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1343, in <lambda>
2023-09-06T16:19:59.1145463Z     lambda: run_node(tx.output, node, args, kwargs, nnmodule)
2023-09-06T16:19:59.1146033Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1415, in run_node
2023-09-06T16:19:59.1146476Z     raise RuntimeError(fn_str + str(e)).with_traceback(e.__traceback__) from e
2023-09-06T16:19:59.1147075Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1402, in run_node
2023-09-06T16:19:59.1147466Z     return node.target(*args, **kwargs)
2023-09-06T16:19:59.1147993Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/einops/einops.py", line 483, in rearrange
2023-09-06T16:19:59.1148580Z     return reduce(cast(Tensor, tensor), pattern, reduction='rearrange', **axes_lengths)
2023-09-06T16:19:59.1149317Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/einops/einops.py", line 412, in reduce
2023-09-06T16:19:59.1149765Z     return _apply_recipe(recipe, tensor, reduction_type=reduction)
2023-09-06T16:19:59.1150350Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/einops/einops.py", line 235, in _apply_recipe
2023-09-06T16:19:59.1150782Z     _reconstruct_from_shape(recipe, backend.shape(tensor))
2023-09-06T16:19:59.1151753Z torch._dynamo.exc.TorchRuntimeError: Failed running call_function <function rearrange at 0x7f8feb2e9f30>(*(FakeTensor(..., device='cuda:0', size=(1, 128, s0, s1)), 'b c ... -> b ... c'), **{}):
2023-09-06T16:19:59.1152324Z unhashable type: 'SymInt'
2023-09-06T16:19:59.1152499Z 
2023-09-06T16:19:59.1152593Z from user code:
2023-09-06T16:19:59.1153133Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 1670, in forward
2023-09-06T16:19:59.1153656Z     h = rearrange(h, 'b c ... -> b ... c')
2023-09-06T16:19:59.1153841Z 
2023-09-06T16:19:59.1154018Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T16:19:59.1154255Z 
2023-09-06T16:19:59.1154261Z 
2023-09-06T16:19:59.1154450Z You can suppress this exception and fall back to eager by setting:
2023-09-06T16:19:59.1154786Z     import torch._dynamo
2023-09-06T16:19:59.1155098Z     torch._dynamo.config.suppress_errors = True
2023-09-06T16:19:59.1155303Z 
2023-09-06T16:20:01.7610845Z ERROR
2023-09-06T16:20:05.3913706Z 
2023-09-06T16:20:09.8305977Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:20:09.8306599Z loading model: 0it [00:04, ?it/s]
2023-09-06T16:20:09.8325175Z cuda eval  LearningToPaint                    
2023-09-06T16:20:26.4977090Z 
2023-09-06T16:20:26.6014998Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:20:26.7072929Z running benchmark:  30% 9/30 [00:00<00:00, 86.79it/s]
2023-09-06T16:20:26.8133307Z running benchmark:  63% 19/30 [00:00<00:00, 91.34it/s]
2023-09-06T16:20:26.8242428Z running benchmark:  97% 29/30 [00:00<00:00, 92.62it/s]
2023-09-06T16:20:26.8242796Z running benchmark: 100% 30/30 [00:00<00:00, 91.80it/s]
2023-09-06T16:20:26.8259753Z 1.745x
2023-09-06T16:20:32.1489966Z 
2023-09-06T16:20:34.6297824Z loading model: 0it [00:00, ?it/s]WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T16:20:35.1191790Z 
2023-09-06T16:20:35.1192703Z loading model: 0it [00:02, ?it/s]
2023-09-06T16:20:35.1193284Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T16:20:35.1193680Z cuda eval  Super_SloMo                        
2023-09-06T16:20:35.1302184Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T16:21:25.4688838Z 
2023-09-06T16:21:25.6239234Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:21:25.7698008Z running benchmark:   3% 1/30 [00:00<00:04,  6.44it/s]
2023-09-06T16:21:25.9153689Z running benchmark:  10% 3/30 [00:00<00:02, 10.60it/s]
2023-09-06T16:21:26.0614293Z running benchmark:  17% 5/30 [00:00<00:02, 12.00it/s]
2023-09-06T16:21:26.2074296Z running benchmark:  23% 7/30 [00:00<00:01, 12.66it/s]
2023-09-06T16:21:26.3531996Z running benchmark:  30% 9/30 [00:00<00:01, 13.04it/s]
2023-09-06T16:21:26.4990278Z running benchmark:  37% 11/30 [00:00<00:01, 13.27it/s]
2023-09-06T16:21:26.6592071Z running benchmark:  43% 13/30 [00:01<00:01, 13.41it/s]
2023-09-06T16:21:26.8050422Z running benchmark:  50% 15/30 [00:01<00:01, 13.10it/s]
2023-09-06T16:21:26.9508093Z running benchmark:  57% 17/30 [00:01<00:00, 13.29it/s]
2023-09-06T16:21:27.0964712Z running benchmark:  63% 19/30 [00:01<00:00, 13.42it/s]
2023-09-06T16:21:27.2419608Z running benchmark:  70% 21/30 [00:01<00:00, 13.51it/s]
2023-09-06T16:21:27.3876498Z running benchmark:  77% 23/30 [00:01<00:00, 13.58it/s]
2023-09-06T16:21:27.5336601Z running benchmark:  83% 25/30 [00:01<00:00, 13.63it/s]
2023-09-06T16:21:27.6855741Z running benchmark:  90% 27/30 [00:02<00:00, 13.65it/s]
2023-09-06T16:21:27.7587024Z running benchmark:  97% 29/30 [00:02<00:00, 13.50it/s]
2023-09-06T16:21:27.7587413Z running benchmark: 100% 30/30 [00:02<00:00, 13.10it/s]
2023-09-06T16:21:27.7610194Z 1.545x
2023-09-06T16:21:33.5225754Z 
2023-09-06T16:21:37.0648649Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:21:37.0649356Z loading model: 0it [00:03, ?it/s]
2023-09-06T16:21:37.0652589Z cuda eval  alexnet                            
2023-09-06T16:21:48.4951722Z 
2023-09-06T16:21:48.6351216Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:21:48.7659761Z running benchmark:   7% 2/30 [00:00<00:01, 14.32it/s]
2023-09-06T16:21:48.8965350Z running benchmark:  17% 5/30 [00:00<00:01, 19.23it/s]
2023-09-06T16:21:49.0263612Z running benchmark:  27% 8/30 [00:00<00:01, 20.92it/s]
2023-09-06T16:21:49.1564496Z running benchmark:  37% 11/30 [00:00<00:00, 21.76it/s]
2023-09-06T16:21:49.2866500Z running benchmark:  47% 14/30 [00:00<00:00, 22.23it/s]
2023-09-06T16:21:49.4170309Z running benchmark:  57% 17/30 [00:00<00:00, 22.50it/s]
2023-09-06T16:21:49.5476292Z running benchmark:  67% 20/30 [00:00<00:00, 22.67it/s]
2023-09-06T16:21:49.6783083Z running benchmark:  77% 23/30 [00:01<00:00, 22.77it/s]
2023-09-06T16:21:49.8086557Z running benchmark:  87% 26/30 [00:01<00:00, 22.83it/s]
2023-09-06T16:21:49.8524021Z running benchmark:  97% 29/30 [00:01<00:00, 22.88it/s]
2023-09-06T16:21:49.8524418Z running benchmark: 100% 30/30 [00:01<00:00, 22.10it/s]
2023-09-06T16:21:49.8543825Z 1.384x
2023-09-06T16:21:54.8561499Z 
2023-09-06T16:22:01.7467160Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:22:01.7467736Z loading model: 0it [00:06, ?it/s]
2023-09-06T16:22:01.7509292Z cuda eval  attention_is_all_you_need_pytorch  
2023-09-06T16:22:35.2215584Z 
2023-09-06T16:22:35.3253142Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:22:35.4264718Z running benchmark:  10% 3/30 [00:00<00:00, 28.67it/s]
2023-09-06T16:22:35.5505717Z running benchmark:  23% 7/30 [00:00<00:00, 34.96it/s]
2023-09-06T16:22:35.6715366Z running benchmark:  40% 12/30 [00:00<00:00, 37.64it/s]
2023-09-06T16:22:35.7949296Z running benchmark:  57% 17/30 [00:00<00:00, 39.15it/s]
2023-09-06T16:22:35.8954414Z running benchmark:  73% 22/30 [00:00<00:00, 39.67it/s]
2023-09-06T16:22:35.9950052Z running benchmark:  87% 26/30 [00:00<00:00, 39.71it/s]
2023-09-06T16:22:35.9950415Z running benchmark: 100% 30/30 [00:00<00:00, 38.73it/s]
2023-09-06T16:22:35.9972274Z 1.612x
2023-09-06T16:22:41.4738645Z 
2023-09-06T16:22:44.3987791Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:22:44.3988544Z loading model: 0it [00:02, ?it/s]
2023-09-06T16:22:44.3993118Z cuda eval  basic_gnn_edgecnn                  
2023-09-06T16:22:44.8893306Z [2023-09-06 16:22:44,888] [0/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T16:22:52.8782471Z 
2023-09-06T16:22:52.9813867Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:22:53.0176201Z running benchmark:  73% 22/30 [00:00<00:00, 213.63it/s]
2023-09-06T16:22:53.0176795Z running benchmark: 100% 30/30 [00:00<00:00, 215.24it/s]
2023-09-06T16:22:53.0194310Z 1.696x
2023-09-06T16:22:57.8935058Z 
2023-09-06T16:23:00.7014218Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:23:00.7014581Z loading model: 0it [00:02, ?it/s]
2023-09-06T16:23:00.7016695Z cuda eval  basic_gnn_gcn                      
2023-09-06T16:23:09.9540173Z ERROR:common:Backend dynamo failed in warmup()
2023-09-06T16:23:09.9540545Z Traceback (most recent call last):
2023-09-06T16:23:09.9541358Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2288, in warmup
2023-09-06T16:23:09.9541730Z     fn(model, example_inputs)
2023-09-06T16:23:09.9549375Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T16:23:09.9549957Z     return fn(*args, **kwargs)
2023-09-06T16:23:09.9550745Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:23:09.9551474Z     return mod(*inputs)
2023-09-06T16:23:09.9552254Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:23:09.9552669Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:23:09.9553236Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:23:09.9554032Z     return forward_call(*args, **kwargs)
2023-09-06T16:23:09.9555498Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/models/basic_gnn.py", line 222, in forward
2023-09-06T16:23:09.9556311Z     x = self.convs[i](x, edge_index, edge_weight=edge_weight)
2023-09-06T16:23:09.9557226Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:23:09.9557643Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:23:09.9558379Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:23:09.9559153Z     return forward_call(*args, **kwargs)
2023-09-06T16:23:09.9559751Z   File "/tmp/jenkins_pyg/tmp94qrt64h.py", line 221, in forward
2023-09-06T16:23:09.9560358Z     edge_index, edge_weight = gcn_norm(  # yapf: disable
2023-09-06T16:23:09.9561601Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/gcn_conv.py", line 91, in gcn_norm
2023-09-06T16:23:09.9562186Z     edge_index, edge_weight = add_remaining_self_loops(
2023-09-06T16:23:09.9563055Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T16:23:09.9564034Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T16:23:09.9564838Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 634, in _convert_frame
2023-09-06T16:23:09.9565308Z     result = inner_convert(frame, cache_entry, hooks, frame_state)
2023-09-06T16:23:09.9565892Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T16:23:09.9566283Z     return fn(*args, **kwargs)
2023-09-06T16:23:09.9566874Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T16:23:09.9567263Z     return _compile(
2023-09-06T16:23:09.9567791Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T16:23:09.9568572Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T16:23:09.9569170Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:23:09.9569532Z     r = func(*args, **kwargs)
2023-09-06T16:23:09.9570087Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T16:23:09.9570526Z     out_code = transform_code_object(code, transform)
2023-09-06T16:23:09.9571173Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T16:23:09.9571642Z     transformations(instructions, code_options)
2023-09-06T16:23:09.9572204Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T16:23:09.9572573Z     tracer.run()
2023-09-06T16:23:09.9573111Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T16:23:09.9573541Z     super().run()
2023-09-06T16:23:09.9574037Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T16:23:09.9574406Z     and self.step()
2023-09-06T16:23:09.9574925Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T16:23:09.9575323Z     getattr(self, inst.opname)(inst)
2023-09-06T16:23:09.9575877Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2166, in RETURN_VALUE
2023-09-06T16:23:09.9576292Z     self.output.compile_subgraph(
2023-09-06T16:23:09.9576868Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 905, in compile_subgraph
2023-09-06T16:23:09.9577345Z     self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
2023-09-06T16:23:09.9577914Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T16:23:09.9578248Z     return func(*args, **kwds)
2023-09-06T16:23:09.9578838Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1008, in compile_and_call_fx_graph
2023-09-06T16:23:09.9579280Z     compiled_fn = self.call_user_compiler(gm)
2023-09-06T16:23:09.9579836Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:23:09.9580200Z     r = func(*args, **kwargs)
2023-09-06T16:23:09.9580753Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1075, in call_user_compiler
2023-09-06T16:23:09.9581245Z     raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
2023-09-06T16:23:09.9581887Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1060, in call_user_compiler
2023-09-06T16:23:09.9582349Z     compiled_fn = compiler_fn(gm, self.example_inputs())
2023-09-06T16:23:09.9582943Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper
2023-09-06T16:23:09.9583431Z     compiled_gm = compiler_fn(gm, example_inputs)
2023-09-06T16:23:09.9584028Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/inductor.py", line 9, in inductor
2023-09-06T16:23:09.9584436Z     return compile_fx(*args, **kwargs)
2023-09-06T16:23:09.9584984Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1167, in compile_fx
2023-09-06T16:23:09.9585369Z     return aot_autograd(
2023-09-06T16:23:09.9585912Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 55, in compiler_fn
2023-09-06T16:23:09.9586372Z     cg = aot_module_simplified(gm, example_inputs, **kwargs)
2023-09-06T16:23:09.9587011Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3891, in aot_module_simplified
2023-09-06T16:23:09.9587588Z     compiled_fn = create_aot_dispatcher_function(
2023-09-06T16:23:09.9588157Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:23:09.9588542Z     r = func(*args, **kwargs)
2023-09-06T16:23:09.9589341Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3429, in create_aot_dispatcher_function
2023-09-06T16:23:09.9589993Z     compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:23:09.9590665Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2212, in aot_wrapper_dedupe
2023-09-06T16:23:09.9591175Z     return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:23:09.9591865Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2392, in aot_wrapper_synthetic_base
2023-09-06T16:23:09.9592374Z     return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:23:09.9593017Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1573, in aot_dispatch_base
2023-09-06T16:23:09.9593489Z     compiled_fw = compiler(fw_module, flat_args)
2023-09-06T16:23:09.9594055Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:23:09.9594435Z     r = func(*args, **kwargs)
2023-09-06T16:23:09.9594988Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1109, in fw_compiler_base
2023-09-06T16:23:09.9595372Z     return inner_compile(
2023-09-06T16:23:09.9595921Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 80, in debug_wrapper
2023-09-06T16:23:09.9596552Z     inner_compiled_fn = compiler_fn(gm, example_inputs)
2023-09-06T16:23:09.9597131Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 228, in inner
2023-09-06T16:23:09.9597499Z     return fn(*args, **kwargs)
2023-09-06T16:23:09.9597863Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T16:23:09.9598216Z     return func(*args, **kwds)
2023-09-06T16:23:09.9598761Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 55, in newFunction
2023-09-06T16:23:09.9599164Z     return old_func(*args, **kwargs)
2023-09-06T16:23:09.9599722Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 348, in compile_fx_inner
2023-09-06T16:23:09.9600196Z     compiled_graph: CompiledFxGraph = fx_codegen_and_compile(
2023-09-06T16:23:09.9600830Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 574, in fx_codegen_and_compile
2023-09-06T16:23:09.9601273Z     compiled_fn = graph.compile_to_fn()
2023-09-06T16:23:09.9601832Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 991, in compile_to_fn
2023-09-06T16:23:09.9602257Z     return self.compile_to_module().call
2023-09-06T16:23:09.9602822Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:23:09.9603217Z     r = func(*args, **kwargs)
2023-09-06T16:23:09.9603806Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 946, in compile_to_module
2023-09-06T16:23:09.9604268Z     mod = PyCodeCache.load_by_key_path(key, path, linemap=linemap)
2023-09-06T16:23:09.9604897Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1161, in load_by_key_path
2023-09-06T16:23:09.9605319Z     exec(code, mod.__dict__, mod.__dict__)
2023-09-06T16:23:09.9605817Z   File "/tmp/tmpkvfheag0/le/cletkfvccml5x23cglc4d46ihk7jl7phdlitihrwa3o6uegy7pq7.py", line 131, in <module>
2023-09-06T16:23:09.9606516Z     async_compile.wait(globals())
2023-09-06T16:23:09.9607062Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1440, in wait
2023-09-06T16:23:09.9607457Z     scope[key] = result.result()
2023-09-06T16:23:09.9608002Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1299, in result
2023-09-06T16:23:09.9608399Z     self.future.result()
2023-09-06T16:23:09.9608764Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 458, in result
2023-09-06T16:23:09.9609130Z     return self.__get_result()
2023-09-06T16:23:09.9609522Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
2023-09-06T16:23:09.9609889Z     raise self._exception
2023-09-06T16:23:09.9610343Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
2023-09-06T16:23:09.9610903Z CompilationError: at 11:70:def triton_(in_ptr0, out_ptr0, xnumel, XBLOCK : tl.constexpr):
2023-09-06T16:23:09.9611264Z     xnumel = 209982
2023-09-06T16:23:09.9611537Z     xoffset = tl.program_id(0) * XBLOCK
2023-09-06T16:23:09.9611841Z     xindex = xoffset + tl.arange(0, XBLOCK)[:]
2023-09-06T16:23:09.9612133Z     xmask = xindex < xnumel
2023-09-06T16:23:09.9612379Z     x0 = xindex
2023-09-06T16:23:09.9612655Z     tmp0 = tl.load(in_ptr0 + (209982 + x0), xmask)
2023-09-06T16:23:09.9612963Z     tmp1 = tl.where(tmp0 < 0, tmp0 + 10000, tmp0)
2023-09-06T16:23:09.9613420Z     tl.device_assert(((0 <= tmp1) & (tmp1 < 10000)) | ~xmask, "index out of bounds: 0 <= tmp1 < 10000")
2023-09-06T16:23:09.9613768Z     tmp2 = 1.0
2023-09-06T16:23:09.9614106Z     tl.atomic_add(out_ptr0 + (tl.broadcast_to(tmp1, [XBLOCK])), tmp2, xmask)
2023-09-06T16:23:09.9614448Z                                                                       ^
2023-09-06T16:23:09.9614855Z ValueError('atomic_add does not support bf16')
2023-09-06T16:23:09.9615070Z 
2023-09-06T16:23:09.9615450Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T16:23:09.9615698Z 
2023-09-06T16:23:09.9615704Z 
2023-09-06T16:23:09.9615904Z You can suppress this exception and fall back to eager by setting:
2023-09-06T16:23:09.9616247Z     import torch._dynamo
2023-09-06T16:23:09.9616553Z     torch._dynamo.config.suppress_errors = True
2023-09-06T16:23:09.9616761Z 
2023-09-06T16:23:11.4071845Z ERROR
2023-09-06T16:23:15.0100493Z 
2023-09-06T16:23:17.9960745Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:23:17.9961321Z loading model: 0it [00:02, ?it/s]
2023-09-06T16:23:17.9965270Z cuda eval  basic_gnn_gin                      
2023-09-06T16:23:18.5262179Z [2023-09-06 16:23:18,525] [0/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T16:23:24.9640710Z ERROR:common:Backend dynamo failed in warmup()
2023-09-06T16:23:24.9641528Z Traceback (most recent call last):
2023-09-06T16:23:24.9642110Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2288, in warmup
2023-09-06T16:23:24.9642635Z     fn(model, example_inputs)
2023-09-06T16:23:24.9643788Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T16:23:24.9644396Z     return fn(*args, **kwargs)
2023-09-06T16:23:24.9645303Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T16:23:24.9645984Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T16:23:24.9646950Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 634, in _convert_frame
2023-09-06T16:23:24.9647688Z     result = inner_convert(frame, cache_entry, hooks, frame_state)
2023-09-06T16:23:24.9649267Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T16:23:24.9649923Z     return fn(*args, **kwargs)
2023-09-06T16:23:24.9650803Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T16:23:24.9651398Z     return _compile(
2023-09-06T16:23:24.9652180Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T16:23:24.9652935Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T16:23:24.9653931Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:23:24.9654594Z     r = func(*args, **kwargs)
2023-09-06T16:23:24.9655584Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T16:23:24.9656381Z     out_code = transform_code_object(code, transform)
2023-09-06T16:23:24.9657581Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T16:23:24.9658381Z     transformations(instructions, code_options)
2023-09-06T16:23:24.9659457Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T16:23:24.9660148Z     tracer.run()
2023-09-06T16:23:24.9661095Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T16:23:24.9661683Z     super().run()
2023-09-06T16:23:24.9662576Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T16:23:24.9663372Z     and self.step()
2023-09-06T16:23:24.9664297Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T16:23:24.9665401Z     getattr(self, inst.opname)(inst)
2023-09-06T16:23:24.9666500Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2166, in RETURN_VALUE
2023-09-06T16:23:24.9667201Z     self.output.compile_subgraph(
2023-09-06T16:23:24.9668244Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 881, in compile_subgraph
2023-09-06T16:23:24.9669058Z     self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
2023-09-06T16:23:24.9670083Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T16:23:24.9670741Z     return func(*args, **kwds)
2023-09-06T16:23:24.9671760Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1008, in compile_and_call_fx_graph
2023-09-06T16:23:24.9672592Z     compiled_fn = self.call_user_compiler(gm)
2023-09-06T16:23:24.9673547Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:23:24.9674258Z     r = func(*args, **kwargs)
2023-09-06T16:23:24.9675273Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1075, in call_user_compiler
2023-09-06T16:23:24.9676120Z     raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
2023-09-06T16:23:24.9677179Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1060, in call_user_compiler
2023-09-06T16:23:24.9677994Z     compiled_fn = compiler_fn(gm, self.example_inputs())
2023-09-06T16:23:24.9679054Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper
2023-09-06T16:23:24.9679868Z     compiled_gm = compiler_fn(gm, example_inputs)
2023-09-06T16:23:24.9680883Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/inductor.py", line 9, in inductor
2023-09-06T16:23:24.9681623Z     return compile_fx(*args, **kwargs)
2023-09-06T16:23:24.9682938Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1167, in compile_fx
2023-09-06T16:23:24.9683593Z     return aot_autograd(
2023-09-06T16:23:24.9684540Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 55, in compiler_fn
2023-09-06T16:23:24.9685319Z     cg = aot_module_simplified(gm, example_inputs, **kwargs)
2023-09-06T16:23:24.9686408Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3891, in aot_module_simplified
2023-09-06T16:23:24.9687189Z     compiled_fn = create_aot_dispatcher_function(
2023-09-06T16:23:24.9688183Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:23:24.9688840Z     r = func(*args, **kwargs)
2023-09-06T16:23:24.9689918Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3429, in create_aot_dispatcher_function
2023-09-06T16:23:24.9690881Z     compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:23:24.9692081Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2212, in aot_wrapper_dedupe
2023-09-06T16:23:24.9692961Z     return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:23:24.9693991Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2392, in aot_wrapper_synthetic_base
2023-09-06T16:23:24.9694654Z     return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:23:24.9695564Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1573, in aot_dispatch_base
2023-09-06T16:23:24.9696145Z     compiled_fw = compiler(fw_module, flat_args)
2023-09-06T16:23:24.9696926Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:23:24.9697333Z     r = func(*args, **kwargs)
2023-09-06T16:23:24.9698123Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1109, in fw_compiler_base
2023-09-06T16:23:24.9698503Z     return inner_compile(
2023-09-06T16:23:24.9699054Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 80, in debug_wrapper
2023-09-06T16:23:24.9699562Z     inner_compiled_fn = compiler_fn(gm, example_inputs)
2023-09-06T16:23:24.9700130Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 228, in inner
2023-09-06T16:23:24.9869850Z     return fn(*args, **kwargs)
2023-09-06T16:23:24.9870787Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T16:23:24.9871267Z     return func(*args, **kwds)
2023-09-06T16:23:24.9872349Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 55, in newFunction
2023-09-06T16:23:24.9873019Z     return old_func(*args, **kwargs)
2023-09-06T16:23:24.9873858Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 348, in compile_fx_inner
2023-09-06T16:23:24.9874536Z     compiled_graph: CompiledFxGraph = fx_codegen_and_compile(
2023-09-06T16:23:24.9875434Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 574, in fx_codegen_and_compile
2023-09-06T16:23:24.9876096Z     compiled_fn = graph.compile_to_fn()
2023-09-06T16:23:24.9876994Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 991, in compile_to_fn
2023-09-06T16:23:24.9877594Z     return self.compile_to_module().call
2023-09-06T16:23:24.9878334Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:23:24.9878886Z     r = func(*args, **kwargs)
2023-09-06T16:23:24.9880399Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 946, in compile_to_module
2023-09-06T16:23:24.9881121Z     mod = PyCodeCache.load_by_key_path(key, path, linemap=linemap)
2023-09-06T16:23:24.9882102Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1161, in load_by_key_path
2023-09-06T16:23:24.9882751Z     exec(code, mod.__dict__, mod.__dict__)
2023-09-06T16:23:24.9883419Z   File "/tmp/tmp3n8495bn/as/cas424wkcw5vhxbavdfw4mws6kakp6lj6ueru73e2476z2o7ctqc.py", line 239, in <module>
2023-09-06T16:23:24.9884042Z     async_compile.wait(globals())
2023-09-06T16:23:24.9884822Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1440, in wait
2023-09-06T16:23:24.9885392Z     scope[key] = result.result()
2023-09-06T16:23:24.9886166Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1299, in result
2023-09-06T16:23:24.9886755Z     self.future.result()
2023-09-06T16:23:24.9887297Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 451, in result
2023-09-06T16:23:24.9887793Z     return self.__get_result()
2023-09-06T16:23:24.9888338Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
2023-09-06T16:23:24.9888855Z     raise self._exception
2023-09-06T16:23:24.9889612Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
2023-09-06T16:23:24.9890228Z CompilationError: at 15:53:    xindex = xoffset + tl.arange(0, XBLOCK)[:]
2023-09-06T16:23:24.9890736Z     xmask = xindex < xnumel
2023-09-06T16:23:24.9891104Z     x1 = (xindex // 64)
2023-09-06T16:23:24.9891450Z     x0 = xindex % 64
2023-09-06T16:23:24.9892084Z     tmp0 = tl.load(in_ptr0 + (200000 + x1), None, eviction_policy='evict_last')
2023-09-06T16:23:24.9892797Z     tmp2 = tl.load(in_ptr0 + (x1), None, eviction_policy='evict_last')
2023-09-06T16:23:24.9893303Z     tmp1 = tl.where(tmp0 < 0, tmp0 + 10000, tmp0)
2023-09-06T16:23:24.9894271Z     tl.device_assert((0 <= tmp1) & (tmp1 < 10000), "index out of bounds: 0 <= tmp1 < 10000")
2023-09-06T16:23:24.9894830Z     tmp3 = tl.where(tmp2 < 0, tmp2 + 10000, tmp2)
2023-09-06T16:23:24.9895385Z     tl.device_assert((0 <= tmp3) & (tmp3 < 10000), "index out of bounds: 0 <= tmp3 < 10000")
2023-09-06T16:23:24.9895997Z     tmp4 = tl.load(in_ptr1 + (x0 + (64*tmp3)), None).to(tl.float32)
2023-09-06T16:23:24.9896537Z     tl.atomic_add(out_ptr0 + (x0 + (64*tmp1)), tmp4, None)
2023-09-06T16:23:24.9896998Z                                                      ^
2023-09-06T16:23:24.9897572Z ValueError('atomic_add does not support bf16')
2023-09-06T16:23:24.9897889Z 
2023-09-06T16:23:24.9898185Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T16:23:24.9898573Z 
2023-09-06T16:23:24.9898582Z 
2023-09-06T16:23:24.9898898Z You can suppress this exception and fall back to eager by setting:
2023-09-06T16:23:24.9899505Z     import torch._dynamo
2023-09-06T16:23:24.9900027Z     torch._dynamo.config.suppress_errors = True
2023-09-06T16:23:24.9900368Z 
2023-09-06T16:23:26.1952235Z ERROR
2023-09-06T16:23:29.8118326Z 
2023-09-06T16:23:32.7173714Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:23:32.7174312Z loading model: 0it [00:02, ?it/s]
2023-09-06T16:23:32.7179049Z cuda eval  basic_gnn_sage                     
2023-09-06T16:23:33.2453292Z [2023-09-06 16:23:33,244] [0/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T16:23:39.6252905Z ERROR:common:Backend dynamo failed in warmup()
2023-09-06T16:23:39.6253858Z Traceback (most recent call last):
2023-09-06T16:23:39.6254539Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2288, in warmup
2023-09-06T16:23:39.6255164Z     fn(model, example_inputs)
2023-09-06T16:23:39.6256969Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T16:23:39.6257728Z     return fn(*args, **kwargs)
2023-09-06T16:23:39.6258710Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T16:23:39.6259617Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T16:23:39.6260747Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 634, in _convert_frame
2023-09-06T16:23:39.6261219Z     result = inner_convert(frame, cache_entry, hooks, frame_state)
2023-09-06T16:23:39.6261793Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T16:23:39.6262175Z     return fn(*args, **kwargs)
2023-09-06T16:23:39.6262947Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T16:23:39.6263374Z     return _compile(
2023-09-06T16:23:39.6263935Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T16:23:39.6264406Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T16:23:39.6265000Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:23:39.6265381Z     r = func(*args, **kwargs)
2023-09-06T16:23:39.6265934Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T16:23:39.6266364Z     out_code = transform_code_object(code, transform)
2023-09-06T16:23:39.6267014Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T16:23:39.6267486Z     transformations(instructions, code_options)
2023-09-06T16:23:39.6268369Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T16:23:39.6268736Z     tracer.run()
2023-09-06T16:23:39.6269611Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T16:23:39.6270044Z     super().run()
2023-09-06T16:23:39.6270578Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T16:23:39.6270959Z     and self.step()
2023-09-06T16:23:39.6271470Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T16:23:39.6271867Z     getattr(self, inst.opname)(inst)
2023-09-06T16:23:39.6272442Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2166, in RETURN_VALUE
2023-09-06T16:23:39.6272857Z     self.output.compile_subgraph(
2023-09-06T16:23:39.6273465Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 881, in compile_subgraph
2023-09-06T16:23:39.6274007Z     self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
2023-09-06T16:23:39.6274433Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T16:23:39.6274780Z     return func(*args, **kwds)
2023-09-06T16:23:39.6275344Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1008, in compile_and_call_fx_graph
2023-09-06T16:23:39.6275784Z     compiled_fn = self.call_user_compiler(gm)
2023-09-06T16:23:39.6276338Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:23:39.6276715Z     r = func(*args, **kwargs)
2023-09-06T16:23:39.6277277Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1075, in call_user_compiler
2023-09-06T16:23:39.6277759Z     raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
2023-09-06T16:23:39.6278589Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1060, in call_user_compiler
2023-09-06T16:23:39.6279045Z     compiled_fn = compiler_fn(gm, self.example_inputs())
2023-09-06T16:23:39.6279653Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper
2023-09-06T16:23:39.6280093Z     compiled_gm = compiler_fn(gm, example_inputs)
2023-09-06T16:23:39.6280655Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/inductor.py", line 9, in inductor
2023-09-06T16:23:39.6281061Z     return compile_fx(*args, **kwargs)
2023-09-06T16:23:39.6281616Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1167, in compile_fx
2023-09-06T16:23:39.6282000Z     return aot_autograd(
2023-09-06T16:23:39.6282534Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 55, in compiler_fn
2023-09-06T16:23:39.6282998Z     cg = aot_module_simplified(gm, example_inputs, **kwargs)
2023-09-06T16:23:39.6283630Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3891, in aot_module_simplified
2023-09-06T16:23:39.6284134Z     compiled_fn = create_aot_dispatcher_function(
2023-09-06T16:23:39.6284700Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:23:39.6285064Z     r = func(*args, **kwargs)
2023-09-06T16:23:39.6285664Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3429, in create_aot_dispatcher_function
2023-09-06T16:23:39.6286197Z     compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:23:39.6286861Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2212, in aot_wrapper_dedupe
2023-09-06T16:23:39.6287479Z     return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:23:39.6288171Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2392, in aot_wrapper_synthetic_base
2023-09-06T16:23:39.6288685Z     return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:23:39.6289327Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1573, in aot_dispatch_base
2023-09-06T16:23:39.6289767Z     compiled_fw = compiler(fw_module, flat_args)
2023-09-06T16:23:39.6290310Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:23:39.6290690Z     r = func(*args, **kwargs)
2023-09-06T16:23:39.6291246Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1109, in fw_compiler_base
2023-09-06T16:23:39.6291646Z     return inner_compile(
2023-09-06T16:23:39.6292180Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 80, in debug_wrapper
2023-09-06T16:23:39.6292630Z     inner_compiled_fn = compiler_fn(gm, example_inputs)
2023-09-06T16:23:39.6293188Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 228, in inner
2023-09-06T16:23:39.6293563Z     return fn(*args, **kwargs)
2023-09-06T16:23:39.6293966Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T16:23:39.6294295Z     return func(*args, **kwds)
2023-09-06T16:23:39.6294836Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 55, in newFunction
2023-09-06T16:23:39.6295234Z     return old_func(*args, **kwargs)
2023-09-06T16:23:39.6295792Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 348, in compile_fx_inner
2023-09-06T16:23:39.6296394Z     compiled_graph: CompiledFxGraph = fx_codegen_and_compile(
2023-09-06T16:23:39.6297024Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 574, in fx_codegen_and_compile
2023-09-06T16:23:39.6297453Z     compiled_fn = graph.compile_to_fn()
2023-09-06T16:23:39.6298010Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 991, in compile_to_fn
2023-09-06T16:23:39.6298424Z     return self.compile_to_module().call
2023-09-06T16:23:39.6298959Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:23:39.6299340Z     r = func(*args, **kwargs)
2023-09-06T16:23:39.6299884Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 946, in compile_to_module
2023-09-06T16:23:39.6300361Z     mod = PyCodeCache.load_by_key_path(key, path, linemap=linemap)
2023-09-06T16:23:39.6300978Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1161, in load_by_key_path
2023-09-06T16:23:39.6301398Z     exec(code, mod.__dict__, mod.__dict__)
2023-09-06T16:23:39.6301905Z   File "/tmp/tmpij8clyza/my/cmyknr3jcbozhaifdt7jnixzvhwtidcnhpwzxgqynmxicfbcacvw.py", line 247, in <module>
2023-09-06T16:23:39.6302381Z     async_compile.wait(globals())
2023-09-06T16:23:39.6302915Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1440, in wait
2023-09-06T16:23:39.6303292Z     scope[key] = result.result()
2023-09-06T16:23:39.6303869Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1299, in result
2023-09-06T16:23:39.6304255Z     self.future.result()
2023-09-06T16:23:39.6304634Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 451, in result
2023-09-06T16:23:39.6304981Z     return self.__get_result()
2023-09-06T16:23:39.6305515Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
2023-09-06T16:23:39.6305890Z     raise self._exception
2023-09-06T16:23:39.6306356Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
2023-09-06T16:23:39.6306783Z CompilationError: at 15:53:    xindex = xoffset + tl.arange(0, XBLOCK)[:]
2023-09-06T16:23:39.6307127Z     xmask = xindex < xnumel
2023-09-06T16:23:39.6307386Z     x1 = (xindex // 64)
2023-09-06T16:23:39.6307635Z     x0 = xindex % 64
2023-09-06T16:23:39.6308193Z     tmp0 = tl.load(in_ptr0 + (200000 + x1), None, eviction_policy='evict_last')
2023-09-06T16:23:39.6308695Z     tmp2 = tl.load(in_ptr0 + (x1), None, eviction_policy='evict_last')
2023-09-06T16:23:39.6309054Z     tmp1 = tl.where(tmp0 < 0, tmp0 + 10000, tmp0)
2023-09-06T16:23:39.6309785Z     tl.device_assert((0 <= tmp1) & (tmp1 < 10000), "index out of bounds: 0 <= tmp1 < 10000")
2023-09-06T16:23:39.6310158Z     tmp3 = tl.where(tmp2 < 0, tmp2 + 10000, tmp2)
2023-09-06T16:23:39.6310554Z     tl.device_assert((0 <= tmp3) & (tmp3 < 10000), "index out of bounds: 0 <= tmp3 < 10000")
2023-09-06T16:23:39.6310964Z     tmp4 = tl.load(in_ptr1 + (x0 + (64*tmp3)), None).to(tl.float32)
2023-09-06T16:23:39.6311324Z     tl.atomic_add(out_ptr0 + (x0 + (64*tmp1)), tmp4, None)
2023-09-06T16:23:39.6311635Z                                                      ^
2023-09-06T16:23:39.6312022Z ValueError('atomic_add does not support bf16')
2023-09-06T16:23:39.6312230Z 
2023-09-06T16:23:39.6312424Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T16:23:39.6312660Z 
2023-09-06T16:23:39.6312666Z 
2023-09-06T16:23:39.6312859Z You can suppress this exception and fall back to eager by setting:
2023-09-06T16:23:39.6313290Z     import torch._dynamo
2023-09-06T16:23:39.6313760Z     torch._dynamo.config.suppress_errors = True
2023-09-06T16:23:39.6314071Z 
2023-09-06T16:23:40.8638995Z ERROR
2023-09-06T16:23:44.4517098Z 
2023-09-06T16:23:48.6459568Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:23:48.6460350Z loading model: 0it [00:04, ?it/s]
2023-09-06T16:23:48.6544795Z cuda eval  cm3leon_generate                   
2023-09-06T16:28:31.1401513Z 
2023-09-06T16:28:32.4460932Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:28:33.7327893Z running benchmark:   3% 1/30 [00:01<00:37,  1.31s/it]
2023-09-06T16:28:35.0265979Z running benchmark:   7% 2/30 [00:02<00:36,  1.29s/it]
2023-09-06T16:28:36.3234301Z running benchmark:  10% 3/30 [00:03<00:34,  1.29s/it]
2023-09-06T16:28:37.6182124Z running benchmark:  13% 4/30 [00:05<00:33,  1.30s/it]
2023-09-06T16:28:38.9180812Z running benchmark:  17% 5/30 [00:06<00:32,  1.30s/it]
2023-09-06T16:28:40.2175330Z running benchmark:  20% 6/30 [00:07<00:31,  1.30s/it]
2023-09-06T16:28:41.5056566Z running benchmark:  23% 7/30 [00:09<00:29,  1.30s/it]
2023-09-06T16:28:42.8003962Z running benchmark:  27% 8/30 [00:10<00:28,  1.29s/it]
2023-09-06T16:28:44.1002310Z running benchmark:  30% 9/30 [00:11<00:27,  1.29s/it]
2023-09-06T16:28:45.3858000Z running benchmark:  33% 10/30 [00:12<00:25,  1.30s/it]
2023-09-06T16:28:46.6845260Z running benchmark:  37% 11/30 [00:14<00:24,  1.29s/it]
2023-09-06T16:28:47.9729646Z running benchmark:  40% 12/30 [00:15<00:23,  1.29s/it]
2023-09-06T16:28:49.2566441Z running benchmark:  43% 13/30 [00:16<00:21,  1.29s/it]
2023-09-06T16:28:50.5419739Z running benchmark:  47% 14/30 [00:18<00:20,  1.29s/it]
2023-09-06T16:28:51.8326215Z running benchmark:  50% 15/30 [00:19<00:19,  1.29s/it]
2023-09-06T16:28:53.1294210Z running benchmark:  53% 16/30 [00:20<00:18,  1.29s/it]
2023-09-06T16:28:54.4300536Z running benchmark:  57% 17/30 [00:21<00:16,  1.29s/it]
2023-09-06T16:28:55.7345617Z running benchmark:  60% 18/30 [00:23<00:15,  1.29s/it]
2023-09-06T16:28:57.0184029Z running benchmark:  63% 19/30 [00:24<00:14,  1.30s/it]
2023-09-06T16:28:58.3099341Z running benchmark:  67% 20/30 [00:25<00:12,  1.29s/it]
2023-09-06T16:28:59.6128523Z running benchmark:  70% 21/30 [00:27<00:11,  1.29s/it]
2023-09-06T16:29:00.9047437Z running benchmark:  73% 22/30 [00:28<00:10,  1.30s/it]
2023-09-06T16:29:02.2027462Z running benchmark:  77% 23/30 [00:29<00:09,  1.29s/it]
2023-09-06T16:29:03.4886380Z running benchmark:  80% 24/30 [00:31<00:07,  1.30s/it]
2023-09-06T16:29:04.7726778Z running benchmark:  83% 25/30 [00:32<00:06,  1.29s/it]
2023-09-06T16:29:06.0597469Z running benchmark:  87% 26/30 [00:33<00:05,  1.29s/it]
2023-09-06T16:29:07.3499074Z running benchmark:  90% 27/30 [00:34<00:03,  1.29s/it]
2023-09-06T16:29:08.6511310Z running benchmark:  93% 28/30 [00:36<00:02,  1.29s/it]
2023-09-06T16:29:09.9517181Z running benchmark:  97% 29/30 [00:37<00:01,  1.29s/it]
2023-09-06T16:29:09.9518097Z running benchmark: 100% 30/30 [00:38<00:00,  1.30s/it]
2023-09-06T16:29:09.9518473Z running benchmark: 100% 30/30 [00:38<00:00,  1.29s/it]
2023-09-06T16:29:09.9555506Z 4.721x
2023-09-06T16:29:16.4415463Z 
2023-09-06T16:29:17.3191768Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:29:17.3192125Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:29:17.3196311Z cuda eval  dcgan                              
2023-09-06T16:29:27.0539201Z 
2023-09-06T16:29:27.1600900Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:29:27.2653242Z running benchmark:  47% 14/30 [00:00<00:00, 131.78it/s]
2023-09-06T16:29:27.2724144Z running benchmark:  97% 29/30 [00:00<00:00, 137.99it/s]
2023-09-06T16:29:27.2724506Z running benchmark: 100% 30/30 [00:00<00:00, 137.02it/s]
2023-09-06T16:29:27.2741534Z 1.155x
2023-09-06T16:29:32.1868253Z 
2023-09-06T16:29:35.8363753Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:29:35.8364096Z loading model: 0it [00:03, ?it/s]
2023-09-06T16:29:35.8364373Z Eager model failed to run
2023-09-06T16:29:35.8377459Z Traceback (most recent call last):
2023-09-06T16:29:35.8377942Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T16:29:35.8378362Z     self.model_iter_fn(model, example_inputs)
2023-09-06T16:29:35.8378790Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:29:35.8379159Z     return mod(*inputs)
2023-09-06T16:29:35.8381372Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:29:35.8381828Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:29:35.8382401Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:29:35.8382813Z     return forward_call(*args, **kwargs)
2023-09-06T16:29:35.8383249Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/__init__.py", line 31, in forward
2023-09-06T16:29:35.8383961Z     return sources, self.model(mix)
2023-09-06T16:29:35.8384679Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:29:35.8385109Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:29:35.8385674Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:29:35.8386102Z     return forward_call(*args, **kwargs)
2023-09-06T16:29:35.8386525Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/demucs/model.py", line 209, in forward
2023-09-06T16:29:35.8386913Z     x = self.lstm(x)
2023-09-06T16:29:35.8387457Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:29:35.8387882Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:29:35.8388459Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:29:35.8388862Z     return forward_call(*args, **kwargs)
2023-09-06T16:29:35.8389453Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/demucs/model.py", line 27, in forward
2023-09-06T16:29:35.8389854Z     x = self.lstm(x)[0]
2023-09-06T16:29:35.8390389Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:29:35.8391084Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:29:35.8391654Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:29:35.8392057Z     return forward_call(*args, **kwargs)
2023-09-06T16:29:35.8392596Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/rnn.py", line 879, in forward
2023-09-06T16:29:35.8393058Z     result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
2023-09-06T16:29:35.8393645Z RuntimeError: "_thnn_fused_lstm_cell_cuda" not implemented for 'BFloat16'
2023-09-06T16:29:35.8393889Z 
2023-09-06T16:29:35.8394088Z The above exception was the direct cause of the following exception:
2023-09-06T16:29:35.8394334Z 
2023-09-06T16:29:35.8394464Z Traceback (most recent call last):
2023-09-06T16:29:35.8394845Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T16:29:35.8395180Z     ) = runner.load_model(
2023-09-06T16:29:35.8395572Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T16:29:35.8395986Z     self.validate_model(model, example_inputs)
2023-09-06T16:29:35.8396397Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T16:29:35.8396825Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T16:29:35.8397208Z NotImplementedError: Eager model failed to run
2023-09-06T16:29:35.8397418Z 
2023-09-06T16:29:35.8397543Z WARNING:root:demucs failed to load
2023-09-06T16:29:40.3641234Z 
2023-09-06T16:29:42.4494881Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:29:42.4495406Z loading model: 0it [00:02, ?it/s]
2023-09-06T16:29:42.4624388Z cuda eval  densenet121                        
2023-09-06T16:30:42.1019462Z 
2023-09-06T16:30:42.2089467Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:30:42.3124405Z running benchmark:   7% 2/30 [00:00<00:01, 18.61it/s]
2023-09-06T16:30:42.4165889Z running benchmark:  17% 5/30 [00:00<00:01, 24.60it/s]
2023-09-06T16:30:42.5206298Z running benchmark:  27% 8/30 [00:00<00:00, 26.51it/s]
2023-09-06T16:30:42.6405358Z running benchmark:  37% 11/30 [00:00<00:00, 27.42it/s]
2023-09-06T16:30:42.7451575Z running benchmark:  47% 14/30 [00:00<00:00, 26.48it/s]
2023-09-06T16:30:42.8480410Z running benchmark:  57% 17/30 [00:00<00:00, 27.21it/s]
2023-09-06T16:30:42.9508140Z running benchmark:  67% 20/30 [00:00<00:00, 27.81it/s]
2023-09-06T16:30:43.0545796Z running benchmark:  77% 23/30 [00:00<00:00, 28.23it/s]
2023-09-06T16:30:43.1597847Z running benchmark:  87% 26/30 [00:00<00:00, 28.45it/s]
2023-09-06T16:30:43.1945885Z running benchmark:  97% 29/30 [00:01<00:00, 28.48it/s]
2023-09-06T16:30:43.1946255Z running benchmark: 100% 30/30 [00:01<00:00, 27.44it/s]
2023-09-06T16:30:43.1970148Z 1.735x
2023-09-06T16:30:49.0968855Z 
2023-09-06T16:30:53.7844813Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:30:53.7845808Z loading model: 0it [00:04, ?it/s]
2023-09-06T16:30:53.7846931Z Eager model failed to run
2023-09-06T16:30:53.7857859Z Traceback (most recent call last):
2023-09-06T16:30:53.7859084Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T16:30:53.7859732Z     self.model_iter_fn(model, example_inputs)
2023-09-06T16:30:53.7860162Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:30:53.7863494Z     return mod(*inputs)
2023-09-06T16:30:53.7865027Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:30:53.7865474Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:30:53.7866093Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:30:53.7866507Z     return forward_call(*args, **kwargs)
2023-09-06T16:30:53.7867515Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T16:30:53.7867973Z     return self.inference(batched_inputs)
2023-09-06T16:30:53.7868567Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T16:30:53.7869043Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T16:30:53.7869852Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:30:53.7870255Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:30:53.7870816Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:30:53.7871222Z     return forward_call(*args, **kwargs)
2023-09-06T16:30:53.7871802Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2023-09-06T16:30:53.7872233Z     box_features = self._shared_roi_transform(
2023-09-06T16:30:53.7872863Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2023-09-06T16:30:53.7873359Z     x = self.pooler(features, boxes)
2023-09-06T16:30:53.7873932Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:30:53.7874353Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:30:53.7874889Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:30:53.7875295Z     return forward_call(*args, **kwargs)
2023-09-06T16:30:53.7875853Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T16:30:53.7876286Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T16:30:53.7876890Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:30:53.7877501Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:30:53.7878057Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:30:53.7878457Z     return forward_call(*args, **kwargs)
2023-09-06T16:30:53.7879006Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T16:30:53.7879363Z     return roi_align(
2023-09-06T16:30:53.7879888Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T16:30:53.7880326Z     return torch.ops.torchvision.roi_align(
2023-09-06T16:30:53.7880864Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T16:30:53.7881232Z     return self._op(*args, **kwargs or {})
2023-09-06T16:30:53.7881699Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T16:30:53.7881958Z 
2023-09-06T16:30:53.7882155Z The above exception was the direct cause of the following exception:
2023-09-06T16:30:53.7882402Z 
2023-09-06T16:30:53.7882528Z Traceback (most recent call last):
2023-09-06T16:30:53.7882904Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T16:30:53.7883237Z     ) = runner.load_model(
2023-09-06T16:30:53.7883658Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T16:30:53.7884066Z     self.validate_model(model, example_inputs)
2023-09-06T16:30:53.7884485Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T16:30:53.7884917Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T16:30:53.7885301Z NotImplementedError: Eager model failed to run
2023-09-06T16:30:53.7885511Z 
2023-09-06T16:30:53.7885678Z WARNING:root:detectron2_fasterrcnn_r_101_c4 failed to load
2023-09-06T16:30:58.4863597Z 
2023-09-06T16:31:05.6318824Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:31:05.6319175Z loading model: 0it [00:07, ?it/s]
2023-09-06T16:31:05.6319457Z Eager model failed to run
2023-09-06T16:31:05.6336029Z Traceback (most recent call last):
2023-09-06T16:31:05.6336796Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T16:31:05.6337607Z     self.model_iter_fn(model, example_inputs)
2023-09-06T16:31:05.6338169Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:31:05.6338594Z     return mod(*inputs)
2023-09-06T16:31:05.6340148Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:31:05.6340584Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:31:05.6341340Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:31:05.6341871Z     return forward_call(*args, **kwargs)
2023-09-06T16:31:05.6342738Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T16:31:05.6343494Z     return self.inference(batched_inputs)
2023-09-06T16:31:05.6344636Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T16:31:05.6346604Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T16:31:05.6347710Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:31:05.6348395Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:31:05.6349816Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:31:05.6350613Z     return forward_call(*args, **kwargs)
2023-09-06T16:31:05.6351728Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T16:31:05.6353387Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T16:31:05.6354099Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T16:31:05.6354628Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T16:31:05.6355264Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:31:05.6355687Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:31:05.6356241Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:31:05.6356643Z     return forward_call(*args, **kwargs)
2023-09-06T16:31:05.6357213Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T16:31:05.6357687Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T16:31:05.6358302Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:31:05.6358724Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:31:05.6359279Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:31:05.6359674Z     return forward_call(*args, **kwargs)
2023-09-06T16:31:05.6360205Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T16:31:05.6360579Z     return roi_align(
2023-09-06T16:31:05.6361103Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T16:31:05.6361790Z     return torch.ops.torchvision.roi_align(
2023-09-06T16:31:05.6362659Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T16:31:05.6363129Z     return self._op(*args, **kwargs or {})
2023-09-06T16:31:05.6363612Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T16:31:05.6363859Z 
2023-09-06T16:31:05.6364056Z The above exception was the direct cause of the following exception:
2023-09-06T16:31:05.6364300Z 
2023-09-06T16:31:05.6364414Z Traceback (most recent call last):
2023-09-06T16:31:05.6364797Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T16:31:05.6365152Z     ) = runner.load_model(
2023-09-06T16:31:05.6365526Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T16:31:05.6365925Z     self.validate_model(model, example_inputs)
2023-09-06T16:31:05.6366319Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T16:31:05.6366759Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T16:31:05.6367159Z NotImplementedError: Eager model failed to run
2023-09-06T16:31:05.6367375Z 
2023-09-06T16:31:05.6367543Z WARNING:root:detectron2_fasterrcnn_r_101_dc5 failed to load
2023-09-06T16:31:10.3158525Z 
2023-09-06T16:31:14.7553135Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:31:14.7553930Z loading model: 0it [00:04, ?it/s]
2023-09-06T16:31:14.7554392Z Eager model failed to run
2023-09-06T16:31:14.7569424Z Traceback (most recent call last):
2023-09-06T16:31:14.7571634Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T16:31:14.7572454Z     self.model_iter_fn(model, example_inputs)
2023-09-06T16:31:14.7573296Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:31:14.7573935Z     return mod(*inputs)
2023-09-06T16:31:14.7575768Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:31:14.7577489Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:31:14.7578482Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:31:14.7579237Z     return forward_call(*args, **kwargs)
2023-09-06T16:31:14.7580251Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T16:31:14.7581006Z     return self.inference(batched_inputs)
2023-09-06T16:31:14.7582024Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T16:31:14.7582936Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T16:31:14.7583996Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:31:14.7584782Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:31:14.7585781Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:31:14.7586493Z     return forward_call(*args, **kwargs)
2023-09-06T16:31:14.7587539Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T16:31:14.7588362Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T16:31:14.7589694Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T16:31:14.7590560Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T16:31:14.7591308Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:31:14.7592049Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:31:14.7592963Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:31:14.7593388Z     return forward_call(*args, **kwargs)
2023-09-06T16:31:14.7594135Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2023-09-06T16:31:14.7594603Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2023-09-06T16:31:14.7595227Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:31:14.7595643Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:31:14.7596201Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:31:14.7596585Z     return forward_call(*args, **kwargs)
2023-09-06T16:31:14.7597133Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T16:31:14.7597560Z     return roi_align(
2023-09-06T16:31:14.7598249Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T16:31:14.7598671Z     return torch.ops.torchvision.roi_align(
2023-09-06T16:31:14.7599205Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T16:31:14.7599594Z     return self._op(*args, **kwargs or {})
2023-09-06T16:31:14.7600067Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T16:31:14.7600317Z 
2023-09-06T16:31:14.7600513Z The above exception was the direct cause of the following exception:
2023-09-06T16:31:14.7600741Z 
2023-09-06T16:31:14.7600868Z Traceback (most recent call last):
2023-09-06T16:31:14.7601247Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T16:31:14.7601594Z     ) = runner.load_model(
2023-09-06T16:31:14.7601971Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T16:31:14.7602549Z     self.validate_model(model, example_inputs)
2023-09-06T16:31:14.7603029Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T16:31:14.7603477Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T16:31:14.7603860Z NotImplementedError: Eager model failed to run
2023-09-06T16:31:14.7604074Z 
2023-09-06T16:31:14.7604273Z WARNING:root:detectron2_fasterrcnn_r_101_fpn failed to load
2023-09-06T16:31:19.4058037Z 
2023-09-06T16:31:23.5217168Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:31:23.5217863Z loading model: 0it [00:04, ?it/s]
2023-09-06T16:31:23.5218142Z Eager model failed to run
2023-09-06T16:31:23.5232905Z Traceback (most recent call last):
2023-09-06T16:31:23.5233924Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T16:31:23.5235875Z     self.model_iter_fn(model, example_inputs)
2023-09-06T16:31:23.5237447Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:31:23.5237854Z     return mod(*inputs)
2023-09-06T16:31:23.5238626Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:31:23.5239064Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:31:23.5239635Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:31:23.5240084Z     return forward_call(*args, **kwargs)
2023-09-06T16:31:23.5240667Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T16:31:23.5241085Z     return self.inference(batched_inputs)
2023-09-06T16:31:23.5241685Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T16:31:23.5242166Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T16:31:23.5243295Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:31:23.5243796Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:31:23.5244368Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:31:23.5244777Z     return forward_call(*args, **kwargs)
2023-09-06T16:31:23.5245363Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2023-09-06T16:31:23.5245805Z     box_features = self._shared_roi_transform(
2023-09-06T16:31:23.5246420Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2023-09-06T16:31:23.5246866Z     x = self.pooler(features, boxes)
2023-09-06T16:31:23.5247439Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:31:23.5247871Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:31:23.5248411Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:31:23.5248808Z     return forward_call(*args, **kwargs)
2023-09-06T16:31:23.5249365Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T16:31:23.5249795Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T16:31:23.5250392Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:31:23.5250799Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:31:23.5251355Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:31:23.5251761Z     return forward_call(*args, **kwargs)
2023-09-06T16:31:23.5252313Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T16:31:23.5252860Z     return roi_align(
2023-09-06T16:31:23.5253401Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T16:31:23.5253890Z     return torch.ops.torchvision.roi_align(
2023-09-06T16:31:23.5254443Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T16:31:23.5254833Z     return self._op(*args, **kwargs or {})
2023-09-06T16:31:23.5255294Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T16:31:23.5255542Z 
2023-09-06T16:31:23.5255743Z The above exception was the direct cause of the following exception:
2023-09-06T16:31:23.5255984Z 
2023-09-06T16:31:23.5256118Z Traceback (most recent call last):
2023-09-06T16:31:23.5256501Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T16:31:23.5256831Z     ) = runner.load_model(
2023-09-06T16:31:23.5257225Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T16:31:23.5257623Z     self.validate_model(model, example_inputs)
2023-09-06T16:31:23.5258033Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T16:31:23.5258463Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T16:31:23.5258846Z NotImplementedError: Eager model failed to run
2023-09-06T16:31:23.5259059Z 
2023-09-06T16:31:23.5259224Z WARNING:root:detectron2_fasterrcnn_r_50_c4 failed to load
2023-09-06T16:31:28.1444061Z 
2023-09-06T16:31:34.5909331Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:31:34.5909726Z loading model: 0it [00:06, ?it/s]
2023-09-06T16:31:34.5910006Z Eager model failed to run
2023-09-06T16:31:34.5924065Z Traceback (most recent call last):
2023-09-06T16:31:34.5924839Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T16:31:34.5926210Z     self.model_iter_fn(model, example_inputs)
2023-09-06T16:31:34.5927008Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:31:34.5927674Z     return mod(*inputs)
2023-09-06T16:31:34.5929888Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:31:34.5930332Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:31:34.5930903Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:31:34.5931311Z     return forward_call(*args, **kwargs)
2023-09-06T16:31:34.5931886Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T16:31:34.5932296Z     return self.inference(batched_inputs)
2023-09-06T16:31:34.5932973Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T16:31:34.5933466Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T16:31:34.5934087Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:31:34.5934496Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:31:34.5935050Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:31:34.5935456Z     return forward_call(*args, **kwargs)
2023-09-06T16:31:34.5936041Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T16:31:34.5936505Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T16:31:34.5937126Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T16:31:34.5937652Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T16:31:34.5938567Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:31:34.5938991Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:31:34.5939527Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:31:34.5939930Z     return forward_call(*args, **kwargs)
2023-09-06T16:31:34.5940493Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T16:31:34.5940930Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T16:31:34.5941568Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:31:34.5941975Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:31:34.5942536Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:31:34.5942999Z     return forward_call(*args, **kwargs)
2023-09-06T16:31:34.5943556Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T16:31:34.5943918Z     return roi_align(
2023-09-06T16:31:34.5944451Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T16:31:34.5944885Z     return torch.ops.torchvision.roi_align(
2023-09-06T16:31:34.5945423Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T16:31:34.5945811Z     return self._op(*args, **kwargs or {})
2023-09-06T16:31:34.5946265Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T16:31:34.5946510Z 
2023-09-06T16:31:34.5946710Z The above exception was the direct cause of the following exception:
2023-09-06T16:31:34.5946962Z 
2023-09-06T16:31:34.5947227Z Traceback (most recent call last):
2023-09-06T16:31:34.5947618Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T16:31:34.5947950Z     ) = runner.load_model(
2023-09-06T16:31:34.5948334Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T16:31:34.5948743Z     self.validate_model(model, example_inputs)
2023-09-06T16:31:34.5949367Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T16:31:34.5949795Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T16:31:34.5950182Z NotImplementedError: Eager model failed to run
2023-09-06T16:31:34.5950389Z 
2023-09-06T16:31:34.5950557Z WARNING:root:detectron2_fasterrcnn_r_50_dc5 failed to load
2023-09-06T16:31:39.2165052Z 
2023-09-06T16:31:43.2021168Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:31:43.2021778Z loading model: 0it [00:03, ?it/s]
2023-09-06T16:31:43.2022261Z Eager model failed to run
2023-09-06T16:31:43.2037759Z Traceback (most recent call last):
2023-09-06T16:31:43.2038607Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T16:31:43.2039328Z     self.model_iter_fn(model, example_inputs)
2023-09-06T16:31:43.2040077Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:31:43.2040708Z     return mod(*inputs)
2023-09-06T16:31:43.2042284Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:31:43.2042720Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:31:43.2043294Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:31:43.2043748Z     return forward_call(*args, **kwargs)
2023-09-06T16:31:43.2044337Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T16:31:43.2045223Z     return self.inference(batched_inputs)
2023-09-06T16:31:43.2045814Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T16:31:43.2046289Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T16:31:43.2046892Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:31:43.2047313Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:31:43.2047864Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:31:43.2048301Z     return forward_call(*args, **kwargs)
2023-09-06T16:31:43.2048885Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T16:31:43.2049333Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T16:31:43.2050014Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T16:31:43.2050531Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T16:31:43.2051178Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:31:43.2051593Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:31:43.2052129Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:31:43.2052531Z     return forward_call(*args, **kwargs)
2023-09-06T16:31:43.2053082Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2023-09-06T16:31:43.2053581Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2023-09-06T16:31:43.2054358Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:31:43.2054794Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:31:43.2055353Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:31:43.2055752Z     return forward_call(*args, **kwargs)
2023-09-06T16:31:43.2056301Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T16:31:43.2056663Z     return roi_align(
2023-09-06T16:31:43.2057190Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T16:31:43.2057623Z     return torch.ops.torchvision.roi_align(
2023-09-06T16:31:43.2058161Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T16:31:43.2058536Z     return self._op(*args, **kwargs or {})
2023-09-06T16:31:43.2059019Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T16:31:43.2059264Z 
2023-09-06T16:31:43.2059495Z The above exception was the direct cause of the following exception:
2023-09-06T16:31:43.2059735Z 
2023-09-06T16:31:43.2059937Z Traceback (most recent call last):
2023-09-06T16:31:43.2060322Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T16:31:43.2060686Z     ) = runner.load_model(
2023-09-06T16:31:43.2061063Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T16:31:43.2061463Z     self.validate_model(model, example_inputs)
2023-09-06T16:31:43.2061875Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T16:31:43.2062299Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T16:31:43.2062849Z NotImplementedError: Eager model failed to run
2023-09-06T16:31:43.2063061Z 
2023-09-06T16:31:43.2063361Z WARNING:root:detectron2_fasterrcnn_r_50_fpn failed to load
2023-09-06T16:31:47.8454511Z 
2023-09-06T16:31:51.9214412Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:31:51.9215675Z loading model: 0it [00:04, ?it/s]
2023-09-06T16:31:51.9300960Z cuda eval  detectron2_fcos_r_50_fpn           
2023-09-06T16:32:07.5612174Z skipping cudagraphs for unknown reason
2023-09-06T16:32:09.8967693Z skipping cudagraphs due to input mutation
2023-09-06T16:32:52.7043147Z [2023-09-06 16:32:52,702] torch._dynamo.convert_frame: [WARNING] torch._dynamo hit config.cache_size_limit (8)
2023-09-06T16:32:52.7044542Z [2023-09-06 16:32:52,702] torch._dynamo.convert_frame: [WARNING]    function: 'forward' (/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/batch_norm.py:318)
2023-09-06T16:32:52.7045932Z [2023-09-06 16:32:52,702] torch._dynamo.convert_frame: [WARNING] to diagnose recompilation issues, set env variable TORCHDYNAMO_REPORT_GUARD_FAILURES=1 and also see https://pytorch.org/docs/master/compile/troubleshooting.html.
2023-09-06T16:33:11.9079473Z skipping cudagraphs due to input mutation
2023-09-06T16:33:12.2573099Z skipping cudagraphs for unknown reason
2023-09-06T16:33:13.3320445Z skipping cudagraphs due to input mutation
2023-09-06T16:33:13.5270244Z skipping cudagraphs for unknown reason
2023-09-06T16:33:14.5216274Z skipping cudagraphs due to input mutation
2023-09-06T16:33:15.5992850Z skipping cudagraphs due to input mutation
2023-09-06T16:34:05.7758811Z skipping cudagraphs due to input mutation
2023-09-06T16:34:37.7327200Z 
2023-09-06T16:34:37.8671661Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:34:37.9979460Z running benchmark:   3% 1/30 [00:00<00:03,  7.41it/s]
2023-09-06T16:34:38.1359442Z running benchmark:   7% 2/30 [00:00<00:03,  7.54it/s]
2023-09-06T16:34:38.2663318Z running benchmark:  10% 3/30 [00:00<00:03,  7.40it/s]
2023-09-06T16:34:38.3959744Z running benchmark:  13% 4/30 [00:00<00:03,  7.50it/s]
2023-09-06T16:34:38.5262050Z running benchmark:  17% 5/30 [00:00<00:03,  7.58it/s]
2023-09-06T16:34:38.6559857Z running benchmark:  20% 6/30 [00:00<00:03,  7.61it/s]
2023-09-06T16:34:38.7856360Z running benchmark:  23% 7/30 [00:00<00:03,  7.64it/s]
2023-09-06T16:34:38.9154592Z running benchmark:  27% 8/30 [00:01<00:02,  7.66it/s]
2023-09-06T16:34:39.0504933Z running benchmark:  30% 9/30 [00:01<00:02,  7.68it/s]
2023-09-06T16:34:39.1888386Z running benchmark:  33% 10/30 [00:01<00:02,  7.59it/s]
2023-09-06T16:34:39.3181039Z running benchmark:  37% 11/30 [00:01<00:02,  7.48it/s]
2023-09-06T16:34:39.4489507Z running benchmark:  40% 12/30 [00:01<00:02,  7.55it/s]
2023-09-06T16:34:39.5793882Z running benchmark:  43% 13/30 [00:01<00:02,  7.58it/s]
2023-09-06T16:34:39.7106032Z running benchmark:  47% 14/30 [00:01<00:02,  7.61it/s]
2023-09-06T16:34:39.8412513Z running benchmark:  50% 15/30 [00:01<00:01,  7.61it/s]
2023-09-06T16:34:39.9713167Z running benchmark:  53% 16/30 [00:02<00:01,  7.62it/s]
2023-09-06T16:34:40.1024106Z running benchmark:  57% 17/30 [00:02<00:01,  7.64it/s]
2023-09-06T16:34:40.2534855Z running benchmark:  60% 18/30 [00:02<00:01,  7.64it/s]
2023-09-06T16:34:40.3828133Z running benchmark:  63% 19/30 [00:02<00:01,  7.30it/s]
2023-09-06T16:34:40.5116063Z running benchmark:  67% 20/30 [00:02<00:01,  7.42it/s]
2023-09-06T16:34:40.6398180Z running benchmark:  70% 21/30 [00:02<00:01,  7.52it/s]
2023-09-06T16:34:40.7703183Z running benchmark:  73% 22/30 [00:02<00:01,  7.60it/s]
2023-09-06T16:34:40.9012530Z running benchmark:  77% 23/30 [00:03<00:00,  7.62it/s]
2023-09-06T16:34:41.0332271Z running benchmark:  80% 24/30 [00:03<00:00,  7.63it/s]
2023-09-06T16:34:41.1620311Z running benchmark:  83% 25/30 [00:03<00:00,  7.61it/s]
2023-09-06T16:34:41.2963559Z running benchmark:  87% 26/30 [00:03<00:00,  7.66it/s]
2023-09-06T16:34:41.4232299Z running benchmark:  90% 27/30 [00:03<00:00,  7.59it/s]
2023-09-06T16:34:41.5512465Z running benchmark:  93% 28/30 [00:03<00:00,  7.68it/s]
2023-09-06T16:34:41.6816760Z running benchmark:  97% 29/30 [00:03<00:00,  7.72it/s]
2023-09-06T16:34:41.6817369Z running benchmark: 100% 30/30 [00:03<00:00,  7.70it/s]
2023-09-06T16:34:41.6818629Z running benchmark: 100% 30/30 [00:03<00:00,  7.60it/s]
2023-09-06T16:34:41.6856591Z 1.344x
2023-09-06T16:34:44.1969647Z speedup             gmean=0.00x mean=1.380x
2023-09-06T16:34:44.1970036Z abs_latency         gmean=0.00x mean=24.894x
2023-09-06T16:34:44.1970343Z compilation_latency mean=45.245 seconds
2023-09-06T16:34:44.1971496Z compression_ratio   mean=0.539x
2023-09-06T16:34:44.1973268Z eager_peak_mem      gmean=0.00x mean=0.722x
2023-09-06T16:34:44.1975153Z dynamo_peak_mem     gmean=0.00x mean=0.978x
2023-09-06T16:34:44.1977493Z calls_captured      gmean=0.00x mean=433.733x
2023-09-06T16:34:44.1979778Z unique_graphs       gmean=0.00x mean=5.733x
2023-09-06T16:34:44.1981738Z graph_breaks        gmean=0.00x mean=3.667x
2023-09-06T16:34:44.1984097Z unique_graph_breaks gmean=0.00x mean=0.733x
2023-09-06T16:34:44.9664018Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *dynamic-true* ]]
2023-09-06T16:34:44.9666551Z + python benchmarks/dynamo/torchbench.py --performance --cold-start-latency --inference --bfloat16 --backend inductor --dynamic-shapes --dynamic-batch-only --device cuda --total-partitions 4 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inductor_dynamic_torchbench_bfloat16_inference_cuda_performance.csv
2023-09-06T16:34:52.2127839Z 
2023-09-06T16:34:53.1280750Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:34:53.1281112Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:34:53.1281439Z Eager model failed to run
2023-09-06T16:34:53.1297277Z Traceback (most recent call last):
2023-09-06T16:34:53.1297941Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T16:34:53.1300522Z     self.model_iter_fn(model, example_inputs)
2023-09-06T16:34:53.1301028Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:34:53.1301403Z     return mod(*inputs)
2023-09-06T16:34:53.1302745Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:34:53.1303194Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:34:53.1307383Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:34:53.1308289Z     return forward_call(*args, **kwargs)
2023-09-06T16:34:53.1309330Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 893, in forward
2023-09-06T16:34:53.1310183Z     logits = self.model(batch.dense_features, batch.sparse_features)
2023-09-06T16:34:53.1310853Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:34:53.1311277Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:34:53.1312201Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:34:53.1312966Z     return forward_call(*args, **kwargs)
2023-09-06T16:34:53.1314117Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 570, in forward
2023-09-06T16:34:53.1314549Z     embedded_dense = self.dense_arch(dense_features)
2023-09-06T16:34:53.1315143Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:34:53.1315570Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:34:53.1316123Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:34:53.1316529Z     return forward_call(*args, **kwargs)
2023-09-06T16:34:53.1317044Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 149, in forward
2023-09-06T16:34:53.1317436Z     return self.model(features)
2023-09-06T16:34:53.1318178Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:34:53.1319876Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:34:53.1320736Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:34:53.1321141Z     return forward_call(*args, **kwargs)
2023-09-06T16:34:53.1321668Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/mlp.py", line 172, in forward
2023-09-06T16:34:53.1322048Z     return self._mlp(input)
2023-09-06T16:34:53.1322600Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:34:53.1323016Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:34:53.1323557Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:34:53.1323958Z     return forward_call(*args, **kwargs)
2023-09-06T16:34:53.1324523Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/container.py", line 215, in forward
2023-09-06T16:34:53.1324915Z     input = module(input)
2023-09-06T16:34:53.1325449Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:34:53.1325865Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:34:53.1326418Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:34:53.1326819Z     return forward_call(*args, **kwargs)
2023-09-06T16:34:53.1327351Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/mlp.py", line 73, in forward
2023-09-06T16:34:53.1327756Z     return self._activation_fn(self._linear(input))
2023-09-06T16:34:53.1328340Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:34:53.1328807Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:34:53.1329563Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:34:53.1329958Z     return forward_call(*args, **kwargs)
2023-09-06T16:34:53.1330503Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
2023-09-06T16:34:53.1330935Z     return F.linear(input, self.weight, self.bias)
2023-09-06T16:34:53.1331350Z RuntimeError: mat1 and mat2 must have the same dtype, but got Float and BFloat16
2023-09-06T16:34:53.1331615Z 
2023-09-06T16:34:53.1331814Z The above exception was the direct cause of the following exception:
2023-09-06T16:34:53.1332055Z 
2023-09-06T16:34:53.1332169Z Traceback (most recent call last):
2023-09-06T16:34:53.1332547Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T16:34:53.1332902Z     ) = runner.load_model(
2023-09-06T16:34:53.1333295Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T16:34:53.1333697Z     self.validate_model(model, example_inputs)
2023-09-06T16:34:53.1334118Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T16:34:53.1334570Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T16:34:53.1334956Z NotImplementedError: Eager model failed to run
2023-09-06T16:34:53.1335165Z 
2023-09-06T16:34:53.1335302Z WARNING:root:torchrec_dlrm failed to load
2023-09-06T16:34:57.6384136Z 
2023-09-06T16:35:00.8295650Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:35:00.8296002Z loading model: 0it [00:03, ?it/s]
2023-09-06T16:35:00.8347303Z cuda eval  BERT_pytorch                       
2023-09-06T16:35:40.5430928Z 
2023-09-06T16:35:40.6577980Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:35:40.7667293Z running benchmark:  13% 4/30 [00:00<00:00, 34.99it/s]
2023-09-06T16:35:40.8753503Z running benchmark:  33% 10/30 [00:00<00:00, 46.45it/s]
2023-09-06T16:35:40.9857840Z running benchmark:  53% 16/30 [00:00<00:00, 50.44it/s]
2023-09-06T16:35:41.0951020Z running benchmark:  73% 22/30 [00:00<00:00, 51.96it/s]
2023-09-06T16:35:41.1319701Z running benchmark:  93% 28/30 [00:00<00:00, 53.02it/s]
2023-09-06T16:35:41.1320269Z running benchmark: 100% 30/30 [00:00<00:00, 50.93it/s]
2023-09-06T16:35:41.1342334Z 2.063x
2023-09-06T16:35:46.4007632Z 
2023-09-06T16:35:50.1483569Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:35:50.1484192Z loading model: 0it [00:03, ?it/s]
2023-09-06T16:35:50.1539418Z cuda eval  Background_Matting                 
2023-09-06T16:36:15.1314061Z 
2023-09-06T16:36:15.2346703Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:36:15.3379428Z running benchmark:  10% 3/30 [00:00<00:00, 28.97it/s]
2023-09-06T16:36:15.4591275Z running benchmark:  23% 7/30 [00:00<00:00, 34.67it/s]
2023-09-06T16:36:15.5632679Z running benchmark:  37% 11/30 [00:00<00:00, 33.88it/s]
2023-09-06T16:36:15.6671804Z running benchmark:  50% 15/30 [00:00<00:00, 35.56it/s]
2023-09-06T16:36:15.7713741Z running benchmark:  63% 19/30 [00:00<00:00, 36.60it/s]
2023-09-06T16:36:15.8757580Z running benchmark:  77% 23/30 [00:00<00:00, 37.20it/s]
2023-09-06T16:36:15.9540998Z running benchmark:  90% 27/30 [00:00<00:00, 37.55it/s]
2023-09-06T16:36:15.9541385Z running benchmark: 100% 30/30 [00:00<00:00, 36.44it/s]
2023-09-06T16:36:15.9565738Z 1.634x
2023-09-06T16:36:21.3383035Z 
2023-09-06T16:36:31.0746111Z loading model: 0it [00:00, ?it/s]WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T16:36:32.5456799Z 
2023-09-06T16:36:32.5457587Z loading model: 0it [00:11, ?it/s]
2023-09-06T16:36:32.5458391Z WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T16:36:32.5458943Z cuda eval  DALLE2_pytorch                     
2023-09-06T16:36:32.7050924Z WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T16:36:45.8072721Z [2023-09-06 16:36:45,805] [2/1] torch._dynamo.output_graph: [WARNING] nn.Module forward/_pre hooks are only partially supported, and were detected in your model. In particular, if you do not change/remove hooks after calling .compile(), you can disregard this warning, and otherwise you may need to set torch._dynamo.config.skip_nnmodule_hook_guards=False to ensure recompiling after changing hooks.See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations. 
2023-09-06T16:37:01.1480018Z [2023-09-06 16:37:01,146] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:37:01.4811582Z [2023-09-06 16:37:01,480] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:37:01.5942976Z [2023-09-06 16:37:01,593] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:37:01.7054853Z [2023-09-06 16:37:01,704] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:37:01.8184984Z [2023-09-06 16:37:01,817] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:37:01.9271964Z [2023-09-06 16:37:01,926] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:37:02.0383129Z [2023-09-06 16:37:02,037] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:37:02.1698711Z [2023-09-06 16:37:02,169] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:37:02.2795071Z [2023-09-06 16:37:02,278] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:37:02.3895152Z [2023-09-06 16:37:02,388] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:37:02.5092742Z [2023-09-06 16:37:02,508] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:37:02.6213627Z [2023-09-06 16:37:02,620] [4/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:37:14.9371193Z skipping cudagraphs due to multiple devices
2023-09-06T16:38:05.1137208Z [2023-09-06 16:38:05,111] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:38:05.2123152Z [2023-09-06 16:38:05,211] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:38:05.3075107Z [2023-09-06 16:38:05,306] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:38:05.4042590Z [2023-09-06 16:38:05,403] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:38:05.5006052Z [2023-09-06 16:38:05,499] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:38:05.5970299Z [2023-09-06 16:38:05,596] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:38:05.6941715Z [2023-09-06 16:38:05,693] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:38:05.7886339Z [2023-09-06 16:38:05,787] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:38:05.8833850Z [2023-09-06 16:38:05,882] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:38:05.9910036Z [2023-09-06 16:38:05,990] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:38:06.0862718Z [2023-09-06 16:38:06,085] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:38:06.1830131Z [2023-09-06 16:38:06,182] [28/0] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-09-06T16:38:10.0122963Z skipping cudagraphs due to multiple devices
2023-09-06T16:38:36.4892758Z ERROR:common:Backend dynamo failed in warmup()
2023-09-06T16:38:36.4893389Z Traceback (most recent call last):
2023-09-06T16:38:36.4894130Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2288, in warmup
2023-09-06T16:38:36.4899006Z     fn(model, example_inputs)
2023-09-06T16:38:36.4900802Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T16:38:36.4901244Z     return fn(*args, **kwargs)
2023-09-06T16:38:36.4902264Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:38:36.4903051Z     return mod(*inputs)
2023-09-06T16:38:36.4904122Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:38:36.4904567Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:38:36.4905207Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:38:36.4905617Z     return forward_call(*args, **kwargs)
2023-09-06T16:38:36.4906182Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T16:38:36.4906584Z     return func(*args, **kwargs)
2023-09-06T16:38:36.4907108Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 95, in inner
2023-09-06T16:38:36.4907487Z     model.eval()
2023-09-06T16:38:36.4908061Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 96, in <resume in inner>
2023-09-06T16:38:36.4908481Z     out = fn(model, *args, **kwargs)
2023-09-06T16:38:36.4909044Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3329, in forward
2023-09-06T16:38:36.4909927Z     image_embed = self.prior.sample(text, num_samples_per_batch = self.prior_num_samples, cond_scale = prior_cond_scale)
2023-09-06T16:38:36.4910681Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3332, in <resume in forward>
2023-09-06T16:38:36.4911229Z     images = self.decoder.sample(image_embed = image_embed, text = text_cond, cond_scale = cond_scale)
2023-09-06T16:38:36.4911895Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T16:38:36.4912298Z     return func(*args, **kwargs)
2023-09-06T16:38:36.4913231Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 95, in inner
2023-09-06T16:38:36.4913619Z     model.eval()
2023-09-06T16:38:36.4914174Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 96, in <resume in inner>
2023-09-06T16:38:36.4914589Z     out = fn(model, *args, **kwargs)
2023-09-06T16:38:36.4915130Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3199, in sample
2023-09-06T16:38:36.4915531Z     img = self.p_sample_loop(
2023-09-06T16:38:36.4916082Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T16:38:36.4916484Z     return func(*args, **kwargs)
2023-09-06T16:38:36.4917046Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3028, in p_sample_loop
2023-09-06T16:38:36.4917558Z     return self.p_sample_loop_ddpm(*args, noise_scheduler = noise_scheduler, **kwargs)
2023-09-06T16:38:36.4918217Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T16:38:36.4918618Z     return func(*args, **kwargs)
2023-09-06T16:38:36.4919192Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2885, in p_sample_loop_ddpm
2023-09-06T16:38:36.4919593Z     img, x_start = self.p_sample(
2023-09-06T16:38:36.4920150Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T16:38:36.4920550Z     return func(*args, **kwargs)
2023-09-06T16:38:36.4921102Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2822, in p_sample
2023-09-06T16:38:36.4922268Z     model_mean, _, model_log_variance, x_start = self.p_mean_variance(unet, x = x, t = t, image_embed = image_embed, text_encodings = text_encodings, cond_scale = cond_scale, lowres_cond_img = lowres_cond_img, self_cond = self_cond, clip_denoised = clip_denoised, predict_x_start = predict_x_start, predict_v = predict_v, noise_scheduler = noise_scheduler, learned_variance = learned_variance, lowres_noise_level = lowres_noise_level)
2023-09-06T16:38:36.4923586Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2787, in p_mean_variance
2023-09-06T16:38:36.4924373Z     model_output = default(model_output, lambda: unet.forward_with_cond_scale(x, t, image_embed = image_embed, text_encodings = text_encodings, cond_scale = cond_scale, lowres_cond_img = lowres_cond_img, self_cond = self_cond, lowres_noise_level = lowres_noise_level))
2023-09-06T16:38:36.4925244Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 65, in default
2023-09-06T16:38:36.4925655Z     return d() if callable(d) else d
2023-09-06T16:38:36.4926236Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2787, in <lambda>
2023-09-06T16:38:36.4927010Z     model_output = default(model_output, lambda: unet.forward_with_cond_scale(x, t, image_embed = image_embed, text_encodings = text_encodings, cond_scale = cond_scale, lowres_cond_img = lowres_cond_img, self_cond = self_cond, lowres_noise_level = lowres_noise_level))
2023-09-06T16:38:36.4927928Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2159, in forward_with_cond_scale
2023-09-06T16:38:36.4928351Z     logits = self.forward(*args, **kwargs)
2023-09-06T16:38:36.4928923Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2340, in forward
2023-09-06T16:38:36.4929314Z     x = resnet_block(x, t, c)
2023-09-06T16:38:36.4929871Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:38:36.4930296Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:38:36.4930977Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:38:36.4931383Z     return forward_call(*args, **kwargs)
2023-09-06T16:38:36.4931948Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T16:38:36.4932405Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T16:38:36.4933031Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 634, in _convert_frame
2023-09-06T16:38:36.4933548Z     result = inner_convert(frame, cache_entry, hooks, frame_state)
2023-09-06T16:38:36.4934136Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T16:38:36.4934528Z     return fn(*args, **kwargs)
2023-09-06T16:38:36.4935103Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T16:38:36.4935505Z     return _compile(
2023-09-06T16:38:36.4936043Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T16:38:36.4936505Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T16:38:36.4937098Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:38:36.4937462Z     r = func(*args, **kwargs)
2023-09-06T16:38:36.4938012Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T16:38:36.4938451Z     out_code = transform_code_object(code, transform)
2023-09-06T16:38:36.4939097Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T16:38:36.4939572Z     transformations(instructions, code_options)
2023-09-06T16:38:36.4940275Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T16:38:36.4940656Z     tracer.run()
2023-09-06T16:38:36.4941182Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T16:38:36.4941549Z     super().run()
2023-09-06T16:38:36.4942054Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T16:38:36.4942432Z     and self.step()
2023-09-06T16:38:36.4943004Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T16:38:36.4943409Z     getattr(self, inst.opname)(inst)
2023-09-06T16:38:36.4943971Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T16:38:36.4944353Z     return inner_fn(self, inst)
2023-09-06T16:38:36.4944930Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1119, in CALL_FUNCTION
2023-09-06T16:38:36.4945349Z     self.call_function(fn, args, {})
2023-09-06T16:38:36.4945917Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 565, in call_function
2023-09-06T16:38:36.4946340Z     self.push(fn.call_function(self, args, kwargs))
2023-09-06T16:38:36.4946930Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/torch.py", line 731, in call_function
2023-09-06T16:38:36.4947341Z     tensor_variable = wrap_fx_proxy(
2023-09-06T16:38:36.4947918Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1216, in wrap_fx_proxy
2023-09-06T16:38:36.4948324Z     return wrap_fx_proxy_cls(
2023-09-06T16:38:36.4948887Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1303, in wrap_fx_proxy_cls
2023-09-06T16:38:36.4949839Z     example_value = get_fake_value(proxy.node, tx)
2023-09-06T16:38:36.4950443Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1381, in get_fake_value
2023-09-06T16:38:36.4950926Z     raise TorchRuntimeError(str(e)).with_traceback(e.__traceback__) from None
2023-09-06T16:38:36.4951522Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1342, in get_fake_value
2023-09-06T16:38:36.4951916Z     return wrap_fake_exception(
2023-09-06T16:38:36.4952464Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 917, in wrap_fake_exception
2023-09-06T16:38:36.4952897Z     return fn()
2023-09-06T16:38:36.4953386Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1343, in <lambda>
2023-09-06T16:38:36.4953828Z     lambda: run_node(tx.output, node, args, kwargs, nnmodule)
2023-09-06T16:38:36.4954406Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1415, in run_node
2023-09-06T16:38:36.4954873Z     raise RuntimeError(fn_str + str(e)).with_traceback(e.__traceback__) from e
2023-09-06T16:38:36.4955471Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1402, in run_node
2023-09-06T16:38:36.4955855Z     return node.target(*args, **kwargs)
2023-09-06T16:38:36.4956379Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/einops/einops.py", line 483, in rearrange
2023-09-06T16:38:36.4956966Z     return reduce(cast(Tensor, tensor), pattern, reduction='rearrange', **axes_lengths)
2023-09-06T16:38:36.4957571Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/einops/einops.py", line 412, in reduce
2023-09-06T16:38:36.4958011Z     return _apply_recipe(recipe, tensor, reduction_type=reduction)
2023-09-06T16:38:36.4958580Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/einops/einops.py", line 235, in _apply_recipe
2023-09-06T16:38:36.4959270Z     _reconstruct_from_shape(recipe, backend.shape(tensor))
2023-09-06T16:38:36.4960097Z torch._dynamo.exc.TorchRuntimeError: Failed running call_function <function rearrange at 0x7faad7f1df30>(*(FakeTensor(..., device='cuda:0', size=(1, 128, s0, s1)), 'b c ... -> b ... c'), **{}):
2023-09-06T16:38:36.4960649Z unhashable type: 'SymInt'
2023-09-06T16:38:36.4960832Z 
2023-09-06T16:38:36.4960912Z from user code:
2023-09-06T16:38:36.4961458Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 1670, in forward
2023-09-06T16:38:36.4961945Z     h = rearrange(h, 'b c ... -> b ... c')
2023-09-06T16:38:36.4962129Z 
2023-09-06T16:38:36.4962361Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T16:38:36.4962598Z 
2023-09-06T16:38:36.4962604Z 
2023-09-06T16:38:36.4962843Z You can suppress this exception and fall back to eager by setting:
2023-09-06T16:38:36.4963173Z     import torch._dynamo
2023-09-06T16:38:36.4963493Z     torch._dynamo.config.suppress_errors = True
2023-09-06T16:38:36.4963708Z 
2023-09-06T16:38:39.1671336Z ERROR
2023-09-06T16:38:42.8037352Z 
2023-09-06T16:38:47.2928254Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:38:47.2928604Z loading model: 0it [00:04, ?it/s]
2023-09-06T16:38:47.2948448Z cuda eval  LearningToPaint                    
2023-09-06T16:39:03.9626320Z 
2023-09-06T16:39:04.0667571Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:39:04.1719216Z running benchmark:  30% 9/30 [00:00<00:00, 86.38it/s]
2023-09-06T16:39:04.2774521Z running benchmark:  63% 19/30 [00:00<00:00, 91.45it/s]
2023-09-06T16:39:04.2880138Z running benchmark:  97% 29/30 [00:00<00:00, 92.96it/s]
2023-09-06T16:39:04.2880596Z running benchmark: 100% 30/30 [00:00<00:00, 92.09it/s]
2023-09-06T16:39:04.2901120Z 1.748x
2023-09-06T16:39:09.6818559Z 
2023-09-06T16:39:12.1544403Z loading model: 0it [00:00, ?it/s]WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T16:39:12.6740019Z 
2023-09-06T16:39:12.6740579Z loading model: 0it [00:02, ?it/s]
2023-09-06T16:39:12.6742269Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T16:39:12.6743271Z cuda eval  Super_SloMo                        
2023-09-06T16:39:12.6881196Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T16:40:18.8630937Z 
2023-09-06T16:40:19.0196359Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:40:19.1735575Z running benchmark:   3% 1/30 [00:00<00:04,  6.38it/s]
2023-09-06T16:40:19.3196771Z running benchmark:  10% 3/30 [00:00<00:02, 10.24it/s]
2023-09-06T16:40:19.4659996Z running benchmark:  17% 5/30 [00:00<00:02, 11.75it/s]
2023-09-06T16:40:19.6126613Z running benchmark:  23% 7/30 [00:00<00:01, 12.50it/s]
2023-09-06T16:40:19.7597710Z running benchmark:  30% 9/30 [00:00<00:01, 12.90it/s]
2023-09-06T16:40:19.9065008Z running benchmark:  37% 11/30 [00:00<00:01, 13.14it/s]
2023-09-06T16:40:20.0528784Z running benchmark:  43% 13/30 [00:01<00:01, 13.30it/s]
2023-09-06T16:40:20.2000264Z running benchmark:  50% 15/30 [00:01<00:01, 13.41it/s]
2023-09-06T16:40:20.3464219Z running benchmark:  57% 17/30 [00:01<00:00, 13.47it/s]
2023-09-06T16:40:20.4929571Z running benchmark:  63% 19/30 [00:01<00:00, 13.53it/s]
2023-09-06T16:40:20.6398438Z running benchmark:  70% 21/30 [00:01<00:00, 13.57it/s]
2023-09-06T16:40:20.7865208Z running benchmark:  77% 23/30 [00:01<00:00, 13.58it/s]
2023-09-06T16:40:20.9330345Z running benchmark:  83% 25/30 [00:01<00:00, 13.60it/s]
2023-09-06T16:40:21.0798289Z running benchmark:  90% 27/30 [00:02<00:00, 13.61it/s]
2023-09-06T16:40:21.1537094Z running benchmark:  97% 29/30 [00:02<00:00, 13.62it/s]
2023-09-06T16:40:21.1537484Z running benchmark: 100% 30/30 [00:02<00:00, 13.10it/s]
2023-09-06T16:40:21.1555572Z 1.522x
2023-09-06T16:40:26.9845344Z 
2023-09-06T16:40:30.5423903Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:40:30.5424481Z loading model: 0it [00:03, ?it/s]
2023-09-06T16:40:30.5426405Z cuda eval  alexnet                            
2023-09-06T16:40:41.9960990Z 
2023-09-06T16:40:42.1359245Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:40:42.2659517Z running benchmark:   7% 2/30 [00:00<00:01, 14.34it/s]
2023-09-06T16:40:42.3961432Z running benchmark:  17% 5/30 [00:00<00:01, 19.29it/s]
2023-09-06T16:40:42.5266214Z running benchmark:  27% 8/30 [00:00<00:01, 20.98it/s]
2023-09-06T16:40:42.6807507Z running benchmark:  37% 11/30 [00:00<00:00, 21.77it/s]
2023-09-06T16:40:42.8110630Z running benchmark:  47% 14/30 [00:00<00:00, 20.87it/s]
2023-09-06T16:40:42.9414023Z running benchmark:  57% 17/30 [00:00<00:00, 21.55it/s]
2023-09-06T16:40:43.0719301Z running benchmark:  67% 20/30 [00:00<00:00, 22.01it/s]
2023-09-06T16:40:43.2023114Z running benchmark:  77% 23/30 [00:01<00:00, 22.32it/s]
2023-09-06T16:40:43.3323462Z running benchmark:  87% 26/30 [00:01<00:00, 22.53it/s]
2023-09-06T16:40:43.3759592Z running benchmark:  97% 29/30 [00:01<00:00, 22.69it/s]
2023-09-06T16:40:43.3760191Z running benchmark: 100% 30/30 [00:01<00:00, 21.74it/s]
2023-09-06T16:40:43.3779174Z 1.383x
2023-09-06T16:40:48.3992671Z 
2023-09-06T16:40:55.2753524Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:40:55.2753922Z loading model: 0it [00:06, ?it/s]
2023-09-06T16:40:55.2796491Z cuda eval  attention_is_all_you_need_pytorch  
2023-09-06T16:41:44.2756022Z 
2023-09-06T16:41:44.3841140Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:41:44.4856797Z running benchmark:  10% 3/30 [00:00<00:00, 27.57it/s]
2023-09-06T16:41:44.5873304Z running benchmark:  23% 7/30 [00:00<00:00, 34.29it/s]
2023-09-06T16:41:44.6912146Z running benchmark:  37% 11/30 [00:00<00:00, 36.57it/s]
2023-09-06T16:41:44.7941989Z running benchmark:  50% 15/30 [00:00<00:00, 37.34it/s]
2023-09-06T16:41:44.8961185Z running benchmark:  63% 19/30 [00:00<00:00, 37.88it/s]
2023-09-06T16:41:44.9987673Z running benchmark:  77% 23/30 [00:00<00:00, 38.34it/s]
2023-09-06T16:41:45.0755258Z running benchmark:  90% 27/30 [00:00<00:00, 38.55it/s]
2023-09-06T16:41:45.0755702Z running benchmark: 100% 30/30 [00:00<00:00, 37.48it/s]
2023-09-06T16:41:45.0778948Z 1.458x
2023-09-06T16:41:50.7613664Z 
2023-09-06T16:41:53.6365164Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:41:53.6366049Z loading model: 0it [00:02, ?it/s]
2023-09-06T16:41:53.6370852Z cuda eval  basic_gnn_edgecnn                  
2023-09-06T16:41:54.1352567Z [2023-09-06 16:41:54,134] [0/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T16:42:02.1928964Z 
2023-09-06T16:42:02.2931854Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:42:02.3291084Z running benchmark:  73% 22/30 [00:00<00:00, 219.64it/s]
2023-09-06T16:42:02.3307227Z running benchmark: 100% 30/30 [00:00<00:00, 220.19it/s]
2023-09-06T16:42:02.3307522Z 1.759x
2023-09-06T16:42:07.2169418Z 
2023-09-06T16:42:10.0809547Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:42:10.0810183Z loading model: 0it [00:02, ?it/s]
2023-09-06T16:42:10.0811000Z cuda eval  basic_gnn_gcn                      
2023-09-06T16:42:19.3545706Z ERROR:common:Backend dynamo failed in warmup()
2023-09-06T16:42:19.3546400Z Traceback (most recent call last):
2023-09-06T16:42:19.3547043Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2288, in warmup
2023-09-06T16:42:19.3548431Z     fn(model, example_inputs)
2023-09-06T16:42:19.3555060Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T16:42:19.3555497Z     return fn(*args, **kwargs)
2023-09-06T16:42:19.3555968Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:42:19.3556617Z     return mod(*inputs)
2023-09-06T16:42:19.3557333Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:42:19.3557768Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:42:19.3559402Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:42:19.3560051Z     return forward_call(*args, **kwargs)
2023-09-06T16:42:19.3560990Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/models/basic_gnn.py", line 222, in forward
2023-09-06T16:42:19.3561905Z     x = self.convs[i](x, edge_index, edge_weight=edge_weight)
2023-09-06T16:42:19.3562911Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:42:19.3563646Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:42:19.3564755Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:42:19.3565561Z     return forward_call(*args, **kwargs)
2023-09-06T16:42:19.3565942Z   File "/tmp/jenkins_pyg/tmpx2vzk66b.py", line 221, in forward
2023-09-06T16:42:19.3566323Z     edge_index, edge_weight = gcn_norm(  # yapf: disable
2023-09-06T16:42:19.3566926Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/gcn_conv.py", line 91, in gcn_norm
2023-09-06T16:42:19.3567374Z     edge_index, edge_weight = add_remaining_self_loops(
2023-09-06T16:42:19.3567957Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T16:42:19.3568549Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T16:42:19.3569636Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 634, in _convert_frame
2023-09-06T16:42:19.3570491Z     result = inner_convert(frame, cache_entry, hooks, frame_state)
2023-09-06T16:42:19.3571545Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T16:42:19.3572249Z     return fn(*args, **kwargs)
2023-09-06T16:42:19.3573241Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T16:42:19.3573650Z     return _compile(
2023-09-06T16:42:19.3574184Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T16:42:19.3574719Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T16:42:19.3575316Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:42:19.3575702Z     r = func(*args, **kwargs)
2023-09-06T16:42:19.3576242Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T16:42:19.3576682Z     out_code = transform_code_object(code, transform)
2023-09-06T16:42:19.3577331Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T16:42:19.3577818Z     transformations(instructions, code_options)
2023-09-06T16:42:19.3578385Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T16:42:19.3578761Z     tracer.run()
2023-09-06T16:42:19.3579278Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T16:42:19.3579644Z     super().run()
2023-09-06T16:42:19.3580158Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T16:42:19.3580516Z     and self.step()
2023-09-06T16:42:19.3581053Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T16:42:19.3581460Z     getattr(self, inst.opname)(inst)
2023-09-06T16:42:19.3582033Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2166, in RETURN_VALUE
2023-09-06T16:42:19.3582586Z     self.output.compile_subgraph(
2023-09-06T16:42:19.3583223Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 905, in compile_subgraph
2023-09-06T16:42:19.3583711Z     self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
2023-09-06T16:42:19.3584139Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T16:42:19.3584479Z     return func(*args, **kwds)
2023-09-06T16:42:19.3585080Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1008, in compile_and_call_fx_graph
2023-09-06T16:42:19.3585544Z     compiled_fn = self.call_user_compiler(gm)
2023-09-06T16:42:19.3586118Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:42:19.3586496Z     r = func(*args, **kwargs)
2023-09-06T16:42:19.3587072Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1075, in call_user_compiler
2023-09-06T16:42:19.3587573Z     raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
2023-09-06T16:42:19.3588219Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1060, in call_user_compiler
2023-09-06T16:42:19.3588683Z     compiled_fn = compiler_fn(gm, self.example_inputs())
2023-09-06T16:42:19.3589478Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper
2023-09-06T16:42:19.3589926Z     compiled_gm = compiler_fn(gm, example_inputs)
2023-09-06T16:42:19.3590518Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/inductor.py", line 9, in inductor
2023-09-06T16:42:19.3590931Z     return compile_fx(*args, **kwargs)
2023-09-06T16:42:19.3591480Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1167, in compile_fx
2023-09-06T16:42:19.3591867Z     return aot_autograd(
2023-09-06T16:42:19.3592611Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 55, in compiler_fn
2023-09-06T16:42:19.3593109Z     cg = aot_module_simplified(gm, example_inputs, **kwargs)
2023-09-06T16:42:19.3593738Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3891, in aot_module_simplified
2023-09-06T16:42:19.3594185Z     compiled_fn = create_aot_dispatcher_function(
2023-09-06T16:42:19.3594754Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:42:19.3595139Z     r = func(*args, **kwargs)
2023-09-06T16:42:19.3595739Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3429, in create_aot_dispatcher_function
2023-09-06T16:42:19.3596275Z     compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:42:19.3596929Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2212, in aot_wrapper_dedupe
2023-09-06T16:42:19.3597461Z     return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:42:19.3598144Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2392, in aot_wrapper_synthetic_base
2023-09-06T16:42:19.3598649Z     return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:42:19.3599280Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1573, in aot_dispatch_base
2023-09-06T16:42:19.3599719Z     compiled_fw = compiler(fw_module, flat_args)
2023-09-06T16:42:19.3631815Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:42:19.3632331Z     r = func(*args, **kwargs)
2023-09-06T16:42:19.3633608Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1109, in fw_compiler_base
2023-09-06T16:42:19.3634039Z     return inner_compile(
2023-09-06T16:42:19.3634584Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 80, in debug_wrapper
2023-09-06T16:42:19.3635036Z     inner_compiled_fn = compiler_fn(gm, example_inputs)
2023-09-06T16:42:19.3635599Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 228, in inner
2023-09-06T16:42:19.3635975Z     return fn(*args, **kwargs)
2023-09-06T16:42:19.3636328Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T16:42:19.3636658Z     return func(*args, **kwds)
2023-09-06T16:42:19.3637199Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 55, in newFunction
2023-09-06T16:42:19.3637596Z     return old_func(*args, **kwargs)
2023-09-06T16:42:19.3638162Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 348, in compile_fx_inner
2023-09-06T16:42:19.3638623Z     compiled_graph: CompiledFxGraph = fx_codegen_and_compile(
2023-09-06T16:42:19.3639246Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 574, in fx_codegen_and_compile
2023-09-06T16:42:19.3639714Z     compiled_fn = graph.compile_to_fn()
2023-09-06T16:42:19.3640278Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 991, in compile_to_fn
2023-09-06T16:42:19.3640731Z     return self.compile_to_module().call
2023-09-06T16:42:19.3641263Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:42:19.3641645Z     r = func(*args, **kwargs)
2023-09-06T16:42:19.3642189Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 946, in compile_to_module
2023-09-06T16:42:19.3642665Z     mod = PyCodeCache.load_by_key_path(key, path, linemap=linemap)
2023-09-06T16:42:19.3643534Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1161, in load_by_key_path
2023-09-06T16:42:19.3643952Z     exec(code, mod.__dict__, mod.__dict__)
2023-09-06T16:42:19.3644433Z   File "/tmp/tmpunyotejr/pl/cpl5xlkgmdmiejqifd36q6tsqkgg3u3idrs57laz56xz5wbuz7sl.py", line 131, in <module>
2023-09-06T16:42:19.3644893Z     async_compile.wait(globals())
2023-09-06T16:42:19.3645419Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1440, in wait
2023-09-06T16:42:19.3645811Z     scope[key] = result.result()
2023-09-06T16:42:19.3646355Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1299, in result
2023-09-06T16:42:19.3646739Z     self.future.result()
2023-09-06T16:42:19.3647113Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 458, in result
2023-09-06T16:42:19.3647455Z     return self.__get_result()
2023-09-06T16:42:19.3647849Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
2023-09-06T16:42:19.3648208Z     raise self._exception
2023-09-06T16:42:19.3648664Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
2023-09-06T16:42:19.3649118Z CompilationError: at 11:70:def triton_(in_ptr0, out_ptr0, xnumel, XBLOCK : tl.constexpr):
2023-09-06T16:42:19.3649470Z     xnumel = 209982
2023-09-06T16:42:19.3649739Z     xoffset = tl.program_id(0) * XBLOCK
2023-09-06T16:42:19.3650052Z     xindex = xoffset + tl.arange(0, XBLOCK)[:]
2023-09-06T16:42:19.3650333Z     xmask = xindex < xnumel
2023-09-06T16:42:19.3650578Z     x0 = xindex
2023-09-06T16:42:19.3650852Z     tmp0 = tl.load(in_ptr0 + (209982 + x0), xmask)
2023-09-06T16:42:19.3651174Z     tmp1 = tl.where(tmp0 < 0, tmp0 + 10000, tmp0)
2023-09-06T16:42:19.3651562Z     tl.device_assert(((0 <= tmp1) & (tmp1 < 10000)) | ~xmask, "index out of bounds: 0 <= tmp1 < 10000")
2023-09-06T16:42:19.3651924Z     tmp2 = 1.0
2023-09-06T16:42:19.3652380Z     tl.atomic_add(out_ptr0 + (tl.broadcast_to(tmp1, [XBLOCK])), tmp2, xmask)
2023-09-06T16:42:19.3652751Z                                                                       ^
2023-09-06T16:42:19.3653203Z ValueError('atomic_add does not support bf16')
2023-09-06T16:42:19.3653408Z 
2023-09-06T16:42:19.3653608Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T16:42:19.3653841Z 
2023-09-06T16:42:19.3653848Z 
2023-09-06T16:42:19.3654044Z You can suppress this exception and fall back to eager by setting:
2023-09-06T16:42:19.3654383Z     import torch._dynamo
2023-09-06T16:42:19.3654695Z     torch._dynamo.config.suppress_errors = True
2023-09-06T16:42:19.3654889Z 
2023-09-06T16:42:20.7801490Z ERROR
2023-09-06T16:42:24.3926570Z 
2023-09-06T16:42:27.1922886Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:42:27.1923521Z loading model: 0it [00:02, ?it/s]
2023-09-06T16:42:27.1931432Z cuda eval  basic_gnn_gin                      
2023-09-06T16:42:27.7005378Z [2023-09-06 16:42:27,699] [0/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T16:42:34.1682258Z ERROR:common:Backend dynamo failed in warmup()
2023-09-06T16:42:34.1683116Z Traceback (most recent call last):
2023-09-06T16:42:34.1683705Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2288, in warmup
2023-09-06T16:42:34.1684277Z     fn(model, example_inputs)
2023-09-06T16:42:34.1685574Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T16:42:34.1686230Z     return fn(*args, **kwargs)
2023-09-06T16:42:34.1687379Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T16:42:34.1688169Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T16:42:34.1690101Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 634, in _convert_frame
2023-09-06T16:42:34.1690943Z     result = inner_convert(frame, cache_entry, hooks, frame_state)
2023-09-06T16:42:34.1691932Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T16:42:34.1692627Z     return fn(*args, **kwargs)
2023-09-06T16:42:34.1693631Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T16:42:34.1694288Z     return _compile(
2023-09-06T16:42:34.1695231Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T16:42:34.1696030Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T16:42:34.1697090Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:42:34.1697711Z     r = func(*args, **kwargs)
2023-09-06T16:42:34.1698671Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T16:42:34.1699436Z     out_code = transform_code_object(code, transform)
2023-09-06T16:42:34.1700724Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T16:42:34.1701539Z     transformations(instructions, code_options)
2023-09-06T16:42:34.1702574Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T16:42:34.1703218Z     tracer.run()
2023-09-06T16:42:34.1704136Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T16:42:34.1704718Z     super().run()
2023-09-06T16:42:34.1705954Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T16:42:34.1706589Z     and self.step()
2023-09-06T16:42:34.1707528Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T16:42:34.1708213Z     getattr(self, inst.opname)(inst)
2023-09-06T16:42:34.1709429Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2166, in RETURN_VALUE
2023-09-06T16:42:34.1710254Z     self.output.compile_subgraph(
2023-09-06T16:42:34.1711292Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 881, in compile_subgraph
2023-09-06T16:42:34.1712112Z     self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
2023-09-06T16:42:34.1712812Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T16:42:34.1713461Z     return func(*args, **kwds)
2023-09-06T16:42:34.1714477Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1008, in compile_and_call_fx_graph
2023-09-06T16:42:34.1715316Z     compiled_fn = self.call_user_compiler(gm)
2023-09-06T16:42:34.1716270Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:42:34.1716978Z     r = func(*args, **kwargs)
2023-09-06T16:42:34.1717952Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1075, in call_user_compiler
2023-09-06T16:42:34.1718634Z     raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
2023-09-06T16:42:34.1719455Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1060, in call_user_compiler
2023-09-06T16:42:34.1720149Z     compiled_fn = compiler_fn(gm, self.example_inputs())
2023-09-06T16:42:34.1720952Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper
2023-09-06T16:42:34.1721656Z     compiled_gm = compiler_fn(gm, example_inputs)
2023-09-06T16:42:34.1722407Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/inductor.py", line 9, in inductor
2023-09-06T16:42:34.1722809Z     return compile_fx(*args, **kwargs)
2023-09-06T16:42:34.1723388Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1167, in compile_fx
2023-09-06T16:42:34.1723776Z     return aot_autograd(
2023-09-06T16:42:34.1724322Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 55, in compiler_fn
2023-09-06T16:42:34.1724762Z     cg = aot_module_simplified(gm, example_inputs, **kwargs)
2023-09-06T16:42:34.1725394Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3891, in aot_module_simplified
2023-09-06T16:42:34.1725851Z     compiled_fn = create_aot_dispatcher_function(
2023-09-06T16:42:34.1726599Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:42:34.1726979Z     r = func(*args, **kwargs)
2023-09-06T16:42:34.1727564Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3429, in create_aot_dispatcher_function
2023-09-06T16:42:34.1728096Z     compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:42:34.1728766Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2212, in aot_wrapper_dedupe
2023-09-06T16:42:34.1729275Z     return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:42:34.1730001Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2392, in aot_wrapper_synthetic_base
2023-09-06T16:42:34.1730501Z     return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:42:34.1731355Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1573, in aot_dispatch_base
2023-09-06T16:42:34.1731814Z     compiled_fw = compiler(fw_module, flat_args)
2023-09-06T16:42:34.1732381Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:42:34.1732744Z     r = func(*args, **kwargs)
2023-09-06T16:42:34.1733464Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1109, in fw_compiler_base
2023-09-06T16:42:34.1733867Z     return inner_compile(
2023-09-06T16:42:34.1734422Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 80, in debug_wrapper
2023-09-06T16:42:34.1734873Z     inner_compiled_fn = compiler_fn(gm, example_inputs)
2023-09-06T16:42:34.1735414Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 228, in inner
2023-09-06T16:42:34.1735809Z     return fn(*args, **kwargs)
2023-09-06T16:42:34.1736167Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T16:42:34.1736508Z     return func(*args, **kwds)
2023-09-06T16:42:34.1737037Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 55, in newFunction
2023-09-06T16:42:34.1737435Z     return old_func(*args, **kwargs)
2023-09-06T16:42:34.1738001Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 348, in compile_fx_inner
2023-09-06T16:42:34.1738467Z     compiled_graph: CompiledFxGraph = fx_codegen_and_compile(
2023-09-06T16:42:34.1739092Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 574, in fx_codegen_and_compile
2023-09-06T16:42:34.1739499Z     compiled_fn = graph.compile_to_fn()
2023-09-06T16:42:34.1740116Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 991, in compile_to_fn
2023-09-06T16:42:34.1740771Z     return self.compile_to_module().call
2023-09-06T16:42:34.1741335Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:42:34.1741699Z     r = func(*args, **kwargs)
2023-09-06T16:42:34.1742472Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 946, in compile_to_module
2023-09-06T16:42:34.1743149Z     mod = PyCodeCache.load_by_key_path(key, path, linemap=linemap)
2023-09-06T16:42:34.1744044Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1161, in load_by_key_path
2023-09-06T16:42:34.1744474Z     exec(code, mod.__dict__, mod.__dict__)
2023-09-06T16:42:34.1745103Z   File "/tmp/tmptmt20it8/vl/cvl6twpea3ikjocsfiuflqnu32rawonozy7bw3cehr2gdrs3s6pp.py", line 239, in <module>
2023-09-06T16:42:34.1745765Z     async_compile.wait(globals())
2023-09-06T16:42:34.1746649Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1440, in wait
2023-09-06T16:42:34.1747286Z     scope[key] = result.result()
2023-09-06T16:42:34.1748096Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1299, in result
2023-09-06T16:42:34.1748669Z     self.future.result()
2023-09-06T16:42:34.1749425Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 451, in result
2023-09-06T16:42:34.1750155Z     return self.__get_result()
2023-09-06T16:42:34.1750537Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
2023-09-06T16:42:34.1750903Z     raise self._exception
2023-09-06T16:42:34.1751425Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
2023-09-06T16:42:34.1751879Z CompilationError: at 15:53:    xindex = xoffset + tl.arange(0, XBLOCK)[:]
2023-09-06T16:42:34.1752209Z     xmask = xindex < xnumel
2023-09-06T16:42:34.1752474Z     x1 = (xindex // 64)
2023-09-06T16:42:34.1753083Z     x0 = xindex % 64
2023-09-06T16:42:34.1753544Z     tmp0 = tl.load(in_ptr0 + (200000 + x1), None, eviction_policy='evict_last')
2023-09-06T16:42:34.1754032Z     tmp2 = tl.load(in_ptr0 + (x1), None, eviction_policy='evict_last')
2023-09-06T16:42:34.1754393Z     tmp1 = tl.where(tmp0 < 0, tmp0 + 10000, tmp0)
2023-09-06T16:42:34.1754793Z     tl.device_assert((0 <= tmp1) & (tmp1 < 10000), "index out of bounds: 0 <= tmp1 < 10000")
2023-09-06T16:42:34.1755176Z     tmp3 = tl.where(tmp2 < 0, tmp2 + 10000, tmp2)
2023-09-06T16:42:34.1755569Z     tl.device_assert((0 <= tmp3) & (tmp3 < 10000), "index out of bounds: 0 <= tmp3 < 10000")
2023-09-06T16:42:34.1755961Z     tmp4 = tl.load(in_ptr1 + (x0 + (64*tmp3)), None).to(tl.float32)
2023-09-06T16:42:34.1756328Z     tl.atomic_add(out_ptr0 + (x0 + (64*tmp1)), tmp4, None)
2023-09-06T16:42:34.1756644Z                                                      ^
2023-09-06T16:42:34.1757026Z ValueError('atomic_add does not support bf16')
2023-09-06T16:42:34.1757242Z 
2023-09-06T16:42:34.1757430Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T16:42:34.1757669Z 
2023-09-06T16:42:34.1757675Z 
2023-09-06T16:42:34.1757871Z You can suppress this exception and fall back to eager by setting:
2023-09-06T16:42:34.1758212Z     import torch._dynamo
2023-09-06T16:42:34.1758524Z     torch._dynamo.config.suppress_errors = True
2023-09-06T16:42:34.1758734Z 
2023-09-06T16:42:35.4261935Z ERROR
2023-09-06T16:42:39.0052369Z 
2023-09-06T16:42:41.8166231Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:42:41.8166776Z loading model: 0it [00:02, ?it/s]
2023-09-06T16:42:41.8167306Z cuda eval  basic_gnn_sage                     
2023-09-06T16:42:42.3330449Z [2023-09-06 16:42:42,332] [0/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T16:42:48.9037531Z ERROR:common:Backend dynamo failed in warmup()
2023-09-06T16:42:48.9038694Z Traceback (most recent call last):
2023-09-06T16:42:48.9039101Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2288, in warmup
2023-09-06T16:42:48.9039473Z     fn(model, example_inputs)
2023-09-06T16:42:48.9040440Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T16:42:48.9040810Z     return fn(*args, **kwargs)
2023-09-06T16:42:48.9041365Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T16:42:48.9041815Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T16:42:48.9042420Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 634, in _convert_frame
2023-09-06T16:42:48.9042873Z     result = inner_convert(frame, cache_entry, hooks, frame_state)
2023-09-06T16:42:48.9043474Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T16:42:48.9043859Z     return fn(*args, **kwargs)
2023-09-06T16:42:48.9044430Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T16:42:48.9044825Z     return _compile(
2023-09-06T16:42:48.9045338Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T16:42:48.9045800Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T16:42:48.9046392Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:42:48.9046769Z     r = func(*args, **kwargs)
2023-09-06T16:42:48.9047300Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T16:42:48.9047743Z     out_code = transform_code_object(code, transform)
2023-09-06T16:42:48.9048924Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T16:42:48.9049404Z     transformations(instructions, code_options)
2023-09-06T16:42:48.9050085Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T16:42:48.9050460Z     tracer.run()
2023-09-06T16:42:48.9050967Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T16:42:48.9051339Z     super().run()
2023-09-06T16:42:48.9051857Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T16:42:48.9052234Z     and self.step()
2023-09-06T16:42:48.9052750Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T16:42:48.9053168Z     getattr(self, inst.opname)(inst)
2023-09-06T16:42:48.9053750Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2166, in RETURN_VALUE
2023-09-06T16:42:48.9054168Z     self.output.compile_subgraph(
2023-09-06T16:42:48.9054745Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 881, in compile_subgraph
2023-09-06T16:42:48.9055207Z     self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
2023-09-06T16:42:48.9055635Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T16:42:48.9055983Z     return func(*args, **kwds)
2023-09-06T16:42:48.9056569Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1008, in compile_and_call_fx_graph
2023-09-06T16:42:48.9057002Z     compiled_fn = self.call_user_compiler(gm)
2023-09-06T16:42:48.9057559Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:42:48.9058197Z     r = func(*args, **kwargs)
2023-09-06T16:42:48.9058769Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1075, in call_user_compiler
2023-09-06T16:42:48.9059258Z     raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
2023-09-06T16:42:48.9059881Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1060, in call_user_compiler
2023-09-06T16:42:48.9060338Z     compiled_fn = compiler_fn(gm, self.example_inputs())
2023-09-06T16:42:48.9060947Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper
2023-09-06T16:42:48.9061382Z     compiled_gm = compiler_fn(gm, example_inputs)
2023-09-06T16:42:48.9061953Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/inductor.py", line 9, in inductor
2023-09-06T16:42:48.9062363Z     return compile_fx(*args, **kwargs)
2023-09-06T16:42:48.9063130Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1167, in compile_fx
2023-09-06T16:42:48.9063515Z     return aot_autograd(
2023-09-06T16:42:48.9064061Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 55, in compiler_fn
2023-09-06T16:42:48.9064509Z     cg = aot_module_simplified(gm, example_inputs, **kwargs)
2023-09-06T16:42:48.9065138Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3891, in aot_module_simplified
2023-09-06T16:42:48.9065593Z     compiled_fn = create_aot_dispatcher_function(
2023-09-06T16:42:48.9066155Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:42:48.9066516Z     r = func(*args, **kwargs)
2023-09-06T16:42:48.9067240Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3429, in create_aot_dispatcher_function
2023-09-06T16:42:48.9067782Z     compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:42:48.9068501Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2212, in aot_wrapper_dedupe
2023-09-06T16:42:48.9069009Z     return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:42:48.9070032Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2392, in aot_wrapper_synthetic_base
2023-09-06T16:42:48.9070543Z     return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:42:48.9071194Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1573, in aot_dispatch_base
2023-09-06T16:42:48.9071641Z     compiled_fw = compiler(fw_module, flat_args)
2023-09-06T16:42:48.9072215Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:42:48.9072593Z     r = func(*args, **kwargs)
2023-09-06T16:42:48.9073142Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1109, in fw_compiler_base
2023-09-06T16:42:48.9073531Z     return inner_compile(
2023-09-06T16:42:48.9074069Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 80, in debug_wrapper
2023-09-06T16:42:48.9074499Z     inner_compiled_fn = compiler_fn(gm, example_inputs)
2023-09-06T16:42:48.9075059Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 228, in inner
2023-09-06T16:42:48.9075433Z     return fn(*args, **kwargs)
2023-09-06T16:42:48.9075784Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T16:42:48.9076124Z     return func(*args, **kwds)
2023-09-06T16:42:48.9076655Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 55, in newFunction
2023-09-06T16:42:48.9077235Z     return old_func(*args, **kwargs)
2023-09-06T16:42:48.9077801Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 348, in compile_fx_inner
2023-09-06T16:42:48.9078319Z     compiled_graph: CompiledFxGraph = fx_codegen_and_compile(
2023-09-06T16:42:48.9078937Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 574, in fx_codegen_and_compile
2023-09-06T16:42:48.9079365Z     compiled_fn = graph.compile_to_fn()
2023-09-06T16:42:48.9079922Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 991, in compile_to_fn
2023-09-06T16:42:48.9080333Z     return self.compile_to_module().call
2023-09-06T16:42:48.9080869Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:42:48.9081243Z     r = func(*args, **kwargs)
2023-09-06T16:42:48.9081805Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 946, in compile_to_module
2023-09-06T16:42:48.9082267Z     mod = PyCodeCache.load_by_key_path(key, path, linemap=linemap)
2023-09-06T16:42:48.9082893Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1161, in load_by_key_path
2023-09-06T16:42:48.9083292Z     exec(code, mod.__dict__, mod.__dict__)
2023-09-06T16:42:48.9083775Z   File "/tmp/tmpu259xzg7/2w/c2w45klm6swzml3hauo7i2lcowfxdbrhjfsbhmrodea23spqm27p.py", line 247, in <module>
2023-09-06T16:42:48.9084225Z     async_compile.wait(globals())
2023-09-06T16:42:48.9084763Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1440, in wait
2023-09-06T16:42:48.9085143Z     scope[key] = result.result()
2023-09-06T16:42:48.9085688Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1299, in result
2023-09-06T16:42:48.9086074Z     self.future.result()
2023-09-06T16:42:48.9086620Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 451, in result
2023-09-06T16:42:48.9087173Z     return self.__get_result()
2023-09-06T16:42:48.9087741Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
2023-09-06T16:42:48.9088366Z     raise self._exception
2023-09-06T16:42:48.9089132Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
2023-09-06T16:42:48.9089809Z CompilationError: at 15:53:    xindex = xoffset + tl.arange(0, XBLOCK)[:]
2023-09-06T16:42:48.9090313Z     xmask = xindex < xnumel
2023-09-06T16:42:48.9090708Z     x1 = (xindex // 64)
2023-09-06T16:42:48.9091081Z     x0 = xindex % 64
2023-09-06T16:42:48.9091793Z     tmp0 = tl.load(in_ptr0 + (200000 + x1), None, eviction_policy='evict_last')
2023-09-06T16:42:48.9092547Z     tmp2 = tl.load(in_ptr0 + (x1), None, eviction_policy='evict_last')
2023-09-06T16:42:48.9093106Z     tmp1 = tl.where(tmp0 < 0, tmp0 + 10000, tmp0)
2023-09-06T16:42:48.9093734Z     tl.device_assert((0 <= tmp1) & (tmp1 < 10000), "index out of bounds: 0 <= tmp1 < 10000")
2023-09-06T16:42:48.9094308Z     tmp3 = tl.where(tmp2 < 0, tmp2 + 10000, tmp2)
2023-09-06T16:42:48.9094878Z     tl.device_assert((0 <= tmp3) & (tmp3 < 10000), "index out of bounds: 0 <= tmp3 < 10000")
2023-09-06T16:42:48.9095497Z     tmp4 = tl.load(in_ptr1 + (x0 + (64*tmp3)), None).to(tl.float32)
2023-09-06T16:42:48.9096041Z     tl.atomic_add(out_ptr0 + (x0 + (64*tmp1)), tmp4, None)
2023-09-06T16:42:48.9096518Z                                                      ^
2023-09-06T16:42:48.9097137Z ValueError('atomic_add does not support bf16')
2023-09-06T16:42:48.9097448Z 
2023-09-06T16:42:48.9097743Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T16:42:48.9098120Z 
2023-09-06T16:42:48.9098226Z 
2023-09-06T16:42:48.9098534Z You can suppress this exception and fall back to eager by setting:
2023-09-06T16:42:48.9099029Z     import torch._dynamo
2023-09-06T16:42:48.9099946Z     torch._dynamo.config.suppress_errors = True
2023-09-06T16:42:48.9100262Z 
2023-09-06T16:42:50.0963008Z ERROR
2023-09-06T16:42:53.6913084Z 
2023-09-06T16:42:57.8657378Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:42:57.8657954Z loading model: 0it [00:04, ?it/s]
2023-09-06T16:42:57.8742846Z cuda eval  cm3leon_generate                   
2023-09-06T16:47:45.8276844Z 
2023-09-06T16:47:47.1454894Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:47:48.4552684Z running benchmark:   3% 1/30 [00:01<00:38,  1.32s/it]
2023-09-06T16:47:49.7536091Z running benchmark:   7% 2/30 [00:02<00:36,  1.31s/it]
2023-09-06T16:47:51.0683840Z running benchmark:  10% 3/30 [00:03<00:35,  1.31s/it]
2023-09-06T16:47:52.3778085Z running benchmark:  13% 4/30 [00:05<00:34,  1.31s/it]
2023-09-06T16:47:53.6793479Z running benchmark:  17% 5/30 [00:06<00:32,  1.31s/it]
2023-09-06T16:47:54.9841806Z running benchmark:  20% 6/30 [00:07<00:31,  1.31s/it]
2023-09-06T16:47:56.2759956Z running benchmark:  23% 7/30 [00:09<00:30,  1.31s/it]
2023-09-06T16:47:57.5704521Z running benchmark:  27% 8/30 [00:10<00:28,  1.30s/it]
2023-09-06T16:47:58.8551824Z running benchmark:  30% 9/30 [00:11<00:27,  1.30s/it]
2023-09-06T16:48:00.1712036Z running benchmark:  33% 10/30 [00:13<00:25,  1.29s/it]
2023-09-06T16:48:01.4800926Z running benchmark:  37% 11/30 [00:14<00:24,  1.30s/it]
2023-09-06T16:48:02.7939612Z running benchmark:  40% 12/30 [00:15<00:23,  1.30s/it]
2023-09-06T16:48:04.1024851Z running benchmark:  43% 13/30 [00:16<00:22,  1.31s/it]
2023-09-06T16:48:05.3940556Z running benchmark:  47% 14/30 [00:18<00:20,  1.31s/it]
2023-09-06T16:48:06.6852491Z running benchmark:  50% 15/30 [00:19<00:19,  1.30s/it]
2023-09-06T16:48:07.9935462Z running benchmark:  53% 16/30 [00:20<00:18,  1.30s/it]
2023-09-06T16:48:09.2942006Z running benchmark:  57% 17/30 [00:22<00:16,  1.30s/it]
2023-09-06T16:48:10.5966546Z running benchmark:  60% 18/30 [00:23<00:15,  1.30s/it]
2023-09-06T16:48:11.8950887Z running benchmark:  63% 19/30 [00:24<00:14,  1.30s/it]
2023-09-06T16:48:13.1923336Z running benchmark:  67% 20/30 [00:26<00:13,  1.30s/it]
2023-09-06T16:48:14.4958025Z running benchmark:  70% 21/30 [00:27<00:11,  1.30s/it]
2023-09-06T16:48:15.7990028Z running benchmark:  73% 22/30 [00:28<00:10,  1.30s/it]
2023-09-06T16:48:17.1095100Z running benchmark:  77% 23/30 [00:29<00:09,  1.30s/it]
2023-09-06T16:48:18.4078409Z running benchmark:  80% 24/30 [00:31<00:07,  1.30s/it]
2023-09-06T16:48:19.6995124Z running benchmark:  83% 25/30 [00:32<00:06,  1.30s/it]
2023-09-06T16:48:20.9950985Z running benchmark:  87% 26/30 [00:33<00:05,  1.30s/it]
2023-09-06T16:48:22.3030831Z running benchmark:  90% 27/30 [00:35<00:03,  1.30s/it]
2023-09-06T16:48:23.5919880Z running benchmark:  93% 28/30 [00:36<00:02,  1.30s/it]
2023-09-06T16:48:24.8990294Z running benchmark:  97% 29/30 [00:37<00:01,  1.30s/it]
2023-09-06T16:48:24.8994033Z running benchmark: 100% 30/30 [00:39<00:00,  1.30s/it]
2023-09-06T16:48:24.8994456Z running benchmark: 100% 30/30 [00:39<00:00,  1.30s/it]
2023-09-06T16:48:24.9030069Z 4.795x
2023-09-06T16:48:31.3936506Z 
2023-09-06T16:48:32.2539494Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:48:32.2540085Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:48:32.2546541Z cuda eval  dcgan                              
2023-09-06T16:48:41.8081533Z 
2023-09-06T16:48:41.9118423Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:48:42.0123600Z running benchmark:  37% 11/30 [00:00<00:00, 105.80it/s]
2023-09-06T16:48:42.0482331Z running benchmark:  83% 25/30 [00:00<00:00, 125.17it/s]
2023-09-06T16:48:42.0482779Z running benchmark: 100% 30/30 [00:00<00:00, 124.73it/s]
2023-09-06T16:48:42.0499097Z 1.144x
2023-09-06T16:48:46.9517542Z 
2023-09-06T16:48:50.6167710Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:48:50.6168488Z loading model: 0it [00:03, ?it/s]
2023-09-06T16:48:50.6168769Z Eager model failed to run
2023-09-06T16:48:50.6178421Z Traceback (most recent call last):
2023-09-06T16:48:50.6179499Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T16:48:50.6181221Z     self.model_iter_fn(model, example_inputs)
2023-09-06T16:48:50.6181944Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:48:50.6182715Z     return mod(*inputs)
2023-09-06T16:48:50.6184451Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:48:50.6185087Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:48:50.6185668Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:48:50.6186083Z     return forward_call(*args, **kwargs)
2023-09-06T16:48:50.6186522Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/__init__.py", line 31, in forward
2023-09-06T16:48:50.6186918Z     return sources, self.model(mix)
2023-09-06T16:48:50.6187518Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:48:50.6187970Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:48:50.6188531Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:48:50.6189007Z     return forward_call(*args, **kwargs)
2023-09-06T16:48:50.6189728Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/demucs/model.py", line 209, in forward
2023-09-06T16:48:50.6190322Z     x = self.lstm(x)
2023-09-06T16:48:50.6191001Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:48:50.6191427Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:48:50.6192266Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:48:50.6193038Z     return forward_call(*args, **kwargs)
2023-09-06T16:48:50.6193801Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/demucs/model.py", line 27, in forward
2023-09-06T16:48:50.6194220Z     x = self.lstm(x)[0]
2023-09-06T16:48:50.6194828Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:48:50.6195234Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:48:50.6195819Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:48:50.6196222Z     return forward_call(*args, **kwargs)
2023-09-06T16:48:50.6196778Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/rnn.py", line 879, in forward
2023-09-06T16:48:50.6197384Z     result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
2023-09-06T16:48:50.6198153Z RuntimeError: "_thnn_fused_lstm_cell_cuda" not implemented for 'BFloat16'
2023-09-06T16:48:50.6198402Z 
2023-09-06T16:48:50.6198608Z The above exception was the direct cause of the following exception:
2023-09-06T16:48:50.6198944Z 
2023-09-06T16:48:50.6199089Z Traceback (most recent call last):
2023-09-06T16:48:50.6199472Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T16:48:50.6199805Z     ) = runner.load_model(
2023-09-06T16:48:50.6200189Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T16:48:50.6200592Z     self.validate_model(model, example_inputs)
2023-09-06T16:48:50.6201006Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T16:48:50.6201440Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T16:48:50.6201828Z NotImplementedError: Eager model failed to run
2023-09-06T16:48:50.6202039Z 
2023-09-06T16:48:50.6202162Z WARNING:root:demucs failed to load
2023-09-06T16:48:55.1599643Z 
2023-09-06T16:48:57.3116070Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:48:57.3116431Z loading model: 0it [00:02, ?it/s]
2023-09-06T16:48:57.3234631Z cuda eval  densenet121                        
2023-09-06T16:49:57.1030717Z 
2023-09-06T16:49:57.2114352Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:49:57.3135673Z running benchmark:   7% 2/30 [00:00<00:01, 18.43it/s]
2023-09-06T16:49:57.4172327Z running benchmark:  17% 5/30 [00:00<00:01, 24.67it/s]
2023-09-06T16:49:57.5203098Z running benchmark:  27% 8/30 [00:00<00:00, 26.62it/s]
2023-09-06T16:49:57.6223067Z running benchmark:  37% 11/30 [00:00<00:00, 27.59it/s]
2023-09-06T16:49:57.7270379Z running benchmark:  47% 14/30 [00:00<00:00, 28.24it/s]
2023-09-06T16:49:57.8300403Z running benchmark:  57% 17/30 [00:00<00:00, 28.38it/s]
2023-09-06T16:49:57.9340140Z running benchmark:  67% 20/30 [00:00<00:00, 28.62it/s]
2023-09-06T16:49:58.0383880Z running benchmark:  77% 23/30 [00:00<00:00, 28.69it/s]
2023-09-06T16:49:58.1432541Z running benchmark:  87% 26/30 [00:00<00:00, 28.70it/s]
2023-09-06T16:49:58.1784819Z running benchmark:  97% 29/30 [00:01<00:00, 28.68it/s]
2023-09-06T16:49:58.1785526Z running benchmark: 100% 30/30 [00:01<00:00, 27.88it/s]
2023-09-06T16:49:58.1810898Z 1.726x
2023-09-06T16:50:04.1194866Z 
2023-09-06T16:50:08.7292294Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:50:08.7292832Z loading model: 0it [00:04, ?it/s]
2023-09-06T16:50:08.7293110Z Eager model failed to run
2023-09-06T16:50:08.7310054Z Traceback (most recent call last):
2023-09-06T16:50:08.7310756Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T16:50:08.7311172Z     self.model_iter_fn(model, example_inputs)
2023-09-06T16:50:08.7311595Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:50:08.7311968Z     return mod(*inputs)
2023-09-06T16:50:08.7313826Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:50:08.7314300Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:50:08.7315522Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:50:08.7315963Z     return forward_call(*args, **kwargs)
2023-09-06T16:50:08.7316532Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T16:50:08.7317038Z     return self.inference(batched_inputs)
2023-09-06T16:50:08.7317816Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T16:50:08.7318389Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T16:50:08.7319029Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:50:08.7319659Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:50:08.7320513Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:50:08.7321252Z     return forward_call(*args, **kwargs)
2023-09-06T16:50:08.7322187Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2023-09-06T16:50:08.7322615Z     box_features = self._shared_roi_transform(
2023-09-06T16:50:08.7323248Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2023-09-06T16:50:08.7323689Z     x = self.pooler(features, boxes)
2023-09-06T16:50:08.7324299Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:50:08.7324839Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:50:08.7325414Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:50:08.7325816Z     return forward_call(*args, **kwargs)
2023-09-06T16:50:08.7326696Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T16:50:08.7327139Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T16:50:08.7327721Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:50:08.7328144Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:50:08.7328763Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:50:08.7329167Z     return forward_call(*args, **kwargs)
2023-09-06T16:50:08.7329718Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T16:50:08.7330083Z     return roi_align(
2023-09-06T16:50:08.7330612Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T16:50:08.7331049Z     return torch.ops.torchvision.roi_align(
2023-09-06T16:50:08.7331607Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T16:50:08.7331978Z     return self._op(*args, **kwargs or {})
2023-09-06T16:50:08.7332448Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T16:50:08.7332691Z 
2023-09-06T16:50:08.7332888Z The above exception was the direct cause of the following exception:
2023-09-06T16:50:08.7333131Z 
2023-09-06T16:50:08.7333257Z Traceback (most recent call last):
2023-09-06T16:50:08.7333641Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T16:50:08.7333975Z     ) = runner.load_model(
2023-09-06T16:50:08.7334359Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T16:50:08.7334761Z     self.validate_model(model, example_inputs)
2023-09-06T16:50:08.7335177Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T16:50:08.7335750Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T16:50:08.7336152Z NotImplementedError: Eager model failed to run
2023-09-06T16:50:08.7336364Z 
2023-09-06T16:50:08.7336529Z WARNING:root:detectron2_fasterrcnn_r_101_c4 failed to load
2023-09-06T16:50:13.3632559Z 
2023-09-06T16:50:20.4180415Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:50:20.4180952Z loading model: 0it [00:07, ?it/s]
2023-09-06T16:50:20.4181397Z Eager model failed to run
2023-09-06T16:50:20.4199248Z Traceback (most recent call last):
2023-09-06T16:50:20.4199968Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T16:50:20.4200565Z     self.model_iter_fn(model, example_inputs)
2023-09-06T16:50:20.4201201Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:50:20.4201769Z     return mod(*inputs)
2023-09-06T16:50:20.4203966Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:50:20.4204667Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:50:20.4205540Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:50:20.4206172Z     return forward_call(*args, **kwargs)
2023-09-06T16:50:20.4207054Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T16:50:20.4207665Z     return self.inference(batched_inputs)
2023-09-06T16:50:20.4208626Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T16:50:20.4209328Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T16:50:20.4210290Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:50:20.4211618Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:50:20.4212571Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:50:20.4213202Z     return forward_call(*args, **kwargs)
2023-09-06T16:50:20.4214112Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T16:50:20.4214843Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T16:50:20.4215833Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T16:50:20.4216615Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T16:50:20.4217648Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:50:20.4218394Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:50:20.4219319Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:50:20.4219920Z     return forward_call(*args, **kwargs)
2023-09-06T16:50:20.4220786Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T16:50:20.4221461Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T16:50:20.4222364Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:50:20.4222998Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:50:20.4223847Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:50:20.4224447Z     return forward_call(*args, **kwargs)
2023-09-06T16:50:20.4225271Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T16:50:20.4225856Z     return roi_align(
2023-09-06T16:50:20.4227045Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T16:50:20.4227750Z     return torch.ops.torchvision.roi_align(
2023-09-06T16:50:20.4228723Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T16:50:20.4229542Z     return self._op(*args, **kwargs or {})
2023-09-06T16:50:20.4230266Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T16:50:20.4230627Z 
2023-09-06T16:50:20.4230926Z The above exception was the direct cause of the following exception:
2023-09-06T16:50:20.4231298Z 
2023-09-06T16:50:20.4231498Z Traceback (most recent call last):
2023-09-06T16:50:20.4232077Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T16:50:20.4232621Z     ) = runner.load_model(
2023-09-06T16:50:20.4233182Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T16:50:20.4233813Z     self.validate_model(model, example_inputs)
2023-09-06T16:50:20.4234439Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T16:50:20.4235121Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T16:50:20.4235693Z NotImplementedError: Eager model failed to run
2023-09-06T16:50:20.4236008Z 
2023-09-06T16:50:20.4236267Z WARNING:root:detectron2_fasterrcnn_r_101_dc5 failed to load
2023-09-06T16:50:24.9831456Z 
2023-09-06T16:50:29.4693651Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:50:29.4693998Z loading model: 0it [00:04, ?it/s]
2023-09-06T16:50:29.4694284Z Eager model failed to run
2023-09-06T16:50:29.4710944Z Traceback (most recent call last):
2023-09-06T16:50:29.4711453Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T16:50:29.4712111Z     self.model_iter_fn(model, example_inputs)
2023-09-06T16:50:29.4713857Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:50:29.4714230Z     return mod(*inputs)
2023-09-06T16:50:29.4715465Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:50:29.4715901Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:50:29.4716465Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:50:29.4716855Z     return forward_call(*args, **kwargs)
2023-09-06T16:50:29.4717430Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T16:50:29.4717856Z     return self.inference(batched_inputs)
2023-09-06T16:50:29.4718437Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T16:50:29.4718905Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T16:50:29.4719530Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:50:29.4719948Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:50:29.4720505Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:50:29.4720906Z     return forward_call(*args, **kwargs)
2023-09-06T16:50:29.4721478Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T16:50:29.4721937Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T16:50:29.4722571Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T16:50:29.4723192Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T16:50:29.4724036Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:50:29.4724451Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:50:29.4725017Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:50:29.4725422Z     return forward_call(*args, **kwargs)
2023-09-06T16:50:29.4725978Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2023-09-06T16:50:29.4726425Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2023-09-06T16:50:29.4727038Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:50:29.4727449Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:50:29.4727997Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:50:29.4728411Z     return forward_call(*args, **kwargs)
2023-09-06T16:50:29.4728941Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T16:50:29.4729308Z     return roi_align(
2023-09-06T16:50:29.4729832Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T16:50:29.4730268Z     return torch.ops.torchvision.roi_align(
2023-09-06T16:50:29.4730788Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T16:50:29.4731173Z     return self._op(*args, **kwargs or {})
2023-09-06T16:50:29.4731639Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T16:50:29.4731880Z 
2023-09-06T16:50:29.4732075Z The above exception was the direct cause of the following exception:
2023-09-06T16:50:29.4732317Z 
2023-09-06T16:50:29.4732441Z Traceback (most recent call last):
2023-09-06T16:50:29.4732975Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T16:50:29.4733318Z     ) = runner.load_model(
2023-09-06T16:50:29.4733690Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T16:50:29.4734089Z     self.validate_model(model, example_inputs)
2023-09-06T16:50:29.4734487Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T16:50:29.4734929Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T16:50:29.4735311Z NotImplementedError: Eager model failed to run
2023-09-06T16:50:29.4735520Z 
2023-09-06T16:50:29.4735688Z WARNING:root:detectron2_fasterrcnn_r_101_fpn failed to load
2023-09-06T16:50:34.0757967Z 
2023-09-06T16:50:38.1673870Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:50:38.1674283Z loading model: 0it [00:04, ?it/s]
2023-09-06T16:50:38.1674564Z Eager model failed to run
2023-09-06T16:50:38.1688863Z Traceback (most recent call last):
2023-09-06T16:50:38.1689766Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T16:50:38.1690537Z     self.model_iter_fn(model, example_inputs)
2023-09-06T16:50:38.1691315Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:50:38.1691997Z     return mod(*inputs)
2023-09-06T16:50:38.1693176Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:50:38.1693592Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:50:38.1694190Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:50:38.1694601Z     return forward_call(*args, **kwargs)
2023-09-06T16:50:38.1695169Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T16:50:38.1695595Z     return self.inference(batched_inputs)
2023-09-06T16:50:38.1696655Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T16:50:38.1697144Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T16:50:38.1697770Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:50:38.1698195Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:50:38.1698742Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:50:38.1699128Z     return forward_call(*args, **kwargs)
2023-09-06T16:50:38.1699709Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2023-09-06T16:50:38.1700143Z     box_features = self._shared_roi_transform(
2023-09-06T16:50:38.1700781Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2023-09-06T16:50:38.1701220Z     x = self.pooler(features, boxes)
2023-09-06T16:50:38.1701786Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:50:38.1702208Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:50:38.1702760Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:50:38.1703220Z     return forward_call(*args, **kwargs)
2023-09-06T16:50:38.1703760Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T16:50:38.1704196Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T16:50:38.1704787Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:50:38.1705228Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:50:38.1705980Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:50:38.1706379Z     return forward_call(*args, **kwargs)
2023-09-06T16:50:38.1706929Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T16:50:38.1707301Z     return roi_align(
2023-09-06T16:50:38.1707827Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T16:50:38.1708247Z     return torch.ops.torchvision.roi_align(
2023-09-06T16:50:38.1708779Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T16:50:38.1709388Z     return self._op(*args, **kwargs or {})
2023-09-06T16:50:38.1709869Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T16:50:38.1710115Z 
2023-09-06T16:50:38.1710304Z The above exception was the direct cause of the following exception:
2023-09-06T16:50:38.1710555Z 
2023-09-06T16:50:38.1710682Z Traceback (most recent call last):
2023-09-06T16:50:38.1711057Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T16:50:38.1711403Z     ) = runner.load_model(
2023-09-06T16:50:38.1711776Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T16:50:38.1712166Z     self.validate_model(model, example_inputs)
2023-09-06T16:50:38.1712580Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T16:50:38.1713017Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T16:50:38.1713444Z NotImplementedError: Eager model failed to run
2023-09-06T16:50:38.1713637Z 
2023-09-06T16:50:38.1713803Z WARNING:root:detectron2_fasterrcnn_r_50_c4 failed to load
2023-09-06T16:50:42.8514426Z 
2023-09-06T16:50:49.1887884Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:50:49.1889222Z loading model: 0it [00:06, ?it/s]
2023-09-06T16:50:49.1889725Z Eager model failed to run
2023-09-06T16:50:49.1905513Z Traceback (most recent call last):
2023-09-06T16:50:49.1906225Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T16:50:49.1906845Z     self.model_iter_fn(model, example_inputs)
2023-09-06T16:50:49.1907496Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:50:49.1908034Z     return mod(*inputs)
2023-09-06T16:50:49.1910567Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:50:49.1911218Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:50:49.1912133Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:50:49.1912782Z     return forward_call(*args, **kwargs)
2023-09-06T16:50:49.1913722Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T16:50:49.1914438Z     return self.inference(batched_inputs)
2023-09-06T16:50:49.1915371Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T16:50:49.1916104Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T16:50:49.1917085Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:50:49.1917716Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:50:49.1918576Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:50:49.1919283Z     return forward_call(*args, **kwargs)
2023-09-06T16:50:49.1920209Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T16:50:49.1921551Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T16:50:49.1922566Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T16:50:49.1923613Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T16:50:49.1924650Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:50:49.1925286Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:50:49.1926119Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:50:49.1926750Z     return forward_call(*args, **kwargs)
2023-09-06T16:50:49.1927611Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T16:50:49.1928275Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T16:50:49.1929341Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:50:49.1929960Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:50:49.1930802Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:50:49.1931419Z     return forward_call(*args, **kwargs)
2023-09-06T16:50:49.1932250Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T16:50:49.1932799Z     return roi_align(
2023-09-06T16:50:49.1933615Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T16:50:49.1934293Z     return torch.ops.torchvision.roi_align(
2023-09-06T16:50:49.1935196Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T16:50:49.1936245Z     return self._op(*args, **kwargs or {})
2023-09-06T16:50:49.1937085Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T16:50:49.1937463Z 
2023-09-06T16:50:49.1937754Z The above exception was the direct cause of the following exception:
2023-09-06T16:50:49.1938139Z 
2023-09-06T16:50:49.1938336Z Traceback (most recent call last):
2023-09-06T16:50:49.1939111Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T16:50:49.1939646Z     ) = runner.load_model(
2023-09-06T16:50:49.1940199Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T16:50:49.1940784Z     self.validate_model(model, example_inputs)
2023-09-06T16:50:49.1941511Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T16:50:49.1942327Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T16:50:49.1943037Z NotImplementedError: Eager model failed to run
2023-09-06T16:50:49.1943457Z 
2023-09-06T16:50:49.1943778Z WARNING:root:detectron2_fasterrcnn_r_50_dc5 failed to load
2023-09-06T16:50:53.8986490Z 
2023-09-06T16:50:57.9036490Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:50:57.9036847Z loading model: 0it [00:04, ?it/s]
2023-09-06T16:50:57.9037110Z Eager model failed to run
2023-09-06T16:50:57.9053081Z Traceback (most recent call last):
2023-09-06T16:50:57.9053995Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T16:50:57.9054483Z     self.model_iter_fn(model, example_inputs)
2023-09-06T16:50:57.9054906Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:50:57.9055259Z     return mod(*inputs)
2023-09-06T16:50:57.9056472Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:50:57.9056908Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:50:57.9058862Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:50:57.9059579Z     return forward_call(*args, **kwargs)
2023-09-06T16:50:57.9060640Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T16:50:57.9061070Z     return self.inference(batched_inputs)
2023-09-06T16:50:57.9061685Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T16:50:57.9062193Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T16:50:57.9063026Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:50:57.9063437Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:50:57.9064002Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:50:57.9064431Z     return forward_call(*args, **kwargs)
2023-09-06T16:50:57.9065030Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T16:50:57.9065484Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T16:50:57.9066119Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T16:50:57.9091547Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T16:50:57.9092402Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:50:57.9092903Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:50:57.9093463Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:50:57.9093864Z     return forward_call(*args, **kwargs)
2023-09-06T16:50:57.9094862Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2023-09-06T16:50:57.9095332Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2023-09-06T16:50:57.9095970Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:50:57.9096390Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:50:57.9096931Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:50:57.9097337Z     return forward_call(*args, **kwargs)
2023-09-06T16:50:57.9097887Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T16:50:57.9098994Z     return roi_align(
2023-09-06T16:50:57.9099539Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T16:50:57.9099972Z     return torch.ops.torchvision.roi_align(
2023-09-06T16:50:57.9100510Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T16:50:57.9100901Z     return self._op(*args, **kwargs or {})
2023-09-06T16:50:57.9101370Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T16:50:57.9101621Z 
2023-09-06T16:50:57.9101809Z The above exception was the direct cause of the following exception:
2023-09-06T16:50:57.9102054Z 
2023-09-06T16:50:57.9102182Z Traceback (most recent call last):
2023-09-06T16:50:57.9102562Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T16:50:57.9102962Z     ) = runner.load_model(
2023-09-06T16:50:57.9103341Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T16:50:57.9103737Z     self.validate_model(model, example_inputs)
2023-09-06T16:50:57.9104157Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T16:50:57.9104783Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T16:50:57.9105170Z NotImplementedError: Eager model failed to run
2023-09-06T16:50:57.9105386Z 
2023-09-06T16:50:57.9105541Z WARNING:root:detectron2_fasterrcnn_r_50_fpn failed to load
2023-09-06T16:51:02.5595674Z 
2023-09-06T16:51:06.5865694Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:51:06.5866105Z loading model: 0it [00:04, ?it/s]
2023-09-06T16:51:06.5888364Z Traceback (most recent call last):
2023-09-06T16:51:06.5889056Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 495, in <module>
2023-09-06T16:51:06.5890106Z     torchbench_main()
2023-09-06T16:51:06.5890874Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 491, in torchbench_main
2023-09-06T16:51:06.5891354Z     main(TorchBenchmarkRunner(), original_dir)
2023-09-06T16:51:06.5891784Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3031, in main
2023-09-06T16:51:06.5894440Z     process_entry(0, runner, original_dir, args)
2023-09-06T16:51:06.5894860Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2988, in process_entry
2023-09-06T16:51:06.5898543Z     return maybe_fresh_cache(
2023-09-06T16:51:06.5899184Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1659, in inner
2023-09-06T16:51:06.5900799Z     return fn(*args, **kwargs)
2023-09-06T16:51:06.5901175Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3477, in run
2023-09-06T16:51:06.5905816Z     assert marked, f"nothing in example_inputs had a dim with {batch_size}"
2023-09-06T16:51:06.5906243Z AssertionError: nothing in example_inputs had a dim with 4
2023-09-06T16:51:07.7014580Z ERROR
2023-09-06T16:51:07.7062213Z speedup             gmean=0.00x mean=1.282x
2023-09-06T16:51:07.7062666Z abs_latency         gmean=0.00x mean=21.192x
2023-09-06T16:51:07.7066124Z compilation_latency mean=37.392 seconds
2023-09-06T16:51:07.7067345Z compression_ratio   mean=0.485x
2023-09-06T16:51:07.7067970Z eager_peak_mem      gmean=0.00x mean=0.667x
2023-09-06T16:51:07.7069408Z dynamo_peak_mem     gmean=0.00x mean=0.908x
2023-09-06T16:51:07.7072169Z calls_captured      gmean=0.00x mean=359.067x
2023-09-06T16:51:07.7074793Z unique_graphs       gmean=0.00x mean=1.200x
2023-09-06T16:51:07.7076150Z graph_breaks        gmean=0.00x mean=0.400x
2023-09-06T16:51:07.7078491Z unique_graph_breaks gmean=0.00x mean=0.133x
2023-09-06T16:51:08.4667349Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *cppwrapper-true* ]]
2023-09-06T16:51:08.4669755Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *freezing_cudagraphs-true* ]]
2023-09-06T16:51:08.4670531Z + [[ inference == \i\n\f\e\r\e\n\c\e ]]
2023-09-06T16:51:08.4671689Z + python benchmarks/dynamo/torchbench.py --performance --cold-start-latency --inference --bfloat16 --backend inductor --device cuda --total-partitions 4 --partition-id 0 --freezing --output /var/lib/jenkins/workspace/test/test-reports/inductor_with_cudagraphs_freezing_torchbench_bfloat16_inference_cuda_performance.csv
2023-09-06T16:51:15.6569600Z 
2023-09-06T16:51:16.5863470Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:51:16.5863852Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:51:16.5864139Z Eager model failed to run
2023-09-06T16:51:16.5878996Z Traceback (most recent call last):
2023-09-06T16:51:16.5879671Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T16:51:16.5880410Z     self.model_iter_fn(model, example_inputs)
2023-09-06T16:51:16.5881144Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:51:16.5881809Z     return mod(*inputs)
2023-09-06T16:51:16.5884175Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:51:16.5885179Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:51:16.5885766Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:51:16.5886181Z     return forward_call(*args, **kwargs)
2023-09-06T16:51:16.5886735Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 893, in forward
2023-09-06T16:51:16.5887189Z     logits = self.model(batch.dense_features, batch.sparse_features)
2023-09-06T16:51:16.5887815Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:51:16.5888267Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:51:16.5888927Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:51:16.5889610Z     return forward_call(*args, **kwargs)
2023-09-06T16:51:16.5890587Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 570, in forward
2023-09-06T16:51:16.5891438Z     embedded_dense = self.dense_arch(dense_features)
2023-09-06T16:51:16.5892053Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:51:16.5892480Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:51:16.5893040Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:51:16.5893434Z     return forward_call(*args, **kwargs)
2023-09-06T16:51:16.5893992Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 149, in forward
2023-09-06T16:51:16.5894386Z     return self.model(features)
2023-09-06T16:51:16.5894947Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:51:16.5895358Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:51:16.5896160Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:51:16.5896575Z     return forward_call(*args, **kwargs)
2023-09-06T16:51:16.5897121Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/mlp.py", line 172, in forward
2023-09-06T16:51:16.5897486Z     return self._mlp(input)
2023-09-06T16:51:16.5898040Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:51:16.5898453Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:51:16.5899066Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:51:16.5899471Z     return forward_call(*args, **kwargs)
2023-09-06T16:51:16.5900004Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/container.py", line 215, in forward
2023-09-06T16:51:16.5900398Z     input = module(input)
2023-09-06T16:51:16.5900950Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:51:16.5901366Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:51:16.5901898Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:51:16.5902330Z     return forward_call(*args, **kwargs)
2023-09-06T16:51:16.5902863Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/mlp.py", line 73, in forward
2023-09-06T16:51:16.5903289Z     return self._activation_fn(self._linear(input))
2023-09-06T16:51:16.5903900Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:51:16.5904302Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:51:16.5904859Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:51:16.5905403Z     return forward_call(*args, **kwargs)
2023-09-06T16:51:16.5905946Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
2023-09-06T16:51:16.5906384Z     return F.linear(input, self.weight, self.bias)
2023-09-06T16:51:16.5906782Z RuntimeError: mat1 and mat2 must have the same dtype, but got Float and BFloat16
2023-09-06T16:51:16.5907047Z 
2023-09-06T16:51:16.5907243Z The above exception was the direct cause of the following exception:
2023-09-06T16:51:16.5907483Z 
2023-09-06T16:51:16.5907611Z Traceback (most recent call last):
2023-09-06T16:51:16.5907987Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T16:51:16.5908319Z     ) = runner.load_model(
2023-09-06T16:51:16.5908696Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T16:51:16.5909332Z     self.validate_model(model, example_inputs)
2023-09-06T16:51:16.5909773Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T16:51:16.5910217Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T16:51:16.5910586Z NotImplementedError: Eager model failed to run
2023-09-06T16:51:16.5910794Z 
2023-09-06T16:51:16.5910932Z WARNING:root:torchrec_dlrm failed to load
2023-09-06T16:51:21.0633686Z 
2023-09-06T16:51:24.2842712Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:51:24.2843077Z loading model: 0it [00:03, ?it/s]
2023-09-06T16:51:24.2893630Z cuda eval  BERT_pytorch                       
2023-09-06T16:52:02.7900607Z 
2023-09-06T16:52:02.8968611Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:52:03.0076696Z running benchmark:  17% 5/30 [00:00<00:00, 46.73it/s]
2023-09-06T16:52:03.1185508Z running benchmark:  37% 11/30 [00:00<00:00, 51.09it/s]
2023-09-06T16:52:03.2296838Z running benchmark:  57% 17/30 [00:00<00:00, 52.48it/s]
2023-09-06T16:52:03.3407818Z running benchmark:  77% 23/30 [00:00<00:00, 53.08it/s]
2023-09-06T16:52:03.3594413Z running benchmark:  97% 29/30 [00:00<00:00, 53.42it/s]
2023-09-06T16:52:03.3594804Z running benchmark: 100% 30/30 [00:00<00:00, 52.64it/s]
2023-09-06T16:52:03.3616370Z 2.220x
2023-09-06T16:52:08.5157342Z 
2023-09-06T16:52:12.2287624Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:52:12.2288376Z loading model: 0it [00:03, ?it/s]
2023-09-06T16:52:12.2341957Z cuda eval  Background_Matting                 
2023-09-06T16:52:42.7081879Z 
2023-09-06T16:52:42.8100651Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:52:42.9121718Z running benchmark:  10% 3/30 [00:00<00:00, 29.39it/s]
2023-09-06T16:52:43.0144304Z running benchmark:  23% 7/30 [00:00<00:00, 35.10it/s]
2023-09-06T16:52:43.1164209Z running benchmark:  37% 11/30 [00:00<00:00, 36.94it/s]
2023-09-06T16:52:43.2182633Z running benchmark:  50% 15/30 [00:00<00:00, 37.83it/s]
2023-09-06T16:52:43.3205279Z running benchmark:  63% 19/30 [00:00<00:00, 38.34it/s]
2023-09-06T16:52:43.4226790Z running benchmark:  77% 23/30 [00:00<00:00, 38.62it/s]
2023-09-06T16:52:43.4994528Z running benchmark:  90% 27/30 [00:00<00:00, 38.80it/s]
2023-09-06T16:52:43.4998048Z running benchmark: 100% 30/30 [00:00<00:00, 37.89it/s]
2023-09-06T16:52:43.5018056Z 1.722x
2023-09-06T16:52:48.8737130Z 
2023-09-06T16:52:58.5686995Z loading model: 0it [00:00, ?it/s]WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T16:53:00.0640442Z 
2023-09-06T16:53:00.2236264Z loading model: 0it [00:11, ?it/s]
2023-09-06T16:53:00.2236741Z WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T16:53:00.2237155Z cuda eval  DALLE2_pytorch                     
2023-09-06T16:53:00.2237557Z WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T16:53:13.3510302Z [2023-09-06 16:53:13,349] [2/1] torch._dynamo.output_graph: [WARNING] nn.Module forward/_pre hooks are only partially supported, and were detected in your model. In particular, if you do not change/remove hooks after calling .compile(), you can disregard this warning, and otherwise you may need to set torch._dynamo.config.skip_nnmodule_hook_guards=False to ensure recompiling after changing hooks.See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations. 
2023-09-06T16:53:49.9324894Z skipping cudagraphs due to multiple devices
2023-09-06T16:54:59.0816974Z skipping cudagraphs due to multiple devices
2023-09-06T16:55:28.3344042Z ERROR:common:Backend dynamo failed in warmup()
2023-09-06T16:55:28.3344668Z Traceback (most recent call last):
2023-09-06T16:55:28.3345409Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2288, in warmup
2023-09-06T16:55:28.3346121Z     fn(model, example_inputs)
2023-09-06T16:55:28.3347898Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T16:55:28.3348619Z     return fn(*args, **kwargs)
2023-09-06T16:55:28.3349642Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:55:28.3350349Z     return mod(*inputs)
2023-09-06T16:55:28.3351408Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:55:28.3352184Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:55:28.3353193Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:55:28.3353969Z     return forward_call(*args, **kwargs)
2023-09-06T16:55:28.3354979Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T16:55:28.3355737Z     return func(*args, **kwargs)
2023-09-06T16:55:28.3356717Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 95, in inner
2023-09-06T16:55:28.3357396Z     model.eval()
2023-09-06T16:55:28.3358880Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 96, in <resume in inner>
2023-09-06T16:55:28.3359667Z     out = fn(model, *args, **kwargs)
2023-09-06T16:55:28.3360608Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3329, in forward
2023-09-06T16:55:28.3361472Z     image_embed = self.prior.sample(text, num_samples_per_batch = self.prior_num_samples, cond_scale = prior_cond_scale)
2023-09-06T16:55:28.3362782Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3332, in <resume in forward>
2023-09-06T16:55:28.3363852Z     images = self.decoder.sample(image_embed = image_embed, text = text_cond, cond_scale = cond_scale)
2023-09-06T16:55:28.3365068Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T16:55:28.3365745Z     return func(*args, **kwargs)
2023-09-06T16:55:28.3366766Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 95, in inner
2023-09-06T16:55:28.3367387Z     model.eval()
2023-09-06T16:55:28.3368370Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 96, in <resume in inner>
2023-09-06T16:55:28.3369030Z     out = fn(model, *args, **kwargs)
2023-09-06T16:55:28.3370026Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3199, in sample
2023-09-06T16:55:28.3370710Z     img = self.p_sample_loop(
2023-09-06T16:55:28.3371665Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T16:55:28.3372488Z     return func(*args, **kwargs)
2023-09-06T16:55:28.3373476Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 3028, in p_sample_loop
2023-09-06T16:55:28.3374471Z     return self.p_sample_loop_ddpm(*args, noise_scheduler = noise_scheduler, **kwargs)
2023-09-06T16:55:28.3376058Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T16:55:28.3376764Z     return func(*args, **kwargs)
2023-09-06T16:55:28.3377805Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2885, in p_sample_loop_ddpm
2023-09-06T16:55:28.3378535Z     img, x_start = self.p_sample(
2023-09-06T16:55:28.3379506Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T16:55:28.3380197Z     return func(*args, **kwargs)
2023-09-06T16:55:28.3381163Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2822, in p_sample
2023-09-06T16:55:28.3382944Z     model_mean, _, model_log_variance, x_start = self.p_mean_variance(unet, x = x, t = t, image_embed = image_embed, text_encodings = text_encodings, cond_scale = cond_scale, lowres_cond_img = lowres_cond_img, self_cond = self_cond, clip_denoised = clip_denoised, predict_x_start = predict_x_start, predict_v = predict_v, noise_scheduler = noise_scheduler, learned_variance = learned_variance, lowres_noise_level = lowres_noise_level)
2023-09-06T16:55:28.3385038Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2787, in p_mean_variance
2023-09-06T16:55:28.3386392Z     model_output = default(model_output, lambda: unet.forward_with_cond_scale(x, t, image_embed = image_embed, text_encodings = text_encodings, cond_scale = cond_scale, lowres_cond_img = lowres_cond_img, self_cond = self_cond, lowres_noise_level = lowres_noise_level))
2023-09-06T16:55:28.3387967Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 65, in default
2023-09-06T16:55:28.3388662Z     return d() if callable(d) else d
2023-09-06T16:55:28.3390217Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2787, in <lambda>
2023-09-06T16:55:28.3391601Z     model_output = default(model_output, lambda: unet.forward_with_cond_scale(x, t, image_embed = image_embed, text_encodings = text_encodings, cond_scale = cond_scale, lowres_cond_img = lowres_cond_img, self_cond = self_cond, lowres_noise_level = lowres_noise_level))
2023-09-06T16:55:28.3393262Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2159, in forward_with_cond_scale
2023-09-06T16:55:28.3394148Z     logits = self.forward(*args, **kwargs)
2023-09-06T16:55:28.3395147Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 2340, in forward
2023-09-06T16:55:28.3395812Z     x = resnet_block(x, t, c)
2023-09-06T16:55:28.3396820Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:55:28.3397563Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:55:28.3398589Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:55:28.3399287Z     return forward_call(*args, **kwargs)
2023-09-06T16:55:28.3400355Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T16:55:28.3401164Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T16:55:28.3402252Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 634, in _convert_frame
2023-09-06T16:55:28.3403039Z     result = inner_convert(frame, cache_entry, hooks, frame_state)
2023-09-06T16:55:28.3404184Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T16:55:28.3404854Z     return fn(*args, **kwargs)
2023-09-06T16:55:28.3405889Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T16:55:28.3406909Z     return _compile(
2023-09-06T16:55:28.3407868Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T16:55:28.3408673Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T16:55:28.3409742Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:55:28.3410486Z     r = func(*args, **kwargs)
2023-09-06T16:55:28.3411513Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T16:55:28.3412272Z     out_code = transform_code_object(code, transform)
2023-09-06T16:55:28.3413301Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T16:55:28.3414207Z     transformations(instructions, code_options)
2023-09-06T16:55:28.3415230Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T16:55:28.3415893Z     tracer.run()
2023-09-06T16:55:28.3416789Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T16:55:28.3417434Z     super().run()
2023-09-06T16:55:28.3418343Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T16:55:28.3418962Z     and self.step()
2023-09-06T16:55:28.3419903Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T16:55:28.3420590Z     getattr(self, inst.opname)(inst)
2023-09-06T16:55:28.3421585Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T16:55:28.3422243Z     return inner_fn(self, inst)
2023-09-06T16:55:28.3423611Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1119, in CALL_FUNCTION
2023-09-06T16:55:28.3424355Z     self.call_function(fn, args, {})
2023-09-06T16:55:28.3425378Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 565, in call_function
2023-09-06T16:55:28.3426138Z     self.push(fn.call_function(self, args, kwargs))
2023-09-06T16:55:28.3427183Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/torch.py", line 731, in call_function
2023-09-06T16:55:28.3427915Z     tensor_variable = wrap_fx_proxy(
2023-09-06T16:55:28.3428942Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1216, in wrap_fx_proxy
2023-09-06T16:55:28.3429829Z     return wrap_fx_proxy_cls(
2023-09-06T16:55:28.3430853Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1303, in wrap_fx_proxy_cls
2023-09-06T16:55:28.3431647Z     example_value = get_fake_value(proxy.node, tx)
2023-09-06T16:55:28.3432675Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1381, in get_fake_value
2023-09-06T16:55:28.3433461Z     raise TorchRuntimeError(str(e)).with_traceback(e.__traceback__) from None
2023-09-06T16:55:28.3434617Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1342, in get_fake_value
2023-09-06T16:55:28.3435285Z     return wrap_fake_exception(
2023-09-06T16:55:28.3436270Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 917, in wrap_fake_exception
2023-09-06T16:55:28.3436925Z     return fn()
2023-09-06T16:55:28.3437800Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1343, in <lambda>
2023-09-06T16:55:28.3438565Z     lambda: run_node(tx.output, node, args, kwargs, nnmodule)
2023-09-06T16:55:28.3439544Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1415, in run_node
2023-09-06T16:55:28.3440699Z     raise RuntimeError(fn_str + str(e)).with_traceback(e.__traceback__) from e
2023-09-06T16:55:28.3441781Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1402, in run_node
2023-09-06T16:55:28.3442444Z     return node.target(*args, **kwargs)
2023-09-06T16:55:28.3443350Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/einops/einops.py", line 483, in rearrange
2023-09-06T16:55:28.3444504Z     return reduce(cast(Tensor, tensor), pattern, reduction='rearrange', **axes_lengths)
2023-09-06T16:55:28.3445506Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/einops/einops.py", line 412, in reduce
2023-09-06T16:55:28.3446273Z     return _apply_recipe(recipe, tensor, reduction_type=reduction)
2023-09-06T16:55:28.3447295Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/einops/einops.py", line 235, in _apply_recipe
2023-09-06T16:55:28.3448052Z     _reconstruct_from_shape(recipe, backend.shape(tensor))
2023-09-06T16:55:28.3449451Z torch._dynamo.exc.TorchRuntimeError: Failed running call_function <function rearrange at 0x7f1e49791fc0>(*(FakeTensor(..., device='cuda:0', size=(1, 128, s0, s1)), 'b c ... -> b ... c'), **{}):
2023-09-06T16:55:28.3450427Z unhashable type: 'SymInt'
2023-09-06T16:55:28.3450712Z 
2023-09-06T16:55:28.3450856Z from user code:
2023-09-06T16:55:28.3451774Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 1670, in forward
2023-09-06T16:55:28.3452610Z     h = rearrange(h, 'b c ... -> b ... c')
2023-09-06T16:55:28.3452930Z 
2023-09-06T16:55:28.3453267Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T16:55:28.3453770Z 
2023-09-06T16:55:28.3453779Z 
2023-09-06T16:55:28.3454108Z You can suppress this exception and fall back to eager by setting:
2023-09-06T16:55:28.3454670Z     import torch._dynamo
2023-09-06T16:55:28.3455197Z     torch._dynamo.config.suppress_errors = True
2023-09-06T16:55:28.3455569Z 
2023-09-06T16:55:31.0910848Z ERROR
2023-09-06T16:55:34.7503701Z 
2023-09-06T16:55:39.2388240Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:55:39.2388612Z loading model: 0it [00:04, ?it/s]
2023-09-06T16:55:39.2411172Z cuda eval  LearningToPaint                    
2023-09-06T16:55:58.5526018Z 
2023-09-06T16:55:58.6551774Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:55:58.7575012Z running benchmark:  30% 9/30 [00:00<00:00, 87.75it/s]
2023-09-06T16:55:58.8617658Z running benchmark:  63% 19/30 [00:00<00:00, 93.42it/s]
2023-09-06T16:55:58.8723953Z running benchmark:  97% 29/30 [00:00<00:00, 94.59it/s]
2023-09-06T16:55:58.8728583Z running benchmark: 100% 30/30 [00:00<00:00, 93.68it/s]
2023-09-06T16:55:58.8746892Z 1.851x
2023-09-06T16:56:04.1797235Z 
2023-09-06T16:56:06.6966612Z loading model: 0it [00:00, ?it/s]WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T16:56:07.1781818Z 
2023-09-06T16:56:07.1782414Z loading model: 0it [00:02, ?it/s]
2023-09-06T16:56:07.1782918Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T16:56:07.1783309Z cuda eval  Super_SloMo                        
2023-09-06T16:56:07.1899632Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T16:57:02.0414276Z 
2023-09-06T16:57:02.1953779Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:57:02.3401442Z running benchmark:   3% 1/30 [00:00<00:04,  6.48it/s]
2023-09-06T16:57:02.4851153Z running benchmark:  10% 3/30 [00:00<00:02, 10.68it/s]
2023-09-06T16:57:02.6302738Z running benchmark:  17% 5/30 [00:00<00:02, 12.08it/s]
2023-09-06T16:57:02.7752213Z running benchmark:  23% 7/30 [00:00<00:01, 12.75it/s]
2023-09-06T16:57:02.9201077Z running benchmark:  30% 9/30 [00:00<00:01, 13.12it/s]
2023-09-06T16:57:03.0651600Z running benchmark:  37% 11/30 [00:00<00:01, 13.35it/s]
2023-09-06T16:57:03.2100324Z running benchmark:  43% 13/30 [00:01<00:01, 13.50it/s]
2023-09-06T16:57:03.3548754Z running benchmark:  50% 15/30 [00:01<00:01, 13.59it/s]
2023-09-06T16:57:03.4998158Z running benchmark:  57% 17/30 [00:01<00:00, 13.66it/s]
2023-09-06T16:57:03.6450370Z running benchmark:  63% 19/30 [00:01<00:00, 13.70it/s]
2023-09-06T16:57:03.7903467Z running benchmark:  70% 21/30 [00:01<00:00, 13.73it/s]
2023-09-06T16:57:03.9353416Z running benchmark:  77% 23/30 [00:01<00:00, 13.74it/s]
2023-09-06T16:57:04.0803142Z running benchmark:  83% 25/30 [00:01<00:00, 13.75it/s]
2023-09-06T16:57:04.2250661Z running benchmark:  90% 27/30 [00:02<00:00, 13.77it/s]
2023-09-06T16:57:04.2976258Z running benchmark:  97% 29/30 [00:02<00:00, 13.78it/s]
2023-09-06T16:57:04.2976889Z running benchmark: 100% 30/30 [00:02<00:00, 13.29it/s]
2023-09-06T16:57:04.2998544Z 1.570x
2023-09-06T16:57:09.9715740Z 
2023-09-06T16:57:13.5008315Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:57:13.5008669Z loading model: 0it [00:03, ?it/s]
2023-09-06T16:57:13.5011681Z cuda eval  alexnet                            
2023-09-06T16:57:25.7400825Z 
2023-09-06T16:57:25.8799253Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:57:26.0100896Z running benchmark:   7% 2/30 [00:00<00:01, 14.29it/s]
2023-09-06T16:57:26.1403006Z running benchmark:  17% 5/30 [00:00<00:01, 19.29it/s]
2023-09-06T16:57:26.2857818Z running benchmark:  27% 8/30 [00:00<00:01, 20.96it/s]
2023-09-06T16:57:26.4159334Z running benchmark:  37% 11/30 [00:00<00:00, 20.83it/s]
2023-09-06T16:57:26.5462353Z running benchmark:  47% 14/30 [00:00<00:00, 21.60it/s]
2023-09-06T16:57:26.6766085Z running benchmark:  57% 17/30 [00:00<00:00, 22.07it/s]
2023-09-06T16:57:26.8066927Z running benchmark:  67% 20/30 [00:00<00:00, 22.38it/s]
2023-09-06T16:57:26.9368878Z running benchmark:  77% 23/30 [00:01<00:00, 22.59it/s]
2023-09-06T16:57:27.0671828Z running benchmark:  87% 26/30 [00:01<00:00, 22.73it/s]
2023-09-06T16:57:27.1108374Z running benchmark:  97% 29/30 [00:01<00:00, 22.82it/s]
2023-09-06T16:57:27.1112865Z running benchmark: 100% 30/30 [00:01<00:00, 21.88it/s]
2023-09-06T16:57:27.1126154Z 1.387x
2023-09-06T16:57:32.0853959Z 
2023-09-06T16:57:38.9632366Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:57:38.9632707Z loading model: 0it [00:06, ?it/s]
2023-09-06T16:57:38.9676133Z cuda eval  attention_is_all_you_need_pytorch  
2023-09-06T16:58:21.6060328Z 
2023-09-06T16:58:21.7275321Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:58:21.8473190Z running benchmark:  13% 4/30 [00:00<00:00, 32.83it/s]
2023-09-06T16:58:21.9661444Z running benchmark:  30% 9/30 [00:00<00:00, 38.02it/s]
2023-09-06T16:58:22.0845549Z running benchmark:  47% 14/30 [00:00<00:00, 39.85it/s]
2023-09-06T16:58:22.2051014Z running benchmark:  63% 19/30 [00:00<00:00, 40.79it/s]
2023-09-06T16:58:22.3241450Z running benchmark:  80% 24/30 [00:00<00:00, 41.03it/s]
2023-09-06T16:58:22.3478677Z running benchmark:  97% 29/30 [00:00<00:00, 41.36it/s]
2023-09-06T16:58:22.3479318Z running benchmark: 100% 30/30 [00:00<00:00, 40.41it/s]
2023-09-06T16:58:22.3502626Z 1.693x
2023-09-06T16:58:27.8374492Z 
2023-09-06T16:58:30.6677343Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:58:30.6677696Z loading model: 0it [00:02, ?it/s]
2023-09-06T16:58:30.6683835Z cuda eval  basic_gnn_edgecnn                  
2023-09-06T16:58:31.1975801Z [2023-09-06 16:58:31,196] [0/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T16:58:39.8146877Z 
2023-09-06T16:58:39.9179477Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T16:58:39.9815179Z running benchmark:  53% 16/30 [00:00<00:00, 155.67it/s]
2023-09-06T16:58:39.9815603Z running benchmark: 100% 30/30 [00:00<00:00, 179.71it/s]
2023-09-06T16:58:39.9832789Z 1.703x
2023-09-06T16:58:44.8370915Z 
2023-09-06T16:58:47.6160023Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:58:47.6160527Z loading model: 0it [00:02, ?it/s]
2023-09-06T16:58:47.6169792Z cuda eval  basic_gnn_gcn                      
2023-09-06T16:58:56.9579735Z ERROR:common:Backend dynamo failed in warmup()
2023-09-06T16:58:56.9580216Z Traceback (most recent call last):
2023-09-06T16:58:56.9580719Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2288, in warmup
2023-09-06T16:58:56.9581544Z     fn(model, example_inputs)
2023-09-06T16:58:56.9583763Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T16:58:56.9590406Z     return fn(*args, **kwargs)
2023-09-06T16:58:56.9591180Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T16:58:56.9591918Z     return mod(*inputs)
2023-09-06T16:58:56.9593292Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:58:56.9593969Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:58:56.9595165Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:58:56.9595911Z     return forward_call(*args, **kwargs)
2023-09-06T16:58:56.9596926Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/models/basic_gnn.py", line 222, in forward
2023-09-06T16:58:56.9597400Z     x = self.convs[i](x, edge_index, edge_weight=edge_weight)
2023-09-06T16:58:56.9598011Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T16:58:56.9598429Z     return self._call_impl(*args, **kwargs)
2023-09-06T16:58:56.9598974Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T16:58:56.9599586Z     return forward_call(*args, **kwargs)
2023-09-06T16:58:56.9600244Z   File "/tmp/jenkins_pyg/tmpx2ej0v0t.py", line 221, in forward
2023-09-06T16:58:56.9600937Z     edge_index, edge_weight = gcn_norm(  # yapf: disable
2023-09-06T16:58:56.9602467Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/gcn_conv.py", line 91, in gcn_norm
2023-09-06T16:58:56.9603575Z     edge_index, edge_weight = add_remaining_self_loops(
2023-09-06T16:58:56.9604253Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T16:58:56.9604705Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T16:58:56.9605306Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 634, in _convert_frame
2023-09-06T16:58:56.9605772Z     result = inner_convert(frame, cache_entry, hooks, frame_state)
2023-09-06T16:58:56.9606341Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T16:58:56.9606723Z     return fn(*args, **kwargs)
2023-09-06T16:58:56.9607297Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T16:58:56.9607698Z     return _compile(
2023-09-06T16:58:56.9608210Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T16:58:56.9608670Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T16:58:56.9609261Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:58:56.9609637Z     r = func(*args, **kwargs)
2023-09-06T16:58:56.9610183Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T16:58:56.9610606Z     out_code = transform_code_object(code, transform)
2023-09-06T16:58:56.9611246Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T16:58:56.9611711Z     transformations(instructions, code_options)
2023-09-06T16:58:56.9612541Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T16:58:56.9612954Z     tracer.run()
2023-09-06T16:58:56.9613527Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T16:58:56.9613897Z     super().run()
2023-09-06T16:58:56.9614414Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T16:58:56.9614785Z     and self.step()
2023-09-06T16:58:56.9615292Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T16:58:56.9615689Z     getattr(self, inst.opname)(inst)
2023-09-06T16:58:56.9616261Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2166, in RETURN_VALUE
2023-09-06T16:58:56.9616674Z     self.output.compile_subgraph(
2023-09-06T16:58:56.9617243Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 905, in compile_subgraph
2023-09-06T16:58:56.9617726Z     self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
2023-09-06T16:58:56.9618150Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T16:58:56.9618493Z     return func(*args, **kwds)
2023-09-06T16:58:56.9619078Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1008, in compile_and_call_fx_graph
2023-09-06T16:58:56.9619505Z     compiled_fn = self.call_user_compiler(gm)
2023-09-06T16:58:56.9620066Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:58:56.9620441Z     r = func(*args, **kwargs)
2023-09-06T16:58:56.9620995Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1075, in call_user_compiler
2023-09-06T16:58:56.9621589Z     raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
2023-09-06T16:58:56.9622241Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1060, in call_user_compiler
2023-09-06T16:58:56.9622725Z     compiled_fn = compiler_fn(gm, self.example_inputs())
2023-09-06T16:58:56.9623362Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper
2023-09-06T16:58:56.9623800Z     compiled_gm = compiler_fn(gm, example_inputs)
2023-09-06T16:58:56.9624366Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/inductor.py", line 9, in inductor
2023-09-06T16:58:56.9624772Z     return compile_fx(*args, **kwargs)
2023-09-06T16:58:56.9625329Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1167, in compile_fx
2023-09-06T16:58:56.9625711Z     return aot_autograd(
2023-09-06T16:58:56.9626247Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 55, in compiler_fn
2023-09-06T16:58:56.9626712Z     cg = aot_module_simplified(gm, example_inputs, **kwargs)
2023-09-06T16:58:56.9627341Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3891, in aot_module_simplified
2023-09-06T16:58:56.9627797Z     compiled_fn = create_aot_dispatcher_function(
2023-09-06T16:58:56.9628360Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:58:56.9628723Z     r = func(*args, **kwargs)
2023-09-06T16:58:56.9629644Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3429, in create_aot_dispatcher_function
2023-09-06T16:58:56.9630193Z     compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:58:56.9630873Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2212, in aot_wrapper_dedupe
2023-09-06T16:58:56.9631550Z     return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:58:56.9632233Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2392, in aot_wrapper_synthetic_base
2023-09-06T16:58:56.9632740Z     return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:58:56.9633444Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1573, in aot_dispatch_base
2023-09-06T16:58:56.9633888Z     compiled_fw = compiler(fw_module, flat_args)
2023-09-06T16:58:56.9634470Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 938, in fw_compiler_freezing
2023-09-06T16:58:56.9634893Z     optimized_function = inner_compile(
2023-09-06T16:58:56.9635461Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 80, in debug_wrapper
2023-09-06T16:58:56.9635920Z     inner_compiled_fn = compiler_fn(gm, example_inputs)
2023-09-06T16:58:56.9636487Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 228, in inner
2023-09-06T16:58:56.9636849Z     return fn(*args, **kwargs)
2023-09-06T16:58:56.9637208Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T16:58:56.9637555Z     return func(*args, **kwds)
2023-09-06T16:58:56.9638098Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 55, in newFunction
2023-09-06T16:58:56.9638486Z     return old_func(*args, **kwargs)
2023-09-06T16:58:56.9639048Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 348, in compile_fx_inner
2023-09-06T16:58:56.9639516Z     compiled_graph: CompiledFxGraph = fx_codegen_and_compile(
2023-09-06T16:58:56.9640274Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 574, in fx_codegen_and_compile
2023-09-06T16:58:56.9640711Z     compiled_fn = graph.compile_to_fn()
2023-09-06T16:58:56.9641267Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 991, in compile_to_fn
2023-09-06T16:58:56.9641682Z     return self.compile_to_module().call
2023-09-06T16:58:56.9642233Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:58:56.9642609Z     r = func(*args, **kwargs)
2023-09-06T16:58:56.9643175Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 946, in compile_to_module
2023-09-06T16:58:56.9643649Z     mod = PyCodeCache.load_by_key_path(key, path, linemap=linemap)
2023-09-06T16:58:56.9644270Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1161, in load_by_key_path
2023-09-06T16:58:56.9644686Z     exec(code, mod.__dict__, mod.__dict__)
2023-09-06T16:58:56.9645166Z   File "/tmp/tmpjz3b0znw/jf/cjfjuxh47i625ktgrzghx2bcjrw7lflb7cv4afpvd2jknhjftwop.py", line 131, in <module>
2023-09-06T16:58:56.9645622Z     async_compile.wait(globals())
2023-09-06T16:58:56.9646159Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1440, in wait
2023-09-06T16:58:56.9646549Z     scope[key] = result.result()
2023-09-06T16:58:56.9647087Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1299, in result
2023-09-06T16:58:56.9647458Z     self.future.result()
2023-09-06T16:58:56.9647834Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 458, in result
2023-09-06T16:58:56.9648192Z     return self.__get_result()
2023-09-06T16:58:56.9648578Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
2023-09-06T16:58:56.9648919Z     raise self._exception
2023-09-06T16:58:56.9649373Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
2023-09-06T16:58:56.9650004Z CompilationError: at 11:70:def triton_(in_ptr0, out_ptr0, xnumel, XBLOCK : tl.constexpr):
2023-09-06T16:58:56.9650358Z     xnumel = 209982
2023-09-06T16:58:56.9650615Z     xoffset = tl.program_id(0) * XBLOCK
2023-09-06T16:58:56.9650931Z     xindex = xoffset + tl.arange(0, XBLOCK)[:]
2023-09-06T16:58:56.9651224Z     xmask = xindex < xnumel
2023-09-06T16:58:56.9651467Z     x0 = xindex
2023-09-06T16:58:56.9651724Z     tmp0 = tl.load(in_ptr0 + (209982 + x0), xmask)
2023-09-06T16:58:56.9652049Z     tmp1 = tl.where(tmp0 < 0, tmp0 + 10000, tmp0)
2023-09-06T16:58:56.9652457Z     tl.device_assert(((0 <= tmp1) & (tmp1 < 10000)) | ~xmask, "index out of bounds: 0 <= tmp1 < 10000")
2023-09-06T16:58:56.9652797Z     tmp2 = 1.0
2023-09-06T16:58:56.9653151Z     tl.atomic_add(out_ptr0 + (tl.broadcast_to(tmp1, [XBLOCK])), tmp2, xmask)
2023-09-06T16:58:56.9653508Z                                                                       ^
2023-09-06T16:58:56.9653915Z ValueError('atomic_add does not support bf16')
2023-09-06T16:58:56.9654136Z 
2023-09-06T16:58:56.9654329Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T16:58:56.9654565Z 
2023-09-06T16:58:56.9654571Z 
2023-09-06T16:58:56.9654767Z You can suppress this exception and fall back to eager by setting:
2023-09-06T16:58:56.9655089Z     import torch._dynamo
2023-09-06T16:58:56.9655397Z     torch._dynamo.config.suppress_errors = True
2023-09-06T16:58:56.9655603Z 
2023-09-06T16:58:58.4002135Z ERROR
2023-09-06T16:59:02.0385075Z 
2023-09-06T16:59:05.0510025Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:59:05.0510905Z loading model: 0it [00:03, ?it/s]
2023-09-06T16:59:05.0515775Z cuda eval  basic_gnn_gin                      
2023-09-06T16:59:05.5874621Z [2023-09-06 16:59:05,586] [0/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T16:59:12.5975750Z ERROR:common:Backend dynamo failed in warmup()
2023-09-06T16:59:12.5976413Z Traceback (most recent call last):
2023-09-06T16:59:12.5977034Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2288, in warmup
2023-09-06T16:59:12.5977578Z     fn(model, example_inputs)
2023-09-06T16:59:12.5978693Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T16:59:12.5979531Z     return fn(*args, **kwargs)
2023-09-06T16:59:12.5980390Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T16:59:12.5981125Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T16:59:12.5982517Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 634, in _convert_frame
2023-09-06T16:59:12.5983337Z     result = inner_convert(frame, cache_entry, hooks, frame_state)
2023-09-06T16:59:12.5984428Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T16:59:12.5984984Z     return fn(*args, **kwargs)
2023-09-06T16:59:12.5985570Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T16:59:12.5985966Z     return _compile(
2023-09-06T16:59:12.5986476Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T16:59:12.5986941Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T16:59:12.5987524Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:59:12.5987900Z     r = func(*args, **kwargs)
2023-09-06T16:59:12.5988446Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T16:59:12.5989697Z     out_code = transform_code_object(code, transform)
2023-09-06T16:59:12.5990387Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T16:59:12.5990860Z     transformations(instructions, code_options)
2023-09-06T16:59:12.5991435Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T16:59:12.5991792Z     tracer.run()
2023-09-06T16:59:12.5992305Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T16:59:12.5992668Z     super().run()
2023-09-06T16:59:12.5993179Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T16:59:12.5993547Z     and self.step()
2023-09-06T16:59:12.5994050Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T16:59:12.5994457Z     getattr(self, inst.opname)(inst)
2023-09-06T16:59:12.5995023Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2166, in RETURN_VALUE
2023-09-06T16:59:12.5995435Z     self.output.compile_subgraph(
2023-09-06T16:59:12.5995996Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 881, in compile_subgraph
2023-09-06T16:59:12.5996472Z     self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
2023-09-06T16:59:12.5996893Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T16:59:12.5997234Z     return func(*args, **kwds)
2023-09-06T16:59:12.5997798Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1008, in compile_and_call_fx_graph
2023-09-06T16:59:12.5998236Z     compiled_fn = self.call_user_compiler(gm)
2023-09-06T16:59:12.5999066Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:59:12.5999469Z     r = func(*args, **kwargs)
2023-09-06T16:59:12.6000031Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1075, in call_user_compiler
2023-09-06T16:59:12.6000502Z     raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
2023-09-06T16:59:12.6001132Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1060, in call_user_compiler
2023-09-06T16:59:12.6001587Z     compiled_fn = compiler_fn(gm, self.example_inputs())
2023-09-06T16:59:12.6002185Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper
2023-09-06T16:59:12.6002603Z     compiled_gm = compiler_fn(gm, example_inputs)
2023-09-06T16:59:12.6003177Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/inductor.py", line 9, in inductor
2023-09-06T16:59:12.6003590Z     return compile_fx(*args, **kwargs)
2023-09-06T16:59:12.6004145Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1167, in compile_fx
2023-09-06T16:59:12.6004526Z     return aot_autograd(
2023-09-06T16:59:12.6005049Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 55, in compiler_fn
2023-09-06T16:59:12.6005497Z     cg = aot_module_simplified(gm, example_inputs, **kwargs)
2023-09-06T16:59:12.6006123Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3891, in aot_module_simplified
2023-09-06T16:59:12.6006576Z     compiled_fn = create_aot_dispatcher_function(
2023-09-06T16:59:12.6007119Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:59:12.6007495Z     r = func(*args, **kwargs)
2023-09-06T16:59:12.6008102Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3429, in create_aot_dispatcher_function
2023-09-06T16:59:12.6008811Z     compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:59:12.6009483Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2212, in aot_wrapper_dedupe
2023-09-06T16:59:12.6009970Z     return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:59:12.6010647Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2392, in aot_wrapper_synthetic_base
2023-09-06T16:59:12.6011149Z     return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:59:12.6011788Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1573, in aot_dispatch_base
2023-09-06T16:59:12.6012221Z     compiled_fw = compiler(fw_module, flat_args)
2023-09-06T16:59:12.6012808Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 938, in fw_compiler_freezing
2023-09-06T16:59:12.6013233Z     optimized_function = inner_compile(
2023-09-06T16:59:12.6013799Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 80, in debug_wrapper
2023-09-06T16:59:12.6014241Z     inner_compiled_fn = compiler_fn(gm, example_inputs)
2023-09-06T16:59:12.6014784Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 228, in inner
2023-09-06T16:59:12.6015157Z     return fn(*args, **kwargs)
2023-09-06T16:59:12.6015506Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T16:59:12.6015849Z     return func(*args, **kwds)
2023-09-06T16:59:12.6016387Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 55, in newFunction
2023-09-06T16:59:12.6016768Z     return old_func(*args, **kwargs)
2023-09-06T16:59:12.6017447Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 348, in compile_fx_inner
2023-09-06T16:59:12.6017921Z     compiled_graph: CompiledFxGraph = fx_codegen_and_compile(
2023-09-06T16:59:12.6018547Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 574, in fx_codegen_and_compile
2023-09-06T16:59:12.6018991Z     compiled_fn = graph.compile_to_fn()
2023-09-06T16:59:12.6019546Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 991, in compile_to_fn
2023-09-06T16:59:12.6019953Z     return self.compile_to_module().call
2023-09-06T16:59:12.6020496Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:59:12.6020866Z     r = func(*args, **kwargs)
2023-09-06T16:59:12.6021394Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 946, in compile_to_module
2023-09-06T16:59:12.6021878Z     mod = PyCodeCache.load_by_key_path(key, path, linemap=linemap)
2023-09-06T16:59:12.6022496Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1161, in load_by_key_path
2023-09-06T16:59:12.6022905Z     exec(code, mod.__dict__, mod.__dict__)
2023-09-06T16:59:12.6023365Z   File "/tmp/tmpbjyyo5qo/32/c32dhwhhjzhfr33iqnakm6hbvlfnbqmeifrs2ojkz3kt352cmmhu.py", line 240, in <module>
2023-09-06T16:59:12.6023822Z     async_compile.wait(globals())
2023-09-06T16:59:12.6024352Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1440, in wait
2023-09-06T16:59:12.6024740Z     scope[key] = result.result()
2023-09-06T16:59:12.6025279Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1299, in result
2023-09-06T16:59:12.6025649Z     self.future.result()
2023-09-06T16:59:12.6026022Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 451, in result
2023-09-06T16:59:12.6026511Z     return self.__get_result()
2023-09-06T16:59:12.6026894Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
2023-09-06T16:59:12.6027243Z     raise self._exception
2023-09-06T16:59:12.6027697Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
2023-09-06T16:59:12.6028136Z CompilationError: at 15:53:    xindex = xoffset + tl.arange(0, XBLOCK)[:]
2023-09-06T16:59:12.6028525Z     xmask = xindex < xnumel
2023-09-06T16:59:12.6028834Z     x1 = (xindex // 64)
2023-09-06T16:59:12.6029341Z     x0 = xindex % 64
2023-09-06T16:59:12.6029882Z     tmp0 = tl.load(in_ptr0 + (200000 + x1), None, eviction_policy='evict_last')
2023-09-06T16:59:12.6030385Z     tmp2 = tl.load(in_ptr0 + (x1), None, eviction_policy='evict_last')
2023-09-06T16:59:12.6030730Z     tmp1 = tl.where(tmp0 < 0, tmp0 + 10000, tmp0)
2023-09-06T16:59:12.6031116Z     tl.device_assert((0 <= tmp1) & (tmp1 < 10000), "index out of bounds: 0 <= tmp1 < 10000")
2023-09-06T16:59:12.6031508Z     tmp3 = tl.where(tmp2 < 0, tmp2 + 10000, tmp2)
2023-09-06T16:59:12.6031895Z     tl.device_assert((0 <= tmp3) & (tmp3 < 10000), "index out of bounds: 0 <= tmp3 < 10000")
2023-09-06T16:59:12.6032298Z     tmp4 = tl.load(in_ptr1 + (x0 + (64*tmp3)), None).to(tl.float32)
2023-09-06T16:59:12.6032642Z     tl.atomic_add(out_ptr0 + (x0 + (64*tmp1)), tmp4, None)
2023-09-06T16:59:12.6032951Z                                                      ^
2023-09-06T16:59:12.6033332Z ValueError('atomic_add does not support bf16')
2023-09-06T16:59:12.6033533Z 
2023-09-06T16:59:12.6033724Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T16:59:12.6033957Z 
2023-09-06T16:59:12.6033963Z 
2023-09-06T16:59:12.6034141Z You can suppress this exception and fall back to eager by setting:
2023-09-06T16:59:12.6034475Z     import torch._dynamo
2023-09-06T16:59:12.6034997Z     torch._dynamo.config.suppress_errors = True
2023-09-06T16:59:12.6035377Z 
2023-09-06T16:59:13.8565748Z ERROR
2023-09-06T16:59:17.4502749Z 
2023-09-06T16:59:20.4281623Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:59:20.4282216Z loading model: 0it [00:02, ?it/s]
2023-09-06T16:59:20.4282974Z cuda eval  basic_gnn_sage                     
2023-09-06T16:59:20.9555245Z [2023-09-06 16:59:20,954] [0/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T16:59:27.6876270Z ERROR:common:Backend dynamo failed in warmup()
2023-09-06T16:59:27.6876671Z Traceback (most recent call last):
2023-09-06T16:59:27.6877144Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2288, in warmup
2023-09-06T16:59:27.6877828Z     fn(model, example_inputs)
2023-09-06T16:59:27.6879533Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T16:59:27.6880543Z     return fn(*args, **kwargs)
2023-09-06T16:59:27.6881119Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T16:59:27.6881570Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T16:59:27.6882169Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 634, in _convert_frame
2023-09-06T16:59:27.6882622Z     result = inner_convert(frame, cache_entry, hooks, frame_state)
2023-09-06T16:59:27.6883222Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T16:59:27.6883600Z     return fn(*args, **kwargs)
2023-09-06T16:59:27.6884170Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T16:59:27.6884567Z     return _compile(
2023-09-06T16:59:27.6885090Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T16:59:27.6886120Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T16:59:27.6886729Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:59:27.6887110Z     r = func(*args, **kwargs)
2023-09-06T16:59:27.6887644Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T16:59:27.6888090Z     out_code = transform_code_object(code, transform)
2023-09-06T16:59:27.6888827Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T16:59:27.6889400Z     transformations(instructions, code_options)
2023-09-06T16:59:27.6889980Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T16:59:27.6890360Z     tracer.run()
2023-09-06T16:59:27.6890867Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T16:59:27.6891240Z     super().run()
2023-09-06T16:59:27.6891752Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T16:59:27.6892123Z     and self.step()
2023-09-06T16:59:27.6892624Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T16:59:27.6893023Z     getattr(self, inst.opname)(inst)
2023-09-06T16:59:27.6893597Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2166, in RETURN_VALUE
2023-09-06T16:59:27.6974037Z     self.output.compile_subgraph(
2023-09-06T16:59:27.6975180Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 881, in compile_subgraph
2023-09-06T16:59:27.6976471Z     self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
2023-09-06T16:59:27.6977126Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T16:59:27.6977622Z     return func(*args, **kwds)
2023-09-06T16:59:27.6978543Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1008, in compile_and_call_fx_graph
2023-09-06T16:59:27.6979168Z     compiled_fn = self.call_user_compiler(gm)
2023-09-06T16:59:27.6979962Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:59:27.6980509Z     r = func(*args, **kwargs)
2023-09-06T16:59:27.6981340Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1075, in call_user_compiler
2023-09-06T16:59:27.6982012Z     raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
2023-09-06T16:59:27.6982945Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 1060, in call_user_compiler
2023-09-06T16:59:27.6983614Z     compiled_fn = compiler_fn(gm, self.example_inputs())
2023-09-06T16:59:27.6984444Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper
2023-09-06T16:59:27.6985053Z     compiled_gm = compiler_fn(gm, example_inputs)
2023-09-06T16:59:27.6985834Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/inductor.py", line 9, in inductor
2023-09-06T16:59:27.6986404Z     return compile_fx(*args, **kwargs)
2023-09-06T16:59:27.6987187Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1167, in compile_fx
2023-09-06T16:59:27.6987725Z     return aot_autograd(
2023-09-06T16:59:27.6988533Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 55, in compiler_fn
2023-09-06T16:59:27.6989404Z     cg = aot_module_simplified(gm, example_inputs, **kwargs)
2023-09-06T16:59:27.6990709Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3891, in aot_module_simplified
2023-09-06T16:59:27.6991372Z     compiled_fn = create_aot_dispatcher_function(
2023-09-06T16:59:27.6992197Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:59:27.6992730Z     r = func(*args, **kwargs)
2023-09-06T16:59:27.6993598Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3429, in create_aot_dispatcher_function
2023-09-06T16:59:27.6994360Z     compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:59:27.6995318Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2212, in aot_wrapper_dedupe
2023-09-06T16:59:27.6996047Z     return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:59:27.6997042Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2392, in aot_wrapper_synthetic_base
2023-09-06T16:59:27.6997773Z     return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T16:59:27.6998770Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1573, in aot_dispatch_base
2023-09-06T16:59:27.6999398Z     compiled_fw = compiler(fw_module, flat_args)
2023-09-06T16:59:27.7000270Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 938, in fw_compiler_freezing
2023-09-06T16:59:27.7000876Z     optimized_function = inner_compile(
2023-09-06T16:59:27.7001704Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 80, in debug_wrapper
2023-09-06T16:59:27.7002348Z     inner_compiled_fn = compiler_fn(gm, example_inputs)
2023-09-06T16:59:27.7003406Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 228, in inner
2023-09-06T16:59:27.7003999Z     return fn(*args, **kwargs)
2023-09-06T16:59:27.7004502Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T16:59:27.7004979Z     return func(*args, **kwds)
2023-09-06T16:59:27.7005742Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 55, in newFunction
2023-09-06T16:59:27.7006297Z     return old_func(*args, **kwargs)
2023-09-06T16:59:27.7007070Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 348, in compile_fx_inner
2023-09-06T16:59:27.7007756Z     compiled_graph: CompiledFxGraph = fx_codegen_and_compile(
2023-09-06T16:59:27.7008662Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 574, in fx_codegen_and_compile
2023-09-06T16:59:27.7009225Z     compiled_fn = graph.compile_to_fn()
2023-09-06T16:59:27.7010009Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 991, in compile_to_fn
2023-09-06T16:59:27.7010582Z     return self.compile_to_module().call
2023-09-06T16:59:27.7011368Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T16:59:27.7011901Z     r = func(*args, **kwargs)
2023-09-06T16:59:27.7012716Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 946, in compile_to_module
2023-09-06T16:59:27.7013368Z     mod = PyCodeCache.load_by_key_path(key, path, linemap=linemap)
2023-09-06T16:59:27.7014291Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1161, in load_by_key_path
2023-09-06T16:59:27.7014873Z     exec(code, mod.__dict__, mod.__dict__)
2023-09-06T16:59:27.7015527Z   File "/tmp/tmp30z_gl9q/zr/czrploy62vzjqnp3kv423t5fasvgqbifboe4sc2b5claut7rlybg.py", line 248, in <module>
2023-09-06T16:59:27.7016178Z     async_compile.wait(globals())
2023-09-06T16:59:27.7017269Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1440, in wait
2023-09-06T16:59:27.7017879Z     scope[key] = result.result()
2023-09-06T16:59:27.7018782Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1299, in result
2023-09-06T16:59:27.7019363Z     self.future.result()
2023-09-06T16:59:27.7019914Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 451, in result
2023-09-06T16:59:27.7020437Z     return self.__get_result()
2023-09-06T16:59:27.7020978Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
2023-09-06T16:59:27.7021504Z     raise self._exception
2023-09-06T16:59:27.7022232Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
2023-09-06T16:59:27.7022894Z CompilationError: at 15:53:    xindex = xoffset + tl.arange(0, XBLOCK)[:]
2023-09-06T16:59:27.7023398Z     xmask = xindex < xnumel
2023-09-06T16:59:27.7023796Z     x1 = (xindex // 64)
2023-09-06T16:59:27.7024156Z     x0 = xindex % 64
2023-09-06T16:59:27.7024840Z     tmp0 = tl.load(in_ptr0 + (200000 + x1), None, eviction_policy='evict_last')
2023-09-06T16:59:27.7025608Z     tmp2 = tl.load(in_ptr0 + (x1), None, eviction_policy='evict_last')
2023-09-06T16:59:27.7026136Z     tmp1 = tl.where(tmp0 < 0, tmp0 + 10000, tmp0)
2023-09-06T16:59:27.7026710Z     tl.device_assert((0 <= tmp1) & (tmp1 < 10000), "index out of bounds: 0 <= tmp1 < 10000")
2023-09-06T16:59:27.7027255Z     tmp3 = tl.where(tmp2 < 0, tmp2 + 10000, tmp2)
2023-09-06T16:59:27.7027810Z     tl.device_assert((0 <= tmp3) & (tmp3 < 10000), "index out of bounds: 0 <= tmp3 < 10000")
2023-09-06T16:59:27.7028399Z     tmp4 = tl.load(in_ptr1 + (x0 + (64*tmp3)), None).to(tl.float32)
2023-09-06T16:59:27.7029054Z     tl.atomic_add(out_ptr0 + (x0 + (64*tmp1)), tmp4, None)
2023-09-06T16:59:27.7029716Z                                                      ^
2023-09-06T16:59:27.7030629Z ValueError('atomic_add does not support bf16')
2023-09-06T16:59:27.7030956Z 
2023-09-06T16:59:27.7031211Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T16:59:27.7031545Z 
2023-09-06T16:59:27.7031554Z 
2023-09-06T16:59:27.7031830Z You can suppress this exception and fall back to eager by setting:
2023-09-06T16:59:27.7032305Z     import torch._dynamo
2023-09-06T16:59:27.7032769Z     torch._dynamo.config.suppress_errors = True
2023-09-06T16:59:27.7033138Z 
2023-09-06T16:59:28.9470294Z ERROR
2023-09-06T16:59:32.5836078Z 
2023-09-06T16:59:36.6751791Z loading model: 0it [00:00, ?it/s]
2023-09-06T16:59:36.6752312Z loading model: 0it [00:04, ?it/s]
2023-09-06T16:59:36.6840533Z cuda eval  cm3leon_generate                   
2023-09-06T17:05:14.8456474Z 
2023-09-06T17:05:16.1281284Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T17:05:17.4137964Z running benchmark:   3% 1/30 [00:01<00:37,  1.28s/it]
2023-09-06T17:05:18.6981615Z running benchmark:   7% 2/30 [00:02<00:35,  1.28s/it]
2023-09-06T17:05:19.9863393Z running benchmark:  10% 3/30 [00:03<00:34,  1.28s/it]
2023-09-06T17:05:21.2561536Z running benchmark:  13% 4/30 [00:05<00:33,  1.29s/it]
2023-09-06T17:05:22.5226819Z running benchmark:  17% 5/30 [00:06<00:32,  1.28s/it]
2023-09-06T17:05:23.7893111Z running benchmark:  20% 6/30 [00:07<00:30,  1.28s/it]
2023-09-06T17:05:25.0684410Z running benchmark:  23% 7/30 [00:08<00:29,  1.27s/it]
2023-09-06T17:05:26.3462729Z running benchmark:  27% 8/30 [00:10<00:28,  1.27s/it]
2023-09-06T17:05:27.6273372Z running benchmark:  30% 9/30 [00:11<00:26,  1.28s/it]
2023-09-06T17:05:28.8912523Z running benchmark:  33% 10/30 [00:12<00:25,  1.28s/it]
2023-09-06T17:05:30.1679671Z running benchmark:  37% 11/30 [00:14<00:24,  1.27s/it]
2023-09-06T17:05:31.4369671Z running benchmark:  40% 12/30 [00:15<00:22,  1.27s/it]
2023-09-06T17:05:32.7023486Z running benchmark:  43% 13/30 [00:16<00:21,  1.27s/it]
2023-09-06T17:05:33.9762107Z running benchmark:  47% 14/30 [00:17<00:20,  1.27s/it]
2023-09-06T17:05:35.2396577Z running benchmark:  50% 15/30 [00:19<00:19,  1.27s/it]
2023-09-06T17:05:36.5208347Z running benchmark:  53% 16/30 [00:20<00:17,  1.27s/it]
2023-09-06T17:05:37.8022250Z running benchmark:  57% 17/30 [00:21<00:16,  1.27s/it]
2023-09-06T17:05:39.0817784Z running benchmark:  60% 18/30 [00:22<00:15,  1.28s/it]
2023-09-06T17:05:40.3555299Z running benchmark:  63% 19/30 [00:24<00:14,  1.28s/it]
2023-09-06T17:05:41.6245337Z running benchmark:  67% 20/30 [00:25<00:12,  1.28s/it]
2023-09-06T17:05:42.8905200Z running benchmark:  70% 21/30 [00:26<00:11,  1.27s/it]
2023-09-06T17:05:44.1715675Z running benchmark:  73% 22/30 [00:28<00:10,  1.27s/it]
2023-09-06T17:05:45.4458705Z running benchmark:  77% 23/30 [00:29<00:08,  1.27s/it]
2023-09-06T17:05:46.7252728Z running benchmark:  80% 24/30 [00:30<00:07,  1.27s/it]
2023-09-06T17:05:48.0054954Z running benchmark:  83% 25/30 [00:31<00:06,  1.28s/it]
2023-09-06T17:05:49.2797077Z running benchmark:  87% 26/30 [00:33<00:05,  1.28s/it]
2023-09-06T17:05:50.5572164Z running benchmark:  90% 27/30 [00:34<00:03,  1.28s/it]
2023-09-06T17:05:51.8342555Z running benchmark:  93% 28/30 [00:35<00:02,  1.28s/it]
2023-09-06T17:05:53.1007679Z running benchmark:  97% 29/30 [00:36<00:01,  1.28s/it]
2023-09-06T17:05:53.1008079Z running benchmark: 100% 30/30 [00:38<00:00,  1.27s/it]
2023-09-06T17:05:53.1008404Z running benchmark: 100% 30/30 [00:38<00:00,  1.28s/it]
2023-09-06T17:05:53.1047384Z 5.412x
2023-09-06T17:05:59.5688419Z 
2023-09-06T17:06:00.4990702Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:06:00.4991275Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:06:00.4995410Z cuda eval  dcgan                              
2023-09-06T17:06:11.1372401Z 
2023-09-06T17:06:11.2392635Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T17:06:11.3414925Z running benchmark:  43% 13/30 [00:00<00:00, 127.83it/s]
2023-09-06T17:06:11.3633897Z running benchmark:  90% 27/30 [00:00<00:00, 133.02it/s]
2023-09-06T17:06:11.3634415Z running benchmark: 100% 30/30 [00:00<00:00, 132.65it/s]
2023-09-06T17:06:11.3652115Z 1.125x
2023-09-06T17:06:16.3431859Z 
2023-09-06T17:06:20.0316874Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:06:20.0317235Z loading model: 0it [00:03, ?it/s]
2023-09-06T17:06:20.0317511Z Eager model failed to run
2023-09-06T17:06:20.0329805Z Traceback (most recent call last):
2023-09-06T17:06:20.0330289Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T17:06:20.0330703Z     self.model_iter_fn(model, example_inputs)
2023-09-06T17:06:20.0331124Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T17:06:20.0333951Z     return mod(*inputs)
2023-09-06T17:06:20.0335693Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:06:20.0336299Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:06:20.0336932Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:06:20.0337391Z     return forward_call(*args, **kwargs)
2023-09-06T17:06:20.0337808Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/__init__.py", line 31, in forward
2023-09-06T17:06:20.0338214Z     return sources, self.model(mix)
2023-09-06T17:06:20.0338787Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:06:20.0339209Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:06:20.0341621Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:06:20.0342118Z     return forward_call(*args, **kwargs)
2023-09-06T17:06:20.0342830Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/demucs/model.py", line 209, in forward
2023-09-06T17:06:20.0343269Z     x = self.lstm(x)
2023-09-06T17:06:20.0344257Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:06:20.0345556Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:06:20.0346553Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:06:20.0347213Z     return forward_call(*args, **kwargs)
2023-09-06T17:06:20.0347960Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/demucs/model.py", line 27, in forward
2023-09-06T17:06:20.0348763Z     x = self.lstm(x)[0]
2023-09-06T17:06:20.0349947Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:06:20.0350728Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:06:20.0351671Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:06:20.0352411Z     return forward_call(*args, **kwargs)
2023-09-06T17:06:20.0353349Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/rnn.py", line 879, in forward
2023-09-06T17:06:20.0354190Z     result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
2023-09-06T17:06:20.0355114Z RuntimeError: "_thnn_fused_lstm_cell_cuda" not implemented for 'BFloat16'
2023-09-06T17:06:20.0355489Z 
2023-09-06T17:06:20.0355864Z The above exception was the direct cause of the following exception:
2023-09-06T17:06:20.0356229Z 
2023-09-06T17:06:20.0356459Z Traceback (most recent call last):
2023-09-06T17:06:20.0357062Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T17:06:20.0357712Z     ) = runner.load_model(
2023-09-06T17:06:20.0358359Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T17:06:20.0359085Z     self.validate_model(model, example_inputs)
2023-09-06T17:06:20.0359740Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T17:06:20.0360850Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T17:06:20.0361516Z NotImplementedError: Eager model failed to run
2023-09-06T17:06:20.0361916Z 
2023-09-06T17:06:20.0362148Z WARNING:root:demucs failed to load
2023-09-06T17:06:24.5649980Z 
2023-09-06T17:06:26.6706342Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:06:26.6706697Z loading model: 0it [00:02, ?it/s]
2023-09-06T17:06:26.6825754Z cuda eval  densenet121                        
2023-09-06T17:08:03.0171890Z 
2023-09-06T17:08:03.1290573Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T17:08:03.2309737Z running benchmark:   7% 2/30 [00:00<00:01, 17.83it/s]
2023-09-06T17:08:03.3330493Z running benchmark:  17% 5/30 [00:00<00:01, 24.33it/s]
2023-09-06T17:08:03.4364213Z running benchmark:  27% 8/30 [00:00<00:00, 26.59it/s]
2023-09-06T17:08:03.5388190Z running benchmark:  37% 11/30 [00:00<00:00, 27.55it/s]
2023-09-06T17:08:03.6423401Z running benchmark:  47% 14/30 [00:00<00:00, 28.17it/s]
2023-09-06T17:08:03.7486715Z running benchmark:  57% 17/30 [00:00<00:00, 28.46it/s]
2023-09-06T17:08:03.8549681Z running benchmark:  67% 20/30 [00:00<00:00, 28.37it/s]
2023-09-06T17:08:03.9606811Z running benchmark:  77% 23/30 [00:00<00:00, 28.32it/s]
2023-09-06T17:08:04.0643545Z running benchmark:  87% 26/30 [00:00<00:00, 28.34it/s]
2023-09-06T17:08:04.0993508Z running benchmark:  97% 29/30 [00:01<00:00, 28.52it/s]
2023-09-06T17:08:04.1018472Z running benchmark: 100% 30/30 [00:01<00:00, 27.70it/s]
2023-09-06T17:08:04.1018788Z 1.722x
2023-09-06T17:08:09.9683634Z 
2023-09-06T17:08:14.5286876Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:08:14.5288313Z loading model: 0it [00:04, ?it/s]
2023-09-06T17:08:14.5288773Z Eager model failed to run
2023-09-06T17:08:14.5305133Z Traceback (most recent call last):
2023-09-06T17:08:14.5305588Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T17:08:14.5306007Z     self.model_iter_fn(model, example_inputs)
2023-09-06T17:08:14.5307690Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T17:08:14.5308304Z     return mod(*inputs)
2023-09-06T17:08:14.5310442Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:08:14.5310881Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:08:14.5311463Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:08:14.5311875Z     return forward_call(*args, **kwargs)
2023-09-06T17:08:14.5312430Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T17:08:14.5312891Z     return self.inference(batched_inputs)
2023-09-06T17:08:14.5313552Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T17:08:14.5314045Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T17:08:14.5314686Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:08:14.5315093Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:08:14.5315654Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:08:14.5316058Z     return forward_call(*args, **kwargs)
2023-09-06T17:08:14.5316630Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2023-09-06T17:08:14.5317065Z     box_features = self._shared_roi_transform(
2023-09-06T17:08:14.5317677Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2023-09-06T17:08:14.5318112Z     x = self.pooler(features, boxes)
2023-09-06T17:08:14.5318964Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:08:14.5319402Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:08:14.5319955Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:08:14.5320356Z     return forward_call(*args, **kwargs)
2023-09-06T17:08:14.5320911Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T17:08:14.5321341Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T17:08:14.5321930Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:08:14.5322336Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:08:14.5322884Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:08:14.5323281Z     return forward_call(*args, **kwargs)
2023-09-06T17:08:14.5323892Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T17:08:14.5324248Z     return roi_align(
2023-09-06T17:08:14.5324777Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T17:08:14.5325212Z     return torch.ops.torchvision.roi_align(
2023-09-06T17:08:14.5325756Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T17:08:14.5326145Z     return self._op(*args, **kwargs or {})
2023-09-06T17:08:14.5326597Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T17:08:14.5326843Z 
2023-09-06T17:08:14.5327039Z The above exception was the direct cause of the following exception:
2023-09-06T17:08:14.5327281Z 
2023-09-06T17:08:14.5327410Z Traceback (most recent call last):
2023-09-06T17:08:14.5327787Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T17:08:14.5328281Z     ) = runner.load_model(
2023-09-06T17:08:14.5328659Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T17:08:14.5329063Z     self.validate_model(model, example_inputs)
2023-09-06T17:08:14.5329483Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T17:08:14.5329913Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T17:08:14.5330291Z NotImplementedError: Eager model failed to run
2023-09-06T17:08:14.5330501Z 
2023-09-06T17:08:14.5330665Z WARNING:root:detectron2_fasterrcnn_r_101_c4 failed to load
2023-09-06T17:08:19.2628476Z 
2023-09-06T17:08:26.3407929Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:08:26.3408571Z loading model: 0it [00:07, ?it/s]
2023-09-06T17:08:26.3408858Z Eager model failed to run
2023-09-06T17:08:26.3422560Z Traceback (most recent call last):
2023-09-06T17:08:26.3423190Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T17:08:26.3423885Z     self.model_iter_fn(model, example_inputs)
2023-09-06T17:08:26.3424599Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T17:08:26.3425280Z     return mod(*inputs)
2023-09-06T17:08:26.3427912Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:08:26.3428693Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:08:26.3429796Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:08:26.3430217Z     return forward_call(*args, **kwargs)
2023-09-06T17:08:26.3430805Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T17:08:26.3431243Z     return self.inference(batched_inputs)
2023-09-06T17:08:26.3432231Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T17:08:26.3432733Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T17:08:26.3433360Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:08:26.3433867Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:08:26.3434413Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:08:26.3434820Z     return forward_call(*args, **kwargs)
2023-09-06T17:08:26.3435407Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T17:08:26.3436343Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T17:08:26.3437386Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T17:08:26.3438449Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T17:08:26.3439153Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:08:26.3439581Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:08:26.3440144Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:08:26.3440532Z     return forward_call(*args, **kwargs)
2023-09-06T17:08:26.3441095Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T17:08:26.3441525Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T17:08:26.3442122Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:08:26.3442531Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:08:26.3443562Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:08:26.3443965Z     return forward_call(*args, **kwargs)
2023-09-06T17:08:26.3444536Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T17:08:26.3444909Z     return roi_align(
2023-09-06T17:08:26.3445427Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T17:08:26.3445861Z     return torch.ops.torchvision.roi_align(
2023-09-06T17:08:26.3446406Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T17:08:26.3446791Z     return self._op(*args, **kwargs or {})
2023-09-06T17:08:26.3447245Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T17:08:26.3447494Z 
2023-09-06T17:08:26.3447696Z The above exception was the direct cause of the following exception:
2023-09-06T17:08:26.3447947Z 
2023-09-06T17:08:26.3448074Z Traceback (most recent call last):
2023-09-06T17:08:26.3448454Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T17:08:26.3448798Z     ) = runner.load_model(
2023-09-06T17:08:26.3449161Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T17:08:26.3449564Z     self.validate_model(model, example_inputs)
2023-09-06T17:08:26.3449971Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T17:08:26.3450418Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T17:08:26.3450789Z NotImplementedError: Eager model failed to run
2023-09-06T17:08:26.3451000Z 
2023-09-06T17:08:26.3451166Z WARNING:root:detectron2_fasterrcnn_r_101_dc5 failed to load
2023-09-06T17:08:30.9877904Z 
2023-09-06T17:08:35.4735182Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:08:35.4736123Z loading model: 0it [00:04, ?it/s]
2023-09-06T17:08:35.4736413Z Eager model failed to run
2023-09-06T17:08:35.4751987Z Traceback (most recent call last):
2023-09-06T17:08:35.4752811Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T17:08:35.4753643Z     self.model_iter_fn(model, example_inputs)
2023-09-06T17:08:35.4754282Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T17:08:35.4754837Z     return mod(*inputs)
2023-09-06T17:08:35.4757399Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:08:35.4758085Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:08:35.4759161Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:08:35.4759881Z     return forward_call(*args, **kwargs)
2023-09-06T17:08:35.4761045Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T17:08:35.4761863Z     return self.inference(batched_inputs)
2023-09-06T17:08:35.4763006Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T17:08:35.4764476Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T17:08:35.4765323Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:08:35.4765762Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:08:35.4766317Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:08:35.4766787Z     return forward_call(*args, **kwargs)
2023-09-06T17:08:35.4767401Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T17:08:35.4768370Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T17:08:35.4769008Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T17:08:35.4769508Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T17:08:35.4770156Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:08:35.4770580Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:08:35.4771131Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:08:35.4771531Z     return forward_call(*args, **kwargs)
2023-09-06T17:08:35.4772068Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2023-09-06T17:08:35.4772526Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2023-09-06T17:08:35.4773155Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:08:35.4773578Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:08:35.4774209Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:08:35.4774612Z     return forward_call(*args, **kwargs)
2023-09-06T17:08:35.4775165Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T17:08:35.4775543Z     return roi_align(
2023-09-06T17:08:35.4776056Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T17:08:35.4776490Z     return torch.ops.torchvision.roi_align(
2023-09-06T17:08:35.4777035Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T17:08:35.4777597Z     return self._op(*args, **kwargs or {})
2023-09-06T17:08:35.4778083Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T17:08:35.4778315Z 
2023-09-06T17:08:35.4778516Z The above exception was the direct cause of the following exception:
2023-09-06T17:08:35.4778760Z 
2023-09-06T17:08:35.4778889Z Traceback (most recent call last):
2023-09-06T17:08:35.4779271Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T17:08:35.4779627Z     ) = runner.load_model(
2023-09-06T17:08:35.4779996Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T17:08:35.4780403Z     self.validate_model(model, example_inputs)
2023-09-06T17:08:35.4780813Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T17:08:35.4781267Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T17:08:35.4781655Z NotImplementedError: Eager model failed to run
2023-09-06T17:08:35.4781865Z 
2023-09-06T17:08:35.4782036Z WARNING:root:detectron2_fasterrcnn_r_101_fpn failed to load
2023-09-06T17:08:40.1566807Z 
2023-09-06T17:08:44.4090544Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:08:44.4091826Z loading model: 0it [00:04, ?it/s]
2023-09-06T17:08:44.4092297Z Eager model failed to run
2023-09-06T17:08:44.4120261Z Traceback (most recent call last):
2023-09-06T17:08:44.4121068Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T17:08:44.4121781Z     self.model_iter_fn(model, example_inputs)
2023-09-06T17:08:44.4122334Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T17:08:44.4122953Z     return mod(*inputs)
2023-09-06T17:08:44.4124549Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:08:44.4124985Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:08:44.4126482Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:08:44.4126897Z     return forward_call(*args, **kwargs)
2023-09-06T17:08:44.4127499Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T17:08:44.4127922Z     return self.inference(batched_inputs)
2023-09-06T17:08:44.4128513Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T17:08:44.4129096Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T17:08:44.4129718Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:08:44.4130124Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:08:44.4130685Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:08:44.4131105Z     return forward_call(*args, **kwargs)
2023-09-06T17:08:44.4131691Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2023-09-06T17:08:44.4132108Z     box_features = self._shared_roi_transform(
2023-09-06T17:08:44.4132760Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2023-09-06T17:08:44.4133207Z     x = self.pooler(features, boxes)
2023-09-06T17:08:44.4133778Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:08:44.4134200Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:08:44.4134739Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:08:44.4135150Z     return forward_call(*args, **kwargs)
2023-09-06T17:08:44.4137388Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T17:08:44.4137878Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T17:08:44.4138489Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:08:44.4138990Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:08:44.4139553Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:08:44.4139953Z     return forward_call(*args, **kwargs)
2023-09-06T17:08:44.4140505Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T17:08:44.4140867Z     return roi_align(
2023-09-06T17:08:44.4141396Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T17:08:44.4141834Z     return torch.ops.torchvision.roi_align(
2023-09-06T17:08:44.4142380Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T17:08:44.4142754Z     return self._op(*args, **kwargs or {})
2023-09-06T17:08:44.4143222Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T17:08:44.4143481Z 
2023-09-06T17:08:44.4143679Z The above exception was the direct cause of the following exception:
2023-09-06T17:08:44.4143926Z 
2023-09-06T17:08:44.4144054Z Traceback (most recent call last):
2023-09-06T17:08:44.4144420Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T17:08:44.4144803Z     ) = runner.load_model(
2023-09-06T17:08:44.4145182Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T17:08:44.4145586Z     self.validate_model(model, example_inputs)
2023-09-06T17:08:44.4145996Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T17:08:44.4146615Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T17:08:44.4147001Z NotImplementedError: Eager model failed to run
2023-09-06T17:08:44.4147213Z 
2023-09-06T17:08:44.4147383Z WARNING:root:detectron2_fasterrcnn_r_50_c4 failed to load
2023-09-06T17:08:49.0858122Z 
2023-09-06T17:08:55.4821292Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:08:55.4821646Z loading model: 0it [00:06, ?it/s]
2023-09-06T17:08:55.4822059Z Eager model failed to run
2023-09-06T17:08:55.4839973Z Traceback (most recent call last):
2023-09-06T17:08:55.4840860Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T17:08:55.4841704Z     self.model_iter_fn(model, example_inputs)
2023-09-06T17:08:55.4842505Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T17:08:55.4843253Z     return mod(*inputs)
2023-09-06T17:08:55.4844277Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:08:55.4844766Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:08:55.4845373Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:08:55.4845798Z     return forward_call(*args, **kwargs)
2023-09-06T17:08:55.4846502Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T17:08:55.4846927Z     return self.inference(batched_inputs)
2023-09-06T17:08:55.4847513Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T17:08:55.4847979Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T17:08:55.4848601Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:08:55.4849029Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:08:55.4850352Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:08:55.4850788Z     return forward_call(*args, **kwargs)
2023-09-06T17:08:55.4851435Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T17:08:55.4851915Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T17:08:55.4852566Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T17:08:55.4853153Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T17:08:55.4853796Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:08:55.4854221Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:08:55.4854787Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:08:55.4855205Z     return forward_call(*args, **kwargs)
2023-09-06T17:08:55.4855769Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T17:08:55.4856195Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T17:08:55.4856790Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:08:55.4857250Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:08:55.4857805Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:08:55.4858207Z     return forward_call(*args, **kwargs)
2023-09-06T17:08:55.4858746Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T17:08:55.4859118Z     return roi_align(
2023-09-06T17:08:55.4859896Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T17:08:55.4860341Z     return torch.ops.torchvision.roi_align(
2023-09-06T17:08:55.4860882Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T17:08:55.4861271Z     return self._op(*args, **kwargs or {})
2023-09-06T17:08:55.4861753Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T17:08:55.4862000Z 
2023-09-06T17:08:55.4862213Z The above exception was the direct cause of the following exception:
2023-09-06T17:08:55.4862460Z 
2023-09-06T17:08:55.4862805Z Traceback (most recent call last):
2023-09-06T17:08:55.4863177Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T17:08:55.4863536Z     ) = runner.load_model(
2023-09-06T17:08:55.4863918Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T17:08:55.4864335Z     self.validate_model(model, example_inputs)
2023-09-06T17:08:55.4864738Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T17:08:55.4865188Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T17:08:55.4865586Z NotImplementedError: Eager model failed to run
2023-09-06T17:08:55.4865800Z 
2023-09-06T17:08:55.4865978Z WARNING:root:detectron2_fasterrcnn_r_50_dc5 failed to load
2023-09-06T17:09:00.1817590Z 
2023-09-06T17:09:04.1776370Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:09:04.1776952Z loading model: 0it [00:03, ?it/s]
2023-09-06T17:09:04.1777381Z Eager model failed to run
2023-09-06T17:09:04.1793839Z Traceback (most recent call last):
2023-09-06T17:09:04.1794555Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T17:09:04.1795175Z     self.model_iter_fn(model, example_inputs)
2023-09-06T17:09:04.1796432Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T17:09:04.1797027Z     return mod(*inputs)
2023-09-06T17:09:04.1799316Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:09:04.1799974Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:09:04.1800840Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:09:04.1801443Z     return forward_call(*args, **kwargs)
2023-09-06T17:09:04.1802347Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T17:09:04.1803067Z     return self.inference(batched_inputs)
2023-09-06T17:09:04.1803999Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T17:09:04.1804752Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T17:09:04.1805775Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:09:04.1806416Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:09:04.1807347Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:09:04.1807971Z     return forward_call(*args, **kwargs)
2023-09-06T17:09:04.1808865Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T17:09:04.1809575Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T17:09:04.1810536Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T17:09:04.1811320Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T17:09:04.1812376Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:09:04.1813570Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:09:04.1814459Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:09:04.1815055Z     return forward_call(*args, **kwargs)
2023-09-06T17:09:04.1815901Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2023-09-06T17:09:04.1816630Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2023-09-06T17:09:04.1817561Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:09:04.1818200Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:09:04.1819050Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:09:04.1819713Z     return forward_call(*args, **kwargs)
2023-09-06T17:09:04.1820563Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T17:09:04.1821113Z     return roi_align(
2023-09-06T17:09:04.1821889Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T17:09:04.1822550Z     return torch.ops.torchvision.roi_align(
2023-09-06T17:09:04.1823445Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T17:09:04.1824017Z     return self._op(*args, **kwargs or {})
2023-09-06T17:09:04.1824718Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T17:09:04.1825080Z 
2023-09-06T17:09:04.1825379Z The above exception was the direct cause of the following exception:
2023-09-06T17:09:04.1825760Z 
2023-09-06T17:09:04.1825954Z Traceback (most recent call last):
2023-09-06T17:09:04.1826862Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T17:09:04.1827408Z     ) = runner.load_model(
2023-09-06T17:09:04.1827935Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T17:09:04.1828556Z     self.validate_model(model, example_inputs)
2023-09-06T17:09:04.1829462Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T17:09:04.1830144Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T17:09:04.1830788Z NotImplementedError: Eager model failed to run
2023-09-06T17:09:04.1831106Z 
2023-09-06T17:09:04.1831365Z WARNING:root:detectron2_fasterrcnn_r_50_fpn failed to load
2023-09-06T17:09:08.8558858Z 
2023-09-06T17:09:12.9967460Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:09:12.9968038Z loading model: 0it [00:04, ?it/s]
2023-09-06T17:09:13.0035582Z cuda eval  detectron2_fcos_r_50_fpn           
2023-09-06T17:09:28.7910842Z skipping cudagraphs for unknown reason
2023-09-06T17:09:31.2894780Z skipping cudagraphs due to input mutation
2023-09-06T17:10:28.4311986Z [2023-09-06 17:10:28,428] torch._dynamo.convert_frame: [WARNING] torch._dynamo hit config.cache_size_limit (8)
2023-09-06T17:10:28.4313758Z [2023-09-06 17:10:28,428] torch._dynamo.convert_frame: [WARNING]    function: 'forward' (/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/batch_norm.py:318)
2023-09-06T17:10:28.4316133Z [2023-09-06 17:10:28,428] torch._dynamo.convert_frame: [WARNING] to diagnose recompilation issues, set env variable TORCHDYNAMO_REPORT_GUARD_FAILURES=1 and also see https://pytorch.org/docs/master/compile/troubleshooting.html.
2023-09-06T17:10:50.6886604Z skipping cudagraphs due to input mutation
2023-09-06T17:10:51.0841011Z skipping cudagraphs for unknown reason
2023-09-06T17:10:52.6186111Z skipping cudagraphs due to input mutation
2023-09-06T17:10:52.8237479Z skipping cudagraphs for unknown reason
2023-09-06T17:10:54.0518097Z skipping cudagraphs due to input mutation
2023-09-06T17:10:55.3938539Z skipping cudagraphs due to input mutation
2023-09-06T17:11:44.8416113Z skipping cudagraphs due to input mutation
2023-09-06T17:12:15.7467961Z 
2023-09-06T17:12:15.8880889Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T17:12:16.0401109Z running benchmark:   3% 1/30 [00:00<00:04,  7.06it/s]
2023-09-06T17:12:16.1786028Z running benchmark:   7% 2/30 [00:00<00:04,  6.77it/s]
2023-09-06T17:12:16.3174657Z running benchmark:  10% 3/30 [00:00<00:03,  6.97it/s]
2023-09-06T17:12:16.4550825Z running benchmark:  13% 4/30 [00:00<00:03,  7.06it/s]
2023-09-06T17:12:16.5932512Z running benchmark:  17% 5/30 [00:00<00:03,  7.13it/s]
2023-09-06T17:12:16.7302430Z running benchmark:  20% 6/30 [00:00<00:03,  7.17it/s]
2023-09-06T17:12:16.8691244Z running benchmark:  23% 7/30 [00:00<00:03,  7.21it/s]
2023-09-06T17:12:17.0076801Z running benchmark:  27% 8/30 [00:01<00:03,  7.21it/s]
2023-09-06T17:12:17.1470948Z running benchmark:  30% 9/30 [00:01<00:02,  7.21it/s]
2023-09-06T17:12:17.2830660Z running benchmark:  33% 10/30 [00:01<00:02,  7.20it/s]
2023-09-06T17:12:17.4181185Z running benchmark:  37% 11/30 [00:01<00:02,  7.24it/s]
2023-09-06T17:12:17.5560963Z running benchmark:  40% 12/30 [00:01<00:02,  7.29it/s]
2023-09-06T17:12:17.6944667Z running benchmark:  43% 13/30 [00:01<00:02,  7.28it/s]
2023-09-06T17:12:17.8324718Z running benchmark:  47% 14/30 [00:01<00:02,  7.26it/s]
2023-09-06T17:12:17.9693487Z running benchmark:  50% 15/30 [00:02<00:02,  7.26it/s]
2023-09-06T17:12:18.1061199Z running benchmark:  53% 16/30 [00:02<00:01,  7.27it/s]
2023-09-06T17:12:18.2432255Z running benchmark:  57% 17/30 [00:02<00:01,  7.29it/s]
2023-09-06T17:12:18.3799474Z running benchmark:  60% 18/30 [00:02<00:01,  7.29it/s]
2023-09-06T17:12:18.5169805Z running benchmark:  63% 19/30 [00:02<00:01,  7.29it/s]
2023-09-06T17:12:18.6559368Z running benchmark:  67% 20/30 [00:02<00:01,  7.30it/s]
2023-09-06T17:12:18.7938138Z running benchmark:  70% 21/30 [00:02<00:01,  7.27it/s]
2023-09-06T17:12:18.9309767Z running benchmark:  73% 22/30 [00:03<00:01,  7.26it/s]
2023-09-06T17:12:19.0667122Z running benchmark:  77% 23/30 [00:03<00:00,  7.27it/s]
2023-09-06T17:12:19.2040094Z running benchmark:  80% 24/30 [00:03<00:00,  7.30it/s]
2023-09-06T17:12:19.3482182Z running benchmark:  83% 25/30 [00:03<00:00,  7.29it/s]
2023-09-06T17:12:19.4922609Z running benchmark:  87% 26/30 [00:03<00:00,  7.18it/s]
2023-09-06T17:12:19.6301588Z running benchmark:  90% 27/30 [00:03<00:00,  7.11it/s]
2023-09-06T17:12:19.7669629Z running benchmark:  93% 28/30 [00:03<00:00,  7.15it/s]
2023-09-06T17:12:19.9041708Z running benchmark:  97% 29/30 [00:04<00:00,  7.20it/s]
2023-09-06T17:12:19.9042359Z running benchmark: 100% 30/30 [00:04<00:00,  7.22it/s]
2023-09-06T17:12:19.9042687Z running benchmark: 100% 30/30 [00:04<00:00,  7.21it/s]
2023-09-06T17:12:19.9081857Z 1.178x
2023-09-06T17:12:22.3282943Z speedup             gmean=0.00x mean=1.439x
2023-09-06T17:12:22.3283922Z abs_latency         gmean=0.00x mean=23.457x
2023-09-06T17:12:22.3284810Z compilation_latency mean=54.750 seconds
2023-09-06T17:12:22.3285358Z compression_ratio   mean=0.503x
2023-09-06T17:12:22.3285879Z eager_peak_mem      gmean=0.00x mean=0.723x
2023-09-06T17:12:22.3287385Z dynamo_peak_mem     gmean=0.00x mean=1.055x
2023-09-06T17:12:22.3289247Z calls_captured      gmean=0.00x mean=419.267x
2023-09-06T17:12:22.3290847Z unique_graphs       gmean=0.00x mean=5.333x
2023-09-06T17:12:22.3292553Z graph_breaks        gmean=0.00x mean=3.267x
2023-09-06T17:12:22.3294593Z unique_graph_breaks gmean=0.00x mean=0.733x
2023-09-06T17:12:23.0644943Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *aotinductor-true* ]]
2023-09-06T17:12:23.0645955Z + [[ inference == \i\n\f\e\r\e\n\c\e ]]
2023-09-06T17:12:23.0647137Z + python benchmarks/dynamo/torchbench.py --performance --cold-start-latency --inference --bfloat16 --export-aot-inductor --disable-cudagraphs --device cuda --total-partitions 4 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inductor_aot_inductor_torchbench_bfloat16_inference_cuda_performance.csv
2023-09-06T17:12:30.3748002Z 
2023-09-06T17:12:31.2772752Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:12:31.2773338Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:12:31.2773828Z Eager model failed to run
2023-09-06T17:12:31.2791389Z Traceback (most recent call last):
2023-09-06T17:12:31.2792131Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T17:12:31.2792787Z     self.model_iter_fn(model, example_inputs)
2023-09-06T17:12:31.2793412Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T17:12:31.2796046Z     return mod(*inputs)
2023-09-06T17:12:31.2797188Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:12:31.2798059Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:12:31.2799148Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:12:31.2799923Z     return forward_call(*args, **kwargs)
2023-09-06T17:12:31.2800969Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 893, in forward
2023-09-06T17:12:31.2801825Z     logits = self.model(batch.dense_features, batch.sparse_features)
2023-09-06T17:12:31.2803042Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:12:31.2803795Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:12:31.2804830Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:12:31.2805576Z     return forward_call(*args, **kwargs)
2023-09-06T17:12:31.2806506Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 570, in forward
2023-09-06T17:12:31.2807912Z     embedded_dense = self.dense_arch(dense_features)
2023-09-06T17:12:31.2809054Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:12:31.2809831Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:12:31.2810838Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:12:31.2811578Z     return forward_call(*args, **kwargs)
2023-09-06T17:12:31.2812590Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 149, in forward
2023-09-06T17:12:31.2813314Z     return self.model(features)
2023-09-06T17:12:31.2814385Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:12:31.2815173Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:12:31.2816223Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:12:31.2816983Z     return forward_call(*args, **kwargs)
2023-09-06T17:12:31.2818059Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/mlp.py", line 172, in forward
2023-09-06T17:12:31.2818772Z     return self._mlp(input)
2023-09-06T17:12:31.2819773Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:12:31.2820511Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:12:31.2821545Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:12:31.2822293Z     return forward_call(*args, **kwargs)
2023-09-06T17:12:31.2823267Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/container.py", line 215, in forward
2023-09-06T17:12:31.2823806Z     input = module(input)
2023-09-06T17:12:31.2824713Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:12:31.2825748Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:12:31.2826637Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:12:31.2827230Z     return forward_call(*args, **kwargs)
2023-09-06T17:12:31.2828168Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/mlp.py", line 73, in forward
2023-09-06T17:12:31.2828808Z     return self._activation_fn(self._linear(input))
2023-09-06T17:12:31.2830022Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:12:31.2830752Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:12:31.2831620Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:12:31.2832261Z     return forward_call(*args, **kwargs)
2023-09-06T17:12:31.2833198Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
2023-09-06T17:12:31.2833853Z     return F.linear(input, self.weight, self.bias)
2023-09-06T17:12:31.2834483Z RuntimeError: mat1 and mat2 must have the same dtype, but got Float and BFloat16
2023-09-06T17:12:31.2834863Z 
2023-09-06T17:12:31.2835183Z The above exception was the direct cause of the following exception:
2023-09-06T17:12:31.2835542Z 
2023-09-06T17:12:31.2835747Z Traceback (most recent call last):
2023-09-06T17:12:31.2836343Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T17:12:31.2836887Z     ) = runner.load_model(
2023-09-06T17:12:31.2837469Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T17:12:31.2838208Z     self.validate_model(model, example_inputs)
2023-09-06T17:12:31.2838856Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T17:12:31.2839936Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T17:12:31.2840547Z NotImplementedError: Eager model failed to run
2023-09-06T17:12:31.2840849Z 
2023-09-06T17:12:31.2841048Z WARNING:root:torchrec_dlrm failed to load
2023-09-06T17:12:35.7529967Z 
2023-09-06T17:12:39.0887580Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:12:39.0887940Z loading model: 0it [00:03, ?it/s]
2023-09-06T17:12:39.0939413Z cuda eval  BERT_pytorch                       
2023-09-06T17:12:40.8864606Z ERROR:common:Backend dynamo failed in warmup()
2023-09-06T17:12:40.8865005Z Traceback (most recent call last):
2023-09-06T17:12:40.8865400Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2288, in warmup
2023-09-06T17:12:40.8865773Z     fn(model, example_inputs)
2023-09-06T17:12:40.8866168Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1190, in opt_aot_inductor
2023-09-06T17:12:40.8866661Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-09-06T17:12:40.8867162Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1134, in load
2023-09-06T17:12:40.8870880Z     so_path, exported = torch._export.aot_compile(
2023-09-06T17:12:40.8873127Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 635, in aot_compile
2023-09-06T17:12:40.8873560Z     ep = export(f, args, kwargs, constraints)
2023-09-06T17:12:40.8874499Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 307, in export
2023-09-06T17:12:40.8875123Z     gm_torch_level, _ = torch._dynamo.export(
2023-09-06T17:12:40.8876143Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1150, in inner
2023-09-06T17:12:40.8876562Z     result_traced = opt_f(*args, **kwargs)
2023-09-06T17:12:40.8877160Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:12:40.8878133Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:12:40.8878697Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:12:40.8879094Z     return forward_call(*args, **kwargs)
2023-09-06T17:12:40.8879628Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T17:12:40.8880004Z     return fn(*args, **kwargs)
2023-09-06T17:12:40.8880561Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:12:40.8880965Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:12:40.8881514Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:12:40.8881917Z     return forward_call(*args, **kwargs)
2023-09-06T17:12:40.8882469Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T17:12:40.8883007Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T17:12:40.8883565Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T17:12:40.8883947Z     return fn(*args, **kwargs)
2023-09-06T17:12:40.8884523Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T17:12:40.8884972Z     return _compile(
2023-09-06T17:12:40.8885497Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T17:12:40.8885964Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T17:12:40.8886543Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T17:12:40.8886929Z     r = func(*args, **kwargs)
2023-09-06T17:12:40.8887695Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T17:12:40.8888154Z     out_code = transform_code_object(code, transform)
2023-09-06T17:12:40.8888789Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T17:12:40.8889265Z     transformations(instructions, code_options)
2023-09-06T17:12:40.8889845Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T17:12:40.8890217Z     tracer.run()
2023-09-06T17:12:40.8890734Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T17:12:40.8891088Z     super().run()
2023-09-06T17:12:40.8891603Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T17:12:40.8891975Z     and self.step()
2023-09-06T17:12:40.8892515Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T17:12:40.8892955Z     getattr(self, inst.opname)(inst)
2023-09-06T17:12:40.8893512Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T17:12:40.8893914Z     return inner_fn(self, inst)
2023-09-06T17:12:40.8894476Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1119, in CALL_FUNCTION
2023-09-06T17:12:40.8894877Z     self.call_function(fn, args, {})
2023-09-06T17:12:40.8895442Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 565, in call_function
2023-09-06T17:12:40.8895888Z     self.push(fn.call_function(self, args, kwargs))
2023-09-06T17:12:40.8896491Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-09-06T17:12:40.8897100Z     return super().call_function(tx, args, kwargs)
2023-09-06T17:12:40.8897689Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-09-06T17:12:40.8898127Z     return super().call_function(tx, args, kwargs)
2023-09-06T17:12:40.8898723Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-09-06T17:12:40.8899176Z     return tx.inline_user_function_return(
2023-09-06T17:12:40.8899771Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 601, in inline_user_function_return
2023-09-06T17:12:40.8900313Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-09-06T17:12:40.8900976Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2183, in inline_call
2023-09-06T17:12:40.8901421Z     return cls.inline_call_(parent, func, args, kwargs)
2023-09-06T17:12:40.8902028Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2290, in inline_call_
2023-09-06T17:12:40.8902393Z     tracer.run()
2023-09-06T17:12:40.8902948Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T17:12:40.8903322Z     and self.step()
2023-09-06T17:12:40.8903846Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T17:12:40.8904234Z     getattr(self, inst.opname)(inst)
2023-09-06T17:12:40.8904790Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T17:12:40.8905191Z     return inner_fn(self, inst)
2023-09-06T17:12:40.8905757Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1119, in CALL_FUNCTION
2023-09-06T17:12:40.8906175Z     self.call_function(fn, args, {})
2023-09-06T17:12:40.8906848Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 565, in call_function
2023-09-06T17:12:40.8907296Z     self.push(fn.call_function(self, args, kwargs))
2023-09-06T17:12:40.8907899Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-09-06T17:12:40.8908335Z     return super().call_function(tx, args, kwargs)
2023-09-06T17:12:40.8908912Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-09-06T17:12:40.8909523Z     return super().call_function(tx, args, kwargs)
2023-09-06T17:12:40.8910131Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-09-06T17:12:40.8910560Z     return tx.inline_user_function_return(
2023-09-06T17:12:40.8911186Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 601, in inline_user_function_return
2023-09-06T17:12:40.8911717Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-09-06T17:12:40.8912381Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2183, in inline_call
2023-09-06T17:12:40.8912864Z     return cls.inline_call_(parent, func, args, kwargs)
2023-09-06T17:12:40.8913462Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2290, in inline_call_
2023-09-06T17:12:40.8913844Z     tracer.run()
2023-09-06T17:12:40.8914348Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T17:12:40.8914725Z     and self.step()
2023-09-06T17:12:40.8915247Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T17:12:40.8915648Z     getattr(self, inst.opname)(inst)
2023-09-06T17:12:40.8916368Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1207, in STORE_ATTR
2023-09-06T17:12:40.8916739Z     assert (
2023-09-06T17:12:40.8917094Z AssertionError: Mutating module attribute mask during export.
2023-09-06T17:12:40.8917450Z 
2023-09-06T17:12:40.8917589Z from user code:
2023-09-06T17:12:40.8918203Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch/bert_pytorch/model/bert.py", line 47, in forward
2023-09-06T17:12:40.8918895Z     x = transformer.forward(x, mask)
2023-09-06T17:12:40.8919619Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch/bert_pytorch/model/transformer.py", line 46, in forward
2023-09-06T17:12:40.8920315Z     self.lambda_module.set_mask(mask)
2023-09-06T17:12:40.8921040Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch/bert_pytorch/model/transformer.py", line 17, in set_mask
2023-09-06T17:12:40.8921685Z     self.mask = mask
2023-09-06T17:12:40.8921977Z 
2023-09-06T17:12:40.8922276Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T17:12:40.8922732Z 
2023-09-06T17:12:41.9771725Z ERROR
2023-09-06T17:12:45.5234832Z 
2023-09-06T17:12:49.2229019Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:12:49.2230697Z loading model: 0it [00:03, ?it/s]
2023-09-06T17:12:49.2286521Z cuda eval  Background_Matting                 
2023-09-06T17:13:40.4361630Z 
2023-09-06T17:13:40.5439146Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T17:13:40.6486854Z running benchmark:  10% 3/30 [00:00<00:00, 27.73it/s]
2023-09-06T17:13:40.7534619Z running benchmark:  23% 7/30 [00:00<00:00, 33.78it/s]
2023-09-06T17:13:40.8578640Z running benchmark:  37% 11/30 [00:00<00:00, 35.77it/s]
2023-09-06T17:13:40.9620440Z running benchmark:  50% 15/30 [00:00<00:00, 36.76it/s]
2023-09-06T17:13:41.0796632Z running benchmark:  63% 19/30 [00:00<00:00, 37.35it/s]
2023-09-06T17:13:41.1839774Z running benchmark:  77% 23/30 [00:00<00:00, 36.14it/s]
2023-09-06T17:13:41.2624342Z running benchmark:  90% 27/30 [00:00<00:00, 36.82it/s]
2023-09-06T17:13:41.2624756Z running benchmark: 100% 30/30 [00:00<00:00, 36.28it/s]
2023-09-06T17:13:41.2643266Z 1.625x
2023-09-06T17:13:46.7061971Z 
2023-09-06T17:13:56.3194329Z loading model: 0it [00:00, ?it/s]WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T17:13:57.9420731Z 
2023-09-06T17:13:57.9421300Z loading model: 0it [00:11, ?it/s]
2023-09-06T17:13:57.9422031Z WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T17:13:57.9426563Z cuda eval  DALLE2_pytorch                     
2023-09-06T17:13:58.0982056Z WARNING:common:Model DALLE2_pytorch does not support bfloat16, running with amp instead
2023-09-06T17:14:01.9569265Z ERROR:common:Backend dynamo failed in warmup()
2023-09-06T17:14:01.9570176Z Traceback (most recent call last):
2023-09-06T17:14:01.9570618Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2288, in warmup
2023-09-06T17:14:01.9571006Z     fn(model, example_inputs)
2023-09-06T17:14:01.9571406Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1190, in opt_aot_inductor
2023-09-06T17:14:01.9571886Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-09-06T17:14:01.9572377Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1134, in load
2023-09-06T17:14:01.9572862Z     so_path, exported = torch._export.aot_compile(
2023-09-06T17:14:01.9575037Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 635, in aot_compile
2023-09-06T17:14:01.9575812Z     ep = export(f, args, kwargs, constraints)
2023-09-06T17:14:01.9576815Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 307, in export
2023-09-06T17:14:01.9577235Z     gm_torch_level, _ = torch._dynamo.export(
2023-09-06T17:14:01.9577821Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1150, in inner
2023-09-06T17:14:01.9578717Z     result_traced = opt_f(*args, **kwargs)
2023-09-06T17:14:01.9581548Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:14:01.9582309Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:14:01.9583460Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:14:01.9584257Z     return forward_call(*args, **kwargs)
2023-09-06T17:14:01.9585332Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T17:14:01.9586091Z     return fn(*args, **kwargs)
2023-09-06T17:14:01.9587227Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T17:14:01.9588014Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T17:14:01.9589022Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T17:14:01.9590042Z     return fn(*args, **kwargs)
2023-09-06T17:14:01.9591271Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T17:14:01.9591781Z     return _compile(
2023-09-06T17:14:01.9592343Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T17:14:01.9592828Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T17:14:01.9593540Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T17:14:01.9593911Z     r = func(*args, **kwargs)
2023-09-06T17:14:01.9594464Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T17:14:01.9595310Z     out_code = transform_code_object(code, transform)
2023-09-06T17:14:01.9596017Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T17:14:01.9596504Z     transformations(instructions, code_options)
2023-09-06T17:14:01.9597077Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T17:14:01.9597454Z     tracer.run()
2023-09-06T17:14:01.9597982Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T17:14:01.9598358Z     super().run()
2023-09-06T17:14:01.9598862Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T17:14:01.9599235Z     and self.step()
2023-09-06T17:14:01.9599755Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T17:14:01.9600163Z     getattr(self, inst.opname)(inst)
2023-09-06T17:14:01.9600707Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T17:14:01.9601107Z     return inner_fn(self, inst)
2023-09-06T17:14:01.9601691Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1159, in CALL_FUNCTION_EX
2023-09-06T17:14:01.9602172Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-09-06T17:14:01.9602802Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 565, in call_function
2023-09-06T17:14:01.9603280Z     self.push(fn.call_function(self, args, kwargs))
2023-09-06T17:14:01.9603899Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 336, in call_function
2023-09-06T17:14:01.9604334Z     return tx.inline_user_function_return(
2023-09-06T17:14:01.9604968Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 601, in inline_user_function_return
2023-09-06T17:14:01.9605699Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-09-06T17:14:01.9606512Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2183, in inline_call
2023-09-06T17:14:01.9607187Z     return cls.inline_call_(parent, func, args, kwargs)
2023-09-06T17:14:01.9608183Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2290, in inline_call_
2023-09-06T17:14:01.9608777Z     tracer.run()
2023-09-06T17:14:01.9609573Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T17:14:01.9610151Z     and self.step()
2023-09-06T17:14:01.9610987Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T17:14:01.9611757Z     getattr(self, inst.opname)(inst)
2023-09-06T17:14:01.9612638Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T17:14:01.9613333Z     return inner_fn(self, inst)
2023-09-06T17:14:01.9614213Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1159, in CALL_FUNCTION_EX
2023-09-06T17:14:01.9614935Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-09-06T17:14:01.9615893Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 565, in call_function
2023-09-06T17:14:01.9616609Z     self.push(fn.call_function(self, args, kwargs))
2023-09-06T17:14:01.9617295Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-09-06T17:14:01.9617740Z     return super().call_function(tx, args, kwargs)
2023-09-06T17:14:01.9618660Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-09-06T17:14:01.9619119Z     return super().call_function(tx, args, kwargs)
2023-09-06T17:14:01.9619726Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-09-06T17:14:01.9620140Z     return tx.inline_user_function_return(
2023-09-06T17:14:01.9620758Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 601, in inline_user_function_return
2023-09-06T17:14:01.9621606Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-09-06T17:14:01.9622781Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2183, in inline_call
2023-09-06T17:14:01.9623530Z     return cls.inline_call_(parent, func, args, kwargs)
2023-09-06T17:14:01.9624412Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2290, in inline_call_
2023-09-06T17:14:01.9624976Z     tracer.run()
2023-09-06T17:14:01.9625726Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T17:14:01.9626259Z     and self.step()
2023-09-06T17:14:01.9627007Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T17:14:01.9627580Z     getattr(self, inst.opname)(inst)
2023-09-06T17:14:01.9628403Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T17:14:01.9628975Z     return inner_fn(self, inst)
2023-09-06T17:14:01.9630003Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1159, in CALL_FUNCTION_EX
2023-09-06T17:14:01.9630673Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-09-06T17:14:01.9631565Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 565, in call_function
2023-09-06T17:14:01.9632487Z     self.push(fn.call_function(self, args, kwargs))
2023-09-06T17:14:01.9633435Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-09-06T17:14:01.9634075Z     return super().call_function(tx, args, kwargs)
2023-09-06T17:14:01.9634926Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-09-06T17:14:01.9635532Z     return tx.inline_user_function_return(
2023-09-06T17:14:01.9636395Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 601, in inline_user_function_return
2023-09-06T17:14:01.9637142Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-09-06T17:14:01.9638087Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2183, in inline_call
2023-09-06T17:14:01.9638715Z     return cls.inline_call_(parent, func, args, kwargs)
2023-09-06T17:14:01.9639549Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2290, in inline_call_
2023-09-06T17:14:01.9640062Z     tracer.run()
2023-09-06T17:14:01.9640748Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T17:14:01.9641271Z     and self.step()
2023-09-06T17:14:01.9642020Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T17:14:01.9642592Z     getattr(self, inst.opname)(inst)
2023-09-06T17:14:01.9643434Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T17:14:01.9644019Z     return inner_fn(self, inst)
2023-09-06T17:14:01.9645095Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1119, in CALL_FUNCTION
2023-09-06T17:14:01.9645696Z     self.call_function(fn, args, {})
2023-09-06T17:14:01.9646528Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 565, in call_function
2023-09-06T17:14:01.9647128Z     self.push(fn.call_function(self, args, kwargs))
2023-09-06T17:14:01.9647988Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 304, in call_function
2023-09-06T17:14:01.9648566Z     return self.obj.call_method(
2023-09-06T17:14:01.9649366Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 601, in call_method
2023-09-06T17:14:01.9649967Z     return super().call_method(tx, name, args, kwargs)
2023-09-06T17:14:01.9650814Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/base.py", line 329, in call_method
2023-09-06T17:14:01.9651520Z     raise unimplemented(f"call_method {self} {name} {args} {kwargs}")
2023-09-06T17:14:01.9652451Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 176, in unimplemented
2023-09-06T17:14:01.9653118Z     raise Unsupported(msg)
2023-09-06T17:14:01.9653598Z torch._dynamo.exc.Unsupported: call_method NNModuleVariable() eval [] {}
2023-09-06T17:14:01.9653956Z 
2023-09-06T17:14:01.9654084Z from user code:
2023-09-06T17:14:01.9654796Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/external_utils.py", line 17, in inner
2023-09-06T17:14:01.9655329Z     return fn(*args, **kwargs)
2023-09-06T17:14:01.9656056Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:14:01.9656609Z     return forward_call(*args, **kwargs)
2023-09-06T17:14:01.9657376Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2023-09-06T17:14:01.9657913Z     return func(*args, **kwargs)
2023-09-06T17:14:01.9659021Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 95, in inner
2023-09-06T17:14:01.9659545Z     model.eval()
2023-09-06T17:14:01.9659762Z 
2023-09-06T17:14:01.9660043Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T17:14:01.9660381Z 
2023-09-06T17:14:03.4437797Z ERROR
2023-09-06T17:14:07.0460780Z 
2023-09-06T17:14:11.4878767Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:14:11.4879110Z loading model: 0it [00:04, ?it/s]
2023-09-06T17:14:11.4898835Z cuda eval  LearningToPaint                    
2023-09-06T17:14:44.3242115Z 
2023-09-06T17:14:44.4286585Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T17:14:44.5342620Z running benchmark:  30% 9/30 [00:00<00:00, 86.18it/s]
2023-09-06T17:14:44.6389936Z running benchmark:  60% 18/30 [00:00<00:00, 85.55it/s]
2023-09-06T17:14:44.6600641Z running benchmark:  93% 28/30 [00:00<00:00, 90.09it/s]
2023-09-06T17:14:44.6601404Z running benchmark: 100% 30/30 [00:00<00:00, 89.26it/s]
2023-09-06T17:14:44.6619946Z 1.825x
2023-09-06T17:14:49.9724638Z 
2023-09-06T17:14:52.4533762Z loading model: 0it [00:00, ?it/s]WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T17:14:52.9871876Z 
2023-09-06T17:14:52.9872392Z loading model: 0it [00:03, ?it/s]
2023-09-06T17:14:52.9872867Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T17:14:52.9873257Z cuda eval  Super_SloMo                        
2023-09-06T17:14:52.9984611Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2023-09-06T17:16:15.0946955Z 
2023-09-06T17:16:15.3277308Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T17:16:15.5090316Z running benchmark:   3% 1/30 [00:00<00:06,  4.28it/s]
2023-09-06T17:16:15.6910102Z running benchmark:  10% 3/30 [00:00<00:03,  7.83it/s]
2023-09-06T17:16:15.8722880Z running benchmark:  17% 5/30 [00:00<00:02,  9.19it/s]
2023-09-06T17:16:16.0538986Z running benchmark:  23% 7/30 [00:00<00:02,  9.89it/s]
2023-09-06T17:16:16.2351281Z running benchmark:  30% 9/30 [00:00<00:02, 10.29it/s]
2023-09-06T17:16:16.4163517Z running benchmark:  37% 11/30 [00:01<00:01, 10.53it/s]
2023-09-06T17:16:16.5976467Z running benchmark:  43% 13/30 [00:01<00:01, 10.70it/s]
2023-09-06T17:16:16.7791824Z running benchmark:  50% 15/30 [00:01<00:01, 10.80it/s]
2023-09-06T17:16:16.9605195Z running benchmark:  57% 17/30 [00:01<00:01, 10.87it/s]
2023-09-06T17:16:17.1419264Z running benchmark:  63% 19/30 [00:01<00:01, 10.92it/s]
2023-09-06T17:16:17.3232034Z running benchmark:  70% 21/30 [00:02<00:00, 10.95it/s]
2023-09-06T17:16:17.5042601Z running benchmark:  77% 23/30 [00:02<00:00, 10.98it/s]
2023-09-06T17:16:17.6939829Z running benchmark:  83% 25/30 [00:02<00:00, 11.00it/s]
2023-09-06T17:16:17.8751526Z running benchmark:  90% 27/30 [00:02<00:00, 10.86it/s]
2023-09-06T17:16:17.9661175Z running benchmark:  97% 29/30 [00:02<00:00, 10.91it/s]
2023-09-06T17:16:17.9684595Z running benchmark: 100% 30/30 [00:02<00:00, 10.45it/s]
2023-09-06T17:16:17.9684901Z 0.954x
2023-09-06T17:16:23.8820695Z 
2023-09-06T17:16:27.3935514Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:16:27.3936052Z loading model: 0it [00:03, ?it/s]
2023-09-06T17:16:27.3939591Z cuda eval  alexnet                            
2023-09-06T17:16:51.3361964Z 
2023-09-06T17:16:51.4472800Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T17:16:51.5802638Z running benchmark:   3% 1/30 [00:00<00:03,  9.02it/s]
2023-09-06T17:16:51.7088078Z running benchmark:  13% 4/30 [00:00<00:01, 17.55it/s]
2023-09-06T17:16:51.8377578Z running benchmark:  23% 7/30 [00:00<00:01, 20.24it/s]
2023-09-06T17:16:51.9663586Z running benchmark:  33% 10/30 [00:00<00:00, 21.45it/s]
2023-09-06T17:16:52.0953844Z running benchmark:  43% 13/30 [00:00<00:00, 22.13it/s]
2023-09-06T17:16:52.2243027Z running benchmark:  53% 16/30 [00:00<00:00, 22.52it/s]
2023-09-06T17:16:52.3532352Z running benchmark:  63% 19/30 [00:00<00:00, 22.76it/s]
2023-09-06T17:16:52.4815700Z running benchmark:  73% 22/30 [00:01<00:00, 22.93it/s]
2023-09-06T17:16:52.6163459Z running benchmark:  83% 25/30 [00:01<00:00, 23.06it/s]
2023-09-06T17:16:52.7023307Z running benchmark:  93% 28/30 [00:01<00:00, 22.81it/s]
2023-09-06T17:16:52.7023777Z running benchmark: 100% 30/30 [00:01<00:00, 21.96it/s]
2023-09-06T17:16:52.7044020Z 1.426x
2023-09-06T17:16:57.6419271Z 
2023-09-06T17:17:04.5167065Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:17:04.5168015Z loading model: 0it [00:06, ?it/s]
2023-09-06T17:17:04.5212072Z cuda eval  attention_is_all_you_need_pytorch  
2023-09-06T17:18:15.0162316Z 
2023-09-06T17:18:15.1230852Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T17:18:15.2235379Z running benchmark:  10% 3/30 [00:00<00:00, 27.96it/s]
2023-09-06T17:18:15.3467175Z running benchmark:  23% 7/30 [00:00<00:00, 34.70it/s]
2023-09-06T17:18:15.4504011Z running benchmark:  40% 12/30 [00:00<00:00, 37.66it/s]
2023-09-06T17:18:15.5523365Z running benchmark:  53% 16/30 [00:00<00:00, 38.02it/s]
2023-09-06T17:18:15.6535934Z running benchmark:  67% 20/30 [00:00<00:00, 38.43it/s]
2023-09-06T17:18:15.7546666Z running benchmark:  80% 24/30 [00:00<00:00, 38.78it/s]
2023-09-06T17:18:15.8048241Z running benchmark:  93% 28/30 [00:00<00:00, 39.03it/s]
2023-09-06T17:18:15.8048610Z running benchmark: 100% 30/30 [00:00<00:00, 38.01it/s]
2023-09-06T17:18:15.8073365Z 1.600x
2023-09-06T17:18:21.3537426Z 
2023-09-06T17:18:24.1368416Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:18:24.1369209Z loading model: 0it [00:02, ?it/s]
2023-09-06T17:18:24.1373274Z cuda eval  basic_gnn_edgecnn                  
2023-09-06T17:18:24.6259263Z [2023-09-06 17:18:24,624] [0/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T17:18:45.4340527Z 
2023-09-06T17:18:45.5370083Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T17:18:45.5669515Z running benchmark:  77% 23/30 [00:00<00:00, 222.80it/s]
2023-09-06T17:18:45.5670331Z running benchmark: 100% 30/30 [00:00<00:00, 225.13it/s]
2023-09-06T17:18:45.5686253Z 1.905x
2023-09-06T17:18:50.5305875Z 
2023-09-06T17:18:53.2764131Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:18:53.2764934Z loading model: 0it [00:02, ?it/s]
2023-09-06T17:18:53.2766898Z cuda eval  basic_gnn_gcn                      
2023-09-06T17:18:53.9086456Z [2023-09-06 17:18:53,907] [0/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T17:18:56.2785783Z ERROR:common:Backend dynamo failed in warmup()
2023-09-06T17:18:56.2786332Z Traceback (most recent call last):
2023-09-06T17:18:56.2787062Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2288, in warmup
2023-09-06T17:18:56.2787489Z     fn(model, example_inputs)
2023-09-06T17:18:56.2787868Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1190, in opt_aot_inductor
2023-09-06T17:18:56.2788358Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-09-06T17:18:56.2791054Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1134, in load
2023-09-06T17:18:56.2791499Z     so_path, exported = torch._export.aot_compile(
2023-09-06T17:18:56.2792326Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 645, in aot_compile
2023-09-06T17:18:56.2792818Z     so_path = torch._inductor.aot_compile(ep.graph_module, list(all_args), options)
2023-09-06T17:18:56.2793539Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/__init__.py", line 48, in aot_compile
2023-09-06T17:18:56.2793965Z     result = compile_fx_aot(
2023-09-06T17:18:56.2797039Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 874, in compile_fx_aot
2023-09-06T17:18:56.2798349Z     return compile_fx(
2023-09-06T17:18:56.2798994Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 975, in compile_fx
2023-09-06T17:18:56.2799383Z     return compile_fx(
2023-09-06T17:18:56.2799924Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 995, in compile_fx
2023-09-06T17:18:56.2800298Z     return compile_fx(
2023-09-06T17:18:56.2800881Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1167, in compile_fx
2023-09-06T17:18:56.2801269Z     return aot_autograd(
2023-09-06T17:18:56.2801816Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 55, in compiler_fn
2023-09-06T17:18:56.2802281Z     cg = aot_module_simplified(gm, example_inputs, **kwargs)
2023-09-06T17:18:56.2802984Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3891, in aot_module_simplified
2023-09-06T17:18:56.2803443Z     compiled_fn = create_aot_dispatcher_function(
2023-09-06T17:18:56.2804007Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T17:18:56.2804384Z     r = func(*args, **kwargs)
2023-09-06T17:18:56.2804986Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3379, in create_aot_dispatcher_function
2023-09-06T17:18:56.2805461Z     fw_metadata = run_functionalized_fw_and_collect_metadata(
2023-09-06T17:18:56.2806046Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 757, in inner
2023-09-06T17:18:56.2806437Z     flat_f_outs = f(*flat_f_args)
2023-09-06T17:18:56.2807005Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3496, in functional_call
2023-09-06T17:18:56.2807685Z     out = Interpreter(mod).run(*args[params_len:], **kwargs)
2023-09-06T17:18:56.2808268Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 138, in run
2023-09-06T17:18:56.2808659Z     self.env[node] = self.run_node(node)
2023-09-06T17:18:56.2809198Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 195, in run_node
2023-09-06T17:18:56.2809624Z     return getattr(self, n.op)(n.target, args, kwargs)
2023-09-06T17:18:56.2810191Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/interpreter.py", line 267, in call_function
2023-09-06T17:18:56.2810583Z     return target(*args, **kwargs)
2023-09-06T17:18:56.2811082Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 448, in __call__
2023-09-06T17:18:56.2811464Z     return self._op(*args, **kwargs or {})
2023-09-06T17:18:56.2811982Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_stats.py", line 20, in wrapper
2023-09-06T17:18:56.2812362Z     return fn(*args, **kwargs)
2023-09-06T17:18:56.2812984Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1290, in __torch_dispatch__
2023-09-06T17:18:56.2813432Z     return self.dispatch(func, types, args, kwargs)
2023-09-06T17:18:56.2814021Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1532, in dispatch
2023-09-06T17:18:56.2814443Z     op_impl_out = op_impl(self, func, *args, **kwargs)
2023-09-06T17:18:56.2815030Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 634, in index_tensor
2023-09-06T17:18:56.2815444Z     out = meta_index_Tensor(*args, **kwargs)
2023-09-06T17:18:56.2816028Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_meta_registrations.py", line 2668, in meta_index_Tensor
2023-09-06T17:18:56.2816418Z     nonzero = index.nonzero()
2023-09-06T17:18:56.2817084Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_stats.py", line 20, in wrapper
2023-09-06T17:18:56.2817454Z     return fn(*args, **kwargs)
2023-09-06T17:18:56.2818032Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1290, in __torch_dispatch__
2023-09-06T17:18:56.2818478Z     return self.dispatch(func, types, args, kwargs)
2023-09-06T17:18:56.2819053Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1532, in dispatch
2023-09-06T17:18:56.2819485Z     op_impl_out = op_impl(self, func, *args, **kwargs)
2023-09-06T17:18:56.2820053Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 556, in nonzero
2023-09-06T17:18:56.2820491Z     raise DynamicOutputShapeException(func)
2023-09-06T17:18:56.2820927Z torch._subclasses.fake_tensor.DynamicOutputShapeException: aten.nonzero.default
2023-09-06T17:18:56.2821221Z 
2023-09-06T17:18:56.2821528Z While executing %index : [num_users=2] = call_function[target=torch.ops.aten.index.Tensor](args = (%slice_1, [None, %ne]), kwargs = {})
2023-09-06T17:18:56.2821951Z Original traceback:
2023-09-06T17:18:56.2822494Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/models/basic_gnn.py", line 222, in forward
2023-09-06T17:18:56.2823086Z     x = self.convs[i](x, edge_index, edge_weight=edge_weight)
2023-09-06T17:18:56.2824128Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:18:56.2824857Z     return forward_call(*args, **kwargs)
2023-09-06T17:18:56.2825377Z   File "/tmp/jenkins_pyg/tmp23gt9do0.py", line 221, in forward
2023-09-06T17:18:56.2825937Z     edge_index, edge_weight = gcn_norm(  # yapf: disable
2023-09-06T17:18:56.2826782Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/gcn_conv.py", line 91, in gcn_norm
2023-09-06T17:18:56.2827677Z     edge_index, edge_weight = add_remaining_self_loops(
2023-09-06T17:18:56.2828614Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/loop.py", line 370, in add_remaining_self_loops
2023-09-06T17:18:56.2829477Z     edge_index = torch.cat([edge_index[:, mask], loop_index], dim=1)
2023-09-06T17:18:56.2829803Z 
2023-09-06T17:18:57.5125293Z ERROR
2023-09-06T17:19:01.1050608Z 
2023-09-06T17:19:04.0421393Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:19:04.0421738Z loading model: 0it [00:02, ?it/s]
2023-09-06T17:19:04.0428726Z cuda eval  basic_gnn_gin                      
2023-09-06T17:19:04.5528215Z [2023-09-06 17:19:04,551] [0/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T17:19:11.1236892Z ERROR:common:Backend dynamo failed in warmup()
2023-09-06T17:19:11.1237864Z concurrent.futures.process._RemoteTraceback: 
2023-09-06T17:19:11.1238400Z """
2023-09-06T17:19:11.1238788Z Traceback (most recent call last):
2023-09-06T17:19:11.1241493Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/code_generator.py", line 1124, in ast_to_ttir
2023-09-06T17:19:11.1242327Z     generator.visit(fn.parse())
2023-09-06T17:19:11.1243301Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/code_generator.py", line 1017, in visit
2023-09-06T17:19:11.1244043Z     ret = super().visit(node)
2023-09-06T17:19:11.1244658Z   File "/opt/conda/envs/py_3.10/lib/python3.10/ast.py", line 418, in visit
2023-09-06T17:19:11.1245226Z     return visitor(node)
2023-09-06T17:19:11.1431068Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/code_generator.py", line 293, in visit_Module
2023-09-06T17:19:11.1431866Z     ast.NodeVisitor.generic_visit(self, node)
2023-09-06T17:19:11.1432549Z   File "/opt/conda/envs/py_3.10/lib/python3.10/ast.py", line 426, in generic_visit
2023-09-06T17:19:11.1433729Z     self.visit(item)
2023-09-06T17:19:11.1434575Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/code_generator.py", line 1017, in visit
2023-09-06T17:19:11.1435143Z     ret = super().visit(node)
2023-09-06T17:19:11.1435625Z   File "/opt/conda/envs/py_3.10/lib/python3.10/ast.py", line 418, in visit
2023-09-06T17:19:11.1436110Z     return visitor(node)
2023-09-06T17:19:11.1436971Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/code_generator.py", line 362, in visit_FunctionDef
2023-09-06T17:19:11.1437653Z     self.visit_compound_statement(node.body)
2023-09-06T17:19:11.1438722Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/code_generator.py", line 288, in visit_compound_statement
2023-09-06T17:19:11.1439477Z     ret_type = self.visit(stmt)
2023-09-06T17:19:11.1440423Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/code_generator.py", line 1017, in visit
2023-09-06T17:19:11.1441001Z     ret = super().visit(node)
2023-09-06T17:19:11.1441521Z   File "/opt/conda/envs/py_3.10/lib/python3.10/ast.py", line 418, in visit
2023-09-06T17:19:11.1441950Z     return visitor(node)
2023-09-06T17:19:11.1442730Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/code_generator.py", line 983, in visit_Expr
2023-09-06T17:19:11.1443346Z     ast.NodeVisitor.generic_visit(self, node)
2023-09-06T17:19:11.1443916Z   File "/opt/conda/envs/py_3.10/lib/python3.10/ast.py", line 428, in generic_visit
2023-09-06T17:19:11.1444369Z     self.visit(value)
2023-09-06T17:19:11.1445155Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/code_generator.py", line 1017, in visit
2023-09-06T17:19:11.1445697Z     ret = super().visit(node)
2023-09-06T17:19:11.1446166Z   File "/opt/conda/envs/py_3.10/lib/python3.10/ast.py", line 418, in visit
2023-09-06T17:19:11.1446636Z     return visitor(node)
2023-09-06T17:19:11.1448023Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/code_generator.py", line 946, in visit_Call
2023-09-06T17:19:11.1448679Z     return fn(*args, **extra_kwargs, **kws)
2023-09-06T17:19:11.1449495Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/language/core.py", line 30, in wrapper
2023-09-06T17:19:11.1450062Z     return fn(*args, **kwargs)
2023-09-06T17:19:11.1450985Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/language/core.py", line 1137, in atomic_add
2023-09-06T17:19:11.1451791Z     return semantic.atomic_add(pointer, val, mask, sem, _builder)
2023-09-06T17:19:11.1452810Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/language/semantic.py", line 1191, in atomic_add
2023-09-06T17:19:11.1453816Z     ptr, val, mask = atom_red_typechecking_impl(ptr, val, mask, 'add', builder)
2023-09-06T17:19:11.1454977Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/language/semantic.py", line 1087, in atom_red_typechecking_impl
2023-09-06T17:19:11.1455908Z     raise ValueError("atomic_" + op + " does not support " + str(element_ty))
2023-09-06T17:19:11.1456589Z ValueError: atomic_add does not support bf16
2023-09-06T17:19:11.1456933Z 
2023-09-06T17:19:11.1457286Z The above exception was the direct cause of the following exception:
2023-09-06T17:19:11.1457728Z 
2023-09-06T17:19:11.1458003Z Traceback (most recent call last):
2023-09-06T17:19:11.1458646Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/process.py", line 246, in _process_worker
2023-09-06T17:19:11.1459348Z     r = call_item.fn(*call_item.args, **call_item.kwargs)
2023-09-06T17:19:11.1460355Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1278, in _worker_compile
2023-09-06T17:19:11.1461012Z     kernel.precompile(warm_cache_only_with_cc=cc)
2023-09-06T17:19:11.1461911Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 168, in precompile
2023-09-06T17:19:11.1463040Z     self.launchers = [
2023-09-06T17:19:11.1463902Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 169, in <listcomp>
2023-09-06T17:19:11.1464555Z     self._precompile_config(c, warm_cache_only_with_cc)
2023-09-06T17:19:11.1465575Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 189, in _precompile_config
2023-09-06T17:19:11.1466169Z     triton.compile(
2023-09-06T17:19:11.1467005Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/compiler.py", line 476, in compile
2023-09-06T17:19:11.1467619Z     next_module = compile_kernel(module)
2023-09-06T17:19:11.1468677Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/compiler.py", line 381, in <lambda>
2023-09-06T17:19:11.1469810Z     lambda src: optimize_ttir(ast_to_ttir(src, signature, configs[0], constants, debug=debug, arch=arch), arch))
2023-09-06T17:19:11.1471056Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/code_generator.py", line 1133, in ast_to_ttir
2023-09-06T17:19:11.1471876Z     raise CompilationError(fn.src, node, repr(e)) from e
2023-09-06T17:19:11.1472691Z triton.compiler.errors.CompilationError: at 15:53:    xindex = xoffset + tl.arange(0, XBLOCK)[:]
2023-09-06T17:19:11.1473401Z     xmask = xindex < xnumel
2023-09-06T17:19:11.1473859Z     x1 = (xindex // 64)
2023-09-06T17:19:11.1474291Z     x0 = xindex % 64
2023-09-06T17:19:11.1475019Z     tmp0 = tl.load(in_ptr0 + (200000 + x1), None, eviction_policy='evict_last')
2023-09-06T17:19:11.1475833Z     tmp2 = tl.load(in_ptr0 + (x1), None, eviction_policy='evict_last')
2023-09-06T17:19:11.1476441Z     tmp1 = tl.where(tmp0 < 0, tmp0 + 10000, tmp0)
2023-09-06T17:19:11.1477168Z     tl.device_assert((0 <= tmp1) & (tmp1 < 10000), "index out of bounds: 0 <= tmp1 < 10000")
2023-09-06T17:19:11.1477856Z     tmp3 = tl.where(tmp2 < 0, tmp2 + 10000, tmp2)
2023-09-06T17:19:11.1478881Z     tl.device_assert((0 <= tmp3) & (tmp3 < 10000), "index out of bounds: 0 <= tmp3 < 10000")
2023-09-06T17:19:11.1479596Z     tmp4 = tl.load(in_ptr1 + (x0 + (64*tmp3)), None).to(tl.float32)
2023-09-06T17:19:11.1480235Z     tl.atomic_add(out_ptr0 + (x0 + (64*tmp1)), tmp4, None)
2023-09-06T17:19:11.1480792Z                                                      ^
2023-09-06T17:19:11.1481514Z ValueError('atomic_add does not support bf16')
2023-09-06T17:19:11.1481974Z """
2023-09-06T17:19:11.1482200Z 
2023-09-06T17:19:11.1482550Z The above exception was the direct cause of the following exception:
2023-09-06T17:19:11.1482984Z 
2023-09-06T17:19:11.1483196Z Traceback (most recent call last):
2023-09-06T17:19:11.1483858Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2288, in warmup
2023-09-06T17:19:11.1484480Z     fn(model, example_inputs)
2023-09-06T17:19:11.1485121Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1190, in opt_aot_inductor
2023-09-06T17:19:11.1485986Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-09-06T17:19:11.1486739Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1134, in load
2023-09-06T17:19:11.1487377Z     so_path, exported = torch._export.aot_compile(
2023-09-06T17:19:11.1488491Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 645, in aot_compile
2023-09-06T17:19:11.1489363Z     so_path = torch._inductor.aot_compile(ep.graph_module, list(all_args), options)
2023-09-06T17:19:11.1490493Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/__init__.py", line 48, in aot_compile
2023-09-06T17:19:11.1491156Z     result = compile_fx_aot(
2023-09-06T17:19:11.1492174Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 874, in compile_fx_aot
2023-09-06T17:19:11.1492823Z     return compile_fx(
2023-09-06T17:19:11.1493638Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 975, in compile_fx
2023-09-06T17:19:11.1494470Z     return compile_fx(
2023-09-06T17:19:11.1495291Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 995, in compile_fx
2023-09-06T17:19:11.1495895Z     return compile_fx(
2023-09-06T17:19:11.1496719Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1167, in compile_fx
2023-09-06T17:19:11.1497271Z     return aot_autograd(
2023-09-06T17:19:11.1498105Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 55, in compiler_fn
2023-09-06T17:19:11.1498765Z     cg = aot_module_simplified(gm, example_inputs, **kwargs)
2023-09-06T17:19:11.1499673Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3891, in aot_module_simplified
2023-09-06T17:19:11.1500360Z     compiled_fn = create_aot_dispatcher_function(
2023-09-06T17:19:11.1501153Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T17:19:11.1501692Z     r = func(*args, **kwargs)
2023-09-06T17:19:11.1502550Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3429, in create_aot_dispatcher_function
2023-09-06T17:19:11.1503284Z     compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T17:19:11.1504213Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2212, in aot_wrapper_dedupe
2023-09-06T17:19:11.1504890Z     return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T17:19:11.1505842Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2392, in aot_wrapper_synthetic_base
2023-09-06T17:19:11.1506538Z     return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T17:19:11.1507674Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1573, in aot_dispatch_base
2023-09-06T17:19:11.1508357Z     compiled_fw = compiler(fw_module, flat_args)
2023-09-06T17:19:11.1509326Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T17:19:11.1509867Z     r = func(*args, **kwargs)
2023-09-06T17:19:11.1510654Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1109, in fw_compiler_base
2023-09-06T17:19:11.1511225Z     return inner_compile(
2023-09-06T17:19:11.1512051Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 219, in wrapper
2023-09-06T17:19:11.1512741Z     compiled = inner_compile(
2023-09-06T17:19:11.1513383Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T17:19:11.1513994Z     return func(*args, **kwds)
2023-09-06T17:19:11.1514989Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 80, in debug_wrapper
2023-09-06T17:19:11.1515765Z     inner_compiled_fn = compiler_fn(gm, example_inputs)
2023-09-06T17:19:11.1516665Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 228, in inner
2023-09-06T17:19:11.1517212Z     return fn(*args, **kwargs)
2023-09-06T17:19:11.1517704Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T17:19:11.1518282Z     return func(*args, **kwds)
2023-09-06T17:19:11.1519117Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 55, in newFunction
2023-09-06T17:19:11.1519702Z     return old_func(*args, **kwargs)
2023-09-06T17:19:11.1520542Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 348, in compile_fx_inner
2023-09-06T17:19:11.1521270Z     compiled_graph: CompiledFxGraph = fx_codegen_and_compile(
2023-09-06T17:19:11.1522527Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 574, in fx_codegen_and_compile
2023-09-06T17:19:11.1523132Z     compiled_fn = graph.compile_to_fn()
2023-09-06T17:19:11.1523941Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 991, in compile_to_fn
2023-09-06T17:19:11.1524507Z     return self.compile_to_module().call
2023-09-06T17:19:11.1525268Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T17:19:11.1525798Z     r = func(*args, **kwargs)
2023-09-06T17:19:11.1526586Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 946, in compile_to_module
2023-09-06T17:19:11.1527258Z     mod = PyCodeCache.load_by_key_path(key, path, linemap=linemap)
2023-09-06T17:19:11.1528158Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1161, in load_by_key_path
2023-09-06T17:19:11.1528725Z     exec(code, mod.__dict__, mod.__dict__)
2023-09-06T17:19:11.1529341Z   File "/tmp/tmp6z4_dxrj/fe/cfe3utb7wwz657ihqng4gtjywdsnd5dadvyblfn46tep5r36lkve.py", line 239, in <module>
2023-09-06T17:19:11.1529917Z     async_compile.wait(globals())
2023-09-06T17:19:11.1530683Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1440, in wait
2023-09-06T17:19:11.1531272Z     scope[key] = result.result()
2023-09-06T17:19:11.1532075Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1299, in result
2023-09-06T17:19:11.1532655Z     self.future.result()
2023-09-06T17:19:11.1533232Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 451, in result
2023-09-06T17:19:11.1533832Z     return self.__get_result()
2023-09-06T17:19:11.1534508Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
2023-09-06T17:19:11.1535429Z     raise self._exception
2023-09-06T17:19:11.1536192Z triton.compiler.errors.CompilationError: at 15:53:    xindex = xoffset + tl.arange(0, XBLOCK)[:]
2023-09-06T17:19:11.1536875Z     xmask = xindex < xnumel
2023-09-06T17:19:11.1537317Z     x1 = (xindex // 64)
2023-09-06T17:19:11.1537735Z     x0 = xindex % 64
2023-09-06T17:19:11.1538665Z     tmp0 = tl.load(in_ptr0 + (200000 + x1), None, eviction_policy='evict_last')
2023-09-06T17:19:11.1539457Z     tmp2 = tl.load(in_ptr0 + (x1), None, eviction_policy='evict_last')
2023-09-06T17:19:11.1540007Z     tmp1 = tl.where(tmp0 < 0, tmp0 + 10000, tmp0)
2023-09-06T17:19:11.1540619Z     tl.device_assert((0 <= tmp1) & (tmp1 < 10000), "index out of bounds: 0 <= tmp1 < 10000")
2023-09-06T17:19:11.1541221Z     tmp3 = tl.where(tmp2 < 0, tmp2 + 10000, tmp2)
2023-09-06T17:19:11.1541828Z     tl.device_assert((0 <= tmp3) & (tmp3 < 10000), "index out of bounds: 0 <= tmp3 < 10000")
2023-09-06T17:19:11.1542452Z     tmp4 = tl.load(in_ptr1 + (x0 + (64*tmp3)), None).to(tl.float32)
2023-09-06T17:19:11.1543056Z     tl.atomic_add(out_ptr0 + (x0 + (64*tmp1)), tmp4, None)
2023-09-06T17:19:11.1543542Z                                                      ^
2023-09-06T17:19:11.1544147Z ValueError('atomic_add does not support bf16')
2023-09-06T17:19:12.3432756Z ERROR
2023-09-06T17:19:15.9690069Z 
2023-09-06T17:19:18.7719549Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:19:18.7720387Z loading model: 0it [00:02, ?it/s]
2023-09-06T17:19:18.7723524Z cuda eval  basic_gnn_sage                     
2023-09-06T17:19:19.3061344Z [2023-09-06 17:19:19,305] [0/0] torch._dynamo.output_graph: [WARNING] nn.Module state_dict and backward hooks are not yet supported by torch.compile, but were detected in your model and will be silently ignored. See https://pytorch.org/docs/master/compile/nn-module.html for more information and limitations.
2023-09-06T17:19:25.8175541Z ERROR:common:Backend dynamo failed in warmup()
2023-09-06T17:19:25.8175992Z concurrent.futures.process._RemoteTraceback: 
2023-09-06T17:19:25.8177445Z """
2023-09-06T17:19:25.8177880Z Traceback (most recent call last):
2023-09-06T17:19:25.8179203Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/code_generator.py", line 1124, in ast_to_ttir
2023-09-06T17:19:25.8179876Z     generator.visit(fn.parse())
2023-09-06T17:19:25.8181000Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/code_generator.py", line 1017, in visit
2023-09-06T17:19:25.8181444Z     ret = super().visit(node)
2023-09-06T17:19:25.8181877Z   File "/opt/conda/envs/py_3.10/lib/python3.10/ast.py", line 418, in visit
2023-09-06T17:19:25.8182206Z     return visitor(node)
2023-09-06T17:19:25.8182767Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/code_generator.py", line 293, in visit_Module
2023-09-06T17:19:25.8183223Z     ast.NodeVisitor.generic_visit(self, node)
2023-09-06T17:19:25.8183671Z   File "/opt/conda/envs/py_3.10/lib/python3.10/ast.py", line 426, in generic_visit
2023-09-06T17:19:25.8184019Z     self.visit(item)
2023-09-06T17:19:25.8184584Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/code_generator.py", line 1017, in visit
2023-09-06T17:19:25.8184983Z     ret = super().visit(node)
2023-09-06T17:19:25.8185310Z   File "/opt/conda/envs/py_3.10/lib/python3.10/ast.py", line 418, in visit
2023-09-06T17:19:25.8185639Z     return visitor(node)
2023-09-06T17:19:25.8186212Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/code_generator.py", line 362, in visit_FunctionDef
2023-09-06T17:19:25.8186660Z     self.visit_compound_statement(node.body)
2023-09-06T17:19:25.8187269Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/code_generator.py", line 288, in visit_compound_statement
2023-09-06T17:19:25.8187702Z     ret_type = self.visit(stmt)
2023-09-06T17:19:25.8188249Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/code_generator.py", line 1017, in visit
2023-09-06T17:19:25.8188655Z     ret = super().visit(node)
2023-09-06T17:19:25.8189588Z   File "/opt/conda/envs/py_3.10/lib/python3.10/ast.py", line 418, in visit
2023-09-06T17:19:25.8189941Z     return visitor(node)
2023-09-06T17:19:25.8190525Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/code_generator.py", line 983, in visit_Expr
2023-09-06T17:19:25.8190967Z     ast.NodeVisitor.generic_visit(self, node)
2023-09-06T17:19:25.8191364Z   File "/opt/conda/envs/py_3.10/lib/python3.10/ast.py", line 428, in generic_visit
2023-09-06T17:19:25.8191683Z     self.visit(value)
2023-09-06T17:19:25.8192218Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/code_generator.py", line 1017, in visit
2023-09-06T17:19:25.8192612Z     ret = super().visit(node)
2023-09-06T17:19:25.8192956Z   File "/opt/conda/envs/py_3.10/lib/python3.10/ast.py", line 418, in visit
2023-09-06T17:19:25.8193262Z     return visitor(node)
2023-09-06T17:19:25.8193874Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/code_generator.py", line 946, in visit_Call
2023-09-06T17:19:25.8194300Z     return fn(*args, **extra_kwargs, **kws)
2023-09-06T17:19:25.8194850Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/language/core.py", line 30, in wrapper
2023-09-06T17:19:25.8195237Z     return fn(*args, **kwargs)
2023-09-06T17:19:25.8195746Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/language/core.py", line 1137, in atomic_add
2023-09-06T17:19:25.8196213Z     return semantic.atomic_add(pointer, val, mask, sem, _builder)
2023-09-06T17:19:25.8196813Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/language/semantic.py", line 1191, in atomic_add
2023-09-06T17:19:25.8197390Z     ptr, val, mask = atom_red_typechecking_impl(ptr, val, mask, 'add', builder)
2023-09-06T17:19:25.8198028Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/language/semantic.py", line 1087, in atom_red_typechecking_impl
2023-09-06T17:19:25.8198539Z     raise ValueError("atomic_" + op + " does not support " + str(element_ty))
2023-09-06T17:19:25.8199095Z ValueError: atomic_add does not support bf16
2023-09-06T17:19:25.8199299Z 
2023-09-06T17:19:25.8199495Z The above exception was the direct cause of the following exception:
2023-09-06T17:19:25.8199740Z 
2023-09-06T17:19:25.8199869Z Traceback (most recent call last):
2023-09-06T17:19:25.8200271Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/process.py", line 246, in _process_worker
2023-09-06T17:19:25.8200696Z     r = call_item.fn(*call_item.args, **call_item.kwargs)
2023-09-06T17:19:25.8201302Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1278, in _worker_compile
2023-09-06T17:19:25.8201744Z     kernel.precompile(warm_cache_only_with_cc=cc)
2023-09-06T17:19:25.8202330Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 168, in precompile
2023-09-06T17:19:25.8202723Z     self.launchers = [
2023-09-06T17:19:25.8203297Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 169, in <listcomp>
2023-09-06T17:19:25.8203793Z     self._precompile_config(c, warm_cache_only_with_cc)
2023-09-06T17:19:25.8204418Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/triton_heuristics.py", line 189, in _precompile_config
2023-09-06T17:19:25.8204811Z     triton.compile(
2023-09-06T17:19:25.8205332Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/compiler.py", line 476, in compile
2023-09-06T17:19:25.8205745Z     next_module = compile_kernel(module)
2023-09-06T17:19:25.8206300Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/compiler.py", line 381, in <lambda>
2023-09-06T17:19:25.8206805Z     lambda src: optimize_ttir(ast_to_ttir(src, signature, configs[0], constants, debug=debug, arch=arch), arch))
2023-09-06T17:19:25.8207603Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/triton/compiler/code_generator.py", line 1133, in ast_to_ttir
2023-09-06T17:19:25.8208075Z     raise CompilationError(fn.src, node, repr(e)) from e
2023-09-06T17:19:25.8208545Z triton.compiler.errors.CompilationError: at 15:53:    xindex = xoffset + tl.arange(0, XBLOCK)[:]
2023-09-06T17:19:25.8208950Z     xmask = xindex < xnumel
2023-09-06T17:19:25.8209198Z     x1 = (xindex // 64)
2023-09-06T17:19:25.8209442Z     x0 = xindex % 64
2023-09-06T17:19:25.8209887Z     tmp0 = tl.load(in_ptr0 + (200000 + x1), None, eviction_policy='evict_last')
2023-09-06T17:19:25.8210389Z     tmp2 = tl.load(in_ptr0 + (x1), None, eviction_policy='evict_last')
2023-09-06T17:19:25.8210732Z     tmp1 = tl.where(tmp0 < 0, tmp0 + 10000, tmp0)
2023-09-06T17:19:25.8211126Z     tl.device_assert((0 <= tmp1) & (tmp1 < 10000), "index out of bounds: 0 <= tmp1 < 10000")
2023-09-06T17:19:25.8211504Z     tmp3 = tl.where(tmp2 < 0, tmp2 + 10000, tmp2)
2023-09-06T17:19:25.8211890Z     tl.device_assert((0 <= tmp3) & (tmp3 < 10000), "index out of bounds: 0 <= tmp3 < 10000")
2023-09-06T17:19:25.8212291Z     tmp4 = tl.load(in_ptr1 + (x0 + (64*tmp3)), None).to(tl.float32)
2023-09-06T17:19:25.8212651Z     tl.atomic_add(out_ptr0 + (x0 + (64*tmp1)), tmp4, None)
2023-09-06T17:19:25.8212956Z                                                      ^
2023-09-06T17:19:25.8213341Z ValueError('atomic_add does not support bf16')
2023-09-06T17:19:25.8213639Z """
2023-09-06T17:19:25.8213769Z 
2023-09-06T17:19:25.8213966Z The above exception was the direct cause of the following exception:
2023-09-06T17:19:25.8214211Z 
2023-09-06T17:19:25.8214338Z Traceback (most recent call last):
2023-09-06T17:19:25.8214737Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2288, in warmup
2023-09-06T17:19:25.8215102Z     fn(model, example_inputs)
2023-09-06T17:19:25.8215483Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1190, in opt_aot_inductor
2023-09-06T17:19:25.8215970Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-09-06T17:19:25.8216591Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1134, in load
2023-09-06T17:19:25.8216985Z     so_path, exported = torch._export.aot_compile(
2023-09-06T17:19:25.8217553Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 645, in aot_compile
2023-09-06T17:19:25.8218046Z     so_path = torch._inductor.aot_compile(ep.graph_module, list(all_args), options)
2023-09-06T17:19:25.8218667Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/__init__.py", line 48, in aot_compile
2023-09-06T17:19:25.8219056Z     result = compile_fx_aot(
2023-09-06T17:19:25.8219610Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 874, in compile_fx_aot
2023-09-06T17:19:25.8220030Z     return compile_fx(
2023-09-06T17:19:25.8220590Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 975, in compile_fx
2023-09-06T17:19:25.8220992Z     return compile_fx(
2023-09-06T17:19:25.8221526Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 995, in compile_fx
2023-09-06T17:19:25.8221891Z     return compile_fx(
2023-09-06T17:19:25.8222421Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1167, in compile_fx
2023-09-06T17:19:25.8222806Z     return aot_autograd(
2023-09-06T17:19:25.8223355Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/backends/common.py", line 55, in compiler_fn
2023-09-06T17:19:25.8223837Z     cg = aot_module_simplified(gm, example_inputs, **kwargs)
2023-09-06T17:19:25.8224470Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3891, in aot_module_simplified
2023-09-06T17:19:25.8224925Z     compiled_fn = create_aot_dispatcher_function(
2023-09-06T17:19:25.8225492Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T17:19:25.8226047Z     r = func(*args, **kwargs)
2023-09-06T17:19:25.8226650Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3429, in create_aot_dispatcher_function
2023-09-06T17:19:25.8227196Z     compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T17:19:25.8227861Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2212, in aot_wrapper_dedupe
2023-09-06T17:19:25.8228376Z     return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T17:19:25.8229057Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2392, in aot_wrapper_synthetic_base
2023-09-06T17:19:25.8229855Z     return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
2023-09-06T17:19:25.8230527Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1573, in aot_dispatch_base
2023-09-06T17:19:25.8230980Z     compiled_fw = compiler(fw_module, flat_args)
2023-09-06T17:19:25.8231538Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T17:19:25.8231901Z     r = func(*args, **kwargs)
2023-09-06T17:19:25.8232460Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1109, in fw_compiler_base
2023-09-06T17:19:25.8232853Z     return inner_compile(
2023-09-06T17:19:25.8233387Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 219, in wrapper
2023-09-06T17:19:25.8233818Z     compiled = inner_compile(
2023-09-06T17:19:25.8234163Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T17:19:25.8234514Z     return func(*args, **kwds)
2023-09-06T17:19:25.8235083Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 80, in debug_wrapper
2023-09-06T17:19:25.8235710Z     inner_compiled_fn = compiler_fn(gm, example_inputs)
2023-09-06T17:19:25.8236447Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/debug.py", line 228, in inner
2023-09-06T17:19:25.8237172Z     return fn(*args, **kwargs)
2023-09-06T17:19:25.8237797Z   File "/opt/conda/envs/py_3.10/lib/python3.10/contextlib.py", line 79, in inner
2023-09-06T17:19:25.8238322Z     return func(*args, **kwds)
2023-09-06T17:19:25.8239178Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 55, in newFunction
2023-09-06T17:19:25.8239747Z     return old_func(*args, **kwargs)
2023-09-06T17:19:25.8240563Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 348, in compile_fx_inner
2023-09-06T17:19:25.8241221Z     compiled_graph: CompiledFxGraph = fx_codegen_and_compile(
2023-09-06T17:19:25.8242137Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 574, in fx_codegen_and_compile
2023-09-06T17:19:25.8242735Z     compiled_fn = graph.compile_to_fn()
2023-09-06T17:19:25.8243603Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 991, in compile_to_fn
2023-09-06T17:19:25.8244186Z     return self.compile_to_module().call
2023-09-06T17:19:25.8244956Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T17:19:25.8245503Z     r = func(*args, **kwargs)
2023-09-06T17:19:25.8246277Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/graph.py", line 946, in compile_to_module
2023-09-06T17:19:25.8246946Z     mod = PyCodeCache.load_by_key_path(key, path, linemap=linemap)
2023-09-06T17:19:25.8247814Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1161, in load_by_key_path
2023-09-06T17:19:25.8248392Z     exec(code, mod.__dict__, mod.__dict__)
2023-09-06T17:19:25.8249452Z   File "/tmp/tmpu28elv2u/il/cilf2lo4zn4rlzmzsw4cf6x2m5glsbalgsle42aeaxht46querte.py", line 247, in <module>
2023-09-06T17:19:25.8250123Z     async_compile.wait(globals())
2023-09-06T17:19:25.8250916Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1440, in wait
2023-09-06T17:19:25.8251479Z     scope[key] = result.result()
2023-09-06T17:19:25.8252237Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1299, in result
2023-09-06T17:19:25.8252774Z     self.future.result()
2023-09-06T17:19:25.8253301Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 451, in result
2023-09-06T17:19:25.8253871Z     return self.__get_result()
2023-09-06T17:19:25.8254367Z   File "/opt/conda/envs/py_3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
2023-09-06T17:19:25.8254860Z     raise self._exception
2023-09-06T17:19:25.8255433Z triton.compiler.errors.CompilationError: at 15:53:    xindex = xoffset + tl.arange(0, XBLOCK)[:]
2023-09-06T17:19:25.8255982Z     xmask = xindex < xnumel
2023-09-06T17:19:25.8256336Z     x1 = (xindex // 64)
2023-09-06T17:19:25.8256654Z     x0 = xindex % 64
2023-09-06T17:19:25.8257294Z     tmp0 = tl.load(in_ptr0 + (200000 + x1), None, eviction_policy='evict_last')
2023-09-06T17:19:25.8257990Z     tmp2 = tl.load(in_ptr0 + (x1), None, eviction_policy='evict_last')
2023-09-06T17:19:25.8258498Z     tmp1 = tl.where(tmp0 < 0, tmp0 + 10000, tmp0)
2023-09-06T17:19:25.8259045Z     tl.device_assert((0 <= tmp1) & (tmp1 < 10000), "index out of bounds: 0 <= tmp1 < 10000")
2023-09-06T17:19:25.8259577Z     tmp3 = tl.where(tmp2 < 0, tmp2 + 10000, tmp2)
2023-09-06T17:19:25.8260134Z     tl.device_assert((0 <= tmp3) & (tmp3 < 10000), "index out of bounds: 0 <= tmp3 < 10000")
2023-09-06T17:19:25.8260700Z     tmp4 = tl.load(in_ptr1 + (x0 + (64*tmp3)), None).to(tl.float32)
2023-09-06T17:19:25.8261189Z     tl.atomic_add(out_ptr0 + (x0 + (64*tmp1)), tmp4, None)
2023-09-06T17:19:25.8261914Z                                                      ^
2023-09-06T17:19:25.8262482Z ValueError('atomic_add does not support bf16')
2023-09-06T17:19:27.0696986Z ERROR
2023-09-06T17:19:30.7203099Z 
2023-09-06T17:19:34.9179225Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:19:34.9179828Z loading model: 0it [00:04, ?it/s]
2023-09-06T17:19:34.9268117Z cuda eval  cm3leon_generate                   
2023-09-06T17:19:43.5204053Z ERROR:common:Backend dynamo failed in warmup()
2023-09-06T17:19:43.5204644Z Traceback (most recent call last):
2023-09-06T17:19:43.5205048Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2288, in warmup
2023-09-06T17:19:43.5205403Z     fn(model, example_inputs)
2023-09-06T17:19:43.5205795Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1190, in opt_aot_inductor
2023-09-06T17:19:43.5206289Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-09-06T17:19:43.5207170Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1134, in load
2023-09-06T17:19:43.5207761Z     so_path, exported = torch._export.aot_compile(
2023-09-06T17:19:43.5210889Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 635, in aot_compile
2023-09-06T17:19:43.5211685Z     ep = export(f, args, kwargs, constraints)
2023-09-06T17:19:43.5212471Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 307, in export
2023-09-06T17:19:43.5212948Z     gm_torch_level, _ = torch._dynamo.export(
2023-09-06T17:19:43.5213511Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1150, in inner
2023-09-06T17:19:43.5213897Z     result_traced = opt_f(*args, **kwargs)
2023-09-06T17:19:43.5214475Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:19:43.5214894Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:19:43.5215927Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:19:43.5216326Z     return forward_call(*args, **kwargs)
2023-09-06T17:19:43.5216871Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T17:19:43.5217250Z     return fn(*args, **kwargs)
2023-09-06T17:19:43.5217807Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:19:43.5218230Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:19:43.5218767Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:19:43.5219165Z     return forward_call(*args, **kwargs)
2023-09-06T17:19:43.5219719Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T17:19:43.5220178Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T17:19:43.5220737Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T17:19:43.5221121Z     return fn(*args, **kwargs)
2023-09-06T17:19:43.5221689Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T17:19:43.5222086Z     return _compile(
2023-09-06T17:19:43.5222605Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T17:19:43.5223112Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T17:19:43.5223708Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T17:19:43.5224085Z     r = func(*args, **kwargs)
2023-09-06T17:19:43.5224644Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T17:19:43.5225301Z     out_code = transform_code_object(code, transform)
2023-09-06T17:19:43.5225956Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T17:19:43.5226427Z     transformations(instructions, code_options)
2023-09-06T17:19:43.5227004Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T17:19:43.5227377Z     tracer.run()
2023-09-06T17:19:43.5227878Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T17:19:43.5228243Z     super().run()
2023-09-06T17:19:43.5228758Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T17:19:43.5229389Z     and self.step()
2023-09-06T17:19:43.5230070Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T17:19:43.5230467Z     getattr(self, inst.opname)(inst)
2023-09-06T17:19:43.5231029Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T17:19:43.5231428Z     return inner_fn(self, inst)
2023-09-06T17:19:43.5231966Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 168, in impl
2023-09-06T17:19:43.5232398Z     self.push(fn_var.call_function(self, self.popn(nargs), {}))
2023-09-06T17:19:43.5233057Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 618, in call_function
2023-09-06T17:19:43.5233481Z     result = handler(tx, *args, **kwargs)
2023-09-06T17:19:43.5234053Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builtin.py", line 950, in call_getitem
2023-09-06T17:19:43.5234503Z     return args[0].call_method(tx, "__getitem__", args[1:], kwargs)
2023-09-06T17:19:43.5235298Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 302, in call_method
2023-09-06T17:19:43.5235722Z     ).call_function(tx, args, kwargs)
2023-09-06T17:19:43.5236304Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-09-06T17:19:43.5236746Z     return super().call_function(tx, args, kwargs)
2023-09-06T17:19:43.5237322Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-09-06T17:19:43.5237761Z     return super().call_function(tx, args, kwargs)
2023-09-06T17:19:43.5238350Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-09-06T17:19:43.5238781Z     return tx.inline_user_function_return(
2023-09-06T17:19:43.5239397Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 601, in inline_user_function_return
2023-09-06T17:19:43.5239930Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-09-06T17:19:43.5240591Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2183, in inline_call
2023-09-06T17:19:43.5241039Z     return cls.inline_call_(parent, func, args, kwargs)
2023-09-06T17:19:43.5241632Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2236, in inline_call_
2023-09-06T17:19:43.5242120Z     InliningInstructionTranslator.check_inlineable(func)
2023-09-06T17:19:43.5242749Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2217, in check_inlineable
2023-09-06T17:19:43.5243191Z     unimplemented(
2023-09-06T17:19:43.5243711Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 176, in unimplemented
2023-09-06T17:19:43.5244242Z     raise Unsupported(msg)
2023-09-06T17:19:43.5244692Z torch._dynamo.exc.Unsupported: inline in skipfiles: _SpecialForm.__getitem__  | inner /opt/conda/envs/py_3.10/lib/python3.10/typing.py
2023-09-06T17:19:43.5245023Z 
2023-09-06T17:19:43.5245117Z from user code:
2023-09-06T17:19:43.5245531Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate/model.py", line 1116, in forward
2023-09-06T17:19:43.5245967Z     Dict[str, Dict[str, Optional[Tensor]]], {}
2023-09-06T17:19:43.5246160Z 
2023-09-06T17:19:43.5246355Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T17:19:43.5246578Z 
2023-09-06T17:19:44.7248048Z ERROR
2023-09-06T17:19:48.3201232Z 
2023-09-06T17:19:49.1790226Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:19:49.1791676Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:19:49.1792559Z cuda eval  dcgan                              
2023-09-06T17:20:10.7645882Z 
2023-09-06T17:20:10.8647487Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T17:20:10.9704210Z running benchmark:  43% 13/30 [00:00<00:00, 129.02it/s]
2023-09-06T17:20:10.9848263Z running benchmark:  93% 28/30 [00:00<00:00, 136.76it/s]
2023-09-06T17:20:10.9848635Z running benchmark: 100% 30/30 [00:00<00:00, 135.71it/s]
2023-09-06T17:20:10.9869914Z 1.163x
2023-09-06T17:20:15.8504365Z 
2023-09-06T17:20:19.5289115Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:20:19.5290415Z loading model: 0it [00:03, ?it/s]
2023-09-06T17:20:19.5290889Z Eager model failed to run
2023-09-06T17:20:19.5301826Z Traceback (most recent call last):
2023-09-06T17:20:19.5302725Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T17:20:19.5303418Z     self.model_iter_fn(model, example_inputs)
2023-09-06T17:20:19.5303846Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T17:20:19.5304757Z     return mod(*inputs)
2023-09-06T17:20:19.5307605Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:20:19.5308097Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:20:19.5308678Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:20:19.5309308Z     return forward_call(*args, **kwargs)
2023-09-06T17:20:19.5310024Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/__init__.py", line 31, in forward
2023-09-06T17:20:19.5310423Z     return sources, self.model(mix)
2023-09-06T17:20:19.5311021Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:20:19.5311443Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:20:19.5312001Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:20:19.5312404Z     return forward_call(*args, **kwargs)
2023-09-06T17:20:19.5312903Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/demucs/model.py", line 209, in forward
2023-09-06T17:20:19.5313292Z     x = self.lstm(x)
2023-09-06T17:20:19.5313838Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:20:19.5314259Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:20:19.5314799Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:20:19.5315200Z     return forward_call(*args, **kwargs)
2023-09-06T17:20:19.5315631Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs/demucs/model.py", line 27, in forward
2023-09-06T17:20:19.5316021Z     x = self.lstm(x)[0]
2023-09-06T17:20:19.5316572Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:20:19.5316973Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:20:19.5317832Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:20:19.5318254Z     return forward_call(*args, **kwargs)
2023-09-06T17:20:19.5318802Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/rnn.py", line 879, in forward
2023-09-06T17:20:19.5319260Z     result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
2023-09-06T17:20:19.5319797Z RuntimeError: "_thnn_fused_lstm_cell_cuda" not implemented for 'BFloat16'
2023-09-06T17:20:19.5320048Z 
2023-09-06T17:20:19.5320248Z The above exception was the direct cause of the following exception:
2023-09-06T17:20:19.5320491Z 
2023-09-06T17:20:19.5320620Z Traceback (most recent call last):
2023-09-06T17:20:19.5320998Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T17:20:19.5321330Z     ) = runner.load_model(
2023-09-06T17:20:19.5321717Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T17:20:19.5322129Z     self.validate_model(model, example_inputs)
2023-09-06T17:20:19.5322543Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T17:20:19.5323025Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T17:20:19.5323411Z NotImplementedError: Eager model failed to run
2023-09-06T17:20:19.5323633Z 
2023-09-06T17:20:19.5323760Z WARNING:root:demucs failed to load
2023-09-06T17:20:24.0435195Z 
2023-09-06T17:20:26.1127170Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:20:26.1127718Z loading model: 0it [00:02, ?it/s]
2023-09-06T17:20:26.1243598Z cuda eval  densenet121                        
2023-09-06T17:23:08.4794929Z 
2023-09-06T17:23:08.6053268Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-09-06T17:23:08.7143873Z running benchmark:  10% 3/30 [00:00<00:01, 23.81it/s]
2023-09-06T17:23:08.8239062Z running benchmark:  20% 6/30 [00:00<00:00, 25.82it/s]
2023-09-06T17:23:08.9338885Z running benchmark:  30% 9/30 [00:00<00:00, 26.50it/s]
2023-09-06T17:23:09.0432739Z running benchmark:  40% 12/30 [00:00<00:00, 26.81it/s]
2023-09-06T17:23:09.1484813Z running benchmark:  50% 15/30 [00:00<00:00, 27.02it/s]
2023-09-06T17:23:09.2538174Z running benchmark:  60% 18/30 [00:00<00:00, 27.51it/s]
2023-09-06T17:23:09.3663490Z running benchmark:  70% 21/30 [00:00<00:00, 27.82it/s]
2023-09-06T17:23:09.4726995Z running benchmark:  80% 24/30 [00:00<00:00, 27.46it/s]
2023-09-06T17:23:09.5808928Z running benchmark:  90% 27/30 [00:00<00:00, 27.67it/s]
2023-09-06T17:23:09.5809581Z running benchmark: 100% 30/30 [00:01<00:00, 27.69it/s]
2023-09-06T17:23:09.5810110Z running benchmark: 100% 30/30 [00:01<00:00, 27.22it/s]
2023-09-06T17:23:09.5837143Z 1.730x
2023-09-06T17:23:15.6425390Z 
2023-09-06T17:23:20.3732317Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:23:20.3732687Z loading model: 0it [00:04, ?it/s]
2023-09-06T17:23:20.3732969Z Eager model failed to run
2023-09-06T17:23:20.3747781Z Traceback (most recent call last):
2023-09-06T17:23:20.3748458Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T17:23:20.3749295Z     self.model_iter_fn(model, example_inputs)
2023-09-06T17:23:20.3749985Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T17:23:20.3753238Z     return mod(*inputs)
2023-09-06T17:23:20.3755423Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:23:20.3755915Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:23:20.3756505Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:23:20.3757120Z     return forward_call(*args, **kwargs)
2023-09-06T17:23:20.3758075Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T17:23:20.3759695Z     return self.inference(batched_inputs)
2023-09-06T17:23:20.3760658Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T17:23:20.3761138Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T17:23:20.3761753Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:23:20.3762176Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:23:20.3762739Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:23:20.3763142Z     return forward_call(*args, **kwargs)
2023-09-06T17:23:20.3763811Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2023-09-06T17:23:20.3764234Z     box_features = self._shared_roi_transform(
2023-09-06T17:23:20.3764874Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2023-09-06T17:23:20.3765319Z     x = self.pooler(features, boxes)
2023-09-06T17:23:20.3765884Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:23:20.3766292Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:23:20.3766847Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:23:20.3767250Z     return forward_call(*args, **kwargs)
2023-09-06T17:23:20.3767804Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T17:23:20.3768238Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T17:23:20.3768821Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:23:20.3769457Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:23:20.3770032Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:23:20.3770432Z     return forward_call(*args, **kwargs)
2023-09-06T17:23:20.3770962Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T17:23:20.3771342Z     return roi_align(
2023-09-06T17:23:20.3771877Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T17:23:20.3772312Z     return torch.ops.torchvision.roi_align(
2023-09-06T17:23:20.3772856Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T17:23:20.3773244Z     return self._op(*args, **kwargs or {})
2023-09-06T17:23:20.3773774Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T17:23:20.3774030Z 
2023-09-06T17:23:20.3774237Z The above exception was the direct cause of the following exception:
2023-09-06T17:23:20.3774483Z 
2023-09-06T17:23:20.3774610Z Traceback (most recent call last):
2023-09-06T17:23:20.3774985Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T17:23:20.3775334Z     ) = runner.load_model(
2023-09-06T17:23:20.3775713Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T17:23:20.3776116Z     self.validate_model(model, example_inputs)
2023-09-06T17:23:20.3776517Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T17:23:20.3776968Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T17:23:20.3777351Z NotImplementedError: Eager model failed to run
2023-09-06T17:23:20.3777561Z 
2023-09-06T17:23:20.3777728Z WARNING:root:detectron2_fasterrcnn_r_101_c4 failed to load
2023-09-06T17:23:25.0389930Z 
2023-09-06T17:23:32.0664847Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:23:32.0665714Z loading model: 0it [00:07, ?it/s]
2023-09-06T17:23:32.0666174Z Eager model failed to run
2023-09-06T17:23:32.0681008Z Traceback (most recent call last):
2023-09-06T17:23:32.0681739Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T17:23:32.0682554Z     self.model_iter_fn(model, example_inputs)
2023-09-06T17:23:32.0683103Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T17:23:32.0683472Z     return mod(*inputs)
2023-09-06T17:23:32.0685609Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:23:32.0686041Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:23:32.0686596Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:23:32.0687005Z     return forward_call(*args, **kwargs)
2023-09-06T17:23:32.0687616Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T17:23:32.0688046Z     return self.inference(batched_inputs)
2023-09-06T17:23:32.0688612Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T17:23:32.0689094Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T17:23:32.0689726Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:23:32.0690313Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:23:32.0691301Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:23:32.0692041Z     return forward_call(*args, **kwargs)
2023-09-06T17:23:32.0693770Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T17:23:32.0694296Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T17:23:32.0694961Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T17:23:32.0695479Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T17:23:32.0696112Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:23:32.0696532Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:23:32.0697084Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:23:32.0697512Z     return forward_call(*args, **kwargs)
2023-09-06T17:23:32.0698066Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T17:23:32.0698498Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T17:23:32.0699096Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:23:32.0699511Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:23:32.0700060Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:23:32.0700454Z     return forward_call(*args, **kwargs)
2023-09-06T17:23:32.0700985Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T17:23:32.0701355Z     return roi_align(
2023-09-06T17:23:32.0701882Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T17:23:32.0702310Z     return torch.ops.torchvision.roi_align(
2023-09-06T17:23:32.0702836Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T17:23:32.0703492Z     return self._op(*args, **kwargs or {})
2023-09-06T17:23:32.0703972Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T17:23:32.0704216Z 
2023-09-06T17:23:32.0704416Z The above exception was the direct cause of the following exception:
2023-09-06T17:23:32.0704653Z 
2023-09-06T17:23:32.0704765Z Traceback (most recent call last):
2023-09-06T17:23:32.0705146Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T17:23:32.0705491Z     ) = runner.load_model(
2023-09-06T17:23:32.0705873Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T17:23:32.0706279Z     self.validate_model(model, example_inputs)
2023-09-06T17:23:32.0706674Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T17:23:32.0707110Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T17:23:32.0707504Z NotImplementedError: Eager model failed to run
2023-09-06T17:23:32.0707707Z 
2023-09-06T17:23:32.0707873Z WARNING:root:detectron2_fasterrcnn_r_101_dc5 failed to load
2023-09-06T17:23:36.7231528Z 
2023-09-06T17:23:41.1703593Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:23:41.1703937Z loading model: 0it [00:04, ?it/s]
2023-09-06T17:23:41.1704213Z Eager model failed to run
2023-09-06T17:23:41.1715988Z Traceback (most recent call last):
2023-09-06T17:23:41.1716480Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T17:23:41.1717174Z     self.model_iter_fn(model, example_inputs)
2023-09-06T17:23:41.1717858Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T17:23:41.1718538Z     return mod(*inputs)
2023-09-06T17:23:41.1719807Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:23:41.1720245Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:23:41.1722377Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:23:41.1722946Z     return forward_call(*args, **kwargs)
2023-09-06T17:23:41.1723604Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T17:23:41.1724024Z     return self.inference(batched_inputs)
2023-09-06T17:23:41.1724606Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T17:23:41.1725070Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T17:23:41.1725696Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:23:41.1726120Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:23:41.1726680Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:23:41.1727079Z     return forward_call(*args, **kwargs)
2023-09-06T17:23:41.1727667Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T17:23:41.1728135Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T17:23:41.1728766Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T17:23:41.1729282Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T17:23:41.1729915Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:23:41.1730332Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:23:41.1730881Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:23:41.1731507Z     return forward_call(*args, **kwargs)
2023-09-06T17:23:41.1732069Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2023-09-06T17:23:41.1732516Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2023-09-06T17:23:41.1733189Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:23:41.1733607Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:23:41.1734162Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:23:41.1734542Z     return forward_call(*args, **kwargs)
2023-09-06T17:23:41.1735092Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T17:23:41.1735461Z     return roi_align(
2023-09-06T17:23:41.1735996Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T17:23:41.1736422Z     return torch.ops.torchvision.roi_align(
2023-09-06T17:23:41.1736962Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T17:23:41.1737346Z     return self._op(*args, **kwargs or {})
2023-09-06T17:23:41.1737812Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T17:23:41.1738061Z 
2023-09-06T17:23:41.1738259Z The above exception was the direct cause of the following exception:
2023-09-06T17:23:41.1738506Z 
2023-09-06T17:23:41.1738617Z Traceback (most recent call last):
2023-09-06T17:23:41.1738995Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T17:23:41.1739341Z     ) = runner.load_model(
2023-09-06T17:23:41.1739721Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T17:23:41.1740105Z     self.validate_model(model, example_inputs)
2023-09-06T17:23:41.1740649Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T17:23:41.1741096Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T17:23:41.1741482Z NotImplementedError: Eager model failed to run
2023-09-06T17:23:41.1741694Z 
2023-09-06T17:23:41.1741881Z WARNING:root:detectron2_fasterrcnn_r_101_fpn failed to load
2023-09-06T17:23:45.8365396Z 
2023-09-06T17:23:49.9415638Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:23:49.9416226Z loading model: 0it [00:04, ?it/s]
2023-09-06T17:23:49.9416693Z Eager model failed to run
2023-09-06T17:23:49.9427498Z Traceback (most recent call last):
2023-09-06T17:23:49.9428355Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T17:23:49.9429313Z     self.model_iter_fn(model, example_inputs)
2023-09-06T17:23:49.9430029Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T17:23:49.9430636Z     return mod(*inputs)
2023-09-06T17:23:49.9432116Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:23:49.9432575Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:23:49.9433142Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:23:49.9433532Z     return forward_call(*args, **kwargs)
2023-09-06T17:23:49.9434097Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T17:23:49.9434519Z     return self.inference(batched_inputs)
2023-09-06T17:23:49.9435102Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T17:23:49.9435579Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T17:23:49.9436190Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:23:49.9437135Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:23:49.9437697Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:23:49.9438162Z     return forward_call(*args, **kwargs)
2023-09-06T17:23:49.9438734Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2023-09-06T17:23:49.9439176Z     box_features = self._shared_roi_transform(
2023-09-06T17:23:49.9439812Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2023-09-06T17:23:49.9440253Z     x = self.pooler(features, boxes)
2023-09-06T17:23:49.9440824Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:23:49.9441226Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:23:49.9441792Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:23:49.9442224Z     return forward_call(*args, **kwargs)
2023-09-06T17:23:49.9442781Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T17:23:49.9443205Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T17:23:49.9443793Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:23:49.9444207Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:23:49.9444754Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:23:49.9445151Z     return forward_call(*args, **kwargs)
2023-09-06T17:23:49.9445681Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T17:23:49.9446235Z     return roi_align(
2023-09-06T17:23:49.9446777Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T17:23:49.9447214Z     return torch.ops.torchvision.roi_align(
2023-09-06T17:23:49.9447769Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T17:23:49.9448197Z     return self._op(*args, **kwargs or {})
2023-09-06T17:23:49.9448672Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T17:23:49.9448915Z 
2023-09-06T17:23:49.9449113Z The above exception was the direct cause of the following exception:
2023-09-06T17:23:49.9449353Z 
2023-09-06T17:23:49.9449481Z Traceback (most recent call last):
2023-09-06T17:23:49.9449843Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T17:23:49.9450185Z     ) = runner.load_model(
2023-09-06T17:23:49.9450577Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T17:23:49.9450978Z     self.validate_model(model, example_inputs)
2023-09-06T17:23:49.9451378Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T17:23:49.9451814Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T17:23:49.9452193Z NotImplementedError: Eager model failed to run
2023-09-06T17:23:49.9452401Z 
2023-09-06T17:23:49.9452563Z WARNING:root:detectron2_fasterrcnn_r_50_c4 failed to load
2023-09-06T17:23:54.6119987Z 
2023-09-06T17:24:01.0171523Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:24:01.0173134Z loading model: 0it [00:06, ?it/s]
2023-09-06T17:24:01.0173546Z Eager model failed to run
2023-09-06T17:24:01.0190088Z Traceback (most recent call last):
2023-09-06T17:24:01.0190809Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T17:24:01.0191422Z     self.model_iter_fn(model, example_inputs)
2023-09-06T17:24:01.0192712Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T17:24:01.0193236Z     return mod(*inputs)
2023-09-06T17:24:01.0194927Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:24:01.0195586Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:24:01.0196439Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:24:01.0197109Z     return forward_call(*args, **kwargs)
2023-09-06T17:24:01.0198132Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T17:24:01.0198774Z     return self.inference(batched_inputs)
2023-09-06T17:24:01.0199698Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T17:24:01.0200480Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T17:24:01.0201453Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:24:01.0202087Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:24:01.0202952Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:24:01.0203578Z     return forward_call(*args, **kwargs)
2023-09-06T17:24:01.0204481Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T17:24:01.0205189Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T17:24:01.0206183Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T17:24:01.0206996Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T17:24:01.0208511Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:24:01.0209168Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:24:01.0210032Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:24:01.0210637Z     return forward_call(*args, **kwargs)
2023-09-06T17:24:01.0211476Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2023-09-06T17:24:01.0212129Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2023-09-06T17:24:01.0213042Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:24:01.0213661Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:24:01.0214537Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:24:01.0215156Z     return forward_call(*args, **kwargs)
2023-09-06T17:24:01.0215988Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T17:24:01.0216541Z     return roi_align(
2023-09-06T17:24:01.0217341Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T17:24:01.0218069Z     return torch.ops.torchvision.roi_align(
2023-09-06T17:24:01.0218891Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T17:24:01.0219492Z     return self._op(*args, **kwargs or {})
2023-09-06T17:24:01.0220214Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T17:24:01.0220581Z 
2023-09-06T17:24:01.0220851Z The above exception was the direct cause of the following exception:
2023-09-06T17:24:01.0221205Z 
2023-09-06T17:24:01.0221412Z Traceback (most recent call last):
2023-09-06T17:24:01.0222339Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T17:24:01.0222867Z     ) = runner.load_model(
2023-09-06T17:24:01.0223428Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T17:24:01.0224072Z     self.validate_model(model, example_inputs)
2023-09-06T17:24:01.0224731Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T17:24:01.0225408Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T17:24:01.0226121Z NotImplementedError: Eager model failed to run
2023-09-06T17:24:01.0226513Z 
2023-09-06T17:24:01.0226818Z WARNING:root:detectron2_fasterrcnn_r_50_dc5 failed to load
2023-09-06T17:24:05.7036711Z 
2023-09-06T17:24:09.9324586Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:24:09.9325137Z loading model: 0it [00:04, ?it/s]
2023-09-06T17:24:09.9325612Z Eager model failed to run
2023-09-06T17:24:09.9341142Z Traceback (most recent call last):
2023-09-06T17:24:09.9342185Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1859, in validate_model
2023-09-06T17:24:09.9342765Z     self.model_iter_fn(model, example_inputs)
2023-09-06T17:24:09.9343453Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 472, in forward_pass
2023-09-06T17:24:09.9344049Z     return mod(*inputs)
2023-09-06T17:24:09.9345184Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:24:09.9345837Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:24:09.9346682Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:24:09.9347306Z     return forward_call(*args, **kwargs)
2023-09-06T17:24:09.9348291Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2023-09-06T17:24:09.9349768Z     return self.inference(batched_inputs)
2023-09-06T17:24:09.9350829Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2023-09-06T17:24:09.9351554Z     results, _ = self.roi_heads(images, features, proposals, None)
2023-09-06T17:24:09.9352576Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:24:09.9353240Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:24:09.9354120Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:24:09.9354726Z     return forward_call(*args, **kwargs)
2023-09-06T17:24:09.9355668Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2023-09-06T17:24:09.9356371Z     pred_instances = self._forward_box(features, proposals)
2023-09-06T17:24:09.9357379Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2023-09-06T17:24:09.9358232Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2023-09-06T17:24:09.9359227Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:24:09.9359888Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:24:09.9360759Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:24:09.9361382Z     return forward_call(*args, **kwargs)
2023-09-06T17:24:09.9362218Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2023-09-06T17:24:09.9362979Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2023-09-06T17:24:09.9363977Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:24:09.9365015Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:24:09.9365893Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:24:09.9366498Z     return forward_call(*args, **kwargs)
2023-09-06T17:24:09.9367345Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2023-09-06T17:24:09.9367991Z     return roi_align(
2023-09-06T17:24:09.9368798Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 238, in roi_align
2023-09-06T17:24:09.9369433Z     return torch.ops.torchvision.roi_align(
2023-09-06T17:24:09.9370264Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 692, in __call__
2023-09-06T17:24:09.9370839Z     return self._op(*args, **kwargs or {})
2023-09-06T17:24:09.9371539Z RuntimeError: "roi_align_forward_kernel" not implemented for 'BFloat16'
2023-09-06T17:24:09.9371933Z 
2023-09-06T17:24:09.9372225Z The above exception was the direct cause of the following exception:
2023-09-06T17:24:09.9372574Z 
2023-09-06T17:24:09.9372767Z Traceback (most recent call last):
2023-09-06T17:24:09.9373342Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3432, in run
2023-09-06T17:24:09.9373892Z     ) = runner.load_model(
2023-09-06T17:24:09.9374498Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 408, in load_model
2023-09-06T17:24:09.9375087Z     self.validate_model(model, example_inputs)
2023-09-06T17:24:09.9375664Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1861, in validate_model
2023-09-06T17:24:09.9376362Z     raise NotImplementedError("Eager model failed to run") from e
2023-09-06T17:24:09.9376944Z NotImplementedError: Eager model failed to run
2023-09-06T17:24:09.9377245Z 
2023-09-06T17:24:09.9377545Z WARNING:root:detectron2_fasterrcnn_r_50_fpn failed to load
2023-09-06T17:24:14.6147013Z 
2023-09-06T17:24:18.7304783Z loading model: 0it [00:00, ?it/s]
2023-09-06T17:24:18.7305149Z loading model: 0it [00:04, ?it/s]
2023-09-06T17:24:18.7373526Z cuda eval  detectron2_fcos_r_50_fpn           
2023-09-06T17:24:21.5495211Z ERROR:common:Backend dynamo failed in warmup()
2023-09-06T17:24:21.5495861Z Traceback (most recent call last):
2023-09-06T17:24:21.5498050Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1404, in run_node
2023-09-06T17:24:21.5498779Z     return getattr(args[0], node.target)(*args[1:], **kwargs)
2023-09-06T17:24:21.5499759Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_stats.py", line 20, in wrapper
2023-09-06T17:24:21.5500420Z     return fn(*args, **kwargs)
2023-09-06T17:24:21.5501420Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1290, in __torch_dispatch__
2023-09-06T17:24:21.5502183Z     return self.dispatch(func, types, args, kwargs)
2023-09-06T17:24:21.5503579Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1495, in dispatch
2023-09-06T17:24:21.5504354Z     return decomposition_table[func](*args, **kwargs)
2023-09-06T17:24:21.5505329Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_refs/__init__.py", line 4688, in new_full
2023-09-06T17:24:21.5505955Z     return torch.full(
2023-09-06T17:24:21.5506797Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_stats.py", line 20, in wrapper
2023-09-06T17:24:21.5507439Z     return fn(*args, **kwargs)
2023-09-06T17:24:21.5508527Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1290, in __torch_dispatch__
2023-09-06T17:24:21.5509551Z     return self.dispatch(func, types, args, kwargs)
2023-09-06T17:24:21.5510569Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1532, in dispatch
2023-09-06T17:24:21.5511960Z     op_impl_out = op_impl(self, func, *args, **kwargs)
2023-09-06T17:24:21.5513024Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 447, in constructors
2023-09-06T17:24:21.5513738Z     r = func(*args, **new_kwargs)
2023-09-06T17:24:21.5514605Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 448, in __call__
2023-09-06T17:24:21.5515252Z     return self._op(*args, **kwargs or {})
2023-09-06T17:24:21.5516203Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_meta_registrations.py", line 4433, in full
2023-09-06T17:24:21.5516909Z     return torch.empty(size, *args, **kwargs)
2023-09-06T17:24:21.5518018Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 985, in guard_bool
2023-09-06T17:24:21.5518895Z     r = self.shape_env.evaluate_expr(self.expr, self.hint, fx_node=self.fx_node)
2023-09-06T17:24:21.5520077Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 3491, in evaluate_expr
2023-09-06T17:24:21.5520871Z     concrete_val = self.size_hint(orig_expr)
2023-09-06T17:24:21.5521895Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 3292, in size_hint
2023-09-06T17:24:21.5522714Z     raise self._make_data_dependent_error(result_expr, expr)
2023-09-06T17:24:21.5525008Z torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode: It appears that you're trying to get a value out of symbolic int/float whose value is data-dependent (and thus we do not know the true value.)  The expression we were trying to evaluate is Eq(12*i0*i1, 0) (unhinted: Eq(12*i0*i1, 0)).  Scroll up to see where each of these data-dependent accesses originally occurred.
2023-09-06T17:24:21.5526831Z 
2023-09-06T17:24:21.5527178Z The above exception was the direct cause of the following exception:
2023-09-06T17:24:21.5527582Z 
2023-09-06T17:24:21.5528250Z Traceback (most recent call last):
2023-09-06T17:24:21.5529269Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1342, in get_fake_value
2023-09-06T17:24:21.5529980Z     return wrap_fake_exception(
2023-09-06T17:24:21.5531015Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 917, in wrap_fake_exception
2023-09-06T17:24:21.5531660Z     return fn()
2023-09-06T17:24:21.5532554Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1343, in <lambda>
2023-09-06T17:24:21.5533343Z     lambda: run_node(tx.output, node, args, kwargs, nnmodule)
2023-09-06T17:24:21.5534374Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1415, in run_node
2023-09-06T17:24:21.5535193Z     raise RuntimeError(fn_str + str(e)).with_traceback(e.__traceback__) from e
2023-09-06T17:24:21.5536249Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1404, in run_node
2023-09-06T17:24:21.5537035Z     return getattr(args[0], node.target)(*args[1:], **kwargs)
2023-09-06T17:24:21.5538080Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_stats.py", line 20, in wrapper
2023-09-06T17:24:21.5538737Z     return fn(*args, **kwargs)
2023-09-06T17:24:21.5539898Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1290, in __torch_dispatch__
2023-09-06T17:24:21.5540702Z     return self.dispatch(func, types, args, kwargs)
2023-09-06T17:24:21.5541713Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1495, in dispatch
2023-09-06T17:24:21.5542454Z     return decomposition_table[func](*args, **kwargs)
2023-09-06T17:24:21.5543425Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_refs/__init__.py", line 4688, in new_full
2023-09-06T17:24:21.5544073Z     return torch.full(
2023-09-06T17:24:21.5544985Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_stats.py", line 20, in wrapper
2023-09-06T17:24:21.5545964Z     return fn(*args, **kwargs)
2023-09-06T17:24:21.5546996Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1290, in __torch_dispatch__
2023-09-06T17:24:21.5547982Z     return self.dispatch(func, types, args, kwargs)
2023-09-06T17:24:21.5549034Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 1532, in dispatch
2023-09-06T17:24:21.5549969Z     op_impl_out = op_impl(self, func, *args, **kwargs)
2023-09-06T17:24:21.5551014Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py", line 447, in constructors
2023-09-06T17:24:21.5551720Z     r = func(*args, **new_kwargs)
2023-09-06T17:24:21.5552598Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_ops.py", line 448, in __call__
2023-09-06T17:24:21.5553263Z     return self._op(*args, **kwargs or {})
2023-09-06T17:24:21.5554239Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_meta_registrations.py", line 4433, in full
2023-09-06T17:24:21.5554972Z     return torch.empty(size, *args, **kwargs)
2023-09-06T17:24:21.5556036Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 985, in guard_bool
2023-09-06T17:24:21.5556915Z     r = self.shape_env.evaluate_expr(self.expr, self.hint, fx_node=self.fx_node)
2023-09-06T17:24:21.5558116Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 3491, in evaluate_expr
2023-09-06T17:24:21.5558902Z     concrete_val = self.size_hint(orig_expr)
2023-09-06T17:24:21.5559940Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 3292, in size_hint
2023-09-06T17:24:21.5560727Z     raise self._make_data_dependent_error(result_expr, expr)
2023-09-06T17:24:21.5562688Z RuntimeError: Failed running call_method new_full(*(FakeTensor(..., device='cuda:0', size=(3, 800, 1199), dtype=torch.bfloat16), [4, 3, FakeTensor(..., size=(), dtype=torch.int64), FakeTensor(..., size=(), dtype=torch.int64)], 0.0), **{'device': None}):
2023-09-06T17:24:21.5565060Z It appears that you're trying to get a value out of symbolic int/float whose value is data-dependent (and thus we do not know the true value.)  The expression we were trying to evaluate is Eq(12*i0*i1, 0) (unhinted: Eq(12*i0*i1, 0)).  Scroll up to see where each of these data-dependent accesses originally occurred.
2023-09-06T17:24:21.5566082Z 
2023-09-06T17:24:21.5566418Z During handling of the above exception, another exception occurred:
2023-09-06T17:24:21.5566835Z 
2023-09-06T17:24:21.5567056Z Traceback (most recent call last):
2023-09-06T17:24:21.5567712Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2288, in warmup
2023-09-06T17:24:21.5568376Z     fn(model, example_inputs)
2023-09-06T17:24:21.5569035Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1190, in opt_aot_inductor
2023-09-06T17:24:21.5569929Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-09-06T17:24:21.5570741Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1134, in load
2023-09-06T17:24:21.5571424Z     so_path, exported = torch._export.aot_compile(
2023-09-06T17:24:21.5572416Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 635, in aot_compile
2023-09-06T17:24:21.5573093Z     ep = export(f, args, kwargs, constraints)
2023-09-06T17:24:21.5574012Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 307, in export
2023-09-06T17:24:21.5574712Z     gm_torch_level, _ = torch._dynamo.export(
2023-09-06T17:24:21.5575648Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1150, in inner
2023-09-06T17:24:21.5576315Z     result_traced = opt_f(*args, **kwargs)
2023-09-06T17:24:21.5577682Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:24:21.5617335Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:24:21.5618645Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:24:21.5619339Z     return forward_call(*args, **kwargs)
2023-09-06T17:24:21.5620276Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 338, in _fn
2023-09-06T17:24:21.5620916Z     return fn(*args, **kwargs)
2023-09-06T17:24:21.5621881Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-09-06T17:24:21.5622607Z     return self._call_impl(*args, **kwargs)
2023-09-06T17:24:21.5623548Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-09-06T17:24:21.5624277Z     return forward_call(*args, **kwargs)
2023-09-06T17:24:21.5625223Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 500, in catch_errors
2023-09-06T17:24:21.5625982Z     return callback(frame, cache_entry, hooks, frame_state)
2023-09-06T17:24:21.5626942Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 140, in _fn
2023-09-06T17:24:21.5627575Z     return fn(*args, **kwargs)
2023-09-06T17:24:21.5628609Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
2023-09-06T17:24:21.5629480Z     return _compile(
2023-09-06T17:24:21.5630346Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 562, in _compile
2023-09-06T17:24:21.5631142Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-09-06T17:24:21.5632675Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 189, in time_wrapper
2023-09-06T17:24:21.5633362Z     r = func(*args, **kwargs)
2023-09-06T17:24:21.5634332Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 484, in compile_inner
2023-09-06T17:24:21.5635032Z     out_code = transform_code_object(code, transform)
2023-09-06T17:24:21.5636119Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-09-06T17:24:21.5636904Z     transformations(instructions, code_options)
2023-09-06T17:24:21.5637911Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 451, in transform
2023-09-06T17:24:21.5638477Z     tracer.run()
2023-09-06T17:24:21.5639303Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2078, in run
2023-09-06T17:24:21.5639849Z     super().run()
2023-09-06T17:24:21.5640636Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T17:24:21.5641177Z     and self.step()
2023-09-06T17:24:21.5641919Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T17:24:21.5642503Z     getattr(self, inst.opname)(inst)
2023-09-06T17:24:21.5643325Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T17:24:21.5643899Z     return inner_fn(self, inst)
2023-09-06T17:24:21.5644706Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1119, in CALL_FUNCTION
2023-09-06T17:24:21.5645313Z     self.call_function(fn, args, {})
2023-09-06T17:24:21.5646131Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 565, in call_function
2023-09-06T17:24:21.5646756Z     self.push(fn.call_function(self, args, kwargs))
2023-09-06T17:24:21.5648167Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-09-06T17:24:21.5648783Z     return super().call_function(tx, args, kwargs)
2023-09-06T17:24:21.5649648Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-09-06T17:24:21.5650277Z     return super().call_function(tx, args, kwargs)
2023-09-06T17:24:21.5651159Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-09-06T17:24:21.5651752Z     return tx.inline_user_function_return(
2023-09-06T17:24:21.5652653Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 601, in inline_user_function_return
2023-09-06T17:24:21.5653491Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-09-06T17:24:21.5654485Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2183, in inline_call
2023-09-06T17:24:21.5655134Z     return cls.inline_call_(parent, func, args, kwargs)
2023-09-06T17:24:21.5655991Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2290, in inline_call_
2023-09-06T17:24:21.5656536Z     tracer.run()
2023-09-06T17:24:21.5657283Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T17:24:21.5657810Z     and self.step()
2023-09-06T17:24:21.5658654Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T17:24:21.5659223Z     getattr(self, inst.opname)(inst)
2023-09-06T17:24:21.5660032Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T17:24:21.5660587Z     return inner_fn(self, inst)
2023-09-06T17:24:21.5661706Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1171, in CALL_FUNCTION_KW
2023-09-06T17:24:21.5662358Z     self.call_function(fn, args, kwargs)
2023-09-06T17:24:21.5663463Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 565, in call_function
2023-09-06T17:24:21.5664117Z     self.push(fn.call_function(self, args, kwargs))
2023-09-06T17:24:21.5665051Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-09-06T17:24:21.5665665Z     return super().call_function(tx, args, kwargs)
2023-09-06T17:24:21.5666515Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-09-06T17:24:21.5667138Z     return tx.inline_user_function_return(
2023-09-06T17:24:21.5668168Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 601, in inline_user_function_return
2023-09-06T17:24:21.5668996Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-09-06T17:24:21.5670215Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2183, in inline_call
2023-09-06T17:24:21.5670876Z     return cls.inline_call_(parent, func, args, kwargs)
2023-09-06T17:24:21.5671780Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2290, in inline_call_
2023-09-06T17:24:21.5672343Z     tracer.run()
2023-09-06T17:24:21.5673101Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 728, in run
2023-09-06T17:24:21.5673648Z     and self.step()
2023-09-06T17:24:21.5674484Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 691, in step
2023-09-06T17:24:21.5675097Z     getattr(self, inst.opname)(inst)
2023-09-06T17:24:21.5675922Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-09-06T17:24:21.5676913Z     return inner_fn(self, inst)
2023-09-06T17:24:21.5677778Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1171, in CALL_FUNCTION_KW
2023-09-06T17:24:21.5678458Z     self.call_function(fn, args, kwargs)
2023-09-06T17:24:21.5679306Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 565, in call_function
2023-09-06T17:24:21.5679925Z     self.push(fn.call_function(self, args, kwargs))
2023-09-06T17:24:21.5680858Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/misc.py", line 594, in call_function
2023-09-06T17:24:21.5681600Z     return self.obj.call_method(tx, self.name, args, kwargs).add_options(self)
2023-09-06T17:24:21.5682581Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/tensor.py", line 652, in call_method
2023-09-06T17:24:21.5683206Z     return wrap_fx_proxy(
2023-09-06T17:24:21.5684054Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1216, in wrap_fx_proxy
2023-09-06T17:24:21.5684667Z     return wrap_fx_proxy_cls(
2023-09-06T17:24:21.5685551Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1303, in wrap_fx_proxy_cls
2023-09-06T17:24:21.5686234Z     example_value = get_fake_value(proxy.node, tx)
2023-09-06T17:24:21.5687096Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 1371, in get_fake_value
2023-09-06T17:24:21.5687659Z     raise UserError(
2023-09-06T17:24:21.5689403Z torch._dynamo.exc.UserError: Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time.You will need to explicitly give hint to the compiler. Please take a look at constrain_as_value OR constrain_as_size APIs
2023-09-06T17:24:21.5690650Z 
2023-09-06T17:24:21.5690805Z from user code:
2023-09-06T17:24:21.5691704Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/dense_detector.py", line 95, in forward
2023-09-06T17:24:21.5692433Z     images = self.preprocess_image(batched_inputs)
2023-09-06T17:24:21.5693475Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/dense_detector.py", line 129, in preprocess_image
2023-09-06T17:24:21.5694171Z     images = ImageList.from_tensors(
2023-09-06T17:24:21.5695092Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/structures/image_list.py", line 122, in from_tensors
2023-09-06T17:24:21.5695845Z     batched_imgs = tensors[0].new_full(batch_shape, pad_value, device=device)
2023-09-06T17:24:21.5696221Z 
2023-09-06T17:24:21.5696509Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-09-06T17:24:21.5696882Z 
2023-09-06T17:24:22.8997042Z ERROR
2023-09-06T17:24:22.9045608Z speedup             gmean=0.00x mean=0.815x
2023-09-06T17:24:22.9045988Z abs_latency         gmean=0.00x mean=6.913x
2023-09-06T17:24:22.9048092Z compilation_latency mean=30.956 seconds
2023-09-06T17:24:22.9048519Z compression_ratio   mean=0.338x
2023-09-06T17:24:22.9050938Z eager_peak_mem      gmean=0.00x mean=0.445x
2023-09-06T17:24:22.9052490Z dynamo_peak_mem     gmean=0.00x mean=0.721x
2023-09-06T17:24:22.9054827Z calls_captured      gmean=0.00x mean=113.333x
2023-09-06T17:24:22.9056498Z unique_graphs       gmean=0.00x mean=0.533x
2023-09-06T17:24:22.9058256Z graph_breaks        gmean=0.00x mean=0.000x
2023-09-06T17:24:22.9059960Z unique_graph_breaks gmean=0.00x mean=0.000x
2023-09-06T17:24:23.6498553Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *maxautotune-true* ]]
2023-09-06T17:24:23.6667727Z ##[group]Run cat test/**/*.log || true
2023-09-06T17:24:23.6668066Z cat test/**/*.log || true
2023-09-06T17:24:23.6689260Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-09-06T17:24:23.6689573Z env:
2023-09-06T17:24:23.6689818Z   GIT_DEFAULT_BRANCH: main
2023-09-06T17:24:23.6690130Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T17:24:23.6690556Z   DOCKER_CONTAINER_ID: 36c47af667b537196146162b92493090315b5cd7ff3f07f8f64ee4bb99365a29
2023-09-06T17:24:23.6690896Z ##[endgroup]
2023-09-06T17:24:23.6757302Z cat: 'test/**/*.log': No such file or directory
2023-09-06T17:24:23.6791878Z Prepare all required actions
2023-09-06T17:24:23.6820164Z ##[group]Run ./.github/actions/get-workflow-job-id
2023-09-06T17:24:23.6820477Z with:
2023-09-06T17:24:23.6821372Z   github-token: ***
2023-09-06T17:24:23.6821600Z env:
2023-09-06T17:24:23.6821841Z   GIT_DEFAULT_BRANCH: main
2023-09-06T17:24:23.6822166Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T17:24:23.6822584Z   DOCKER_CONTAINER_ID: 36c47af667b537196146162b92493090315b5cd7ff3f07f8f64ee4bb99365a29
2023-09-06T17:24:23.6823129Z ##[endgroup]
2023-09-06T17:24:23.6841675Z ##[group]Run set -eux
2023-09-06T17:24:23.6841970Z set -eux
2023-09-06T17:24:23.6842354Z GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}")
2023-09-06T17:24:23.6842797Z echo "job-id=${GHA_WORKFLOW_JOB_ID}" >> "${GITHUB_OUTPUT}"
2023-09-06T17:24:23.6866431Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-09-06T17:24:23.6866775Z env:
2023-09-06T17:24:23.6867134Z   GIT_DEFAULT_BRANCH: main
2023-09-06T17:24:23.6867448Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T17:24:23.6867866Z   DOCKER_CONTAINER_ID: 36c47af667b537196146162b92493090315b5cd7ff3f07f8f64ee4bb99365a29
2023-09-06T17:24:23.6868441Z   GITHUB_TOKEN: ***
2023-09-06T17:24:23.6868687Z ##[endgroup]
2023-09-06T17:24:23.6916586Z ++ python3 .github/scripts/get_workflow_job_id.py 6093795712 gh-ci-gcp-a100-17
2023-09-06T17:24:24.7299798Z + GHA_WORKFLOW_JOB_ID=16535270177
2023-09-06T17:24:24.7300464Z + echo job-id=16535270177
2023-09-06T17:24:24.7333682Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2023-09-06T17:24:24.7334013Z kill "$MONITOR_SCRIPT_PID"
2023-09-06T17:24:24.7353353Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-09-06T17:24:24.7353721Z env:
2023-09-06T17:24:24.7353949Z   GIT_DEFAULT_BRANCH: main
2023-09-06T17:24:24.7354280Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T17:24:24.7354696Z   DOCKER_CONTAINER_ID: 36c47af667b537196146162b92493090315b5cd7ff3f07f8f64ee4bb99365a29
2023-09-06T17:24:24.7355059Z   MONITOR_SCRIPT_PID: 1951841
2023-09-06T17:24:24.7355298Z ##[endgroup]
2023-09-06T17:24:24.7476081Z Prepare all required actions
2023-09-06T17:24:24.7476748Z Getting action download info
2023-09-06T17:24:24.9984069Z Download action repository 'actions/upload-artifact@v3' (SHA:0b7f8abb1508181956e8e162db84b466c27e18ce)
2023-09-06T17:24:25.5958201Z ##[group]Run ./.github/actions/upload-test-artifacts
2023-09-06T17:24:25.5958539Z with:
2023-09-06T17:24:25.5958874Z   file-suffix: test-inductor_torchbench_perf-1-4-linux.gcp.a100.large_16535270177
2023-09-06T17:24:25.5959277Z   use-gha: anything-non-empty-to-use-gha
2023-09-06T17:24:25.5959541Z env:
2023-09-06T17:24:25.5959769Z   GIT_DEFAULT_BRANCH: main
2023-09-06T17:24:25.5960093Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T17:24:25.5960502Z   DOCKER_CONTAINER_ID: 36c47af667b537196146162b92493090315b5cd7ff3f07f8f64ee4bb99365a29
2023-09-06T17:24:25.5960824Z ##[endgroup]
2023-09-06T17:24:25.6026967Z ##[group]Run actions/upload-artifact@v3
2023-09-06T17:24:25.6027257Z with:
2023-09-06T17:24:25.6027696Z   name: test-jsons-runattempt1-test-inductor_torchbench_perf-1-4-linux.gcp.a100.large_16535270177.zip
2023-09-06T17:24:25.6028113Z   retention-days: 14
2023-09-06T17:24:25.6028366Z   if-no-files-found: warn
2023-09-06T17:24:25.6028629Z   path: test/**/*.json
2023-09-06T17:24:25.6028860Z env:
2023-09-06T17:24:25.6029268Z   GIT_DEFAULT_BRANCH: main
2023-09-06T17:24:25.6029941Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T17:24:25.6030357Z   DOCKER_CONTAINER_ID: 36c47af667b537196146162b92493090315b5cd7ff3f07f8f64ee4bb99365a29
2023-09-06T17:24:25.6030688Z ##[endgroup]
2023-09-06T17:24:25.8446269Z With the provided path, there will be 3 files uploaded
2023-09-06T17:24:25.8449202Z Starting artifact upload
2023-09-06T17:24:25.8450128Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging
2023-09-06T17:24:25.8450761Z Artifact name is valid!
2023-09-06T17:24:25.9662919Z Container for artifact "test-jsons-runattempt1-test-inductor_torchbench_perf-1-4-linux.gcp.a100.large_16535270177.zip" successfully created. Starting upload of file(s)
2023-09-06T17:24:26.3040352Z Total size of all the files uploaded is 29080 bytes
2023-09-06T17:24:26.3041091Z File upload process has finished. Finalizing the artifact upload
2023-09-06T17:24:26.4083708Z Artifact has been finalized. All files have been successfully uploaded!
2023-09-06T17:24:26.4084007Z 
2023-09-06T17:24:26.4084218Z The raw size of all the files that were specified for upload is 300200 bytes
2023-09-06T17:24:26.4084792Z The size of all the files that were uploaded is 29080 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage
2023-09-06T17:24:26.4085156Z 
2023-09-06T17:24:26.4085803Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 
2023-09-06T17:24:26.4086267Z 
2023-09-06T17:24:26.4086758Z Artifact test-jsons-runattempt1-test-inductor_torchbench_perf-1-4-linux.gcp.a100.large_16535270177.zip has been successfully uploaded!
2023-09-06T17:24:26.4210711Z ##[group]Run actions/upload-artifact@v3
2023-09-06T17:24:26.4210999Z with:
2023-09-06T17:24:26.4211404Z   name: test-reports-runattempt1-test-inductor_torchbench_perf-1-4-linux.gcp.a100.large_16535270177.zip
2023-09-06T17:24:26.4211837Z   retention-days: 14
2023-09-06T17:24:26.4212082Z   if-no-files-found: ignore
2023-09-06T17:24:26.4212366Z   path: test/**/*.xml
test/**/*.csv

2023-09-06T17:24:26.4212612Z env:
2023-09-06T17:24:26.4212835Z   GIT_DEFAULT_BRANCH: main
2023-09-06T17:24:26.4213136Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T17:24:26.4213540Z   DOCKER_CONTAINER_ID: 36c47af667b537196146162b92493090315b5cd7ff3f07f8f64ee4bb99365a29
2023-09-06T17:24:26.4213871Z ##[endgroup]
2023-09-06T17:24:26.6833131Z With the provided path, there will be 24 files uploaded
2023-09-06T17:24:26.6836449Z Starting artifact upload
2023-09-06T17:24:26.6837378Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging
2023-09-06T17:24:26.6838007Z Artifact name is valid!
2023-09-06T17:24:26.8993454Z Container for artifact "test-reports-runattempt1-test-inductor_torchbench_perf-1-4-linux.gcp.a100.large_16535270177.zip" successfully created. Starting upload of file(s)
2023-09-06T17:24:28.6538204Z Total size of all the files uploaded is 16265 bytes
2023-09-06T17:24:28.6538699Z File upload process has finished. Finalizing the artifact upload
2023-09-06T17:24:28.7492951Z Artifact has been finalized. All files have been successfully uploaded!
2023-09-06T17:24:28.7493252Z 
2023-09-06T17:24:28.7493513Z The raw size of all the files that were specified for upload is 30047 bytes
2023-09-06T17:24:28.7494130Z The size of all the files that were uploaded is 16265 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage
2023-09-06T17:24:28.7494493Z 
2023-09-06T17:24:28.7495166Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 
2023-09-06T17:24:28.7496144Z 
2023-09-06T17:24:28.7496648Z Artifact test-reports-runattempt1-test-inductor_torchbench_perf-1-4-linux.gcp.a100.large_16535270177.zip has been successfully uploaded!
2023-09-06T17:24:28.7594556Z ##[group]Run actions/upload-artifact@v3
2023-09-06T17:24:28.7594845Z with:
2023-09-06T17:24:28.7595237Z   name: usage-log-runattempt1-test-inductor_torchbench_perf-1-4-linux.gcp.a100.large_16535270177.zip
2023-09-06T17:24:28.7595623Z   retention-days: 14
2023-09-06T17:24:28.7595886Z   if-no-files-found: ignore
2023-09-06T17:24:28.7596169Z   path: usage_log.txt
test/**/*.log

2023-09-06T17:24:28.7596419Z env:
2023-09-06T17:24:28.7596645Z   GIT_DEFAULT_BRANCH: main
2023-09-06T17:24:28.7596964Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T17:24:28.7597368Z   DOCKER_CONTAINER_ID: 36c47af667b537196146162b92493090315b5cd7ff3f07f8f64ee4bb99365a29
2023-09-06T17:24:28.7597704Z ##[endgroup]
2023-09-06T17:24:29.0053224Z Multiple search paths detected. Calculating the least common ancestor of all paths
2023-09-06T17:24:29.0055640Z The least common ancestor is /home/weiwangmeta/actions-runner/_work/pytorch/pytorch. This will be the root directory of the artifact
2023-09-06T17:24:29.0058047Z With the provided path, there will be 1 file uploaded
2023-09-06T17:24:29.0060843Z Starting artifact upload
2023-09-06T17:24:29.0061805Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging
2023-09-06T17:24:29.0062443Z Artifact name is valid!
2023-09-06T17:24:29.1173792Z Container for artifact "usage-log-runattempt1-test-inductor_torchbench_perf-1-4-linux.gcp.a100.large_16535270177.zip" successfully created. Starting upload of file(s)
2023-09-06T17:24:30.3800977Z Total size of all the files uploaded is 1254423 bytes
2023-09-06T17:24:30.3801455Z File upload process has finished. Finalizing the artifact upload
2023-09-06T17:24:30.4852596Z Artifact has been finalized. All files have been successfully uploaded!
2023-09-06T17:24:30.4852920Z 
2023-09-06T17:24:30.4853295Z The raw size of all the files that were specified for upload is 65717852 bytes
2023-09-06T17:24:30.4854144Z The size of all the files that were uploaded is 1254423 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage
2023-09-06T17:24:30.4854519Z 
2023-09-06T17:24:30.4855187Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 
2023-09-06T17:24:30.4855622Z 
2023-09-06T17:24:30.4856516Z Artifact usage-log-runattempt1-test-inductor_torchbench_perf-1-4-linux.gcp.a100.large_16535270177.zip has been successfully uploaded!
2023-09-06T17:24:30.4992079Z ##[group]Run # shellcheck disable=SC2156
2023-09-06T17:24:30.4992430Z # shellcheck disable=SC2156
2023-09-06T17:24:30.4992949Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2023-09-06T17:24:30.5012443Z shell: /usr/bin/bash -e {0}
2023-09-06T17:24:30.5012747Z env:
2023-09-06T17:24:30.5012989Z   GIT_DEFAULT_BRANCH: main
2023-09-06T17:24:30.5013322Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T17:24:30.5013731Z   DOCKER_CONTAINER_ID: 36c47af667b537196146162b92493090315b5cd7ff3f07f8f64ee4bb99365a29
2023-09-06T17:24:30.5014054Z ##[endgroup]
2023-09-06T17:24:30.8080968Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main
2023-09-06T17:24:30.8081314Z with:
2023-09-06T17:24:30.8081522Z env:
2023-09-06T17:24:30.8081740Z   GIT_DEFAULT_BRANCH: main
2023-09-06T17:24:30.8082064Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T17:24:30.8082470Z   DOCKER_CONTAINER_ID: 36c47af667b537196146162b92493090315b5cd7ff3f07f8f64ee4bb99365a29
2023-09-06T17:24:30.8082809Z ##[endgroup]
2023-09-06T17:24:30.8258522Z ##[group]Run set -eou pipefail
2023-09-06T17:24:30.8259096Z set -eou pipefail
2023-09-06T17:24:30.8259342Z 
2023-09-06T17:24:30.8259677Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2023-09-06T17:24:30.8260039Z for _ in $(seq 1440); do
2023-09-06T17:24:30.8260335Z     # Break if no ssh session exists anymore
2023-09-06T17:24:30.8260652Z     if [ "$(who)" = "" ]; then
2023-09-06T17:24:30.8260939Z       break
2023-09-06T17:24:30.8261218Z     fi
2023-09-06T17:24:30.8261448Z     echo "."
2023-09-06T17:24:30.8261669Z     sleep 5
2023-09-06T17:24:30.8261896Z done
2023-09-06T17:24:30.8282038Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-09-06T17:24:30.8282349Z env:
2023-09-06T17:24:30.8282591Z   GIT_DEFAULT_BRANCH: main
2023-09-06T17:24:30.8282918Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T17:24:30.8283314Z   DOCKER_CONTAINER_ID: 36c47af667b537196146162b92493090315b5cd7ff3f07f8f64ee4bb99365a29
2023-09-06T17:24:30.8283734Z ##[endgroup]
2023-09-06T17:24:30.8326887Z Holding runner for 2 hours until all ssh sessions have logged out
2023-09-06T17:24:30.8382886Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2023-09-06T17:24:30.8383513Z # ignore expansion of "docker ps -q" since it could be empty
2023-09-06T17:24:30.8383885Z # shellcheck disable=SC2046
2023-09-06T17:24:30.8384269Z docker stop $(docker ps -q) || true
2023-09-06T17:24:30.8384593Z # Prune all of the docker images
2023-09-06T17:24:30.8384877Z docker system prune -af
2023-09-06T17:24:30.8404261Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-09-06T17:24:30.8404570Z env:
2023-09-06T17:24:30.8404817Z   GIT_DEFAULT_BRANCH: main
2023-09-06T17:24:30.8405146Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-09-06T17:24:30.8405547Z   DOCKER_CONTAINER_ID: 36c47af667b537196146162b92493090315b5cd7ff3f07f8f64ee4bb99365a29
2023-09-06T17:24:30.8405901Z ##[endgroup]
2023-09-06T17:24:31.4509611Z 36c47af667b5
2023-09-06T17:24:35.8214165Z Deleted Containers:
2023-09-06T17:24:35.8214844Z 36c47af667b537196146162b92493090315b5cd7ff3f07f8f64ee4bb99365a29
2023-09-06T17:24:35.8215234Z 
2023-09-06T17:24:43.1517025Z Deleted Images:
2023-09-06T17:24:43.1518417Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:9dd361d1c04129f8eaa9d6b43335917800dd6d24
2023-09-06T17:24:43.1520124Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks@sha256:ea376d3cb14a9400b57a5e0e5bc49830f955ac984dd12fde7eab8e45479d06ee
2023-09-06T17:24:43.1521028Z deleted: sha256:d5a715adf5384149f42b562408a0548773cf7fc344221f5b65a1c1a685e1c2a8
2023-09-06T17:24:43.1521849Z deleted: sha256:a6860550ba817cd2f1793140ea7cf179d23c260ab4072ee325e0ed2a1e249a60
2023-09-06T17:24:43.1522716Z deleted: sha256:0dbc01f49755a72fd098fd6c2d1ade3232b19cf4d36711a268597345f6b4aee9
2023-09-06T17:24:43.1523580Z deleted: sha256:f336a347fbc86157fe5445109f1117eb97c2e74a709b0c32057e9438ea1e23f4
2023-09-06T17:24:43.1524502Z deleted: sha256:93adc0249942a8898738f81fa44190a7fceeb883cf8a737ce7effd628d075132
2023-09-06T17:24:43.1525332Z deleted: sha256:245fc29afb2c682e83abe08c41c9b384903274dc44a62a690c60019f970691cd
2023-09-06T17:24:43.1525792Z deleted: sha256:7c02af11e936ad4089e62628e114ba1c4ee5516e4c32201f976d73adb15a4119
2023-09-06T17:24:43.1526665Z deleted: sha256:40b1aa0a0854c76d066d9bb2902e4d8504cc8eabda71ce1223ec9346a1c872ba
2023-09-06T17:24:43.1527400Z deleted: sha256:3630034928544cb0509cfc2515bcb2d4f92f1ee8fa0bb8ff5394fe9cb870f9d7
2023-09-06T17:24:43.1527914Z deleted: sha256:5861945ff388c9e45a595e11b6380f8967907e2545b8de3d965e62b4d177be63
2023-09-06T17:24:43.1528350Z deleted: sha256:90d0a508d21f8ec158e206f98b9c21f709caefa358bbb755925d7b84ae7d03c3
2023-09-06T17:24:43.1528805Z deleted: sha256:a592721ed0653085f29b9bfdafccaff21a6d2fb48b0b03e6bbba1259b6db436a
2023-09-06T17:24:43.1529475Z deleted: sha256:e99f945232c08867864c9aa87c50f515190f25bc67a229ace189400a05fb31f9
2023-09-06T17:24:43.1529905Z deleted: sha256:3e47d845dde1f8a984423ad9cc8f261305717a9b92e57aa9521f1d6255a89af4
2023-09-06T17:24:43.1530443Z deleted: sha256:30d0cdadfb570c7a543a49b53522ec0f293bc5dcb92cc2c094d50bc6cb68bfb5
2023-09-06T17:24:43.1530918Z deleted: sha256:542209ba17c53f1e699f48e5f90f2c80636abeaebbc373643e6108f545e75ce8
2023-09-06T17:24:43.1531372Z deleted: sha256:05e4dbd1ddd076f76ac590f1f0ae38200ae17c02a22c4aa1a204089ac0171abd
2023-09-06T17:24:43.1531827Z deleted: sha256:e5004623f4683537ea2a9cd21c6a0b3036f569b2620a18d61a24712c3d6317e9
2023-09-06T17:24:43.1532249Z deleted: sha256:4f9e0570e948c4510e75be415389c83c2f36b83ff78a5f4528325e4b1bb6a549
2023-09-06T17:24:43.1532785Z deleted: sha256:5b8c31773008e34484f73b1ca406ef11475238420e2cc93ca8be6f14e97a60ba
2023-09-06T17:24:43.1533316Z deleted: sha256:bc7e36046a4b3e944dd688750bbf14dffbbb423a68f4fe2281d20d3abb109bc8
2023-09-06T17:24:43.1533815Z deleted: sha256:2eb070c73312bf53948c78d07e247816b5cd9349f34719eae28fe0d42643e3a4
2023-09-06T17:24:43.1534268Z deleted: sha256:666690e29b73cba5f2bcff9e87d2bd54af5b3ab82d48f332130c44cb7da4dc24
2023-09-06T17:24:43.1534728Z deleted: sha256:31058a0b4b45b86e010a9da2699e7a8b601fb5a08e1befa48b13474d49ed3306
2023-09-06T17:24:43.1535177Z deleted: sha256:da96c92d4dc247ff972962b17c50229145eecd6786676a41eadc38be1955fbd0
2023-09-06T17:24:43.1535630Z deleted: sha256:ec645b413fcd663d08c12ca4859da2680cc8597cdfeef1bbc22202aa0254c737
2023-09-06T17:24:43.1536069Z deleted: sha256:3713ae5e7ffe5b09185393291002388f51a08126a72077ee527752f440467b75
2023-09-06T17:24:43.1536632Z deleted: sha256:e4ce9691a8267e62e43dad598091568b9500502990cedb210df2ca161025a2a6
2023-09-06T17:24:43.1537270Z deleted: sha256:c4367a260dc8cd805d4315fe8cfddcbd2f7596337273a8af0c586384b71cc929
2023-09-06T17:24:43.1537751Z deleted: sha256:bd8a7a181da8078799b432efaa7eac7a27129b42477c05dfb6bc41ca7e404037
2023-09-06T17:24:43.1538216Z deleted: sha256:6f4107fce1f5c1d3a14ae4dcfe63d1b32bcc0271fe9511c29a7a38a31f782089
2023-09-06T17:24:43.1538677Z deleted: sha256:b9f976d7987f9b5613cfb1f05ab9369e62147ec09e568a265157e0788f302ee0
2023-09-06T17:24:43.1539121Z deleted: sha256:ff4d86462bb0a5d0e899ba7a435a7d86a67f3da04a4e0e58d93d591280c2878f
2023-09-06T17:24:43.1539595Z deleted: sha256:cf1ca9efe25cee8c9c769fe2d61882d3f38a3681b4ee90a553473511925bac14
2023-09-06T17:24:43.1540059Z deleted: sha256:f80bf36cc0645a23594a1965bcb45bc77540d4b13dfe592c63e7678c63a7bece
2023-09-06T17:24:43.1540489Z deleted: sha256:105b6238d3d8fb98ce63c980fe2264e638318e8f722c716a7138d680d48fef34
2023-09-06T17:24:43.1540922Z deleted: sha256:e814e2932ca219f05fa61c9634d66f2ebbc3965a864d39666bd65267e7d1530d
2023-09-06T17:24:43.1541351Z deleted: sha256:48833ded82527862476e593e4552d21ff8b1b3d592db5273d434cbb63680c4a0
2023-09-06T17:24:43.1541804Z deleted: sha256:bd934d82d83dd7d51bf639691acf95a2c16c6639cc309b7ec6f0b4b445070259
2023-09-06T17:24:43.1542244Z deleted: sha256:202f5632f2ca3ac0f1b5ff02866d6dd48b88e076774f86e8607f353a7999e819
2023-09-06T17:24:43.1542696Z deleted: sha256:2b09a5cff18522e53e874f83c8e142cec00c1ecd25e1e80813b5106aa582ed77
2023-09-06T17:24:43.1543137Z deleted: sha256:3461a011917448b5cd441f64b32dfcc74df77cb34d294103fc15b2f499674ace
2023-09-06T17:24:43.1543579Z deleted: sha256:ca1a88204f52a1a7ed0ba3a8d4e727a81c3901adf00a0e82000090747edac77a
2023-09-06T17:24:43.1544073Z deleted: sha256:a03b7e5348e4a9f52a9bc6a7e6a721901f8ff81d9c990be285e9514896e60014
2023-09-06T17:24:43.1544718Z deleted: sha256:81df849664843e70024d3739aafc4ac07671e5b396c997460e01246e3fee8e5c
2023-09-06T17:24:43.1545145Z deleted: sha256:8dc7c12508965fc88f62760ae386c1e33c5632e593321d57d0662b2730c13ef6
2023-09-06T17:24:43.1545585Z deleted: sha256:a1640700f10f70cc1e95f62e6e85b58eedb2bf5667f1c5179541767d284dd3a2
2023-09-06T17:24:43.1546034Z deleted: sha256:6fa796fb0b1943c4085305943a995094fde9c1ee8293fbc08d9c90c88384e8d2
2023-09-06T17:24:43.1546469Z deleted: sha256:9d62fe17bb848700de25ca2d91e90e6b08c6d999ab20a4f2f1090eea5cd20a6e
2023-09-06T17:24:43.1546928Z deleted: sha256:74f7a2dd742da12cad04b173b7cdfa33547293e4cf3f0026a790917143185551
2023-09-06T17:24:43.1547614Z deleted: sha256:0c0b361cfce253427e6ca97978463a1c856229e198ddfaacfc8a7270f1379505
2023-09-06T17:24:43.1548074Z deleted: sha256:97e83f61281bbd15b6fc299c48bc0a291271c9eae6edc8ae779de269f9408a27
2023-09-06T17:24:43.1548511Z deleted: sha256:009bc094f59e67fa581a79042f8dda827bc86bd3645977bc67228fa8d99f6e2b
2023-09-06T17:24:43.1548978Z deleted: sha256:f2f5c8c8a2edc3ae51ad9976703fc85e6b5e34f569b789db6cd40800a8ca9de2
2023-09-06T17:24:43.1549777Z deleted: sha256:4f3e23ed3f49b8139feb925eec6c92e3577aea450dc80c7342a892451c64b9a0
2023-09-06T17:24:43.1550224Z deleted: sha256:4da783b7b3eb1789f68b48b4e6605b0608671188b3dfe9475811c51f835f86a7
2023-09-06T17:24:43.1550681Z deleted: sha256:51cc6bc5a58f435a446deb6a91cfcdce0fb3bf1e33e8ab27cd9bfbbc41bc5ed6
2023-09-06T17:24:43.1551168Z deleted: sha256:dac932e4e5486656bab1762b210caabc344f2b857c4a6e842508f4e7a322defc
2023-09-06T17:24:43.1551619Z deleted: sha256:34f782915cf51fa7bfebb671648424b4e5c60bc1ba619af822de17c622dc0860
2023-09-06T17:24:43.1552070Z deleted: sha256:535334b0e534587a4ca7fbe99074d8848adc6baf3a7d1d0bc5202428ad07a466
2023-09-06T17:24:43.1552521Z deleted: sha256:13f93f55af415a19540beedb91537d8aaeea9f47dde0558cf70883d7f8fb8b5d
2023-09-06T17:24:43.1552973Z deleted: sha256:991a554027e504dc8fdf1bd4ecf3ac527ff97c17ebd97f2b12754ddd224c3e64
2023-09-06T17:24:43.1553433Z deleted: sha256:d1a674dc61f6276fe006a383e273def82d1101a845a3a52e4cd8d424b919161c
2023-09-06T17:24:43.1553926Z deleted: sha256:a0224c222e71e7155aef719d5900e7373b81021b9e569fb84a2b45e2317342b8
2023-09-06T17:24:43.1554371Z deleted: sha256:ace98a62b5d75f64dc251f788a5ad5883e49859785a6db0c31fd79a11908961a
2023-09-06T17:24:43.1554839Z deleted: sha256:a2cd7462e4fdf5bc8c0d39a39fe58f2315e3dfbd66de33bb8bcaf12616a1cb1a
2023-09-06T17:24:43.1555321Z deleted: sha256:6f37ca73c74f2cef0ddefd960260f2033c16c84583c5507a4f37b1cf7631dc20
2023-09-06T17:24:43.1555767Z 
2023-09-06T17:24:43.1555939Z Total reclaimed space: 36.76GB
2023-09-06T17:24:43.1628475Z Post job cleanup.
2023-09-06T17:24:43.1673857Z Post job cleanup.
2023-09-06T17:24:43.2824286Z [command]/usr/bin/git version
2023-09-06T17:24:43.2878797Z git version 2.25.1
2023-09-06T17:24:43.2928957Z Temporarily overriding HOME='/home/weiwangmeta/actions-runner/_work/_temp/fa53127c-e90b-496e-9f28-735ab5c175fb' before making global git config changes
2023-09-06T17:24:43.2929681Z Adding repository directory to the temporary git global config as a safe directory
2023-09-06T17:24:43.2936358Z [command]/usr/bin/git config --global --add safe.directory /home/weiwangmeta/actions-runner/_work/pytorch/pytorch
2023-09-06T17:24:43.2986942Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2023-09-06T17:24:43.3029863Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2023-09-06T17:24:43.3318222Z Entering 'android/libs/fbjni'
2023-09-06T17:24:43.3369823Z Entering 'third_party/FP16'
2023-09-06T17:24:43.3418087Z Entering 'third_party/FXdiv'
2023-09-06T17:24:43.3466003Z Entering 'third_party/NNPACK'
2023-09-06T17:24:43.3512553Z Entering 'third_party/QNNPACK'
2023-09-06T17:24:43.3561186Z Entering 'third_party/VulkanMemoryAllocator'
2023-09-06T17:24:43.3607364Z Entering 'third_party/XNNPACK'
2023-09-06T17:24:43.3674697Z Entering 'third_party/benchmark'
2023-09-06T17:24:43.3721554Z Entering 'third_party/cpuinfo'
2023-09-06T17:24:43.3767014Z Entering 'third_party/cub'
2023-09-06T17:24:43.3816080Z Entering 'third_party/cudnn_frontend'
2023-09-06T17:24:43.3869874Z Entering 'third_party/cutlass'
2023-09-06T17:24:43.3926743Z Entering 'third_party/eigen'
2023-09-06T17:24:43.3977425Z Entering 'third_party/fbgemm'
2023-09-06T17:24:43.4024080Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-09-06T17:24:43.4070634Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-09-06T17:24:43.4117508Z Entering 'third_party/fbgemm/third_party/cutlass'
2023-09-06T17:24:43.4172218Z Entering 'third_party/fbgemm/third_party/googletest'
2023-09-06T17:24:43.4217615Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-09-06T17:24:43.4265161Z Entering 'third_party/flatbuffers'
2023-09-06T17:24:43.4315713Z Entering 'third_party/fmt'
2023-09-06T17:24:43.4363529Z Entering 'third_party/foxi'
2023-09-06T17:24:43.4409934Z Entering 'third_party/gemmlowp/gemmlowp'
2023-09-06T17:24:43.4456915Z Entering 'third_party/gloo'
2023-09-06T17:24:43.4502356Z Entering 'third_party/googletest'
2023-09-06T17:24:43.4549403Z Entering 'third_party/ideep'
2023-09-06T17:24:43.4594311Z Entering 'third_party/ideep/mkl-dnn'
2023-09-06T17:24:43.4648352Z Entering 'third_party/ios-cmake'
2023-09-06T17:24:43.4694725Z Entering 'third_party/ittapi'
2023-09-06T17:24:43.4741913Z Entering 'third_party/kineto'
2023-09-06T17:24:43.4791312Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2023-09-06T17:24:43.4837826Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2023-09-06T17:24:43.4887571Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2023-09-06T17:24:43.4933451Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2023-09-06T17:24:43.4980534Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2023-09-06T17:24:43.5025259Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2023-09-06T17:24:43.5072852Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2023-09-06T17:24:43.5118927Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2023-09-06T17:24:43.5165667Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2023-09-06T17:24:43.5213138Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2023-09-06T17:24:43.5261947Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-09-06T17:24:43.5310468Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-09-06T17:24:43.5489408Z Entering 'third_party/mimalloc'
2023-09-06T17:24:43.5540178Z Entering 'third_party/nccl/nccl'
2023-09-06T17:24:43.5586582Z Entering 'third_party/neon2sse'
2023-09-06T17:24:43.5634054Z Entering 'third_party/nlohmann'
2023-09-06T17:24:43.5681614Z Entering 'third_party/onnx'
2023-09-06T17:24:43.5744461Z Entering 'third_party/onnx/third_party/benchmark'
2023-09-06T17:24:43.5791505Z Entering 'third_party/onnx/third_party/pybind11'
2023-09-06T17:24:43.5840040Z Entering 'third_party/onnx-tensorrt'
2023-09-06T17:24:43.5885016Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-09-06T17:24:43.5935935Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-09-06T17:24:43.5984619Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-09-06T17:24:43.6034353Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-09-06T17:24:43.6089031Z Entering 'third_party/pocketfft'
2023-09-06T17:24:43.6141765Z Entering 'third_party/protobuf'
2023-09-06T17:24:43.6195651Z Entering 'third_party/protobuf/third_party/benchmark'
2023-09-06T17:24:43.6244069Z Entering 'third_party/protobuf/third_party/googletest'
2023-09-06T17:24:43.6293668Z Entering 'third_party/psimd'
2023-09-06T17:24:43.6342274Z Entering 'third_party/pthreadpool'
2023-09-06T17:24:43.6388573Z Entering 'third_party/pybind11'
2023-09-06T17:24:43.6437392Z Entering 'third_party/python-peachpy'
2023-09-06T17:24:43.6485630Z Entering 'third_party/sleef'
2023-09-06T17:24:43.6534269Z Entering 'third_party/tbb'
2023-09-06T17:24:43.6585126Z Entering 'third_party/tensorpipe'
2023-09-06T17:24:43.6635288Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-09-06T17:24:43.6682850Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-09-06T17:24:43.6728452Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-09-06T17:24:43.6776975Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-09-06T17:24:43.6821122Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-09-06T17:24:43.6873435Z Entering 'third_party/zstd'
2023-09-06T17:24:43.6940600Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2023-09-06T17:24:43.6975886Z http.https://github.com/.extraheader
2023-09-06T17:24:43.6989641Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2023-09-06T17:24:43.7035093Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2023-09-06T17:24:43.7325582Z Entering 'android/libs/fbjni'
2023-09-06T17:24:43.7353406Z http.https://github.com/.extraheader
2023-09-06T17:24:43.7392187Z Entering 'third_party/FP16'
2023-09-06T17:24:43.7419336Z http.https://github.com/.extraheader
2023-09-06T17:24:43.7456014Z Entering 'third_party/FXdiv'
2023-09-06T17:24:43.7483857Z http.https://github.com/.extraheader
2023-09-06T17:24:43.7521505Z Entering 'third_party/NNPACK'
2023-09-06T17:24:43.7549439Z http.https://github.com/.extraheader
2023-09-06T17:24:43.7587489Z Entering 'third_party/QNNPACK'
2023-09-06T17:24:43.7615631Z http.https://github.com/.extraheader
2023-09-06T17:24:43.7652832Z Entering 'third_party/VulkanMemoryAllocator'
2023-09-06T17:24:43.7680152Z http.https://github.com/.extraheader
2023-09-06T17:24:43.7717282Z Entering 'third_party/XNNPACK'
2023-09-06T17:24:43.7744287Z http.https://github.com/.extraheader
2023-09-06T17:24:43.7795857Z Entering 'third_party/benchmark'
2023-09-06T17:24:43.7823759Z http.https://github.com/.extraheader
2023-09-06T17:24:43.7860259Z Entering 'third_party/cpuinfo'
2023-09-06T17:24:43.7887974Z http.https://github.com/.extraheader
2023-09-06T17:24:43.7926819Z Entering 'third_party/cub'
2023-09-06T17:24:43.7953155Z http.https://github.com/.extraheader
2023-09-06T17:24:43.7990754Z Entering 'third_party/cudnn_frontend'
2023-09-06T17:24:43.8017583Z http.https://github.com/.extraheader
2023-09-06T17:24:43.8062467Z Entering 'third_party/cutlass'
2023-09-06T17:24:43.8090973Z http.https://github.com/.extraheader
2023-09-06T17:24:43.8140499Z Entering 'third_party/eigen'
2023-09-06T17:24:43.8168152Z http.https://github.com/.extraheader
2023-09-06T17:24:43.8210756Z Entering 'third_party/fbgemm'
2023-09-06T17:24:43.8237426Z http.https://github.com/.extraheader
2023-09-06T17:24:43.8277173Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-09-06T17:24:43.8303519Z http.https://github.com/.extraheader
2023-09-06T17:24:43.8342147Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-09-06T17:24:43.8370156Z http.https://github.com/.extraheader
2023-09-06T17:24:43.8409812Z Entering 'third_party/fbgemm/third_party/cutlass'
2023-09-06T17:24:43.8437088Z http.https://github.com/.extraheader
2023-09-06T17:24:43.8484824Z Entering 'third_party/fbgemm/third_party/googletest'
2023-09-06T17:24:43.8513094Z http.https://github.com/.extraheader
2023-09-06T17:24:43.8552301Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-09-06T17:24:43.8579269Z http.https://github.com/.extraheader
2023-09-06T17:24:43.8619549Z Entering 'third_party/flatbuffers'
2023-09-06T17:24:43.8647791Z http.https://github.com/.extraheader
2023-09-06T17:24:43.8689893Z Entering 'third_party/fmt'
2023-09-06T17:24:43.8717946Z http.https://github.com/.extraheader
2023-09-06T17:24:43.8755493Z Entering 'third_party/foxi'
2023-09-06T17:24:43.8783191Z http.https://github.com/.extraheader
2023-09-06T17:24:43.8820251Z Entering 'third_party/gemmlowp/gemmlowp'
2023-09-06T17:24:43.8846294Z http.https://github.com/.extraheader
2023-09-06T17:24:43.8882218Z Entering 'third_party/gloo'
2023-09-06T17:24:43.8910459Z http.https://github.com/.extraheader
2023-09-06T17:24:43.8946036Z Entering 'third_party/googletest'
2023-09-06T17:24:43.8972876Z http.https://github.com/.extraheader
2023-09-06T17:24:43.9010362Z Entering 'third_party/ideep'
2023-09-06T17:24:43.9037162Z http.https://github.com/.extraheader
2023-09-06T17:24:43.9073854Z Entering 'third_party/ideep/mkl-dnn'
2023-09-06T17:24:43.9101225Z http.https://github.com/.extraheader
2023-09-06T17:24:43.9145855Z Entering 'third_party/ios-cmake'
2023-09-06T17:24:43.9172338Z http.https://github.com/.extraheader
2023-09-06T17:24:43.9208200Z Entering 'third_party/ittapi'
2023-09-06T17:24:43.9236745Z http.https://github.com/.extraheader
2023-09-06T17:24:43.9274567Z Entering 'third_party/kineto'
2023-09-06T17:24:43.9300817Z http.https://github.com/.extraheader
2023-09-06T17:24:43.9339670Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2023-09-06T17:24:43.9367427Z http.https://github.com/.extraheader
2023-09-06T17:24:43.9406782Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2023-09-06T17:24:43.9435099Z http.https://github.com/.extraheader
2023-09-06T17:24:43.9476279Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2023-09-06T17:24:43.9504355Z http.https://github.com/.extraheader
2023-09-06T17:24:43.9542574Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2023-09-06T17:24:43.9569363Z http.https://github.com/.extraheader
2023-09-06T17:24:43.9607657Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2023-09-06T17:24:43.9635598Z http.https://github.com/.extraheader
2023-09-06T17:24:43.9673633Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2023-09-06T17:24:43.9701836Z http.https://github.com/.extraheader
2023-09-06T17:24:43.9742717Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2023-09-06T17:24:43.9769757Z http.https://github.com/.extraheader
2023-09-06T17:24:43.9807816Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2023-09-06T17:24:43.9835161Z http.https://github.com/.extraheader
2023-09-06T17:24:43.9875048Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2023-09-06T17:24:43.9902829Z http.https://github.com/.extraheader
2023-09-06T17:24:43.9942734Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2023-09-06T17:24:43.9969215Z http.https://github.com/.extraheader
2023-09-06T17:24:44.0009512Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-09-06T17:24:44.0035953Z http.https://github.com/.extraheader
2023-09-06T17:24:44.0074266Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-09-06T17:24:44.0099878Z http.https://github.com/.extraheader
2023-09-06T17:24:44.0139778Z Entering 'third_party/mimalloc'
2023-09-06T17:24:44.0167010Z http.https://github.com/.extraheader
2023-09-06T17:24:44.0205586Z Entering 'third_party/nccl/nccl'
2023-09-06T17:24:44.0231996Z http.https://github.com/.extraheader
2023-09-06T17:24:44.0270924Z Entering 'third_party/neon2sse'
2023-09-06T17:24:44.0297450Z http.https://github.com/.extraheader
2023-09-06T17:24:44.0335310Z Entering 'third_party/nlohmann'
2023-09-06T17:24:44.0363334Z http.https://github.com/.extraheader
2023-09-06T17:24:44.0403447Z Entering 'third_party/onnx'
2023-09-06T17:24:44.0430564Z http.https://github.com/.extraheader
2023-09-06T17:24:44.0487676Z Entering 'third_party/onnx/third_party/benchmark'
2023-09-06T17:24:44.0515871Z http.https://github.com/.extraheader
2023-09-06T17:24:44.0554345Z Entering 'third_party/onnx/third_party/pybind11'
2023-09-06T17:24:44.0581627Z http.https://github.com/.extraheader
2023-09-06T17:24:44.0624778Z Entering 'third_party/onnx-tensorrt'
2023-09-06T17:24:44.0653445Z http.https://github.com/.extraheader
2023-09-06T17:24:44.0691861Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-09-06T17:24:44.0720743Z http.https://github.com/.extraheader
2023-09-06T17:24:44.0765249Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-09-06T17:24:44.0793342Z http.https://github.com/.extraheader
2023-09-06T17:24:44.0832966Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-09-06T17:24:44.0859759Z http.https://github.com/.extraheader
2023-09-06T17:24:44.0898088Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-09-06T17:24:44.0925327Z http.https://github.com/.extraheader
2023-09-06T17:24:44.0970122Z Entering 'third_party/pocketfft'
2023-09-06T17:24:44.0996620Z http.https://github.com/.extraheader
2023-09-06T17:24:44.1033673Z Entering 'third_party/protobuf'
2023-09-06T17:24:44.1059769Z http.https://github.com/.extraheader
2023-09-06T17:24:44.1100610Z Entering 'third_party/protobuf/third_party/benchmark'
2023-09-06T17:24:44.1127588Z http.https://github.com/.extraheader
2023-09-06T17:24:44.1165297Z Entering 'third_party/protobuf/third_party/googletest'
2023-09-06T17:24:44.1191089Z http.https://github.com/.extraheader
2023-09-06T17:24:44.1230270Z Entering 'third_party/psimd'
2023-09-06T17:24:44.1256984Z http.https://github.com/.extraheader
2023-09-06T17:24:44.1292572Z Entering 'third_party/pthreadpool'
2023-09-06T17:24:44.1319537Z http.https://github.com/.extraheader
2023-09-06T17:24:44.1356677Z Entering 'third_party/pybind11'
2023-09-06T17:24:44.1382277Z http.https://github.com/.extraheader
2023-09-06T17:24:44.1419756Z Entering 'third_party/python-peachpy'
2023-09-06T17:24:44.1446985Z http.https://github.com/.extraheader
2023-09-06T17:24:44.1483835Z Entering 'third_party/sleef'
2023-09-06T17:24:44.1510482Z http.https://github.com/.extraheader
2023-09-06T17:24:44.1546219Z Entering 'third_party/tbb'
2023-09-06T17:24:44.1573762Z http.https://github.com/.extraheader
2023-09-06T17:24:44.1611859Z Entering 'third_party/tensorpipe'
2023-09-06T17:24:44.1639018Z http.https://github.com/.extraheader
2023-09-06T17:24:44.1675236Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-09-06T17:24:44.1701179Z http.https://github.com/.extraheader
2023-09-06T17:24:44.1737534Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-09-06T17:24:44.1763964Z http.https://github.com/.extraheader
2023-09-06T17:24:44.1799628Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-09-06T17:24:44.1825729Z http.https://github.com/.extraheader
2023-09-06T17:24:44.1862047Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-09-06T17:24:44.1888546Z http.https://github.com/.extraheader
2023-09-06T17:24:44.1923304Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-09-06T17:24:44.1949404Z http.https://github.com/.extraheader
2023-09-06T17:24:44.1987904Z Entering 'third_party/zstd'
2023-09-06T17:24:44.2015106Z http.https://github.com/.extraheader
2023-09-06T17:24:44.2154875Z A job completed hook has been configured by the self-hosted runner administrator
2023-09-06T17:24:44.2183387Z ##[group]Run '/home/weiwangmeta/post-job.sh'
2023-09-06T17:24:44.2201477Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-09-06T17:24:44.2201804Z ##[endgroup]
2023-09-06T17:24:44.2242345Z The approximate cost for this shard is shown below (in $): 
2023-09-06T17:24:44.2271120Z 22.673
2023-09-06T17:24:44.2478217Z Cleaning up orphan processes