2023-08-17T07:28:01.5098842Z Requested labels: linux.gcp.a100.large
2023-08-17T07:28:01.5099067Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/main
2023-08-17T07:28:01.5099311Z Reusable workflow chain:
2023-08-17T07:28:01.5099465Z pytorch/pytorch/.github/workflows/inductor-perf-test-nightly.yml@refs/heads/main (384e0d104fd077d31efafc564129660e9b7a0f25)
2023-08-17T07:28:01.5099964Z -> pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/main (384e0d104fd077d31efafc564129660e9b7a0f25)
2023-08-17T07:28:01.5100473Z Waiting for a runner to pick up this job...
2023-08-17T10:43:59.8719459Z Job is about to start running on the runner: gh-ci-gcp-a100-12 (repository)
2023-08-17T10:44:02.7415212Z Current runner version: '2.308.0'
2023-08-17T10:44:02.7423214Z Runner name: 'gh-ci-gcp-a100-12'
2023-08-17T10:44:02.7423863Z Runner group name: 'Default'
2023-08-17T10:44:02.7424619Z Machine name: 'gh-ci-gcp-a100-12'
2023-08-17T10:44:02.7427247Z ##[group]GITHUB_TOKEN Permissions
2023-08-17T10:44:02.7428201Z Actions: write
2023-08-17T10:44:02.7428552Z Checks: write
2023-08-17T10:44:02.7428975Z Contents: write
2023-08-17T10:44:02.7429364Z Deployments: write
2023-08-17T10:44:02.7429759Z Discussions: write
2023-08-17T10:44:02.7430101Z Issues: write
2023-08-17T10:44:02.7430469Z Metadata: read
2023-08-17T10:44:02.7430878Z Packages: write
2023-08-17T10:44:02.7431258Z Pages: write
2023-08-17T10:44:02.7431601Z PullRequests: write
2023-08-17T10:44:02.7432058Z RepositoryProjects: write
2023-08-17T10:44:02.7432513Z SecurityEvents: write
2023-08-17T10:44:02.7432911Z Statuses: write
2023-08-17T10:44:02.7433235Z ##[endgroup]
2023-08-17T10:44:02.7437058Z Secret source: Actions
2023-08-17T10:44:02.7437952Z Prepare workflow directory
2023-08-17T10:44:03.0223153Z Prepare all required actions
2023-08-17T10:44:03.0441184Z Getting action download info
2023-08-17T10:44:03.2895702Z Download action repository 'pytorch/test-infra@main' (SHA:c5b77a44fcc51819e48db99c37fd3509a61ac4dd)
2023-08-17T10:44:03.9827176Z Download action repository 'pytorch/pytorch@main' (SHA:a98f745c808380e40c37b65f0e27ca7ff1a60390)
2023-08-17T10:44:08.9478131Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2023-08-17T10:44:09.5120283Z Getting action download info
2023-08-17T10:44:09.7314422Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c)
2023-08-17T10:44:10.2500688Z Getting action download info
2023-08-17T10:44:10.3739903Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482)
2023-08-17T10:44:10.7802357Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/main (384e0d104fd077d31efafc564129660e9b7a0f25)
2023-08-17T10:44:10.7805452Z ##[group] Inputs
2023-08-17T10:44:10.7805861Z   build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm80
2023-08-17T10:44:10.7807590Z   test-matrix: {"include": [{"config": "inductor_huggingface_perf", "shard": 1, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_huggingface_perf", "shard": 2, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_huggingface_perf", "shard": 3, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 1, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 2, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 3, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 4, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 5, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 1, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 2, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 3, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 4, "num_shards": 4, "runner": "linux.gcp.a100.large"}]}
2023-08-17T10:44:10.7809639Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:f89d62c8626a24aae4b1a64d42a3caf9cd95bddd
2023-08-17T10:44:10.7810201Z   sync-tag: 
2023-08-17T10:44:10.7811058Z   timeout-minutes: 720
2023-08-17T10:44:10.7811372Z   use-gha: anything-non-empty-to-use-gha
2023-08-17T10:44:10.7812086Z   dashboard-tag: training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true
2023-08-17T10:44:10.7813217Z ##[endgroup]
2023-08-17T10:44:10.7814129Z Complete job name: cuda12.1-py3.10-gcc9-sm80 / test (inductor_huggingface_perf, 3, 3, linux.gcp.a100.large)
2023-08-17T10:44:10.8480115Z A job started hook has been configured by the self-hosted runner administrator
2023-08-17T10:44:10.8659624Z ##[group]Run '/home/weiwangmeta/pre-job.sh'
2023-08-17T10:44:10.8683284Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-08-17T10:44:10.8683623Z ##[endgroup]
2023-08-17T10:44:10.9297296Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main
2023-08-17T10:44:10.9297833Z with:
2023-08-17T10:44:10.9298193Z   submodules: recursive
2023-08-17T10:44:10.9298616Z   fetch-depth: 0
2023-08-17T10:44:10.9298991Z env:
2023-08-17T10:44:10.9299378Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:44:10.9299781Z ##[endgroup]
2023-08-17T10:44:10.9535804Z ##[group]Run retry () {
2023-08-17T10:44:10.9536296Z retry () {
2023-08-17T10:44:10.9536875Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2023-08-17T10:44:10.9537374Z }
2023-08-17T10:44:10.9537778Z echo "${GITHUB_WORKSPACE}"
2023-08-17T10:44:10.9538254Z if [ -z "${NO_SUDO}" ]; then
2023-08-17T10:44:10.9538773Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2023-08-17T10:44:10.9539236Z else
2023-08-17T10:44:10.9539677Z   retry rm -rf "${GITHUB_WORKSPACE}"
2023-08-17T10:44:10.9540117Z fi
2023-08-17T10:44:10.9540613Z mkdir "${GITHUB_WORKSPACE}"
2023-08-17T10:44:10.9560392Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-08-17T10:44:10.9560735Z env:
2023-08-17T10:44:10.9560949Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:44:10.9561188Z   NO_SUDO: 
2023-08-17T10:44:10.9561406Z ##[endgroup]
2023-08-17T10:44:10.9639133Z /home/weiwangmeta/actions-runner/_work/pytorch/pytorch
2023-08-17T10:44:13.6604014Z ##[group]Run malfet/checkout@silent-checkout
2023-08-17T10:44:13.6604361Z with:
2023-08-17T10:44:13.6604609Z   ref: 384e0d104fd077d31efafc564129660e9b7a0f25
2023-08-17T10:44:13.6604905Z   fetch-depth: 0
2023-08-17T10:44:13.6605149Z   submodules: recursive
2023-08-17T10:44:13.6605400Z   quiet-checkout: true
2023-08-17T10:44:13.6605648Z   repository: pytorch/pytorch
2023-08-17T10:44:13.6606132Z   token: ***
2023-08-17T10:44:13.6606352Z   ssh-strict: true
2023-08-17T10:44:13.6606610Z   persist-credentials: true
2023-08-17T10:44:13.6606851Z   clean: true
2023-08-17T10:44:13.6607133Z   sparse-checkout-cone-mode: true
2023-08-17T10:44:13.6607398Z   lfs: false
2023-08-17T10:44:13.6607631Z   set-safe-directory: true
2023-08-17T10:44:13.6607856Z env:
2023-08-17T10:44:13.6608072Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:44:13.6608308Z ##[endgroup]
2023-08-17T10:44:13.7972294Z Syncing repository: pytorch/pytorch
2023-08-17T10:44:13.7974040Z ##[group]Getting Git version info
2023-08-17T10:44:13.7974625Z Working directory is '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch'
2023-08-17T10:44:13.7975416Z Unexpected error attempting to determine if executable file exists '/home/weiwangmeta/.local/bin/git': Error: EACCES: permission denied, stat '/home/weiwangmeta/.local/bin/git'
2023-08-17T10:44:13.7976140Z [command]/usr/bin/git version
2023-08-17T10:44:13.7976391Z git version 2.25.1
2023-08-17T10:44:13.7978136Z ##[endgroup]
2023-08-17T10:44:13.7994706Z Temporarily overriding HOME='/home/weiwangmeta/actions-runner/_work/_temp/0d15183e-3652-413a-ab6e-0c68fb457b84' before making global git config changes
2023-08-17T10:44:13.7995266Z Adding repository directory to the temporary git global config as a safe directory
2023-08-17T10:44:13.8000954Z [command]/usr/bin/git config --global --add safe.directory /home/weiwangmeta/actions-runner/_work/pytorch/pytorch
2023-08-17T10:44:13.8050529Z Deleting the contents of '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch'
2023-08-17T10:44:13.8055568Z ##[group]Initializing the repository
2023-08-17T10:44:13.8058958Z [command]/usr/bin/git init /home/weiwangmeta/actions-runner/_work/pytorch/pytorch
2023-08-17T10:44:13.8102345Z Initialized empty Git repository in /home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/
2023-08-17T10:44:13.8111568Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2023-08-17T10:44:13.8153139Z ##[endgroup]
2023-08-17T10:44:13.8153621Z ##[group]Disabling automatic garbage collection
2023-08-17T10:44:13.8156573Z [command]/usr/bin/git config --local gc.auto 0
2023-08-17T10:44:13.8190383Z ##[endgroup]
2023-08-17T10:44:13.8191149Z ##[group]Setting up auth
2023-08-17T10:44:13.8197064Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2023-08-17T10:44:13.8231213Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2023-08-17T10:44:13.8479550Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2023-08-17T10:44:13.8512651Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2023-08-17T10:44:13.8852993Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2023-08-17T10:44:13.8896200Z ##[endgroup]
2023-08-17T10:44:13.8896763Z ##[group]Fetching the repository
2023-08-17T10:44:13.8905061Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2023-08-17T10:44:21.6031755Z remote: Enumerating objects: 1149674        
2023-08-17T10:44:21.6032136Z remote: Enumerating objects: 1152463, done.        
2023-08-17T10:44:21.6032560Z remote: Counting objects:   0% (1/2789)        
2023-08-17T10:44:21.6032872Z remote: Counting objects:   1% (28/2789)        
2023-08-17T10:44:21.6033191Z remote: Counting objects:   2% (56/2789)        
2023-08-17T10:44:21.6033497Z remote: Counting objects:   3% (84/2789)        
2023-08-17T10:44:21.6033816Z remote: Counting objects:   4% (112/2789)        
2023-08-17T10:44:21.6034111Z remote: Counting objects:   5% (140/2789)        
2023-08-17T10:44:21.6034413Z remote: Counting objects:   6% (168/2789)        
2023-08-17T10:44:21.6039838Z remote: Counting objects:   7% (196/2789)        
2023-08-17T10:44:21.6040272Z remote: Counting objects:   8% (224/2789)        
2023-08-17T10:44:21.6040938Z remote: Counting objects:   9% (252/2789)        
2023-08-17T10:44:21.6041435Z remote: Counting objects:  10% (279/2789)        
2023-08-17T10:44:21.6041736Z remote: Counting objects:  11% (307/2789)        
2023-08-17T10:44:21.6042037Z remote: Counting objects:  12% (335/2789)        
2023-08-17T10:44:21.6042476Z remote: Counting objects:  13% (363/2789)        
2023-08-17T10:44:21.6043249Z remote: Counting objects:  14% (391/2789)        
2023-08-17T10:44:21.6043799Z remote: Counting objects:  15% (419/2789)        
2023-08-17T10:44:21.6044540Z remote: Counting objects:  16% (447/2789)        
2023-08-17T10:44:21.6045124Z remote: Counting objects:  17% (475/2789)        
2023-08-17T10:44:21.6046026Z remote: Counting objects:  18% (503/2789)        
2023-08-17T10:44:21.6046811Z remote: Counting objects:  19% (530/2789)        
2023-08-17T10:44:21.6049284Z remote: Counting objects:  20% (558/2789)        
2023-08-17T10:44:21.6049809Z remote: Counting objects:  21% (586/2789)        
2023-08-17T10:44:21.6050368Z remote: Counting objects:  22% (614/2789)        
2023-08-17T10:44:21.6050945Z remote: Counting objects:  23% (642/2789)        
2023-08-17T10:44:21.6051446Z remote: Counting objects:  24% (670/2789)        
2023-08-17T10:44:21.6051740Z remote: Counting objects:  25% (698/2789)        
2023-08-17T10:44:21.6052644Z remote: Counting objects:  26% (726/2789)        
2023-08-17T10:44:21.6053205Z remote: Counting objects:  27% (754/2789)        
2023-08-17T10:44:21.6053762Z remote: Counting objects:  28% (781/2789)        
2023-08-17T10:44:21.6054308Z remote: Counting objects:  29% (809/2789)        
2023-08-17T10:44:21.6055067Z remote: Counting objects:  30% (837/2789)        
2023-08-17T10:44:21.6055646Z remote: Counting objects:  31% (865/2789)        
2023-08-17T10:44:21.6056146Z remote: Counting objects:  32% (893/2789)        
2023-08-17T10:44:21.6056739Z remote: Counting objects:  33% (921/2789)        
2023-08-17T10:44:21.6057549Z remote: Counting objects:  34% (949/2789)        
2023-08-17T10:44:21.6057855Z remote: Counting objects:  35% (977/2789)        
2023-08-17T10:44:21.6058416Z remote: Counting objects:  36% (1005/2789)        
2023-08-17T10:44:21.6058920Z remote: Counting objects:  37% (1032/2789)        
2023-08-17T10:44:21.6059540Z remote: Counting objects:  38% (1060/2789)        
2023-08-17T10:44:21.6060100Z remote: Counting objects:  39% (1088/2789)        
2023-08-17T10:44:21.6060438Z remote: Counting objects:  40% (1116/2789)        
2023-08-17T10:44:21.6060740Z remote: Counting objects:  41% (1144/2789)        
2023-08-17T10:44:21.6061044Z remote: Counting objects:  42% (1172/2789)        
2023-08-17T10:44:21.6061350Z remote: Counting objects:  43% (1200/2789)        
2023-08-17T10:44:21.6061630Z remote: Counting objects:  44% (1228/2789)        
2023-08-17T10:44:21.6061926Z remote: Counting objects:  45% (1256/2789)        
2023-08-17T10:44:21.6062219Z remote: Counting objects:  46% (1283/2789)        
2023-08-17T10:44:21.6062521Z remote: Counting objects:  47% (1311/2789)        
2023-08-17T10:44:21.6062800Z remote: Counting objects:  48% (1339/2789)        
2023-08-17T10:44:21.6063248Z remote: Counting objects:  49% (1367/2789)        
2023-08-17T10:44:21.6063871Z remote: Counting objects:  50% (1395/2789)        
2023-08-17T10:44:21.6064450Z remote: Counting objects:  51% (1423/2789)        
2023-08-17T10:44:21.6064950Z remote: Counting objects:  52% (1451/2789)        
2023-08-17T10:44:21.6065527Z remote: Counting objects:  53% (1479/2789)        
2023-08-17T10:44:21.6066013Z remote: Counting objects:  54% (1507/2789)        
2023-08-17T10:44:21.6066317Z remote: Counting objects:  55% (1534/2789)        
2023-08-17T10:44:21.6066737Z remote: Counting objects:  56% (1562/2789)        
2023-08-17T10:44:21.6067073Z remote: Counting objects:  57% (1590/2789)        
2023-08-17T10:44:21.6067373Z remote: Counting objects:  58% (1618/2789)        
2023-08-17T10:44:21.6067662Z remote: Counting objects:  59% (1646/2789)        
2023-08-17T10:44:21.6067964Z remote: Counting objects:  60% (1674/2789)        
2023-08-17T10:44:21.6068288Z remote: Counting objects:  61% (1702/2789)        
2023-08-17T10:44:21.6068585Z remote: Counting objects:  62% (1730/2789)        
2023-08-17T10:44:21.6069112Z remote: Counting objects:  63% (1758/2789)        
2023-08-17T10:44:21.6069414Z remote: Counting objects:  64% (1785/2789)        
2023-08-17T10:44:21.6069710Z remote: Counting objects:  65% (1813/2789)        
2023-08-17T10:44:21.6069989Z remote: Counting objects:  66% (1841/2789)        
2023-08-17T10:44:21.6070288Z remote: Counting objects:  67% (1869/2789)        
2023-08-17T10:44:21.6070652Z remote: Counting objects:  68% (1897/2789)        
2023-08-17T10:44:21.6071096Z remote: Counting objects:  69% (1925/2789)        
2023-08-17T10:44:21.6071392Z remote: Counting objects:  70% (1953/2789)        
2023-08-17T10:44:21.6071669Z remote: Counting objects:  71% (1981/2789)        
2023-08-17T10:44:21.6071969Z remote: Counting objects:  72% (2009/2789)        
2023-08-17T10:44:21.6072271Z remote: Counting objects:  73% (2036/2789)        
2023-08-17T10:44:21.6072569Z remote: Counting objects:  74% (2064/2789)        
2023-08-17T10:44:21.6073056Z remote: Counting objects:  75% (2092/2789)        
2023-08-17T10:44:21.6073389Z remote: Counting objects:  76% (2120/2789)        
2023-08-17T10:44:21.6073683Z remote: Counting objects:  77% (2148/2789)        
2023-08-17T10:44:21.6073972Z remote: Counting objects:  78% (2176/2789)        
2023-08-17T10:44:21.6074251Z remote: Counting objects:  79% (2204/2789)        
2023-08-17T10:44:21.6074548Z remote: Counting objects:  80% (2232/2789)        
2023-08-17T10:44:21.6075153Z remote: Counting objects:  81% (2260/2789)        
2023-08-17T10:44:21.6075509Z remote: Counting objects:  82% (2287/2789)        
2023-08-17T10:44:21.6075801Z remote: Counting objects:  83% (2315/2789)        
2023-08-17T10:44:21.6076290Z remote: Counting objects:  84% (2343/2789)        
2023-08-17T10:44:21.6076598Z remote: Counting objects:  85% (2371/2789)        
2023-08-17T10:44:21.6077025Z remote: Counting objects:  86% (2399/2789)        
2023-08-17T10:44:21.6077327Z remote: Counting objects:  87% (2427/2789)        
2023-08-17T10:44:21.6077603Z remote: Counting objects:  88% (2455/2789)        
2023-08-17T10:44:21.6077962Z remote: Counting objects:  89% (2483/2789)        
2023-08-17T10:44:21.6078316Z remote: Counting objects:  90% (2511/2789)        
2023-08-17T10:44:21.6078611Z remote: Counting objects:  91% (2538/2789)        
2023-08-17T10:44:21.6078903Z remote: Counting objects:  92% (2566/2789)        
2023-08-17T10:44:21.6079195Z remote: Counting objects:  93% (2594/2789)        
2023-08-17T10:44:21.6079495Z remote: Counting objects:  94% (2622/2789)        
2023-08-17T10:44:21.6079787Z remote: Counting objects:  95% (2650/2789)        
2023-08-17T10:44:21.6080064Z remote: Counting objects:  96% (2678/2789)        
2023-08-17T10:44:21.6080355Z remote: Counting objects:  97% (2706/2789)        
2023-08-17T10:44:21.6080760Z remote: Counting objects:  98% (2734/2789)        
2023-08-17T10:44:21.6081195Z remote: Counting objects:  99% (2762/2789)        
2023-08-17T10:44:21.6081478Z remote: Counting objects: 100% (2789/2789)        
2023-08-17T10:44:21.6081791Z remote: Counting objects: 100% (2789/2789), done.        
2023-08-17T10:44:21.6627041Z remote: Compressing objects:   0% (1/1488)        
2023-08-17T10:44:21.7672498Z remote: Compressing objects:   1% (15/1488)        
2023-08-17T10:44:22.0693375Z remote: Compressing objects:   2% (30/1488)        
2023-08-17T10:44:22.2524987Z remote: Compressing objects:   3% (45/1488)        
2023-08-17T10:44:22.3608158Z remote: Compressing objects:   4% (60/1488)        
2023-08-17T10:44:22.4412107Z remote: Compressing objects:   5% (75/1488)        
2023-08-17T10:44:22.4954706Z remote: Compressing objects:   6% (90/1488)        
2023-08-17T10:44:22.5534696Z remote: Compressing objects:   7% (105/1488)        
2023-08-17T10:44:22.5993623Z remote: Compressing objects:   8% (120/1488)        
2023-08-17T10:44:22.6089563Z remote: Compressing objects:   9% (134/1488)        
2023-08-17T10:44:22.6318914Z remote: Compressing objects:   9% (143/1488)        
2023-08-17T10:44:22.6535115Z remote: Compressing objects:  10% (149/1488)        
2023-08-17T10:44:22.6672069Z remote: Compressing objects:  11% (164/1488)        
2023-08-17T10:44:22.6908093Z remote: Compressing objects:  12% (179/1488)        
2023-08-17T10:44:22.7093172Z remote: Compressing objects:  13% (194/1488)        
2023-08-17T10:44:22.7243813Z remote: Compressing objects:  14% (209/1488)        
2023-08-17T10:44:22.7334050Z remote: Compressing objects:  15% (224/1488)        
2023-08-17T10:44:22.7473760Z remote: Compressing objects:  16% (239/1488)        
2023-08-17T10:44:22.7609150Z remote: Compressing objects:  17% (253/1488)        
2023-08-17T10:44:22.7649970Z remote: Compressing objects:  18% (268/1488)        
2023-08-17T10:44:22.7670853Z remote: Compressing objects:  19% (283/1488)        
2023-08-17T10:44:22.7716491Z remote: Compressing objects:  20% (298/1488)        
2023-08-17T10:44:22.7776537Z remote: Compressing objects:  21% (313/1488)        
2023-08-17T10:44:22.7831473Z remote: Compressing objects:  22% (328/1488)        
2023-08-17T10:44:22.7904213Z remote: Compressing objects:  23% (343/1488)        
2023-08-17T10:44:22.7950575Z remote: Compressing objects:  24% (358/1488)        
2023-08-17T10:44:22.7973845Z remote: Compressing objects:  25% (372/1488)        
2023-08-17T10:44:22.7979677Z remote: Compressing objects:  26% (387/1488)        
2023-08-17T10:44:22.8004682Z remote: Compressing objects:  27% (402/1488)        
2023-08-17T10:44:22.8013374Z remote: Compressing objects:  28% (417/1488)        
2023-08-17T10:44:22.8014573Z remote: Compressing objects:  29% (432/1488)        
2023-08-17T10:44:22.8047405Z remote: Compressing objects:  30% (447/1488)        
2023-08-17T10:44:22.8063133Z remote: Compressing objects:  31% (462/1488)        
2023-08-17T10:44:22.8085344Z remote: Compressing objects:  32% (477/1488)        
2023-08-17T10:44:22.8108887Z remote: Compressing objects:  33% (492/1488)        
2023-08-17T10:44:22.8124881Z remote: Compressing objects:  34% (506/1488)        
2023-08-17T10:44:22.8143100Z remote: Compressing objects:  35% (521/1488)        
2023-08-17T10:44:22.8178716Z remote: Compressing objects:  36% (536/1488)        
2023-08-17T10:44:22.8191535Z remote: Compressing objects:  37% (551/1488)        
2023-08-17T10:44:22.8203906Z remote: Compressing objects:  38% (566/1488)        
2023-08-17T10:44:22.8230612Z remote: Compressing objects:  39% (581/1488)        
2023-08-17T10:44:22.8251427Z remote: Compressing objects:  40% (596/1488)        
2023-08-17T10:44:22.8263342Z remote: Compressing objects:  41% (611/1488)        
2023-08-17T10:44:22.8276658Z remote: Compressing objects:  42% (625/1488)        
2023-08-17T10:44:22.8284430Z remote: Compressing objects:  43% (640/1488)        
2023-08-17T10:44:22.8307118Z remote: Compressing objects:  44% (655/1488)        
2023-08-17T10:44:22.8312949Z remote: Compressing objects:  45% (670/1488)        
2023-08-17T10:44:22.8314567Z remote: Compressing objects:  46% (685/1488)        
2023-08-17T10:44:22.8333980Z remote: Compressing objects:  47% (700/1488)        
2023-08-17T10:44:22.8347304Z remote: Compressing objects:  48% (715/1488)        
2023-08-17T10:44:22.8356832Z remote: Compressing objects:  49% (730/1488)        
2023-08-17T10:44:22.8368793Z remote: Compressing objects:  50% (744/1488)        
2023-08-17T10:44:22.8374990Z remote: Compressing objects:  51% (759/1488)        
2023-08-17T10:44:22.8387968Z remote: Compressing objects:  52% (774/1488)        
2023-08-17T10:44:22.8394764Z remote: Compressing objects:  53% (789/1488)        
2023-08-17T10:44:22.8404393Z remote: Compressing objects:  54% (804/1488)        
2023-08-17T10:44:22.8419859Z remote: Compressing objects:  55% (819/1488)        
2023-08-17T10:44:22.8420372Z remote: Compressing objects:  56% (834/1488)        
2023-08-17T10:44:22.8422197Z remote: Compressing objects:  57% (849/1488)        
2023-08-17T10:44:22.8425122Z remote: Compressing objects:  58% (864/1488)        
2023-08-17T10:44:22.8427024Z remote: Compressing objects:  59% (878/1488)        
2023-08-17T10:44:22.8429476Z remote: Compressing objects:  60% (893/1488)        
2023-08-17T10:44:22.8431945Z remote: Compressing objects:  61% (908/1488)        
2023-08-17T10:44:22.8434235Z remote: Compressing objects:  62% (923/1488)        
2023-08-17T10:44:22.8435016Z remote: Compressing objects:  63% (938/1488)        
2023-08-17T10:44:22.8437441Z remote: Compressing objects:  64% (953/1488)        
2023-08-17T10:44:22.8439880Z remote: Compressing objects:  65% (968/1488)        
2023-08-17T10:44:22.8440376Z remote: Compressing objects:  66% (983/1488)        
2023-08-17T10:44:22.8441344Z remote: Compressing objects:  67% (997/1488)        
2023-08-17T10:44:22.8441842Z remote: Compressing objects:  68% (1012/1488)        
2023-08-17T10:44:22.8442932Z remote: Compressing objects:  69% (1027/1488)        
2023-08-17T10:44:22.8444740Z remote: Compressing objects:  70% (1042/1488)        
2023-08-17T10:44:22.8445296Z remote: Compressing objects:  71% (1057/1488)        
2023-08-17T10:44:22.8445708Z remote: Compressing objects:  72% (1072/1488)        
2023-08-17T10:44:22.8446017Z remote: Compressing objects:  73% (1087/1488)        
2023-08-17T10:44:22.8458968Z remote: Compressing objects:  74% (1102/1488)        
2023-08-17T10:44:22.8471775Z remote: Compressing objects:  75% (1116/1488)        
2023-08-17T10:44:22.8479831Z remote: Compressing objects:  76% (1131/1488)        
2023-08-17T10:44:22.8525094Z remote: Compressing objects:  77% (1146/1488)        
2023-08-17T10:44:22.8532256Z remote: Compressing objects:  78% (1161/1488)        
2023-08-17T10:44:22.8532761Z remote: Compressing objects:  79% (1176/1488)        
2023-08-17T10:44:22.8533462Z remote: Compressing objects:  80% (1191/1488)        
2023-08-17T10:44:22.8533922Z remote: Compressing objects:  81% (1206/1488)        
2023-08-17T10:44:22.8534236Z remote: Compressing objects:  82% (1221/1488)        
2023-08-17T10:44:22.8535032Z remote: Compressing objects:  83% (1236/1488)        
2023-08-17T10:44:22.8535341Z remote: Compressing objects:  84% (1250/1488)        
2023-08-17T10:44:22.8535889Z remote: Compressing objects:  85% (1265/1488)        
2023-08-17T10:44:22.8536319Z remote: Compressing objects:  86% (1280/1488)        
2023-08-17T10:44:22.8539529Z remote: Compressing objects:  87% (1295/1488)        
2023-08-17T10:44:22.8549371Z remote: Compressing objects:  88% (1310/1488)        
2023-08-17T10:44:22.8553188Z remote: Compressing objects:  89% (1325/1488)        
2023-08-17T10:44:22.8554250Z remote: Compressing objects:  90% (1340/1488)        
2023-08-17T10:44:22.8554634Z remote: Compressing objects:  91% (1355/1488)        
2023-08-17T10:44:22.8554956Z remote: Compressing objects:  92% (1369/1488)        
2023-08-17T10:44:22.8555268Z remote: Compressing objects:  93% (1384/1488)        
2023-08-17T10:44:22.8555582Z remote: Compressing objects:  94% (1399/1488)        
2023-08-17T10:44:22.8555889Z remote: Compressing objects:  95% (1414/1488)        
2023-08-17T10:44:22.8557759Z remote: Compressing objects:  96% (1429/1488)        
2023-08-17T10:44:22.8558647Z remote: Compressing objects:  97% (1444/1488)        
2023-08-17T10:44:22.8560867Z remote: Compressing objects:  98% (1459/1488)        
2023-08-17T10:44:22.8561374Z remote: Compressing objects:  99% (1474/1488)        
2023-08-17T10:44:22.8562026Z remote: Compressing objects: 100% (1488/1488)        
2023-08-17T10:44:22.8562723Z remote: Compressing objects: 100% (1488/1488), done.        
2023-08-17T10:44:46.9294135Z remote: Total 1152463 (delta 1738), reused 2201 (delta 1292), pack-reused 1149674        
2023-08-17T10:45:25.9016009Z [command]/usr/bin/git rev-parse --verify --quiet 384e0d104fd077d31efafc564129660e9b7a0f25^{object}
2023-08-17T10:45:25.9065519Z 384e0d104fd077d31efafc564129660e9b7a0f25
2023-08-17T10:45:25.9074804Z ##[endgroup]
2023-08-17T10:45:25.9075610Z ##[group]Determining the checkout info
2023-08-17T10:45:25.9076384Z ##[endgroup]
2023-08-17T10:45:25.9077122Z ##[group]Checking out the ref
2023-08-17T10:45:25.9082367Z [command]/usr/bin/git checkout --quiet --force 384e0d104fd077d31efafc564129660e9b7a0f25
2023-08-17T10:45:27.6714225Z ##[endgroup]
2023-08-17T10:45:27.6715057Z ##[group]Setting up auth for fetching submodules
2023-08-17T10:45:27.6723532Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2023-08-17T10:45:27.6785763Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2023-08-17T10:45:27.6824026Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2023-08-17T10:45:27.6864281Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2023-08-17T10:45:27.6895557Z ##[endgroup]
2023-08-17T10:45:27.6897685Z ##[group]Fetching submodules
2023-08-17T10:45:27.6901301Z [command]/usr/bin/git submodule sync --recursive
2023-08-17T10:45:27.7174105Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2023-08-17T10:45:27.7440568Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2023-08-17T10:45:27.7445500Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2023-08-17T10:45:27.7451540Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2023-08-17T10:45:27.7457957Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2023-08-17T10:45:27.7463936Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK'
2023-08-17T10:45:27.7469959Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2023-08-17T10:45:27.7475342Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2023-08-17T10:45:27.7481068Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2023-08-17T10:45:27.7488511Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2023-08-17T10:45:27.7496549Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub'
2023-08-17T10:45:27.7504580Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2023-08-17T10:45:27.7511034Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2023-08-17T10:45:27.7518431Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2023-08-17T10:45:27.7525414Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2023-08-17T10:45:27.7532775Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2023-08-17T10:45:27.7538831Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2023-08-17T10:45:27.7546377Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2023-08-17T10:45:27.7552934Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2023-08-17T10:45:27.7559790Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2023-08-17T10:45:27.7566374Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2023-08-17T10:45:27.7574578Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2023-08-17T10:45:27.7581758Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake'
2023-08-17T10:45:27.7589961Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2023-08-17T10:45:27.7597391Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2023-08-17T10:45:27.7605745Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc'
2023-08-17T10:45:27.7613606Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2023-08-17T10:45:27.7622259Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse'
2023-08-17T10:45:27.7629798Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2023-08-17T10:45:27.7637128Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2023-08-17T10:45:27.7644342Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt'
2023-08-17T10:45:27.7652596Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2023-08-17T10:45:27.7661134Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2023-08-17T10:45:27.7668385Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2023-08-17T10:45:27.7676113Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2023-08-17T10:45:27.7684368Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2023-08-17T10:45:27.7692150Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum'
2023-08-17T10:45:27.7700076Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2023-08-17T10:45:27.7708681Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six'
2023-08-17T10:45:27.7727790Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2023-08-17T10:45:27.7729445Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb'
2023-08-17T10:45:27.7732918Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2023-08-17T10:45:27.7741772Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd'
2023-08-17T10:45:27.7815490Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2023-08-17T10:45:28.3278875Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2023-08-17T10:45:28.7427745Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2023-08-17T10:45:29.1072507Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2023-08-17T10:45:29.6189867Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'...
2023-08-17T10:45:30.1188927Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2023-08-17T10:45:33.6507186Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2023-08-17T10:45:45.4536686Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2023-08-17T10:45:46.1637737Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2023-08-17T10:45:47.0419459Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/cub'...
2023-08-17T10:45:48.9084381Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2023-08-17T10:45:50.3090396Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2023-08-17T10:45:52.7474250Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2023-08-17T10:45:58.0387751Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2023-08-17T10:45:59.1192211Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2023-08-17T10:46:01.3445735Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2023-08-17T10:46:03.0104339Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2023-08-17T10:46:03.3980304Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2023-08-17T10:46:04.1916253Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2023-08-17T10:46:04.7834215Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2023-08-17T10:46:06.5763218Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2023-08-17T10:46:07.2045113Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'...
2023-08-17T10:46:07.5875478Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2023-08-17T10:46:08.0916418Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2023-08-17T10:46:09.9047409Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'...
2023-08-17T10:46:10.8386611Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2023-08-17T10:46:11.6012672Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'...
2023-08-17T10:46:12.1649157Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2023-08-17T10:46:19.5060376Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2023-08-17T10:46:22.8676988Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'...
2023-08-17T10:46:23.6839007Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2023-08-17T10:46:24.1474078Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2023-08-17T10:46:32.9591679Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2023-08-17T10:46:33.3387770Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2023-08-17T10:46:33.8005691Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2023-08-17T10:46:35.0497464Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/python-enum'...
2023-08-17T10:46:35.4499198Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2023-08-17T10:46:36.0015051Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/python-six'...
2023-08-17T10:46:36.5653388Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2023-08-17T10:46:37.4108485Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tbb'...
2023-08-17T10:46:40.1552409Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2023-08-17T10:46:40.8925604Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/zstd'...
2023-08-17T10:46:44.2032991Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2023-08-17T10:46:44.2387899Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2023-08-17T10:46:44.2729084Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2023-08-17T10:46:44.3222798Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2023-08-17T10:46:44.3720385Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c'
2023-08-17T10:46:44.4387729Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2023-08-17T10:46:45.2070416Z Submodule path 'third_party/XNNPACK': checked out '51a987591a6fc9f0fc0707077f53d763ac132cbf'
2023-08-17T10:46:45.2626476Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2023-08-17T10:46:45.3975198Z Submodule path 'third_party/cpuinfo': checked out '6481e8bef08f606ddd627e4d3be89f64d62e1b8a'
2023-08-17T10:46:45.4602223Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4'
2023-08-17T10:46:45.8003181Z Submodule path 'third_party/cudnn_frontend': checked out 'a4f05c1edcef453f5fd52f96218c29c7d420e511'
2023-08-17T10:46:46.3403245Z Submodule path 'third_party/cutlass': checked out '6f47420213f757831fae65c686aa471749fa8d60'
2023-08-17T10:46:46.6496281Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2023-08-17T10:46:46.7301334Z Submodule path 'third_party/fbgemm': checked out 'd0ee798b1f198cc51b6ddae20cf6063f6380ba3f'
2023-08-17T10:46:46.7352246Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2023-08-17T10:46:46.7359135Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2023-08-17T10:46:46.7365121Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass'
2023-08-17T10:46:46.7370447Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2023-08-17T10:46:46.7376428Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2023-08-17T10:46:46.7422468Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2023-08-17T10:46:47.8150772Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2023-08-17T10:46:48.6190471Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'...
2023-08-17T10:46:51.0369347Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2023-08-17T10:46:52.6642864Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2023-08-17T10:46:53.2589050Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2023-08-17T10:46:53.3964744Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2023-08-17T10:46:53.8654093Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6'
2023-08-17T10:46:53.9550796Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2023-08-17T10:46:53.9885948Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1'
2023-08-17T10:46:54.1346381Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b'
2023-08-17T10:46:54.1975115Z Submodule path 'third_party/fmt': checked out 'e57ca2e3685b160617d3d95fcd9e789c4e06ca88'
2023-08-17T10:46:54.2312579Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2023-08-17T10:46:54.3015268Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2023-08-17T10:46:54.3514624Z Submodule path 'third_party/gloo': checked out '597accfd79f5b0f9d57b228dec088ca996686475'
2023-08-17T10:46:54.4301354Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2023-08-17T10:46:54.4680645Z Submodule path 'third_party/ideep': checked out 'fe8378249600442043b98f333b8b605bedca5a25'
2023-08-17T10:46:54.4728332Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2023-08-17T10:46:54.4773432Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2023-08-17T10:47:05.8736739Z Submodule path 'third_party/ideep/mkl-dnn': checked out '6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e'
2023-08-17T10:47:05.9123971Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724'
2023-08-17T10:47:05.9536841Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2023-08-17T10:47:06.0852509Z Submodule path 'third_party/kineto': checked out '49e854d805d916b2031e337763928d2f8d2e1fbf'
2023-08-17T10:47:06.0905245Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog'
2023-08-17T10:47:06.0911946Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2023-08-17T10:47:06.0917581Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2023-08-17T10:47:06.0963216Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'...
2023-08-17T10:47:06.9930555Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2023-08-17T10:47:08.6301214Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2023-08-17T10:47:10.3566282Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e'
2023-08-17T10:47:10.3618541Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2023-08-17T10:47:10.3624018Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2023-08-17T10:47:10.3630196Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2023-08-17T10:47:10.3636503Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2023-08-17T10:47:10.3643726Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2023-08-17T10:47:10.3650170Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2023-08-17T10:47:10.3656150Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2023-08-17T10:47:10.3663092Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2023-08-17T10:47:10.3711433Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'...
2023-08-17T10:47:11.6323326Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'...
2023-08-17T10:47:12.2817929Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'...
2023-08-17T10:47:13.9537559Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'...
2023-08-17T10:47:14.6069363Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'...
2023-08-17T10:47:15.4082289Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'...
2023-08-17T10:47:17.0712629Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'...
2023-08-17T10:47:24.4147036Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'...
2023-08-17T10:47:25.1379008Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9'
2023-08-17T10:47:25.1783226Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400'
2023-08-17T10:47:25.2393789Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2023-08-17T10:47:25.2751643Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067'
2023-08-17T10:47:25.2802540Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2023-08-17T10:47:25.2846960Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'...
2023-08-17T10:47:25.8991057Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4'
2023-08-17T10:47:25.9393542Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446'
2023-08-17T10:47:26.0062042Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850'
2023-08-17T10:47:26.1459210Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5'
2023-08-17T10:47:26.1836457Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150'
2023-08-17T10:47:26.2411349Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50'
2023-08-17T10:47:26.3248749Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2023-08-17T10:47:26.3888526Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78'
2023-08-17T10:47:26.4366608Z Submodule path 'third_party/nccl/nccl': checked out '9b7d5edbfcd70a1ab0874e27382d5c31f769aa0f'
2023-08-17T10:47:26.4754986Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a'
2023-08-17T10:47:26.6157252Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2023-08-17T10:47:26.9776142Z Submodule path 'third_party/onnx': checked out 'a0d77f18516d2da7468a96b0de3b737266f23176'
2023-08-17T10:47:26.9848135Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2023-08-17T10:47:26.9855388Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2023-08-17T10:47:26.9919472Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2023-08-17T10:47:27.7272457Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2023-08-17T10:47:29.0585636Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2023-08-17T10:47:29.1185400Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '0bd8896a4010f2d91b2340570c24fa08606ec406'
2023-08-17T10:47:29.1618045Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f'
2023-08-17T10:47:29.1666304Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx'
2023-08-17T10:47:29.1709643Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'...
2023-08-17T10:47:32.6329336Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8'
2023-08-17T10:47:32.6382792Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-08-17T10:47:32.6389837Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-08-17T10:47:32.6441605Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'...
2023-08-17T10:47:33.3879363Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'...
2023-08-17T10:47:34.6700554Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2023-08-17T10:47:34.7686771Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c'
2023-08-17T10:47:34.7737541Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-08-17T10:47:34.7781935Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'...
2023-08-17T10:47:35.2821677Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2023-08-17T10:47:35.3171133Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a'
2023-08-17T10:47:35.6556481Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2023-08-17T10:47:35.6613430Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2023-08-17T10:47:35.6618645Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2023-08-17T10:47:35.6669959Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2023-08-17T10:47:36.3794121Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2023-08-17T10:47:38.0782444Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2023-08-17T10:47:38.1788789Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2023-08-17T10:47:38.2126878Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2023-08-17T10:47:38.2500980Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413'
2023-08-17T10:47:38.3095316Z Submodule path 'third_party/pybind11': checked out '8a099e44b3d5f85b20f05828d919d2332a8de841'
2023-08-17T10:47:38.3420605Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7'
2023-08-17T10:47:38.3989920Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2023-08-17T10:47:38.4320443Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a'
2023-08-17T10:47:38.5049488Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff'
2023-08-17T10:47:38.6506369Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9'
2023-08-17T10:47:38.7013125Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2023-08-17T10:47:38.7059206Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2023-08-17T10:47:38.7065095Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2023-08-17T10:47:38.7072845Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2023-08-17T10:47:38.7078902Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2023-08-17T10:47:38.7124134Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2023-08-17T10:47:40.4793345Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2023-08-17T10:47:41.0220584Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2023-08-17T10:47:42.7335066Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2023-08-17T10:47:44.0725204Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2023-08-17T10:47:44.1090201Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2023-08-17T10:47:44.2015494Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2023-08-17T10:47:44.2529012Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2023-08-17T10:47:44.2579726Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-08-17T10:47:44.2624532Z Cloning into '/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2023-08-17T10:47:44.7259677Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2023-08-17T10:47:44.9088803Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8'
2023-08-17T10:47:44.9208082Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2023-08-17T10:47:44.9491396Z Entering 'android/libs/fbjni'
2023-08-17T10:47:44.9532424Z Entering 'third_party/FP16'
2023-08-17T10:47:44.9571540Z Entering 'third_party/FXdiv'
2023-08-17T10:47:44.9610739Z Entering 'third_party/NNPACK'
2023-08-17T10:47:44.9651121Z Entering 'third_party/QNNPACK'
2023-08-17T10:47:44.9691508Z Entering 'third_party/VulkanMemoryAllocator'
2023-08-17T10:47:44.9731546Z Entering 'third_party/XNNPACK'
2023-08-17T10:47:44.9785449Z Entering 'third_party/benchmark'
2023-08-17T10:47:44.9824641Z Entering 'third_party/cpuinfo'
2023-08-17T10:47:44.9864667Z Entering 'third_party/cub'
2023-08-17T10:47:44.9904022Z Entering 'third_party/cudnn_frontend'
2023-08-17T10:47:44.9949278Z Entering 'third_party/cutlass'
2023-08-17T10:47:44.9997324Z Entering 'third_party/eigen'
2023-08-17T10:47:45.0038630Z Entering 'third_party/fbgemm'
2023-08-17T10:47:45.0099387Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-08-17T10:47:45.0138122Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-08-17T10:47:45.0177064Z Entering 'third_party/fbgemm/third_party/cutlass'
2023-08-17T10:47:45.0222519Z Entering 'third_party/fbgemm/third_party/googletest'
2023-08-17T10:47:45.0261430Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-08-17T10:47:45.0300599Z Entering 'third_party/flatbuffers'
2023-08-17T10:47:45.0343627Z Entering 'third_party/fmt'
2023-08-17T10:47:45.0384368Z Entering 'third_party/foxi'
2023-08-17T10:47:45.0423639Z Entering 'third_party/gemmlowp/gemmlowp'
2023-08-17T10:47:45.0462848Z Entering 'third_party/gloo'
2023-08-17T10:47:45.0500894Z Entering 'third_party/googletest'
2023-08-17T10:47:45.0540327Z Entering 'third_party/ideep'
2023-08-17T10:47:45.0577647Z Entering 'third_party/ideep/mkl-dnn'
2023-08-17T10:47:45.0628690Z Entering 'third_party/ios-cmake'
2023-08-17T10:47:45.0667264Z Entering 'third_party/ittapi'
2023-08-17T10:47:45.0706906Z Entering 'third_party/kineto'
2023-08-17T10:47:45.0745151Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2023-08-17T10:47:45.0782863Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2023-08-17T10:47:45.0821708Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2023-08-17T10:47:45.0861046Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2023-08-17T10:47:45.0899594Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2023-08-17T10:47:45.0936238Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2023-08-17T10:47:45.0976662Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2023-08-17T10:47:45.1014937Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2023-08-17T10:47:45.1053763Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2023-08-17T10:47:45.1092589Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2023-08-17T10:47:45.1132457Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-08-17T10:47:45.1170574Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-08-17T10:47:45.1210693Z Entering 'third_party/mimalloc'
2023-08-17T10:47:45.1249783Z Entering 'third_party/nccl/nccl'
2023-08-17T10:47:45.1288880Z Entering 'third_party/neon2sse'
2023-08-17T10:47:45.1327875Z Entering 'third_party/nlohmann'
2023-08-17T10:47:45.1368929Z Entering 'third_party/onnx'
2023-08-17T10:47:45.1425801Z Entering 'third_party/onnx/third_party/benchmark'
2023-08-17T10:47:45.1465060Z Entering 'third_party/onnx/third_party/pybind11'
2023-08-17T10:47:45.1505460Z Entering 'third_party/onnx-tensorrt'
2023-08-17T10:47:45.1544700Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-08-17T10:47:45.1587018Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-08-17T10:47:45.1626000Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-08-17T10:47:45.1662979Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-08-17T10:47:45.1706496Z Entering 'third_party/pocketfft'
2023-08-17T10:47:45.1744852Z Entering 'third_party/protobuf'
2023-08-17T10:47:45.1787990Z Entering 'third_party/protobuf/third_party/benchmark'
2023-08-17T10:47:45.1825607Z Entering 'third_party/protobuf/third_party/googletest'
2023-08-17T10:47:45.1866233Z Entering 'third_party/psimd'
2023-08-17T10:47:45.1905349Z Entering 'third_party/pthreadpool'
2023-08-17T10:47:45.1943687Z Entering 'third_party/pybind11'
2023-08-17T10:47:45.1982296Z Entering 'third_party/python-enum'
2023-08-17T10:47:45.2021570Z Entering 'third_party/python-peachpy'
2023-08-17T10:47:45.2059702Z Entering 'third_party/python-six'
2023-08-17T10:47:45.2098531Z Entering 'third_party/sleef'
2023-08-17T10:47:45.2137537Z Entering 'third_party/tbb'
2023-08-17T10:47:45.2178928Z Entering 'third_party/tensorpipe'
2023-08-17T10:47:45.2217608Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-08-17T10:47:45.2256401Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-08-17T10:47:45.2294694Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-08-17T10:47:45.2333298Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-08-17T10:47:45.2370153Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-08-17T10:47:45.2411401Z Entering 'third_party/zstd'
2023-08-17T10:47:45.2474766Z ##[endgroup]
2023-08-17T10:47:45.2478054Z ##[group]Persisting credentials for submodules
2023-08-17T10:47:45.2532125Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :"
2023-08-17T10:47:45.2801691Z Entering 'android/libs/fbjni'
2023-08-17T10:47:45.2847008Z Entering 'third_party/FP16'
2023-08-17T10:47:45.2891584Z Entering 'third_party/FXdiv'
2023-08-17T10:47:45.2935417Z Entering 'third_party/NNPACK'
2023-08-17T10:47:45.2980129Z Entering 'third_party/QNNPACK'
2023-08-17T10:47:45.3024077Z Entering 'third_party/VulkanMemoryAllocator'
2023-08-17T10:47:45.3068752Z Entering 'third_party/XNNPACK'
2023-08-17T10:47:45.3127689Z Entering 'third_party/benchmark'
2023-08-17T10:47:45.3172019Z Entering 'third_party/cpuinfo'
2023-08-17T10:47:45.3216242Z Entering 'third_party/cub'
2023-08-17T10:47:45.3262190Z Entering 'third_party/cudnn_frontend'
2023-08-17T10:47:45.3313979Z Entering 'third_party/cutlass'
2023-08-17T10:47:45.3367802Z Entering 'third_party/eigen'
2023-08-17T10:47:45.3415104Z Entering 'third_party/fbgemm'
2023-08-17T10:47:45.3459057Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-08-17T10:47:45.3502091Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-08-17T10:47:45.3545735Z Entering 'third_party/fbgemm/third_party/cutlass'
2023-08-17T10:47:45.3595939Z Entering 'third_party/fbgemm/third_party/googletest'
2023-08-17T10:47:45.3639291Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-08-17T10:47:45.3683706Z Entering 'third_party/flatbuffers'
2023-08-17T10:47:45.3729888Z Entering 'third_party/fmt'
2023-08-17T10:47:45.3774535Z Entering 'third_party/foxi'
2023-08-17T10:47:45.3818151Z Entering 'third_party/gemmlowp/gemmlowp'
2023-08-17T10:47:45.3861843Z Entering 'third_party/gloo'
2023-08-17T10:47:45.3906358Z Entering 'third_party/googletest'
2023-08-17T10:47:45.3950811Z Entering 'third_party/ideep'
2023-08-17T10:47:45.3993368Z Entering 'third_party/ideep/mkl-dnn'
2023-08-17T10:47:45.4046282Z Entering 'third_party/ios-cmake'
2023-08-17T10:47:45.4091265Z Entering 'third_party/ittapi'
2023-08-17T10:47:45.4135611Z Entering 'third_party/kineto'
2023-08-17T10:47:45.4178852Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2023-08-17T10:47:45.4220705Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2023-08-17T10:47:45.4266193Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2023-08-17T10:47:45.4309891Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2023-08-17T10:47:45.4353549Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2023-08-17T10:47:45.4395623Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2023-08-17T10:47:45.4440389Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2023-08-17T10:47:45.4483524Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2023-08-17T10:47:45.4526999Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2023-08-17T10:47:45.4571240Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2023-08-17T10:47:45.4616555Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-08-17T10:47:45.4661000Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-08-17T10:47:45.4705324Z Entering 'third_party/mimalloc'
2023-08-17T10:47:45.4750166Z Entering 'third_party/nccl/nccl'
2023-08-17T10:47:45.4793665Z Entering 'third_party/neon2sse'
2023-08-17T10:47:45.4836897Z Entering 'third_party/nlohmann'
2023-08-17T10:47:45.4881739Z Entering 'third_party/onnx'
2023-08-17T10:47:45.4943510Z Entering 'third_party/onnx/third_party/benchmark'
2023-08-17T10:47:45.4987887Z Entering 'third_party/onnx/third_party/pybind11'
2023-08-17T10:47:45.5033943Z Entering 'third_party/onnx-tensorrt'
2023-08-17T10:47:45.5076411Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-08-17T10:47:45.5124803Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-08-17T10:47:45.5169233Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-08-17T10:47:45.5212033Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-08-17T10:47:45.5260205Z Entering 'third_party/pocketfft'
2023-08-17T10:47:45.5304094Z Entering 'third_party/protobuf'
2023-08-17T10:47:45.5352475Z Entering 'third_party/protobuf/third_party/benchmark'
2023-08-17T10:47:45.5395401Z Entering 'third_party/protobuf/third_party/googletest'
2023-08-17T10:47:45.5441258Z Entering 'third_party/psimd'
2023-08-17T10:47:45.5485337Z Entering 'third_party/pthreadpool'
2023-08-17T10:47:45.5529805Z Entering 'third_party/pybind11'
2023-08-17T10:47:45.5573452Z Entering 'third_party/python-enum'
2023-08-17T10:47:45.5617057Z Entering 'third_party/python-peachpy'
2023-08-17T10:47:45.5661358Z Entering 'third_party/python-six'
2023-08-17T10:47:45.5704505Z Entering 'third_party/sleef'
2023-08-17T10:47:45.5748403Z Entering 'third_party/tbb'
2023-08-17T10:47:45.5794932Z Entering 'third_party/tensorpipe'
2023-08-17T10:47:45.5837689Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-08-17T10:47:45.5880684Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-08-17T10:47:45.5924284Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-08-17T10:47:45.5967385Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-08-17T10:47:45.6009691Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-08-17T10:47:45.6055753Z Entering 'third_party/zstd'
2023-08-17T10:47:45.6111929Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url"
2023-08-17T10:47:45.6380373Z Entering 'android/libs/fbjni'
2023-08-17T10:47:45.6421425Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2023-08-17T10:47:45.6440088Z Entering 'third_party/FP16'
2023-08-17T10:47:45.6480467Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2023-08-17T10:47:45.6499970Z Entering 'third_party/FXdiv'
2023-08-17T10:47:45.6540989Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2023-08-17T10:47:45.6560247Z Entering 'third_party/NNPACK'
2023-08-17T10:47:45.6600874Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2023-08-17T10:47:45.6620815Z Entering 'third_party/QNNPACK'
2023-08-17T10:47:45.6660737Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config	remote.origin.url
2023-08-17T10:47:45.6680013Z Entering 'third_party/VulkanMemoryAllocator'
2023-08-17T10:47:45.6720962Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2023-08-17T10:47:45.6740532Z Entering 'third_party/XNNPACK'
2023-08-17T10:47:45.6780432Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2023-08-17T10:47:45.6814793Z Entering 'third_party/benchmark'
2023-08-17T10:47:45.6855644Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2023-08-17T10:47:45.6874266Z Entering 'third_party/cpuinfo'
2023-08-17T10:47:45.6914950Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2023-08-17T10:47:45.6934602Z Entering 'third_party/cub'
2023-08-17T10:47:45.6975695Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config	remote.origin.url
2023-08-17T10:47:45.6994460Z Entering 'third_party/cudnn_frontend'
2023-08-17T10:47:45.7034573Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2023-08-17T10:47:45.7061352Z Entering 'third_party/cutlass'
2023-08-17T10:47:45.7100846Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2023-08-17T10:47:45.7128992Z Entering 'third_party/eigen'
2023-08-17T10:47:45.7169740Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2023-08-17T10:47:45.7190373Z Entering 'third_party/fbgemm'
2023-08-17T10:47:45.7230428Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2023-08-17T10:47:45.7249710Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-08-17T10:47:45.7289502Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2023-08-17T10:47:45.7307606Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-08-17T10:47:45.7347695Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2023-08-17T10:47:45.7366731Z Entering 'third_party/fbgemm/third_party/cutlass'
2023-08-17T10:47:45.7406584Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config	remote.origin.url
2023-08-17T10:47:45.7432659Z Entering 'third_party/fbgemm/third_party/googletest'
2023-08-17T10:47:45.7472011Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2023-08-17T10:47:45.7490210Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-08-17T10:47:45.7529926Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2023-08-17T10:47:45.7549177Z Entering 'third_party/flatbuffers'
2023-08-17T10:47:45.7589547Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2023-08-17T10:47:45.7611716Z Entering 'third_party/fmt'
2023-08-17T10:47:45.7651823Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2023-08-17T10:47:45.7670112Z Entering 'third_party/foxi'
2023-08-17T10:47:45.7710449Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2023-08-17T10:47:45.7729251Z Entering 'third_party/gemmlowp/gemmlowp'
2023-08-17T10:47:45.7769361Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2023-08-17T10:47:45.7788210Z Entering 'third_party/gloo'
2023-08-17T10:47:45.7828350Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2023-08-17T10:47:45.7847385Z Entering 'third_party/googletest'
2023-08-17T10:47:45.7887545Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2023-08-17T10:47:45.7906178Z Entering 'third_party/ideep'
2023-08-17T10:47:45.7947306Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2023-08-17T10:47:45.7965563Z Entering 'third_party/ideep/mkl-dnn'
2023-08-17T10:47:45.8005279Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2023-08-17T10:47:45.8032478Z Entering 'third_party/ios-cmake'
2023-08-17T10:47:45.8071770Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config	remote.origin.url
2023-08-17T10:47:45.8091215Z Entering 'third_party/ittapi'
2023-08-17T10:47:45.8132210Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2023-08-17T10:47:45.8151338Z Entering 'third_party/kineto'
2023-08-17T10:47:45.8192615Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2023-08-17T10:47:45.8211564Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2023-08-17T10:47:45.8252919Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config	remote.origin.url
2023-08-17T10:47:45.8270400Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2023-08-17T10:47:45.8311190Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config	remote.origin.url
2023-08-17T10:47:45.8331828Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2023-08-17T10:47:45.8371145Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config	remote.origin.url
2023-08-17T10:47:45.8390609Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2023-08-17T10:47:45.8430515Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config	remote.origin.url
2023-08-17T10:47:45.8450015Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2023-08-17T10:47:45.8489426Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config	remote.origin.url
2023-08-17T10:47:45.8506810Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2023-08-17T10:47:45.8547458Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config	remote.origin.url
2023-08-17T10:47:45.8569055Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2023-08-17T10:47:45.8608906Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config	remote.origin.url
2023-08-17T10:47:45.8628273Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2023-08-17T10:47:45.8668657Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config	remote.origin.url
2023-08-17T10:47:45.8688075Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2023-08-17T10:47:45.8728975Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config	remote.origin.url
2023-08-17T10:47:45.8748339Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2023-08-17T10:47:45.8789074Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config	remote.origin.url
2023-08-17T10:47:45.8809206Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-08-17T10:47:45.8849002Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2023-08-17T10:47:45.8867953Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-08-17T10:47:45.8907299Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2023-08-17T10:47:45.8928240Z Entering 'third_party/mimalloc'
2023-08-17T10:47:45.8968164Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config	remote.origin.url
2023-08-17T10:47:45.8987018Z Entering 'third_party/nccl/nccl'
2023-08-17T10:47:45.9026823Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2023-08-17T10:47:45.9046309Z Entering 'third_party/neon2sse'
2023-08-17T10:47:45.9086353Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config	remote.origin.url
2023-08-17T10:47:45.9104526Z Entering 'third_party/nlohmann'
2023-08-17T10:47:45.9145580Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2023-08-17T10:47:45.9166573Z Entering 'third_party/onnx'
2023-08-17T10:47:45.9207076Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2023-08-17T10:47:45.9243289Z Entering 'third_party/onnx/third_party/benchmark'
2023-08-17T10:47:45.9284551Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2023-08-17T10:47:45.9303217Z Entering 'third_party/onnx/third_party/pybind11'
2023-08-17T10:47:45.9343286Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2023-08-17T10:47:45.9365953Z Entering 'third_party/onnx-tensorrt'
2023-08-17T10:47:45.9406418Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config	remote.origin.url
2023-08-17T10:47:45.9425681Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-08-17T10:47:45.9466777Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config	remote.origin.url
2023-08-17T10:47:45.9490240Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-08-17T10:47:45.9530814Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2023-08-17T10:47:45.9548888Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-08-17T10:47:45.9589602Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2023-08-17T10:47:45.9607932Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-08-17T10:47:45.9648176Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2023-08-17T10:47:45.9672661Z Entering 'third_party/pocketfft'
2023-08-17T10:47:45.9712114Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2023-08-17T10:47:45.9731464Z Entering 'third_party/protobuf'
2023-08-17T10:47:45.9771574Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2023-08-17T10:47:45.9793789Z Entering 'third_party/protobuf/third_party/benchmark'
2023-08-17T10:47:45.9833794Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2023-08-17T10:47:45.9853709Z Entering 'third_party/protobuf/third_party/googletest'
2023-08-17T10:47:45.9893208Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2023-08-17T10:47:45.9914216Z Entering 'third_party/psimd'
2023-08-17T10:47:45.9954559Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2023-08-17T10:47:45.9974205Z Entering 'third_party/pthreadpool'
2023-08-17T10:47:46.0014795Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2023-08-17T10:47:46.0033359Z Entering 'third_party/pybind11'
2023-08-17T10:47:46.0073036Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2023-08-17T10:47:46.0092706Z Entering 'third_party/python-enum'
2023-08-17T10:47:46.0132970Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config	remote.origin.url
2023-08-17T10:47:46.0151103Z Entering 'third_party/python-peachpy'
2023-08-17T10:47:46.0191048Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2023-08-17T10:47:46.0211338Z Entering 'third_party/python-six'
2023-08-17T10:47:46.0250201Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config	remote.origin.url
2023-08-17T10:47:46.0268264Z Entering 'third_party/sleef'
2023-08-17T10:47:46.0315282Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2023-08-17T10:47:46.0333008Z Entering 'third_party/tbb'
2023-08-17T10:47:46.0373864Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config	remote.origin.url
2023-08-17T10:47:46.0394708Z Entering 'third_party/tensorpipe'
2023-08-17T10:47:46.0435021Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2023-08-17T10:47:46.0454211Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-08-17T10:47:46.0493830Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2023-08-17T10:47:46.0512859Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-08-17T10:47:46.0553451Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2023-08-17T10:47:46.0572247Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-08-17T10:47:46.0611859Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2023-08-17T10:47:46.0630496Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-08-17T10:47:46.0670188Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2023-08-17T10:47:46.0688063Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-08-17T10:47:46.0728031Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2023-08-17T10:47:46.0749948Z Entering 'third_party/zstd'
2023-08-17T10:47:46.0789177Z file:/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config	remote.origin.url
2023-08-17T10:47:46.1222444Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2023-08-17T10:47:46.1493203Z Entering 'android/libs/fbjni'
2023-08-17T10:47:46.1532182Z Entering 'third_party/FP16'
2023-08-17T10:47:46.1572509Z Entering 'third_party/FXdiv'
2023-08-17T10:47:46.1610869Z Entering 'third_party/NNPACK'
2023-08-17T10:47:46.1651541Z Entering 'third_party/QNNPACK'
2023-08-17T10:47:46.1693457Z Entering 'third_party/VulkanMemoryAllocator'
2023-08-17T10:47:46.1733195Z Entering 'third_party/XNNPACK'
2023-08-17T10:47:46.1789277Z Entering 'third_party/benchmark'
2023-08-17T10:47:46.1828905Z Entering 'third_party/cpuinfo'
2023-08-17T10:47:46.1868961Z Entering 'third_party/cub'
2023-08-17T10:47:46.1908793Z Entering 'third_party/cudnn_frontend'
2023-08-17T10:47:46.1954858Z Entering 'third_party/cutlass'
2023-08-17T10:47:46.2003389Z Entering 'third_party/eigen'
2023-08-17T10:47:46.2045556Z Entering 'third_party/fbgemm'
2023-08-17T10:47:46.2084870Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-08-17T10:47:46.2124613Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-08-17T10:47:46.2163655Z Entering 'third_party/fbgemm/third_party/cutlass'
2023-08-17T10:47:46.2209430Z Entering 'third_party/fbgemm/third_party/googletest'
2023-08-17T10:47:46.2248030Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-08-17T10:47:46.2288640Z Entering 'third_party/flatbuffers'
2023-08-17T10:47:46.2330623Z Entering 'third_party/fmt'
2023-08-17T10:47:46.2370308Z Entering 'third_party/foxi'
2023-08-17T10:47:46.2409948Z Entering 'third_party/gemmlowp/gemmlowp'
2023-08-17T10:47:46.2449563Z Entering 'third_party/gloo'
2023-08-17T10:47:46.2489518Z Entering 'third_party/googletest'
2023-08-17T10:47:46.2530052Z Entering 'third_party/ideep'
2023-08-17T10:47:46.2569059Z Entering 'third_party/ideep/mkl-dnn'
2023-08-17T10:47:46.2616382Z Entering 'third_party/ios-cmake'
2023-08-17T10:47:46.2655707Z Entering 'third_party/ittapi'
2023-08-17T10:47:46.2695847Z Entering 'third_party/kineto'
2023-08-17T10:47:46.2735672Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2023-08-17T10:47:46.2774592Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2023-08-17T10:47:46.2815620Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2023-08-17T10:47:46.2854543Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2023-08-17T10:47:46.2894783Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2023-08-17T10:47:46.2932488Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2023-08-17T10:47:46.2972663Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2023-08-17T10:47:46.3012219Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2023-08-17T10:47:46.3050661Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2023-08-17T10:47:46.3091178Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2023-08-17T10:47:46.3131589Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-08-17T10:47:46.3170608Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-08-17T10:47:46.3211635Z Entering 'third_party/mimalloc'
2023-08-17T10:47:46.3252235Z Entering 'third_party/nccl/nccl'
2023-08-17T10:47:46.3291252Z Entering 'third_party/neon2sse'
2023-08-17T10:47:46.3330730Z Entering 'third_party/nlohmann'
2023-08-17T10:47:46.3370951Z Entering 'third_party/onnx'
2023-08-17T10:47:46.3428072Z Entering 'third_party/onnx/third_party/benchmark'
2023-08-17T10:47:46.3468045Z Entering 'third_party/onnx/third_party/pybind11'
2023-08-17T10:47:46.3508947Z Entering 'third_party/onnx-tensorrt'
2023-08-17T10:47:46.3547085Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-08-17T10:47:46.3589432Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-08-17T10:47:46.3628142Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-08-17T10:47:46.3666135Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-08-17T10:47:46.3710729Z Entering 'third_party/pocketfft'
2023-08-17T10:47:46.3750253Z Entering 'third_party/protobuf'
2023-08-17T10:47:46.3793595Z Entering 'third_party/protobuf/third_party/benchmark'
2023-08-17T10:47:46.3832161Z Entering 'third_party/protobuf/third_party/googletest'
2023-08-17T10:47:46.3872795Z Entering 'third_party/psimd'
2023-08-17T10:47:46.3912092Z Entering 'third_party/pthreadpool'
2023-08-17T10:47:46.3951221Z Entering 'third_party/pybind11'
2023-08-17T10:47:46.3990932Z Entering 'third_party/python-enum'
2023-08-17T10:47:46.4029798Z Entering 'third_party/python-peachpy'
2023-08-17T10:47:46.4069188Z Entering 'third_party/python-six'
2023-08-17T10:47:46.4108655Z Entering 'third_party/sleef'
2023-08-17T10:47:46.4147911Z Entering 'third_party/tbb'
2023-08-17T10:47:46.4189606Z Entering 'third_party/tensorpipe'
2023-08-17T10:47:46.4229036Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-08-17T10:47:46.4267587Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-08-17T10:47:46.4305877Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-08-17T10:47:46.4344018Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-08-17T10:47:46.4381078Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-08-17T10:47:46.4422703Z Entering 'third_party/zstd'
2023-08-17T10:47:46.4477071Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2023-08-17T10:47:46.4749359Z Entering 'android/libs/fbjni'
2023-08-17T10:47:46.4789415Z Entering 'third_party/FP16'
2023-08-17T10:47:46.4828796Z Entering 'third_party/FXdiv'
2023-08-17T10:47:46.4868598Z Entering 'third_party/NNPACK'
2023-08-17T10:47:46.4907473Z Entering 'third_party/QNNPACK'
2023-08-17T10:47:46.4947188Z Entering 'third_party/VulkanMemoryAllocator'
2023-08-17T10:47:46.4986251Z Entering 'third_party/XNNPACK'
2023-08-17T10:47:46.5040719Z Entering 'third_party/benchmark'
2023-08-17T10:47:46.5079492Z Entering 'third_party/cpuinfo'
2023-08-17T10:47:46.5120253Z Entering 'third_party/cub'
2023-08-17T10:47:46.5159397Z Entering 'third_party/cudnn_frontend'
2023-08-17T10:47:46.5205046Z Entering 'third_party/cutlass'
2023-08-17T10:47:46.5252613Z Entering 'third_party/eigen'
2023-08-17T10:47:46.5294317Z Entering 'third_party/fbgemm'
2023-08-17T10:47:46.5333762Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-08-17T10:47:46.5371773Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-08-17T10:47:46.5409787Z Entering 'third_party/fbgemm/third_party/cutlass'
2023-08-17T10:47:46.5454273Z Entering 'third_party/fbgemm/third_party/googletest'
2023-08-17T10:47:46.5491731Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-08-17T10:47:46.5531819Z Entering 'third_party/flatbuffers'
2023-08-17T10:47:46.5573816Z Entering 'third_party/fmt'
2023-08-17T10:47:46.5612382Z Entering 'third_party/foxi'
2023-08-17T10:47:46.5652207Z Entering 'third_party/gemmlowp/gemmlowp'
2023-08-17T10:47:46.5691474Z Entering 'third_party/gloo'
2023-08-17T10:47:46.5730655Z Entering 'third_party/googletest'
2023-08-17T10:47:46.5770261Z Entering 'third_party/ideep'
2023-08-17T10:47:46.5808393Z Entering 'third_party/ideep/mkl-dnn'
2023-08-17T10:47:46.5856491Z Entering 'third_party/ios-cmake'
2023-08-17T10:47:46.5896612Z Entering 'third_party/ittapi'
2023-08-17T10:47:46.5935569Z Entering 'third_party/kineto'
2023-08-17T10:47:46.5975414Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2023-08-17T10:47:46.6014177Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2023-08-17T10:47:46.6054060Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2023-08-17T10:47:46.6093086Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2023-08-17T10:47:46.6133087Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2023-08-17T10:47:46.6170969Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2023-08-17T10:47:46.6211592Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2023-08-17T10:47:46.6250726Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2023-08-17T10:47:46.6289947Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2023-08-17T10:47:46.6330183Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2023-08-17T10:47:46.6370284Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-08-17T10:47:46.6407985Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-08-17T10:47:46.6447368Z Entering 'third_party/mimalloc'
2023-08-17T10:47:46.6487664Z Entering 'third_party/nccl/nccl'
2023-08-17T10:47:46.6526582Z Entering 'third_party/neon2sse'
2023-08-17T10:47:46.6564754Z Entering 'third_party/nlohmann'
2023-08-17T10:47:46.6604686Z Entering 'third_party/onnx'
2023-08-17T10:47:46.6660537Z Entering 'third_party/onnx/third_party/benchmark'
2023-08-17T10:47:46.6699408Z Entering 'third_party/onnx/third_party/pybind11'
2023-08-17T10:47:46.6740879Z Entering 'third_party/onnx-tensorrt'
2023-08-17T10:47:46.6778850Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-08-17T10:47:46.6822681Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-08-17T10:47:46.6861446Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-08-17T10:47:46.6899219Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-08-17T10:47:46.6942837Z Entering 'third_party/pocketfft'
2023-08-17T10:47:46.6981890Z Entering 'third_party/protobuf'
2023-08-17T10:47:46.7023647Z Entering 'third_party/protobuf/third_party/benchmark'
2023-08-17T10:47:46.7061546Z Entering 'third_party/protobuf/third_party/googletest'
2023-08-17T10:47:46.7101586Z Entering 'third_party/psimd'
2023-08-17T10:47:46.7140442Z Entering 'third_party/pthreadpool'
2023-08-17T10:47:46.7179646Z Entering 'third_party/pybind11'
2023-08-17T10:47:46.7219112Z Entering 'third_party/python-enum'
2023-08-17T10:47:46.7258151Z Entering 'third_party/python-peachpy'
2023-08-17T10:47:46.7297648Z Entering 'third_party/python-six'
2023-08-17T10:47:46.7335848Z Entering 'third_party/sleef'
2023-08-17T10:47:46.7375203Z Entering 'third_party/tbb'
2023-08-17T10:47:46.7416420Z Entering 'third_party/tensorpipe'
2023-08-17T10:47:46.7454289Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-08-17T10:47:46.7492435Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-08-17T10:47:46.7529443Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-08-17T10:47:46.7568515Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-08-17T10:47:46.7604288Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-08-17T10:47:46.7646805Z Entering 'third_party/zstd'
2023-08-17T10:47:46.7698658Z ##[endgroup]
2023-08-17T10:47:46.7749961Z [command]/usr/bin/git log -1 --format='%H'
2023-08-17T10:47:46.7784585Z '384e0d104fd077d31efafc564129660e9b7a0f25'
2023-08-17T10:47:46.8044592Z Prepare all required actions
2023-08-17T10:47:46.8045150Z Getting action download info
2023-08-17T10:47:47.0881540Z ##[group]Run ./.github/actions/setup-linux
2023-08-17T10:47:47.0881819Z env:
2023-08-17T10:47:47.0882038Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:47:47.0882277Z ##[endgroup]
2023-08-17T10:47:47.0952572Z ##[group]Run set -euo pipefail
2023-08-17T10:47:47.0952881Z set -euo pipefail
2023-08-17T10:47:47.0953242Z function get_ec2_metadata() {
2023-08-17T10:47:47.0953554Z   # Pulled from instance metadata endpoint for EC2
2023-08-17T10:47:47.0954035Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2023-08-17T10:47:47.0954431Z   category=$1
2023-08-17T10:47:47.0954746Z   # If it is GCP runner (runner name contains gcp), do not run this
2023-08-17T10:47:47.0955082Z   runner_name_str=gh-ci-gcp-a100-12
2023-08-17T10:47:47.0955452Z   if [[ $runner_name_str != *"gcp"* ]]; then
2023-08-17T10:47:47.0955805Z     curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2023-08-17T10:47:47.0956090Z   else
2023-08-17T10:47:47.0956413Z     echo "Runner is from Google Cloud Platform, No info on ec2 metadata"
2023-08-17T10:47:47.0956718Z   fi
2023-08-17T10:47:47.0956920Z }
2023-08-17T10:47:47.0957169Z echo "ami-id: $(get_ec2_metadata ami-id)"
2023-08-17T10:47:47.0957506Z echo "instance-id: $(get_ec2_metadata instance-id)"
2023-08-17T10:47:47.0957868Z echo "instance-type: $(get_ec2_metadata instance-type)"
2023-08-17T10:47:47.0958196Z echo "system info $(uname -a)"
2023-08-17T10:47:47.0978223Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-08-17T10:47:47.0978499Z env:
2023-08-17T10:47:47.0978723Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:47:47.0978961Z ##[endgroup]
2023-08-17T10:47:47.1026802Z ami-id: Runner is from Google Cloud Platform, No info on ec2 metadata
2023-08-17T10:47:47.1032348Z instance-id: Runner is from Google Cloud Platform, No info on ec2 metadata
2023-08-17T10:47:47.1038479Z instance-type: Runner is from Google Cloud Platform, No info on ec2 metadata
2023-08-17T10:47:47.1049627Z system info Linux gh-ci-gcp-a100-12 5.15.0-1037-gcp #45~20.04.1-Ubuntu SMP Thu Jun 22 08:31:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
2023-08-17T10:47:47.1066928Z ##[group]Run if systemctl is-active --quiet docker; then
2023-08-17T10:47:47.1067301Z if systemctl is-active --quiet docker; then
2023-08-17T10:47:47.1067621Z     echo "Docker daemon is running...";
2023-08-17T10:47:47.1067880Z else
2023-08-17T10:47:47.1068169Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2023-08-17T10:47:47.1068548Z fi
2023-08-17T10:47:47.1085652Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-08-17T10:47:47.1085935Z env:
2023-08-17T10:47:47.1086143Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:47:47.1086399Z ##[endgroup]
2023-08-17T10:47:47.1151114Z Docker daemon is running...
2023-08-17T10:47:47.1186443Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2023-08-17T10:47:47.1186764Z with:
2023-08-17T10:47:47.1186973Z   shell: bash
2023-08-17T10:47:47.1187186Z   timeout_minutes: 5
2023-08-17T10:47:47.1187420Z   max_attempts: 3
2023-08-17T10:47:47.1187653Z   retry_wait_seconds: 30
2023-08-17T10:47:47.1188416Z   command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \
    --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"

2023-08-17T10:47:47.1189007Z   polling_interval_seconds: 1
2023-08-17T10:47:47.1189263Z   warning_on_retry: true
2023-08-17T10:47:47.1189494Z   continue_on_error: false
2023-08-17T10:47:47.1189720Z env:
2023-08-17T10:47:47.1189936Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:47:47.1190183Z   AWS_RETRY_MODE: standard
2023-08-17T10:47:47.1190612Z   AWS_MAX_ATTEMPTS: 5
2023-08-17T10:47:47.1190875Z   AWS_DEFAULT_REGION: us-east-1
2023-08-17T10:47:47.1191118Z ##[endgroup]
2023-08-17T10:47:49.2982394Z WARNING! Your password will be stored unencrypted in /home/ubuntu/.docker/config.json.
2023-08-17T10:47:49.2983451Z Configure a credential helper to remove this warning. See
2023-08-17T10:47:49.2984596Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2023-08-17T10:47:49.2985027Z 
2023-08-17T10:47:49.2985166Z Login Succeeded
2023-08-17T10:47:50.1984603Z Command completed after 1 attempt(s).
2023-08-17T10:47:50.2064368Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2023-08-17T10:47:50.2064870Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2023-08-17T10:47:50.2065329Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2023-08-17T10:47:50.2084103Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-08-17T10:47:50.2084422Z env:
2023-08-17T10:47:50.2084761Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:47:50.2085177Z ##[endgroup]
2023-08-17T10:47:50.2179026Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2023-08-17T10:47:50.2179575Z # ignore expansion of "docker ps -q" since it could be empty
2023-08-17T10:47:50.2180031Z # shellcheck disable=SC2046
2023-08-17T10:47:50.2180402Z docker stop $(docker ps -q) || true
2023-08-17T10:47:50.2180775Z # Prune all of the docker images
2023-08-17T10:47:50.2225062Z docker system prune -af
2023-08-17T10:47:50.2242815Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-08-17T10:47:50.2243109Z env:
2023-08-17T10:47:50.2243333Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:47:50.2243576Z ##[endgroup]
2023-08-17T10:47:50.2667418Z "docker stop" requires at least 1 argument.
2023-08-17T10:47:50.2668212Z See 'docker stop --help'.
2023-08-17T10:47:50.2668396Z 
2023-08-17T10:47:50.2668591Z Usage:  docker stop [OPTIONS] CONTAINER [CONTAINER...]
2023-08-17T10:47:50.2668800Z 
2023-08-17T10:47:50.2668926Z Stop one or more running containers
2023-08-17T10:47:50.2891280Z Total reclaimed space: 0B
2023-08-17T10:47:50.2929310Z ##[group]Run set +e
2023-08-17T10:47:50.2929827Z set +e
2023-08-17T10:47:50.2930035Z set -x
2023-08-17T10:47:50.2930247Z 
2023-08-17T10:47:50.2930504Z PT_DOMAIN=download.pytorch.org
2023-08-17T10:47:50.2930939Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400,
2023-08-17T10:47:50.2931417Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last
2023-08-17T10:47:50.2931777Z # one is returned at random
2023-08-17T10:47:50.2932092Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1)
2023-08-17T10:47:50.2932363Z 
2023-08-17T10:47:50.2932588Z if [ -z "${RESOLVED_IP}" ]; then
2023-08-17T10:47:50.2932936Z   echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..."
2023-08-17T10:47:50.2933317Z   RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1)
2023-08-17T10:47:50.2933596Z 
2023-08-17T10:47:50.2933832Z   if [ -z "${RESOLVED_IP}" ]; then
2023-08-17T10:47:50.2934151Z     echo "Couldn't resolve ${PT_DOMAIN}, exiting..."
2023-08-17T10:47:50.2934432Z     exit 1
2023-08-17T10:47:50.2934646Z   fi
2023-08-17T10:47:50.2934836Z fi
2023-08-17T10:47:50.2935041Z 
2023-08-17T10:47:50.2935299Z if grep -r "${PT_DOMAIN}" /etc/hosts; then
2023-08-17T10:47:50.2935602Z   # Clean up any old records first
2023-08-17T10:47:50.2935887Z   sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts
2023-08-17T10:47:50.2936139Z fi
2023-08-17T10:47:50.2936340Z 
2023-08-17T10:47:50.2936619Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts
2023-08-17T10:47:50.2936901Z cat /etc/hosts
2023-08-17T10:47:50.2955057Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-08-17T10:47:50.2955351Z env:
2023-08-17T10:47:50.2955573Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:47:50.2955817Z ##[endgroup]
2023-08-17T10:47:50.2992538Z + PT_DOMAIN=download.pytorch.org
2023-08-17T10:47:50.2998564Z ++ dig -4 +short download.pytorch.org
2023-08-17T10:47:50.2999136Z ++ tail -n1
2023-08-17T10:47:50.3186100Z + RESOLVED_IP=13.226.22.44
2023-08-17T10:47:50.3186789Z + '[' -z 13.226.22.44 ']'
2023-08-17T10:47:50.3187401Z + grep -r download.pytorch.org /etc/hosts
2023-08-17T10:47:50.3197993Z 13.226.22.44 download.pytorch.org
2023-08-17T10:47:50.3199163Z + sudo sed -i /download.pytorch.org/d /etc/hosts
2023-08-17T10:47:50.3295124Z + echo '13.226.22.44 download.pytorch.org'
2023-08-17T10:47:50.3296062Z + sudo tee -a /etc/hosts
2023-08-17T10:47:50.3367157Z 13.226.22.44 download.pytorch.org
2023-08-17T10:47:50.3373838Z + cat /etc/hosts
2023-08-17T10:47:50.3381556Z 127.0.0.1 localhost
2023-08-17T10:47:50.3381828Z 
2023-08-17T10:47:50.3382224Z # The following lines are desirable for IPv6 capable hosts
2023-08-17T10:47:50.3397053Z ::1 ip6-localhost ip6-loopback
2023-08-17T10:47:50.3397403Z fe00::0 ip6-localnet
2023-08-17T10:47:50.3397719Z ff00::0 ip6-mcastprefix
2023-08-17T10:47:50.3398004Z ff02::1 ip6-allnodes
2023-08-17T10:47:50.3398290Z ff02::2 ip6-allrouters
2023-08-17T10:47:50.3398553Z ff02::3 ip6-allhosts
2023-08-17T10:47:50.3398838Z 169.254.169.254 metadata.google.internal metadata
2023-08-17T10:47:50.3399146Z 13.226.22.44 download.pytorch.org
2023-08-17T10:47:50.3475879Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main
2023-08-17T10:47:50.3476203Z with:
2023-08-17T10:47:50.3476780Z   docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:f89d62c8626a24aae4b1a64d42a3caf9cd95bddd
2023-08-17T10:47:50.3477385Z   docker-build-dir: .ci/docker
2023-08-17T10:47:50.3477651Z   working-directory: .
2023-08-17T10:47:50.3477980Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2023-08-17T10:47:50.3478256Z env:
2023-08-17T10:47:50.3478474Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:47:50.3478715Z ##[endgroup]
2023-08-17T10:47:50.3496266Z ##[group]Run set -ex
2023-08-17T10:47:50.3496559Z set -ex
2023-08-17T10:47:50.3496758Z 
2023-08-17T10:47:50.3497083Z # If the docker build directory or the build script doesn't exist, the action will
2023-08-17T10:47:50.3497573Z # gracefully return the docker image name as it is.  Pulling docker image in Linux
2023-08-17T10:47:50.3497976Z # job could then download the pre-built image as usual
2023-08-17T10:47:50.3498348Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then
2023-08-17T10:47:50.3498701Z   echo "skip=true" >> "${GITHUB_OUTPUT}"
2023-08-17T10:47:50.3499051Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2023-08-17T10:47:50.3499345Z 
2023-08-17T10:47:50.3499646Z   echo "There is no Docker build script in ${REPO_NAME} repo, skipping..."
2023-08-17T10:47:50.3499958Z   exit 0
2023-08-17T10:47:50.3500174Z else
2023-08-17T10:47:50.3500446Z   echo "skip=false" >> "${GITHUB_OUTPUT}"
2023-08-17T10:47:50.3500710Z fi
2023-08-17T10:47:50.3500911Z 
2023-08-17T10:47:50.3501202Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then
2023-08-17T10:47:50.3501611Z   # The docker image name already includes the ECR prefix and tag, so we can just
2023-08-17T10:47:50.3501995Z   # use it as it is, but first let's extract the tag
2023-08-17T10:47:50.3502336Z   DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}')
2023-08-17T10:47:50.3502703Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2023-08-17T10:47:50.3503069Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2023-08-17T10:47:50.3503412Z else
2023-08-17T10:47:50.3503691Z   DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}")
2023-08-17T10:47:50.3504052Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2023-08-17T10:47:50.3504482Z   echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2023-08-17T10:47:50.3504828Z fi
2023-08-17T10:47:50.3522270Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-08-17T10:47:50.3522775Z env:
2023-08-17T10:47:50.3523096Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:47:50.3523391Z   REPO_NAME: pytorch
2023-08-17T10:47:50.3523969Z   DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:f89d62c8626a24aae4b1a64d42a3caf9cd95bddd
2023-08-17T10:47:50.3524555Z   DOCKER_BUILD_DIR: .ci/docker
2023-08-17T10:47:50.3524881Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2023-08-17T10:47:50.3525175Z ##[endgroup]
2023-08-17T10:47:50.3562087Z + [[ ! -d .ci/docker ]]
2023-08-17T10:47:50.3562478Z + [[ ! -f .ci/docker/build.sh ]]
2023-08-17T10:47:50.3562971Z + echo skip=false
2023-08-17T10:47:50.3564082Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:f89d62c8626a24aae4b1a64d42a3caf9cd95bddd == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]]
2023-08-17T10:47:50.3570395Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:f89d62c8626a24aae4b1a64d42a3caf9cd95bddd
2023-08-17T10:47:50.3571033Z ++ awk -F '[:,]' '{print $2}'
2023-08-17T10:47:50.3590549Z + DOCKER_TAG=f89d62c8626a24aae4b1a64d42a3caf9cd95bddd
2023-08-17T10:47:50.3591023Z + echo docker-tag=f89d62c8626a24aae4b1a64d42a3caf9cd95bddd
2023-08-17T10:47:50.3591876Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:f89d62c8626a24aae4b1a64d42a3caf9cd95bddd
2023-08-17T10:47:50.3645274Z ##[group]Run set +e
2023-08-17T10:47:50.3645573Z set +e
2023-08-17T10:47:50.3645972Z set -x
2023-08-17T10:47:50.3646170Z 
2023-08-17T10:47:50.3646473Z # Check if image already exists, if it does then skip building it
2023-08-17T10:47:50.3646853Z if docker manifest inspect "${DOCKER_IMAGE}"; then
2023-08-17T10:47:50.3647141Z   exit 0
2023-08-17T10:47:50.3647341Z fi
2023-08-17T10:47:50.3647542Z 
2023-08-17T10:47:50.3647851Z # NB: This part requires a full checkout. Otherwise, the merge base will
2023-08-17T10:47:50.3648264Z # be empty.  The default action would be to continue rebuild the image
2023-08-17T10:47:50.3648694Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then
2023-08-17T10:47:50.3649060Z   # if we're on the base branch then use the parent commit
2023-08-17T10:47:50.3649394Z   MERGE_BASE=$(git rev-parse HEAD~)
2023-08-17T10:47:50.3649655Z else
2023-08-17T10:47:50.3649938Z   # otherwise we're on a PR, so use the most recent base commit
2023-08-17T10:47:50.3650312Z   MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION")
2023-08-17T10:47:50.3650589Z fi
2023-08-17T10:47:50.3650793Z 
2023-08-17T10:47:50.3651020Z if [[ -z "${MERGE_BASE}" ]]; then
2023-08-17T10:47:50.3651329Z   echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2023-08-17T10:47:50.3651587Z 
2023-08-17T10:47:50.3651942Z   echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..."
2023-08-17T10:47:50.3652290Z   exit 0
2023-08-17T10:47:50.3652484Z fi
2023-08-17T10:47:50.3652686Z 
2023-08-17T10:47:50.3652972Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then
2023-08-17T10:47:50.3653418Z   echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit"
2023-08-17T10:47:50.3653762Z   exit 1
2023-08-17T10:47:50.3653979Z fi
2023-08-17T10:47:50.3654179Z 
2023-08-17T10:47:50.3654486Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}")
2023-08-17T10:47:50.3654913Z # If no image exists but the hash is the same as the previous hash then we should error out here
2023-08-17T10:47:50.3655322Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then
2023-08-17T10:47:50.3655769Z   echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch"
2023-08-17T10:47:50.3656249Z   echo "         Will re-build docker image to store in local cache, TTS may be longer"
2023-08-17T10:47:50.3656561Z fi
2023-08-17T10:47:50.3656748Z 
2023-08-17T10:47:50.3657006Z echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2023-08-17T10:47:50.3673468Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-08-17T10:47:50.3673738Z env:
2023-08-17T10:47:50.3673974Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:47:50.3674235Z   DOCKER_BUILD_DIR: .ci/docker
2023-08-17T10:47:50.3674540Z   BASE_REVISION: 384e0d104fd077d31efafc564129660e9b7a0f25
2023-08-17T10:47:50.3675317Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:f89d62c8626a24aae4b1a64d42a3caf9cd95bddd
2023-08-17T10:47:50.3675939Z   DOCKER_TAG: f89d62c8626a24aae4b1a64d42a3caf9cd95bddd
2023-08-17T10:47:50.3676218Z ##[endgroup]
2023-08-17T10:47:50.3712149Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:f89d62c8626a24aae4b1a64d42a3caf9cd95bddd
2023-08-17T10:47:51.0772248Z {
2023-08-17T10:47:51.0772701Z 	"schemaVersion": 2,
2023-08-17T10:47:51.0773429Z 	"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
2023-08-17T10:47:51.0773832Z 	"config": {
2023-08-17T10:47:51.0774165Z 		"mediaType": "application/vnd.docker.container.image.v1+json",
2023-08-17T10:47:51.0774961Z 		"size": 45858,
2023-08-17T10:47:51.0775296Z 		"digest": "sha256:808d86a8c9535f1195b348920a0142956c41b85dbcfdf4cc80d453060e812b6f"
2023-08-17T10:47:51.0775602Z 	},
2023-08-17T10:47:51.0775799Z 	"layers": [
2023-08-17T10:47:51.0775986Z 		{
2023-08-17T10:47:51.0776340Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0776671Z 			"size": 28580519,
2023-08-17T10:47:51.0777845Z 			"digest": "sha256:f0412dfb1aaea4892c13dab6f771bcd79641a4bdcb521ce881f5dcfc0df9a7a5"
2023-08-17T10:47:51.0778628Z 		},
2023-08-17T10:47:51.0779016Z 		{
2023-08-17T10:47:51.0779646Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0780271Z 			"size": 7943183,
2023-08-17T10:47:51.0780832Z 			"digest": "sha256:b0f696c0aebbc48776bf760120174a61b67dd96b6d0ec79c2cf85f09b76fc2cb"
2023-08-17T10:47:51.0781363Z 		},
2023-08-17T10:47:51.0781692Z 		{
2023-08-17T10:47:51.0782287Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0782887Z 			"size": 55728211,
2023-08-17T10:47:51.0783602Z 			"digest": "sha256:e627444df06f688864552aa11965a24023b95cdb3f71995e25184bf507b865ae"
2023-08-17T10:47:51.0784146Z 		},
2023-08-17T10:47:51.0784919Z 		{
2023-08-17T10:47:51.0785310Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0785653Z 			"size": 181,
2023-08-17T10:47:51.0786001Z 			"digest": "sha256:dcf21018e9341e3d453cbb58f229def10eb2974c661617d00137fc301a27e135"
2023-08-17T10:47:51.0786320Z 		},
2023-08-17T10:47:51.0786498Z 		{
2023-08-17T10:47:51.0786822Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0787151Z 			"size": 6881,
2023-08-17T10:47:51.0787509Z 			"digest": "sha256:a2855a2ef2e0d887faef408c1cea9e6be55a3fb00052019da35936d3c0a8308a"
2023-08-17T10:47:51.0788001Z 		},
2023-08-17T10:47:51.0788290Z 		{
2023-08-17T10:47:51.0788793Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0789744Z 			"size": 1290796851,
2023-08-17T10:47:51.0790634Z 			"digest": "sha256:8708e3e52fea95e53818c1e6e78b2dbb06b987e0b61f36a8cc44b06923f9bd6f"
2023-08-17T10:47:51.0791474Z 		},
2023-08-17T10:47:51.0791975Z 		{
2023-08-17T10:47:51.0792928Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0793817Z 			"size": 62491,
2023-08-17T10:47:51.0794729Z 			"digest": "sha256:be2aea6b86ef5dd0440104da131a26bf40e86a8a2fe851aa8077816b74b920d0"
2023-08-17T10:47:51.0795591Z 		},
2023-08-17T10:47:51.0796061Z 		{
2023-08-17T10:47:51.0796925Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0797762Z 			"size": 1682,
2023-08-17T10:47:51.0798782Z 			"digest": "sha256:71ab173e0cbfbeb3f978bb3ee68776c5cef76f435fc15d74980499e0e6e4cac9"
2023-08-17T10:47:51.0799462Z 		},
2023-08-17T10:47:51.0799803Z 		{
2023-08-17T10:47:51.0800397Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0800908Z 			"size": 1521,
2023-08-17T10:47:51.0801403Z 			"digest": "sha256:308bdac48e2c8eb82a667194a286bf81d19f63d04121abca7f7157202199062b"
2023-08-17T10:47:51.0801727Z 		},
2023-08-17T10:47:51.0801938Z 		{
2023-08-17T10:47:51.0802510Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0803290Z 			"size": 2489665476,
2023-08-17T10:47:51.0804341Z 			"digest": "sha256:25a33b062462718545a8d6bd9881f605a30820672c6113cfe225448d85abd9e3"
2023-08-17T10:47:51.0804806Z 		},
2023-08-17T10:47:51.0805073Z 		{
2023-08-17T10:47:51.0805544Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0806001Z 			"size": 86058,
2023-08-17T10:47:51.0806566Z 			"digest": "sha256:7788c362a0e02fccb432057990aabdbcbdbc604f02c3b932006a44d98b14e9c3"
2023-08-17T10:47:51.0807180Z 		},
2023-08-17T10:47:51.0807441Z 		{
2023-08-17T10:47:51.0807766Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0808139Z 			"size": 1800,
2023-08-17T10:47:51.0808487Z 			"digest": "sha256:606cc67f555013472ffbd28ecb7b7d5c7f2f86190e0ae379b40a023641ea99e9"
2023-08-17T10:47:51.0808971Z 		},
2023-08-17T10:47:51.0809162Z 		{
2023-08-17T10:47:51.0809469Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0809798Z 			"size": 240876408,
2023-08-17T10:47:51.0810167Z 			"digest": "sha256:c1e3d3c5b53239d9b7f15bf1baab9c8e419e585e33aaf0e0a09a336579b124dd"
2023-08-17T10:47:51.0810494Z 		},
2023-08-17T10:47:51.0810668Z 		{
2023-08-17T10:47:51.0810988Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0811342Z 			"size": 545,
2023-08-17T10:47:51.0811707Z 			"digest": "sha256:42dfcd7df9dbeb9215bddacfdd3ed351b6d5b692bcb70665d7b42b20aab9694f"
2023-08-17T10:47:51.0812061Z 		},
2023-08-17T10:47:51.0812247Z 		{
2023-08-17T10:47:51.0812565Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0812878Z 			"size": 1278,
2023-08-17T10:47:51.0813225Z 			"digest": "sha256:33a9133686b5907efdfd23eaa02b3ef7518f3e6ea4a428d55afe09b7a60f2297"
2023-08-17T10:47:51.0813551Z 		},
2023-08-17T10:47:51.0813737Z 		{
2023-08-17T10:47:51.0814045Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0814372Z 			"size": 483,
2023-08-17T10:47:51.0814703Z 			"digest": "sha256:2129cd14b507918a9e418a4a0b636f59b05600844567eab2ef8170093de355fa"
2023-08-17T10:47:51.0815001Z 		},
2023-08-17T10:47:51.0815176Z 		{
2023-08-17T10:47:51.0815495Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0815820Z 			"size": 91701511,
2023-08-17T10:47:51.0816145Z 			"digest": "sha256:680ec3688d92666a1a9df00d06e15493c92e4715fcb56765a8290612237b0efd"
2023-08-17T10:47:51.0816429Z 		},
2023-08-17T10:47:51.0816614Z 		{
2023-08-17T10:47:51.0816932Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0817255Z 			"size": 2748,
2023-08-17T10:47:51.0817593Z 			"digest": "sha256:07d72ce662acf1a5c00751070e6cba8bcf3e2a46cefd1a43231a14a516a0d2c9"
2023-08-17T10:47:51.0817961Z 		},
2023-08-17T10:47:51.0818146Z 		{
2023-08-17T10:47:51.0818464Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0818776Z 			"size": 2021,
2023-08-17T10:47:51.0819108Z 			"digest": "sha256:9383e2c9b0209d071e35b8b3d404b61ff09329c7a035dbd2614d682512cfc6ca"
2023-08-17T10:47:51.0819415Z 		},
2023-08-17T10:47:51.0819599Z 		{
2023-08-17T10:47:51.0819905Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0820228Z 			"size": 700,
2023-08-17T10:47:51.0820582Z 			"digest": "sha256:54c5d1abcbe9b53a7155eaa899be6e0401bf3a16e9e9e96cd1615d51a786184e"
2023-08-17T10:47:51.0820919Z 		},
2023-08-17T10:47:51.0821091Z 		{
2023-08-17T10:47:51.0821409Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0821733Z 			"size": 2364302895,
2023-08-17T10:47:51.0822074Z 			"digest": "sha256:7660267bb9416ed44eb8f9f757dbec8a1a536487a8c48d09c55703fa994fe6b3"
2023-08-17T10:47:51.0822368Z 		},
2023-08-17T10:47:51.0822556Z 		{
2023-08-17T10:47:51.0822878Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0823186Z 			"size": 380,
2023-08-17T10:47:51.0823518Z 			"digest": "sha256:970f7a60e31870c71d4fe787a9c9cb18e1e35522c654e95d9f92e7328a72ce73"
2023-08-17T10:47:51.0823827Z 		},
2023-08-17T10:47:51.0824140Z 		{
2023-08-17T10:47:51.0824455Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0824778Z 			"size": 6330,
2023-08-17T10:47:51.0825127Z 			"digest": "sha256:976b5e991b6dbd827deb050b065addd2d66a6f933d316b1aee3f5101350df528"
2023-08-17T10:47:51.0825448Z 		},
2023-08-17T10:47:51.0825621Z 		{
2023-08-17T10:47:51.0825942Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0826263Z 			"size": 803,
2023-08-17T10:47:51.0826605Z 			"digest": "sha256:2d93807632f454adb8c6fe5bee2b258631f94db3eaf30292ce3b8d82e11fdc17"
2023-08-17T10:47:51.0826913Z 		},
2023-08-17T10:47:51.0827101Z 		{
2023-08-17T10:47:51.0827421Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0827930Z 			"size": 106,
2023-08-17T10:47:51.0828263Z 			"digest": "sha256:a08f76bef6ab782f34094c1094f9f95e57e9df55f9df92521ab187d4b70cc98c"
2023-08-17T10:47:51.0828583Z 		},
2023-08-17T10:47:51.0828770Z 		{
2023-08-17T10:47:51.0829074Z + exit 0
2023-08-17T10:47:51.0829397Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0829719Z 			"size": 887,
2023-08-17T10:47:51.0830040Z 			"digest": "sha256:6dda747386b52ee1a44e83805a30220ec2827627dd4e4add20e0d11510ffe8d0"
2023-08-17T10:47:51.0830348Z 		},
2023-08-17T10:47:51.0830533Z 		{
2023-08-17T10:47:51.0830852Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0831219Z 			"size": 121477358,
2023-08-17T10:47:51.0831589Z 			"digest": "sha256:427ceb14dadb87ae0ddbdd8f6541f87b4195ee602f5cd4ebb18cdde4097ba4bc"
2023-08-17T10:47:51.0831930Z 		},
2023-08-17T10:47:51.0832117Z 		{
2023-08-17T10:47:51.0832422Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0832817Z 			"size": 109,
2023-08-17T10:47:51.0833145Z 			"digest": "sha256:193f08489834fc93026baf5ef184472bf9abf06aafa95514637df7227bf22f05"
2023-08-17T10:47:51.0833452Z 		},
2023-08-17T10:47:51.0833623Z 		{
2023-08-17T10:47:51.0833947Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0834269Z 			"size": 558,
2023-08-17T10:47:51.0834615Z 			"digest": "sha256:9d445bb0a609bc06753da0a7dd33ce729e2e129cf047aa0b0b5575e1fa11500e"
2023-08-17T10:47:51.0834921Z 		},
2023-08-17T10:47:51.0835107Z 		{
2023-08-17T10:47:51.0835426Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0835750Z 			"size": 1175295,
2023-08-17T10:47:51.0836083Z 			"digest": "sha256:a91ec5f1152963af4d14da19ad1ba49d816581a0bda76c88e790253fa31dd154"
2023-08-17T10:47:51.0836405Z 		},
2023-08-17T10:47:51.0836589Z 		{
2023-08-17T10:47:51.0836893Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0837217Z 			"size": 103,
2023-08-17T10:47:51.0837573Z 			"digest": "sha256:5c77da10efefce8fe887957ba52b1a51af2b18508b43f7acd61196107a3cd2f0"
2023-08-17T10:47:51.0837949Z 		},
2023-08-17T10:47:51.0838123Z 		{
2023-08-17T10:47:51.0838444Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0838768Z 			"size": 1489,
2023-08-17T10:47:51.0839113Z 			"digest": "sha256:9de12a84efdb2559ed19f70b6f10e6ff238d900d68596ba0b8a4b3317ae998e2"
2023-08-17T10:47:51.0839425Z 		},
2023-08-17T10:47:51.0839610Z 		{
2023-08-17T10:47:51.0839924Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0840248Z 			"size": 437576625,
2023-08-17T10:47:51.0840597Z 			"digest": "sha256:5ef5bbb9550d7ce6b6cb765847dcaf1dcd062fa0a51fa695077c775ee1bbdc5e"
2023-08-17T10:47:51.0840925Z 		},
2023-08-17T10:47:51.0841110Z 		{
2023-08-17T10:47:51.0841428Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0841744Z 			"size": 160,
2023-08-17T10:47:51.0842074Z 			"digest": "sha256:c8050d8f6276d6370bbad9803a43f125cd6172cc6768756ee76409d89918bda3"
2023-08-17T10:47:51.0842379Z 		},
2023-08-17T10:47:51.0842733Z 		{
2023-08-17T10:47:51.0843201Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0843754Z 			"size": 452,
2023-08-17T10:47:51.0844103Z 			"digest": "sha256:4b3155d4eae92b27b82961bdc35e718bea470a8481bf1eda66d1051b5381b73d"
2023-08-17T10:47:51.0844420Z 		},
2023-08-17T10:47:51.0844592Z 		{
2023-08-17T10:47:51.0844912Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0845235Z 			"size": 61500155,
2023-08-17T10:47:51.0845589Z 			"digest": "sha256:29fd9070c65a05535fab23ecfc89ee40e415f06daa946c4da25bcb29f33059dd"
2023-08-17T10:47:51.0845896Z 		},
2023-08-17T10:47:51.0846081Z 		{
2023-08-17T10:47:51.0846399Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0846719Z 			"size": 104,
2023-08-17T10:47:51.0847161Z 			"digest": "sha256:9dc12ea1a262e272534b0d0054ccdb9be89c34692383003deb60c1ce00e66df7"
2023-08-17T10:47:51.0847473Z 		},
2023-08-17T10:47:51.0847657Z 		{
2023-08-17T10:47:51.0848011Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0848335Z 			"size": 403,
2023-08-17T10:47:51.0848691Z 			"digest": "sha256:a0877f65810fba9e0de6f06aa7aa800a6b33739af51a8ea514c07b7dad2032dc"
2023-08-17T10:47:51.0849013Z 		},
2023-08-17T10:47:51.0849186Z 		{
2023-08-17T10:47:51.0849510Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0849831Z 			"size": 20262037,
2023-08-17T10:47:51.0850177Z 			"digest": "sha256:42021fac2a0cb6340cc0a8f624303e7c61ab11ec3a88b20e8050f8fa06b82aa8"
2023-08-17T10:47:51.0850481Z 		},
2023-08-17T10:47:51.0850667Z 		{
2023-08-17T10:47:51.0850986Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0851307Z 			"size": 562,
2023-08-17T10:47:51.0851633Z 			"digest": "sha256:9304db884963ee12a2040bb7cea1d0fb95b5c65f55e8f547c5eb048b9895436d"
2023-08-17T10:47:51.0851952Z 		},
2023-08-17T10:47:51.0852137Z 		{
2023-08-17T10:47:51.0852455Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0852765Z 			"size": 43165481,
2023-08-17T10:47:51.0853112Z 			"digest": "sha256:e8fa116b74bbcae971050fbb7c26356a769536538a2b4f19191f7e90026fad5d"
2023-08-17T10:47:51.0853427Z 		},
2023-08-17T10:47:51.0853614Z 		{
2023-08-17T10:47:51.0853918Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0854243Z 			"size": 106,
2023-08-17T10:47:51.0854612Z 			"digest": "sha256:94c4dbc1ec2288fa0f837e807ab46ed2bffbfbfc7dff3d316b29aca26b1b1b67"
2023-08-17T10:47:51.0854956Z 		},
2023-08-17T10:47:51.0855126Z 		{
2023-08-17T10:47:51.0855443Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0855766Z 			"size": 418,
2023-08-17T10:47:51.0856108Z 			"digest": "sha256:57d789d9281ff1274e6ccb16c8ba398ad18cf21a67cb236dbdf6550eead825b6"
2023-08-17T10:47:51.0856420Z 		},
2023-08-17T10:47:51.0856605Z 		{
2023-08-17T10:47:51.0856927Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0857253Z 			"size": 700,
2023-08-17T10:47:51.0857598Z 			"digest": "sha256:54c5d1abcbe9b53a7155eaa899be6e0401bf3a16e9e9e96cd1615d51a786184e"
2023-08-17T10:47:51.0857983Z 		},
2023-08-17T10:47:51.0858171Z 		{
2023-08-17T10:47:51.0858490Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0858802Z 			"size": 118,
2023-08-17T10:47:51.0859153Z 			"digest": "sha256:886477fddcc1d73a5c33848db4af2b56bbae96fee9654ebccf519bc767564f50"
2023-08-17T10:47:51.0859482Z 		},
2023-08-17T10:47:51.0859670Z 		{
2023-08-17T10:47:51.0859973Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0860295Z 			"size": 138,
2023-08-17T10:47:51.0860649Z 			"digest": "sha256:f2930b2ea93856dbf4c70bba9ed20b93ef8d6027f19f45c34bcfce84c00d59a9"
2023-08-17T10:47:51.0860975Z 		},
2023-08-17T10:47:51.0861159Z 		{
2023-08-17T10:47:51.0861480Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0861808Z 			"size": 122528526,
2023-08-17T10:47:51.0862144Z 			"digest": "sha256:c9b73d6a9c9ecb933046a007139d8dacbc0f5d485b79c1584b54884d59c9faa5"
2023-08-17T10:47:51.0862577Z 		},
2023-08-17T10:47:51.0862768Z 		{
2023-08-17T10:47:51.0863084Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0863393Z 			"size": 190,
2023-08-17T10:47:51.0863738Z 			"digest": "sha256:17e25cf9a6d618d3a0514ac318db118b0e7f7ed12f1c1f4195d9f52b0c5660ab"
2023-08-17T10:47:51.0864056Z 		},
2023-08-17T10:47:51.0864241Z 		{
2023-08-17T10:47:51.0864547Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0864870Z 			"size": 1143,
2023-08-17T10:47:51.0865192Z 			"digest": "sha256:134de79388675f22660fb737d164850a4d230b39360e64f199ba84d843f39e5f"
2023-08-17T10:47:51.0865483Z 		},
2023-08-17T10:47:51.0865765Z 		{
2023-08-17T10:47:51.0866084Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0866451Z 			"size": 700,
2023-08-17T10:47:51.0866808Z 			"digest": "sha256:54c5d1abcbe9b53a7155eaa899be6e0401bf3a16e9e9e96cd1615d51a786184e"
2023-08-17T10:47:51.0867121Z 		},
2023-08-17T10:47:51.0867308Z 		{
2023-08-17T10:47:51.0867629Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0868005Z 			"size": 137,
2023-08-17T10:47:51.0868332Z 			"digest": "sha256:3fc280415fc71a2feec411a9f5e2b496c08c30307a299f7781a057f6527d90ec"
2023-08-17T10:47:51.0868642Z 		},
2023-08-17T10:47:51.0868830Z 		{
2023-08-17T10:47:51.0869150Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0869457Z 			"size": 120,
2023-08-17T10:47:51.0869805Z 			"digest": "sha256:254d0c04a93fd6e18a232bb9b5113b0eb383bf51a62d3e7bc6b2c13d80edc229"
2023-08-17T10:47:51.0870123Z 		},
2023-08-17T10:47:51.0870308Z 		{
2023-08-17T10:47:51.0870612Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0870943Z 			"size": 985648089,
2023-08-17T10:47:51.0871277Z 			"digest": "sha256:8545a2099a4d990fe4498c45453336c802e0bd8ab4ed9e56bab9cc54d63f5fac"
2023-08-17T10:47:51.0871580Z 		},
2023-08-17T10:47:51.0871751Z 		{
2023-08-17T10:47:51.0872078Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0872405Z 			"size": 172,
2023-08-17T10:47:51.0872752Z 			"digest": "sha256:9cd38a38471dea83a05b5549c3fb57c61df2cd28da20c4fa1ba30b7155a0b1d9"
2023-08-17T10:47:51.0873059Z 		},
2023-08-17T10:47:51.0873250Z 		{
2023-08-17T10:47:51.0873573Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0873882Z 			"size": 1840,
2023-08-17T10:47:51.0874216Z 			"digest": "sha256:830bf1ac610414a73ada3f86d2f8c376495bf29720838ad4e7b153339175e3c4"
2023-08-17T10:47:51.0874523Z 		},
2023-08-17T10:47:51.0874708Z 		{
2023-08-17T10:47:51.0875014Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0875345Z 			"size": 7529787,
2023-08-17T10:47:51.0875712Z 			"digest": "sha256:6aa1df30c25e6b1be15bbbf4c0c4f944f48e24bf3dda4e8874a986d709dd2c97"
2023-08-17T10:47:51.0876047Z 		},
2023-08-17T10:47:51.0876218Z 		{
2023-08-17T10:47:51.0876540Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0876865Z 			"size": 163,
2023-08-17T10:47:51.0877197Z 			"digest": "sha256:35225692fc3856b1980eeafc534cb5265b2c8d15dcec109ea352267ffffc00fd"
2023-08-17T10:47:51.0877495Z 		},
2023-08-17T10:47:51.0877682Z 		{
2023-08-17T10:47:51.0878042Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0878365Z 			"size": 7940,
2023-08-17T10:47:51.0878698Z 			"digest": "sha256:77fa5459e4ac1cd5787e9eed9fb0875b0c864c9577375a85c2be115bef27876d"
2023-08-17T10:47:51.0879020Z 		},
2023-08-17T10:47:51.0879204Z 		{
2023-08-17T10:47:51.0879521Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0879836Z 			"size": 8073,
2023-08-17T10:47:51.0880185Z 			"digest": "sha256:60db69b2927fbe2e7c6c8d673594ccb1aa609bfa32816cddb3df4607b89dbd8f"
2023-08-17T10:47:51.0880512Z 		},
2023-08-17T10:47:51.0880695Z 		{
2023-08-17T10:47:51.0881000Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0881434Z 			"size": 304,
2023-08-17T10:47:51.0881798Z 			"digest": "sha256:a2d8e2bb3f9f8f2a9af4971b061aabc1a3b0389ac97a38303faccf532dc23f4b"
2023-08-17T10:47:51.0882131Z 		},
2023-08-17T10:47:51.0882303Z 		{
2023-08-17T10:47:51.0882856Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0883289Z 			"size": 7613605,
2023-08-17T10:47:51.0883650Z 			"digest": "sha256:884ca07df2e289b4ef49c9e4e3de4cafae6d79f102f3c63b00ac4950507f7bce"
2023-08-17T10:47:51.0883965Z 		},
2023-08-17T10:47:51.0884148Z 		{
2023-08-17T10:47:51.0884468Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0884792Z 			"size": 108,
2023-08-17T10:47:51.0885263Z 			"digest": "sha256:31b7988c1f6d67d025acd868ba944d832477bf4c5c882b7cded36795dfdcdd68"
2023-08-17T10:47:51.0885578Z 		},
2023-08-17T10:47:51.0885764Z 		{
2023-08-17T10:47:51.0886069Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0886396Z 			"size": 54145459,
2023-08-17T10:47:51.0886756Z 			"digest": "sha256:64531eb2abcc8a9f9162ea012a4daa2c1fc53d353a49f5a1ab43db1863461697"
2023-08-17T10:47:51.0887084Z 		},
2023-08-17T10:47:51.0887253Z 		{
2023-08-17T10:47:51.0887576Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0887950Z 			"size": 513,
2023-08-17T10:47:51.0888275Z 			"digest": "sha256:70638457ffb83f84754a17019ca4c43e751e5b12abc4a8c6a334b2c3318286b6"
2023-08-17T10:47:51.0888563Z 		},
2023-08-17T10:47:51.0888747Z 		{
2023-08-17T10:47:51.0889063Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0889389Z 			"size": 2993737014,
2023-08-17T10:47:51.0889730Z 			"digest": "sha256:d886d90e960c20293665b21eceed6acdeb16ff758863f8f33d6cccaa3fc35810"
2023-08-17T10:47:51.0890056Z 		},
2023-08-17T10:47:51.0890246Z 		{
2023-08-17T10:47:51.0890568Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2023-08-17T10:47:51.0890879Z 			"size": 106,
2023-08-17T10:47:51.0891223Z 			"digest": "sha256:e5e1d5ac0b31d513fbb474f9a08408b91823a41d47fd118182525a3d42420f08"
2023-08-17T10:47:51.0891541Z 		}
2023-08-17T10:47:51.0891725Z 	]
2023-08-17T10:47:51.0891894Z }
2023-08-17T10:47:51.1012762Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2023-08-17T10:47:51.1013094Z with:
2023-08-17T10:47:51.1013643Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:f89d62c8626a24aae4b1a64d42a3caf9cd95bddd
2023-08-17T10:47:51.1014195Z env:
2023-08-17T10:47:51.1014417Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:47:51.1014656Z ##[endgroup]
2023-08-17T10:47:51.1029105Z ##[group]Run retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2023-08-17T10:47:51.1029473Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2023-08-17T10:47:51.1029820Z # ignore output since only exit code is used for conditional
2023-08-17T10:47:51.1030193Z # only pull docker image if it's not available locally
2023-08-17T10:47:51.1030580Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2023-08-17T10:47:51.1030985Z   retry docker pull "${DOCKER_IMAGE}"
2023-08-17T10:47:51.1031242Z fi
2023-08-17T10:47:51.1049483Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-08-17T10:47:51.1049766Z env:
2023-08-17T10:47:51.1049989Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:47:51.1050582Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:f89d62c8626a24aae4b1a64d42a3caf9cd95bddd
2023-08-17T10:47:51.1051136Z ##[endgroup]
2023-08-17T10:47:51.6679282Z f89d62c8626a24aae4b1a64d42a3caf9cd95bddd: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks
2023-08-17T10:47:51.6680983Z f0412dfb1aae: Pulling fs layer
2023-08-17T10:47:51.6681441Z b0f696c0aebb: Pulling fs layer
2023-08-17T10:47:51.6681871Z e627444df06f: Pulling fs layer
2023-08-17T10:47:51.6682282Z dcf21018e934: Pulling fs layer
2023-08-17T10:47:51.6682858Z a2855a2ef2e0: Pulling fs layer
2023-08-17T10:47:51.6683441Z 8708e3e52fea: Pulling fs layer
2023-08-17T10:47:51.6683920Z be2aea6b86ef: Pulling fs layer
2023-08-17T10:47:51.6684246Z 71ab173e0cbf: Pulling fs layer
2023-08-17T10:47:51.6684550Z 308bdac48e2c: Pulling fs layer
2023-08-17T10:47:51.6685001Z 25a33b062462: Pulling fs layer
2023-08-17T10:47:51.6685378Z 7788c362a0e0: Pulling fs layer
2023-08-17T10:47:51.6685764Z 606cc67f5550: Pulling fs layer
2023-08-17T10:47:51.6686155Z c1e3d3c5b532: Pulling fs layer
2023-08-17T10:47:51.6686524Z dcf21018e934: Waiting
2023-08-17T10:47:51.6686890Z 42dfcd7df9db: Pulling fs layer
2023-08-17T10:47:51.6687890Z 33a9133686b5: Pulling fs layer
2023-08-17T10:47:51.6688326Z 2129cd14b507: Pulling fs layer
2023-08-17T10:47:51.6688770Z a2855a2ef2e0: Waiting
2023-08-17T10:47:51.6689217Z 680ec3688d92: Pulling fs layer
2023-08-17T10:47:51.6689639Z 8708e3e52fea: Waiting
2023-08-17T10:47:51.6690074Z 07d72ce662ac: Pulling fs layer
2023-08-17T10:47:51.6690468Z 9383e2c9b020: Pulling fs layer
2023-08-17T10:47:51.6690711Z 54c5d1abcbe9: Pulling fs layer
2023-08-17T10:47:51.6690970Z be2aea6b86ef: Waiting
2023-08-17T10:47:51.6691213Z 7660267bb941: Pulling fs layer
2023-08-17T10:47:51.6691447Z 71ab173e0cbf: Waiting
2023-08-17T10:47:51.6691673Z 970f7a60e318: Pulling fs layer
2023-08-17T10:47:51.6691922Z 976b5e991b6d: Pulling fs layer
2023-08-17T10:47:51.6692230Z 2d93807632f4: Pulling fs layer
2023-08-17T10:47:51.6692473Z 308bdac48e2c: Waiting
2023-08-17T10:47:51.6692720Z a08f76bef6ab: Pulling fs layer
2023-08-17T10:47:51.6693028Z 6dda747386b5: Pulling fs layer
2023-08-17T10:47:51.6693285Z 427ceb14dadb: Pulling fs layer
2023-08-17T10:47:51.6693531Z 25a33b062462: Waiting
2023-08-17T10:47:51.6693766Z 193f08489834: Pulling fs layer
2023-08-17T10:47:51.6693983Z 2129cd14b507: Waiting
2023-08-17T10:47:51.6694219Z 9d445bb0a609: Pulling fs layer
2023-08-17T10:47:51.6694457Z 7788c362a0e0: Waiting
2023-08-17T10:47:51.6694683Z 680ec3688d92: Waiting
2023-08-17T10:47:51.6694888Z 9383e2c9b020: Waiting
2023-08-17T10:47:51.6695105Z 606cc67f5550: Waiting
2023-08-17T10:47:51.6695582Z a91ec5f11529: Pulling fs layer
2023-08-17T10:47:51.6695845Z 5c77da10efef: Pulling fs layer
2023-08-17T10:47:51.6696090Z 9de12a84efdb: Pulling fs layer
2023-08-17T10:47:51.6696337Z c1e3d3c5b532: Waiting
2023-08-17T10:47:51.6696583Z 5ef5bbb9550d: Pulling fs layer
2023-08-17T10:47:51.6696811Z 07d72ce662ac: Waiting
2023-08-17T10:47:51.6697033Z 7660267bb941: Waiting
2023-08-17T10:47:51.6697266Z c8050d8f6276: Pulling fs layer
2023-08-17T10:47:51.6697518Z 4b3155d4eae9: Pulling fs layer
2023-08-17T10:47:51.6697748Z a08f76bef6ab: Waiting
2023-08-17T10:47:51.6697988Z 54c5d1abcbe9: Waiting
2023-08-17T10:47:51.6698230Z 29fd9070c65a: Pulling fs layer
2023-08-17T10:47:51.6698468Z 9d445bb0a609: Waiting
2023-08-17T10:47:51.6698692Z 9dc12ea1a262: Pulling fs layer
2023-08-17T10:47:51.6698930Z 6dda747386b5: Waiting
2023-08-17T10:47:51.6699152Z a91ec5f11529: Waiting
2023-08-17T10:47:51.6699379Z 42dfcd7df9db: Waiting
2023-08-17T10:47:51.6699603Z a0877f65810f: Pulling fs layer
2023-08-17T10:47:51.6699855Z 42021fac2a0c: Pulling fs layer
2023-08-17T10:47:51.6700103Z 9de12a84efdb: Waiting
2023-08-17T10:47:51.6700318Z 5c77da10efef: Waiting
2023-08-17T10:47:51.6700543Z 427ceb14dadb: Waiting
2023-08-17T10:47:51.6700771Z 5ef5bbb9550d: Waiting
2023-08-17T10:47:51.6701005Z 9304db884963: Pulling fs layer
2023-08-17T10:47:51.6701246Z e8fa116b74bb: Pulling fs layer
2023-08-17T10:47:51.6701484Z 9dc12ea1a262: Waiting
2023-08-17T10:47:51.6701702Z 193f08489834: Waiting
2023-08-17T10:47:51.6701935Z a0877f65810f: Waiting
2023-08-17T10:47:51.6702155Z 94c4dbc1ec22: Pulling fs layer
2023-08-17T10:47:51.6702414Z 57d789d9281f: Pulling fs layer
2023-08-17T10:47:51.6702647Z 9304db884963: Waiting
2023-08-17T10:47:51.6702866Z 94c4dbc1ec22: Waiting
2023-08-17T10:47:51.6703142Z 886477fddcc1: Pulling fs layer
2023-08-17T10:47:51.6703382Z 4b3155d4eae9: Waiting
2023-08-17T10:47:51.6703621Z f2930b2ea938: Pulling fs layer
2023-08-17T10:47:51.6703864Z 29fd9070c65a: Waiting
2023-08-17T10:47:51.6704092Z c9b73d6a9c9e: Pulling fs layer
2023-08-17T10:47:51.6704331Z 886477fddcc1: Waiting
2023-08-17T10:47:51.6704575Z 17e25cf9a6d6: Pulling fs layer
2023-08-17T10:47:51.6704799Z f2930b2ea938: Waiting
2023-08-17T10:47:51.6705035Z 134de7938867: Pulling fs layer
2023-08-17T10:47:51.6705267Z c8050d8f6276: Waiting
2023-08-17T10:47:51.6705486Z 970f7a60e318: Waiting
2023-08-17T10:47:51.6705685Z 134de7938867: Waiting
2023-08-17T10:47:51.6705918Z 3fc280415fc7: Pulling fs layer
2023-08-17T10:47:51.6706151Z 42021fac2a0c: Waiting
2023-08-17T10:47:51.6706372Z e8fa116b74bb: Waiting
2023-08-17T10:47:51.6706594Z 254d0c04a93f: Pulling fs layer
2023-08-17T10:47:51.6706977Z 8545a2099a4d: Pulling fs layer
2023-08-17T10:47:51.6707233Z 9cd38a38471d: Pulling fs layer
2023-08-17T10:47:51.6707487Z 830bf1ac6104: Pulling fs layer
2023-08-17T10:47:51.6707710Z 3fc280415fc7: Waiting
2023-08-17T10:47:51.6708128Z 9cd38a38471d: Waiting
2023-08-17T10:47:51.6708355Z c9b73d6a9c9e: Waiting
2023-08-17T10:47:51.6708580Z 8545a2099a4d: Waiting
2023-08-17T10:47:51.6708803Z 6aa1df30c25e: Pulling fs layer
2023-08-17T10:47:51.6709069Z 35225692fc38: Pulling fs layer
2023-08-17T10:47:51.6709320Z 77fa5459e4ac: Pulling fs layer
2023-08-17T10:47:51.6709558Z 60db69b2927f: Pulling fs layer
2023-08-17T10:47:51.6709794Z 830bf1ac6104: Waiting
2023-08-17T10:47:51.6710012Z 60db69b2927f: Waiting
2023-08-17T10:47:51.6710253Z a2d8e2bb3f9f: Pulling fs layer
2023-08-17T10:47:51.6710474Z 35225692fc38: Waiting
2023-08-17T10:47:51.6710708Z 884ca07df2e2: Pulling fs layer
2023-08-17T10:47:51.6710957Z 31b7988c1f6d: Pulling fs layer
2023-08-17T10:47:51.6711195Z 77fa5459e4ac: Waiting
2023-08-17T10:47:51.6711425Z 64531eb2abcc: Pulling fs layer
2023-08-17T10:47:51.6711663Z 884ca07df2e2: Waiting
2023-08-17T10:47:51.6711898Z 31b7988c1f6d: Waiting
2023-08-17T10:47:51.6712129Z 70638457ffb8: Pulling fs layer
2023-08-17T10:47:51.6712366Z d886d90e960c: Pulling fs layer
2023-08-17T10:47:51.6712601Z 70638457ffb8: Waiting
2023-08-17T10:47:51.6712824Z 64531eb2abcc: Waiting
2023-08-17T10:47:51.6713101Z e5e1d5ac0b31: Pulling fs layer
2023-08-17T10:47:51.6713324Z d886d90e960c: Waiting
2023-08-17T10:47:51.6713679Z e5e1d5ac0b31: Waiting
2023-08-17T10:47:52.4124361Z b0f696c0aebb: Verifying Checksum
2023-08-17T10:47:52.4124709Z b0f696c0aebb: Download complete
2023-08-17T10:47:52.5111275Z f0412dfb1aae: Download complete
2023-08-17T10:47:52.6909761Z dcf21018e934: Verifying Checksum
2023-08-17T10:47:52.6910359Z dcf21018e934: Download complete
2023-08-17T10:47:52.8037413Z a2855a2ef2e0: Verifying Checksum
2023-08-17T10:47:52.8038005Z a2855a2ef2e0: Download complete
2023-08-17T10:47:52.9097793Z e627444df06f: Verifying Checksum
2023-08-17T10:47:52.9098386Z e627444df06f: Download complete
2023-08-17T10:47:53.1565967Z be2aea6b86ef: Download complete
2023-08-17T10:47:53.2022000Z f0412dfb1aae: Pull complete
2023-08-17T10:47:53.2022583Z 71ab173e0cbf: Verifying Checksum
2023-08-17T10:47:53.2022905Z 71ab173e0cbf: Download complete
2023-08-17T10:47:53.4286719Z b0f696c0aebb: Pull complete
2023-08-17T10:47:53.4392056Z 308bdac48e2c: Verifying Checksum
2023-08-17T10:47:53.4392395Z 308bdac48e2c: Download complete
2023-08-17T10:47:53.7745758Z 7788c362a0e0: Verifying Checksum
2023-08-17T10:47:53.7746308Z 7788c362a0e0: Download complete
2023-08-17T10:47:54.0573661Z 606cc67f5550: Verifying Checksum
2023-08-17T10:47:54.0574244Z 606cc67f5550: Download complete
2023-08-17T10:47:54.4499345Z e627444df06f: Pull complete
2023-08-17T10:47:54.5038649Z dcf21018e934: Pull complete
2023-08-17T10:47:54.5573577Z a2855a2ef2e0: Pull complete
2023-08-17T10:47:57.2894478Z c1e3d3c5b532: Verifying Checksum
2023-08-17T10:47:57.2894896Z c1e3d3c5b532: Download complete
2023-08-17T10:47:57.6074542Z 42dfcd7df9db: Verifying Checksum
2023-08-17T10:47:57.6075253Z 42dfcd7df9db: Download complete
2023-08-17T10:47:57.9041161Z 33a9133686b5: Verifying Checksum
2023-08-17T10:47:57.9041697Z 33a9133686b5: Download complete
2023-08-17T10:47:58.2017524Z 2129cd14b507: Download complete
2023-08-17T10:47:59.8415939Z 680ec3688d92: Verifying Checksum
2023-08-17T10:47:59.8416275Z 680ec3688d92: Download complete
2023-08-17T10:48:00.1260358Z 07d72ce662ac: Verifying Checksum
2023-08-17T10:48:00.1260952Z 07d72ce662ac: Download complete
2023-08-17T10:48:00.4148201Z 9383e2c9b020: Verifying Checksum
2023-08-17T10:48:00.4148524Z 9383e2c9b020: Download complete
2023-08-17T10:48:00.7063278Z 54c5d1abcbe9: Download complete
2023-08-17T10:48:12.0820430Z 8708e3e52fea: Verifying Checksum
2023-08-17T10:48:12.0821038Z 8708e3e52fea: Download complete
2023-08-17T10:48:12.3804364Z 970f7a60e318: Download complete
2023-08-17T10:48:12.6889495Z 976b5e991b6d: Verifying Checksum
2023-08-17T10:48:12.6889837Z 976b5e991b6d: Download complete
2023-08-17T10:48:12.9701358Z 2d93807632f4: Verifying Checksum
2023-08-17T10:48:12.9701737Z 2d93807632f4: Download complete
2023-08-17T10:48:13.2489941Z a08f76bef6ab: Verifying Checksum
2023-08-17T10:48:13.2490431Z a08f76bef6ab: Download complete
2023-08-17T10:48:13.5461023Z 6dda747386b5: Download complete
2023-08-17T10:48:15.3113278Z 427ceb14dadb: Verifying Checksum
2023-08-17T10:48:15.3113622Z 427ceb14dadb: Download complete
2023-08-17T10:48:15.6145606Z 193f08489834: Verifying Checksum
2023-08-17T10:48:15.6145929Z 193f08489834: Download complete
2023-08-17T10:48:15.9100267Z 9d445bb0a609: Verifying Checksum
2023-08-17T10:48:15.9100780Z 9d445bb0a609: Download complete
2023-08-17T10:48:16.3916186Z a91ec5f11529: Verifying Checksum
2023-08-17T10:48:16.3916737Z a91ec5f11529: Download complete
2023-08-17T10:48:16.7021198Z 5c77da10efef: Verifying Checksum
2023-08-17T10:48:16.7021770Z 5c77da10efef: Download complete
2023-08-17T10:48:16.9847460Z 9de12a84efdb: Verifying Checksum
2023-08-17T10:48:16.9847895Z 9de12a84efdb: Download complete
2023-08-17T10:48:21.8076836Z 5ef5bbb9550d: Verifying Checksum
2023-08-17T10:48:21.8077661Z 5ef5bbb9550d: Download complete
2023-08-17T10:48:22.1448526Z c8050d8f6276: Verifying Checksum
2023-08-17T10:48:22.1448900Z c8050d8f6276: Download complete
2023-08-17T10:48:22.4565550Z 4b3155d4eae9: Verifying Checksum
2023-08-17T10:48:22.4566099Z 4b3155d4eae9: Download complete
2023-08-17T10:48:23.7989845Z 29fd9070c65a: Verifying Checksum
2023-08-17T10:48:23.7991192Z 29fd9070c65a: Download complete
2023-08-17T10:48:24.1176084Z 9dc12ea1a262: Verifying Checksum
2023-08-17T10:48:24.1176681Z 9dc12ea1a262: Download complete
2023-08-17T10:48:24.4334363Z a0877f65810f: Verifying Checksum
2023-08-17T10:48:24.4334930Z a0877f65810f: Download complete
2023-08-17T10:48:25.1382497Z 42021fac2a0c: Verifying Checksum
2023-08-17T10:48:25.1382999Z 42021fac2a0c: Download complete
2023-08-17T10:48:25.4270932Z 9304db884963: Verifying Checksum
2023-08-17T10:48:25.4271515Z 9304db884963: Download complete
2023-08-17T10:48:25.4758856Z 7660267bb941: Verifying Checksum
2023-08-17T10:48:25.4759403Z 7660267bb941: Download complete
2023-08-17T10:48:25.7790788Z 94c4dbc1ec22: Verifying Checksum
2023-08-17T10:48:25.7791239Z 94c4dbc1ec22: Download complete
2023-08-17T10:48:26.0772637Z 57d789d9281f: Verifying Checksum
2023-08-17T10:48:26.0772975Z 57d789d9281f: Download complete
2023-08-17T10:48:26.3775414Z 886477fddcc1: Verifying Checksum
2023-08-17T10:48:26.3775781Z 886477fddcc1: Download complete
2023-08-17T10:48:26.4983918Z e8fa116b74bb: Verifying Checksum
2023-08-17T10:48:26.4984489Z e8fa116b74bb: Download complete
2023-08-17T10:48:26.6724997Z f2930b2ea938: Verifying Checksum
2023-08-17T10:48:26.6725640Z f2930b2ea938: Download complete
2023-08-17T10:48:26.9630953Z 17e25cf9a6d6: Download complete
2023-08-17T10:48:27.2552946Z 134de7938867: Verifying Checksum
2023-08-17T10:48:27.2553873Z 134de7938867: Download complete
2023-08-17T10:48:27.5648614Z 3fc280415fc7: Download complete
2023-08-17T10:48:27.6567524Z 8708e3e52fea: Pull complete
2023-08-17T10:48:27.8725675Z 254d0c04a93f: Verifying Checksum
2023-08-17T10:48:27.8726184Z 254d0c04a93f: Download complete
2023-08-17T10:48:28.1560295Z be2aea6b86ef: Pull complete
2023-08-17T10:48:28.6458535Z 71ab173e0cbf: Pull complete
2023-08-17T10:48:29.0568606Z 308bdac48e2c: Pull complete
2023-08-17T10:48:29.2508614Z c9b73d6a9c9e: Verifying Checksum
2023-08-17T10:48:29.2508966Z c9b73d6a9c9e: Download complete
2023-08-17T10:48:29.5783322Z 9cd38a38471d: Verifying Checksum
2023-08-17T10:48:29.5783685Z 9cd38a38471d: Download complete
2023-08-17T10:48:29.8813115Z 830bf1ac6104: Download complete
2023-08-17T10:48:30.4383468Z 6aa1df30c25e: Verifying Checksum
2023-08-17T10:48:30.4384033Z 6aa1df30c25e: Download complete
2023-08-17T10:48:30.7819952Z 35225692fc38: Verifying Checksum
2023-08-17T10:48:30.8707717Z 35225692fc38: Download complete
2023-08-17T10:48:30.8708299Z 25a33b062462: Verifying Checksum
2023-08-17T10:48:30.8708707Z 25a33b062462: Download complete
2023-08-17T10:48:31.0682010Z 77fa5459e4ac: Verifying Checksum
2023-08-17T10:48:31.0683085Z 77fa5459e4ac: Download complete
2023-08-17T10:48:31.1776678Z 60db69b2927f: Verifying Checksum
2023-08-17T10:48:31.1777181Z 60db69b2927f: Download complete
2023-08-17T10:48:31.3560822Z a2d8e2bb3f9f: Verifying Checksum
2023-08-17T10:48:31.3561486Z a2d8e2bb3f9f: Download complete
2023-08-17T10:48:31.7236872Z 884ca07df2e2: Verifying Checksum
2023-08-17T10:48:31.7237308Z 884ca07df2e2: Download complete
2023-08-17T10:48:31.9945989Z 70638457ffb8: Verifying Checksum
2023-08-17T10:48:31.9946652Z 70638457ffb8: Download complete
2023-08-17T10:48:32.7966968Z 64531eb2abcc: Verifying Checksum
2023-08-17T10:48:32.7969596Z 64531eb2abcc: Download complete
2023-08-17T10:48:33.0819917Z e5e1d5ac0b31: Verifying Checksum
2023-08-17T10:48:33.0820745Z e5e1d5ac0b31: Download complete
2023-08-17T10:48:40.9611011Z 8545a2099a4d: Verifying Checksum
2023-08-17T10:48:40.9611490Z 8545a2099a4d: Download complete
2023-08-17T10:49:03.7660314Z d886d90e960c: Verifying Checksum
2023-08-17T10:49:03.9798941Z d886d90e960c: Download complete
2023-08-17T10:49:03.9799413Z 25a33b062462: Pull complete
2023-08-17T10:49:04.4683126Z 7788c362a0e0: Pull complete
2023-08-17T10:49:04.9367011Z 606cc67f5550: Pull complete
2023-08-17T10:49:11.0285373Z c1e3d3c5b532: Pull complete
2023-08-17T10:49:11.4056064Z 42dfcd7df9db: Pull complete
2023-08-17T10:49:11.8341996Z 33a9133686b5: Pull complete
2023-08-17T10:49:12.1669787Z 2129cd14b507: Pull complete
2023-08-17T10:49:14.2391244Z 680ec3688d92: Pull complete
2023-08-17T10:49:14.2882938Z 07d72ce662ac: Pull complete
2023-08-17T10:49:14.3365282Z 9383e2c9b020: Pull complete
2023-08-17T10:49:14.3841315Z 54c5d1abcbe9: Pull complete
2023-08-17T10:49:50.4308068Z 7660267bb941: Pull complete
2023-08-17T10:49:50.4832865Z 970f7a60e318: Pull complete
2023-08-17T10:49:50.5382929Z 976b5e991b6d: Pull complete
2023-08-17T10:49:50.5881287Z 2d93807632f4: Pull complete
2023-08-17T10:49:50.6363472Z a08f76bef6ab: Pull complete
2023-08-17T10:49:50.6989955Z 6dda747386b5: Pull complete
2023-08-17T10:49:54.5933412Z 427ceb14dadb: Pull complete
2023-08-17T10:49:54.9031665Z 193f08489834: Pull complete
2023-08-17T10:49:55.3824158Z 9d445bb0a609: Pull complete
2023-08-17T10:49:55.9276259Z a91ec5f11529: Pull complete
2023-08-17T10:49:56.3653309Z 5c77da10efef: Pull complete
2023-08-17T10:49:56.5962437Z 9de12a84efdb: Pull complete
2023-08-17T10:50:03.3345684Z 5ef5bbb9550d: Pull complete
2023-08-17T10:50:03.5327817Z c8050d8f6276: Pull complete
2023-08-17T10:50:03.7251158Z 4b3155d4eae9: Pull complete
2023-08-17T10:50:05.2560106Z 29fd9070c65a: Pull complete
2023-08-17T10:50:05.3082020Z 9dc12ea1a262: Pull complete
2023-08-17T10:50:05.3588172Z a0877f65810f: Pull complete
2023-08-17T10:50:05.6912358Z 42021fac2a0c: Pull complete
2023-08-17T10:50:05.7422085Z 9304db884963: Pull complete
2023-08-17T10:50:06.9176036Z e8fa116b74bb: Pull complete
2023-08-17T10:50:06.9683767Z 94c4dbc1ec22: Pull complete
2023-08-17T10:50:07.0179846Z 57d789d9281f: Pull complete
2023-08-17T10:50:07.1177235Z 886477fddcc1: Pull complete
2023-08-17T10:50:07.1666374Z f2930b2ea938: Pull complete
2023-08-17T10:50:09.9665535Z c9b73d6a9c9e: Pull complete
2023-08-17T10:50:10.0865270Z 17e25cf9a6d6: Pull complete
2023-08-17T10:50:10.2362133Z 134de7938867: Pull complete
2023-08-17T10:50:10.4956620Z 3fc280415fc7: Pull complete
2023-08-17T10:50:10.7501437Z 254d0c04a93f: Pull complete
2023-08-17T10:50:30.1493822Z 8545a2099a4d: Pull complete
2023-08-17T10:50:30.2001563Z 9cd38a38471d: Pull complete
2023-08-17T10:50:30.2511994Z 830bf1ac6104: Pull complete
2023-08-17T10:50:30.4657963Z 6aa1df30c25e: Pull complete
2023-08-17T10:50:30.5159987Z 35225692fc38: Pull complete
2023-08-17T10:50:30.5653560Z 77fa5459e4ac: Pull complete
2023-08-17T10:50:30.6137339Z 60db69b2927f: Pull complete
2023-08-17T10:50:30.6638859Z a2d8e2bb3f9f: Pull complete
2023-08-17T10:50:31.3188466Z 884ca07df2e2: Pull complete
2023-08-17T10:50:31.6319263Z 31b7988c1f6d: Pull complete
2023-08-17T10:50:33.6005545Z 64531eb2abcc: Pull complete
2023-08-17T10:50:33.6521262Z 70638457ffb8: Pull complete
2023-08-17T10:51:08.5121964Z d886d90e960c: Pull complete
2023-08-17T10:51:08.6919199Z e5e1d5ac0b31: Pull complete
2023-08-17T10:51:08.8995368Z Digest: sha256:2fb6aa86ece66a8b83e06455dc98242ec47d8bc0bdb0edad8232d40ed23c8576
2023-08-17T10:51:08.9760744Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:f89d62c8626a24aae4b1a64d42a3caf9cd95bddd
2023-08-17T10:51:09.0319128Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:f89d62c8626a24aae4b1a64d42a3caf9cd95bddd
2023-08-17T10:51:09.0910917Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main
2023-08-17T10:51:09.0911273Z with:
2023-08-17T10:51:09.0911494Z   driver-version: 535.54.03
2023-08-17T10:51:09.0911727Z env:
2023-08-17T10:51:09.0911954Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:51:09.0912184Z ##[endgroup]
2023-08-17T10:51:09.1526240Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2023-08-17T10:51:09.1526559Z with:
2023-08-17T10:51:09.1526784Z   timeout_minutes: 10
2023-08-17T10:51:09.1527023Z   max_attempts: 3
2023-08-17T10:51:09.1537067Z   command: # Is it disgusting to have a full shell script here in this github action? Sure
# But is it the best way to make it so that this action relies on nothing else? Absolutely
set -eou pipefail

DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID)
DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run"
YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo"

install_nvidia_docker2_amzn2() {
    (
        set -x
        # Needed for yum-config-manager
        sudo yum install -y yum-utils
        sudo yum-config-manager --add-repo "${YUM_REPO_URL}"
        sudo yum install -y nvidia-docker2
        sudo systemctl restart docker
    )
}

install_nvidia_docker2_ubuntu20() {
    (
        set -x
        sudo apt-get install -y nvidia-docker2
        sudo systemctl restart docker
    )
}

pre_install_nvidia_driver_amzn2() {
    (
        # Purge any nvidia driver installed from RHEL repo
        sudo yum remove -y nvidia-driver-latest-dkms
    )
}

install_nvidia_driver_common() {
    (
        # Try to gather more information about the runner and its existing NVIDIA driver if any
        echo "Before installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        HAS_NVIDIA_DRIVER=0
        # Check if NVIDIA driver has already been installed
        if [ -x "$(command -v nvidia-smi)" ]; then
            set +e
            # The driver exists, check its version next. Also check only the first GPU if there are more than one of them
            # so that the same driver version is not print over multiple lines
            INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
            NVIDIA_SMI_STATUS=$?

            if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing"
            elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing"
            else
                HAS_NVIDIA_DRIVER=1
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation"
            fi
            set -e
        fi

        if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then
            # CAUTION: this may need to be updated in future
            if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then
                  sudo yum groupinstall -y "Development Tools"
                  # ensure our kernel install is the same as our underlying kernel,
                  # groupinstall "Development Tools" has a habit of mismatching kernel headers
                  sudo yum install -y "kernel-devel-uname-r == $(uname -r)"
                  sudo modprobe backlight
            fi
            sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN"

            set +e
            sudo /bin/bash /tmp/nvidia_driver -s --no-drm
            NVIDIA_INSTALLATION_STATUS=$?

            RESET_GPU=0
            if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then
                sudo cat /var/log/nvidia-installer.log
                # Fail to install NVIDIA driver, try to reset the GPU
                RESET_GPU=1
            elif [ -x "$(command -v nvidia-smi)" ]; then
                # Check again if nvidia-smi works even if the driver installation completes successfully
                INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
                NVIDIA_SMI_STATUS=$?

                if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                    RESET_GPU=1
                fi
            fi

            if [ "$RESET_GPU" -eq 1 ]; then
                NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1)
                # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this
                # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388
                for PCI_ID in $NVIDIA_DEVICES; do
                    DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable)

                    echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)"
                    # This requires sudo permission of course
                    echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset
                    sleep 1
                done
            fi

            sudo rm -fv /tmp/nvidia_driver
            set -e
        fi
    )
}

post_install_nvidia_driver_common() {
    (
        sudo modprobe nvidia || true
        echo "After installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        (
            set +e

            nvidia-smi
            # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in
            # the case where the driver has already crashed as it still can get the driver version
            # and some basic information like the bus ID.  However, the rest of the information
            # would be missing (ERR!), for example:
            #
            # +-----------------------------------------------------------------------------+
            # | NVIDIA-SMI 525.89.02    Driver Version: 525.89.02    CUDA Version: 12.0     |
            # |-------------------------------+----------------------+----------------------+
            # | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
            # | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
            # |                               |                      |               MIG M. |
            # |===============================+======================+======================|
            # |   0  ERR!                Off  | 00000000:00:1E.0 Off |                 ERR! |
            # |ERR!  ERR! ERR!    ERR! / ERR! |   4184MiB / 23028MiB |    ERR!      Default |
            # |                               |                      |                 ERR! |
            # +-------------------------------+----------------------+----------------------+
            #
            # +-----------------------------------------------------------------------------+
            # | Processes:                                                                  |
            # |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
            # |        ID   ID                                                   Usage      |
            # |=============================================================================|
            # +-----------------------------------------------------------------------------+
            #
            # This should be reported as a failure instead as it will guarantee to fail when
            # Docker tries to run with --gpus all
            #
            # So, the correct check here is to query one of the missing piece of info like
            # GPU name, so that the command can fail accordingly
            nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0
            NVIDIA_SMI_STATUS=$?

            # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285
            if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then
                echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}"
            else
                echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}"
                exit ${NVIDIA_SMI_STATUS}
            fi
            set -e
        )
    )
}

install_nvidia_driver_amzn2() {
    (
        set -x
        pre_install_nvidia_driver_amzn2
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

install_nvidia_driver_ubuntu20() {
    (
        set -x
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

echo "== Installing nvidia driver ${DRIVER_FN} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_driver_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_driver_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac

# Install container toolkit based on distribution
echo "== Installing nvidia container toolkit for ${DISTRIBUTION} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_docker2_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_docker2_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac
echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}"

2023-08-17T10:51:09.1547265Z   retry_wait_seconds: 10
2023-08-17T10:51:09.1547530Z   polling_interval_seconds: 1
2023-08-17T10:51:09.1547773Z   warning_on_retry: true
2023-08-17T10:51:09.1548036Z   continue_on_error: false
2023-08-17T10:51:09.1548267Z env:
2023-08-17T10:51:09.1548486Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:51:09.1548722Z   DRIVER_VERSION: 535.54.03
2023-08-17T10:51:09.1548960Z ##[endgroup]
2023-08-17T10:51:09.2499850Z == Installing nvidia driver NVIDIA-Linux-x86_64-535.54.03.run ==
2023-08-17T10:51:09.2500545Z + install_nvidia_driver_common
2023-08-17T10:51:09.2502241Z + echo 'Before installing NVIDIA driver'
2023-08-17T10:51:09.2505506Z Before installing NVIDIA driver
2023-08-17T10:51:09.2506303Z + lspci
2023-08-17T10:51:09.2617316Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
2023-08-17T10:51:09.2617964Z 00:01.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 03)
2023-08-17T10:51:09.2618540Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
2023-08-17T10:51:09.2619631Z 00:03.0 Non-VGA unclassified device: Red Hat, Inc. Virtio SCSI
2023-08-17T10:51:09.2621091Z 00:04.0 3D controller: NVIDIA Corporation Device 20b0 (rev a1)
2023-08-17T10:51:09.2621513Z 00:05.0 Ethernet controller: Red Hat, Inc. Virtio network device
2023-08-17T10:51:09.2621883Z 00:06.0 Unclassified device [00ff]: Red Hat, Inc. Virtio RNG
2023-08-17T10:51:09.2622163Z + lsmod
2023-08-17T10:51:09.2642793Z Module                  Size  Used by
2023-08-17T10:51:09.2643556Z btrfs                1536000  0
2023-08-17T10:51:09.2644013Z blake2b_generic        20480  0
2023-08-17T10:51:09.2644364Z xor                    24576  1 btrfs
2023-08-17T10:51:09.2644637Z zstd_compress         225280  1 btrfs
2023-08-17T10:51:09.2644912Z raid6_pq              122880  1 btrfs
2023-08-17T10:51:09.2645803Z ufs                   106496  0
2023-08-17T10:51:09.2646236Z msdos                  20480  0
2023-08-17T10:51:09.2646623Z xfs                  1753088  0
2023-08-17T10:51:09.2647217Z nvidia_modeset       1302528  0
2023-08-17T10:51:09.2647683Z veth                   32768  0
2023-08-17T10:51:09.2648111Z xt_conntrack           16384  1
2023-08-17T10:51:09.2648803Z xt_MASQUERADE          20480  1
2023-08-17T10:51:09.2649046Z xfrm_user              40960  1
2023-08-17T10:51:09.2649288Z xfrm_algo              16384  1 xfrm_user
2023-08-17T10:51:09.2649547Z xt_addrtype            16384  2
2023-08-17T10:51:09.2649793Z iptable_filter         16384  1
2023-08-17T10:51:09.2650038Z iptable_nat            16384  1
2023-08-17T10:51:09.2650310Z nf_nat                 49152  2 iptable_nat,xt_MASQUERADE
2023-08-17T10:51:09.2650580Z bpfilter               16384  0
2023-08-17T10:51:09.2650826Z br_netfilter           28672  0
2023-08-17T10:51:09.2651247Z bridge                307200  1 br_netfilter
2023-08-17T10:51:09.2651499Z stp                    16384  1 bridge
2023-08-17T10:51:09.2651755Z llc                    16384  2 bridge,stp
2023-08-17T10:51:09.2652012Z nvidia_uvm           1540096  0
2023-08-17T10:51:09.2652251Z aufs                  270336  0
2023-08-17T10:51:09.2652469Z overlay               151552  0
2023-08-17T10:51:09.2652702Z nls_iso8859_1          16384  1
2023-08-17T10:51:09.2652949Z dm_multipath           40960  0
2023-08-17T10:51:09.2653193Z scsi_dh_rdac           16384  0
2023-08-17T10:51:09.2653419Z scsi_dh_emc            16384  0
2023-08-17T10:51:09.2653660Z scsi_dh_alua           20480  0
2023-08-17T10:51:09.2653942Z nvidia              56455168  2 nvidia_uvm,nvidia_modeset
2023-08-17T10:51:09.2654217Z binfmt_misc            24576  1
2023-08-17T10:51:09.2654449Z crct10dif_pclmul       16384  1
2023-08-17T10:51:09.2654693Z crc32_pclmul           16384  0
2023-08-17T10:51:09.2654940Z ghash_clmulni_intel    16384  0
2023-08-17T10:51:09.2655199Z aesni_intel           376832  0
2023-08-17T10:51:09.2655422Z virtio_net             61440  0
2023-08-17T10:51:09.2655660Z psmouse               176128  0
2023-08-17T10:51:09.2655895Z input_leds             16384  0
2023-08-17T10:51:09.2656138Z crypto_simd            16384  1 aesni_intel
2023-08-17T10:51:09.2656412Z net_failover           20480  1 virtio_net
2023-08-17T10:51:09.2656720Z cryptd                 24576  2 crypto_simd,ghash_clmulni_intel
2023-08-17T10:51:09.2657099Z serio_raw              20480  0
2023-08-17T10:51:09.2657346Z failover               16384  1 net_failover
2023-08-17T10:51:09.2657607Z sch_fq_codel           24576  13
2023-08-17T10:51:09.2657855Z efi_pstore             16384  0
2023-08-17T10:51:09.2658100Z drm                   618496  1 nvidia
2023-08-17T10:51:09.2658384Z virtio_rng             16384  0
2023-08-17T10:51:09.2658678Z ip_tables              32768  2 iptable_filter,iptable_nat
2023-08-17T10:51:09.2659110Z x_tables               53248  6 xt_conntrack,iptable_filter,xt_addrtype,ip_tables,iptable_nat,xt_MASQUERADE
2023-08-17T10:51:09.2659471Z autofs4                49152  2
2023-08-17T10:51:09.2659720Z + modinfo nvidia
2023-08-17T10:51:09.2660552Z filename:       /lib/modules/5.15.0-1037-gcp/kernel/drivers/video/nvidia.ko
2023-08-17T10:51:09.2661106Z firmware:       nvidia/535.54.03/gsp_tu10x.bin
2023-08-17T10:51:09.2661641Z firmware:       nvidia/535.54.03/gsp_ga10x.bin
2023-08-17T10:51:09.2662290Z alias:          char-major-195-*
2023-08-17T10:51:09.2662742Z version:        535.54.03
2023-08-17T10:51:09.2663229Z supported:      external
2023-08-17T10:51:09.2663660Z license:        NVIDIA
2023-08-17T10:51:09.2664063Z srcversion:     415DC1001E7D8AD79B6207C
2023-08-17T10:51:09.2664351Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2023-08-17T10:51:09.2664650Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2023-08-17T10:51:09.2664950Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2023-08-17T10:51:09.2665212Z depends:        drm
2023-08-17T10:51:09.2665441Z retpoline:      Y
2023-08-17T10:51:09.2665667Z name:           nvidia
2023-08-17T10:51:09.2666049Z vermagic:       5.15.0-1037-gcp SMP mod_unload modversions 
2023-08-17T10:51:09.2666648Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2023-08-17T10:51:09.2667331Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2023-08-17T10:51:09.2667690Z parm:           NVreg_ResmanDebugLevel:int
2023-08-17T10:51:09.2668248Z parm:           NVreg_RmLogonRC:int
2023-08-17T10:51:09.2668540Z parm:           NVreg_ModifyDeviceFiles:int
2023-08-17T10:51:09.2668909Z parm:           NVreg_DeviceFileUID:int
2023-08-17T10:51:09.2669446Z parm:           NVreg_DeviceFileGID:int
2023-08-17T10:51:09.2669995Z parm:           NVreg_DeviceFileMode:int
2023-08-17T10:51:09.2670564Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2023-08-17T10:51:09.2670917Z parm:           NVreg_UsePageAttributeTable:int
2023-08-17T10:51:09.2671225Z parm:           NVreg_EnablePCIeGen3:int
2023-08-17T10:51:09.2671651Z parm:           NVreg_EnableMSI:int
2023-08-17T10:51:09.2671937Z parm:           NVreg_TCEBypassMode:int
2023-08-17T10:51:09.2672229Z parm:           NVreg_EnableStreamMemOPs:int
2023-08-17T10:51:09.2672829Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2023-08-17T10:51:09.2673508Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2023-08-17T10:51:09.2673880Z parm:           NVreg_EnableS0ixPowerManagement:int
2023-08-17T10:51:09.2674280Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2023-08-17T10:51:09.2674722Z parm:           NVreg_DynamicPowerManagement:int
2023-08-17T10:51:09.2675135Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2023-08-17T10:51:09.2675530Z parm:           NVreg_EnableGpuFirmware:int
2023-08-17T10:51:09.2675841Z parm:           NVreg_EnableGpuFirmwareLogs:int
2023-08-17T10:51:09.2676198Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2023-08-17T10:51:09.2676559Z parm:           NVreg_EnableUserNUMAManagement:int
2023-08-17T10:51:09.2676888Z parm:           NVreg_MemoryPoolSize:int
2023-08-17T10:51:09.2677184Z parm:           NVreg_KMallocHeapMaxSize:int
2023-08-17T10:51:09.2677500Z parm:           NVreg_VMallocHeapMaxSize:int
2023-08-17T10:51:09.2677801Z parm:           NVreg_IgnoreMMIOCheck:int
2023-08-17T10:51:09.2678091Z parm:           NVreg_NvLinkDisable:int
2023-08-17T10:51:09.2678458Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2023-08-17T10:51:09.2678810Z parm:           NVreg_RegisterPCIDriver:int
2023-08-17T10:51:09.2679120Z parm:           NVreg_EnableResizableBar:int
2023-08-17T10:51:09.2679439Z parm:           NVreg_EnableDbgBreakpoint:int
2023-08-17T10:51:09.2679735Z parm:           NVreg_RegistryDwords:charp
2023-08-17T10:51:09.2680063Z parm:           NVreg_RegistryDwordsPerDevice:charp
2023-08-17T10:51:09.2680371Z parm:           NVreg_RmMsg:charp
2023-08-17T10:51:09.2680647Z parm:           NVreg_GpuBlacklist:charp
2023-08-17T10:51:09.2680939Z parm:           NVreg_TemporaryFilePath:charp
2023-08-17T10:51:09.2681244Z parm:           NVreg_ExcludedGpus:charp
2023-08-17T10:51:09.2681539Z parm:           NVreg_DmaRemapPeerMmio:int
2023-08-17T10:51:09.2681852Z parm:           NVreg_RmNvlinkBandwidth:charp
2023-08-17T10:51:09.2682133Z parm:           rm_firmware_active:charp
2023-08-17T10:51:09.2682393Z + HAS_NVIDIA_DRIVER=0
2023-08-17T10:51:09.2682967Z ++ command -v nvidia-smi
2023-08-17T10:51:09.2683300Z + '[' -x /usr/bin/nvidia-smi ']'
2023-08-17T10:51:09.2683564Z + set +e
2023-08-17T10:51:09.2683971Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0
2023-08-17T10:51:09.2881358Z + INSTALLED_DRIVER_VERSION=535.54.03
2023-08-17T10:51:09.2881785Z + NVIDIA_SMI_STATUS=0
2023-08-17T10:51:09.2882287Z + '[' 0 -ne 0 ']'
2023-08-17T10:51:09.2882868Z + '[' 535.54.03 '!=' 535.54.03 ']'
2023-08-17T10:51:09.2883248Z + HAS_NVIDIA_DRIVER=1
2023-08-17T10:51:09.2884156Z + echo 'NVIDIA driver (535.54.03) has already been installed. Skipping NVIDIA driver installation'
2023-08-17T10:51:09.2885593Z + set -e
2023-08-17T10:51:09.2886085Z + '[' 1 -eq 0 ']'
2023-08-17T10:51:09.2886693Z NVIDIA driver (535.54.03) has already been installed. Skipping NVIDIA driver installation
2023-08-17T10:51:09.2887058Z + post_install_nvidia_driver_common
2023-08-17T10:51:09.2887500Z + sudo modprobe nvidia
2023-08-17T10:51:09.3011073Z + echo 'After installing NVIDIA driver'
2023-08-17T10:51:09.3011377Z + lspci
2023-08-17T10:51:09.3011926Z After installing NVIDIA driver
2023-08-17T10:51:09.3121247Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
2023-08-17T10:51:09.3121973Z 00:01.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 03)
2023-08-17T10:51:09.3122711Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
2023-08-17T10:51:09.3123331Z 00:03.0 Non-VGA unclassified device: Red Hat, Inc. Virtio SCSI
2023-08-17T10:51:09.3123698Z 00:04.0 3D controller: NVIDIA Corporation Device 20b0 (rev a1)
2023-08-17T10:51:09.3124315Z 00:05.0 Ethernet controller: Red Hat, Inc. Virtio network device
2023-08-17T10:51:09.3124666Z 00:06.0 Unclassified device [00ff]: Red Hat, Inc. Virtio RNG
2023-08-17T10:51:09.3124940Z + lsmod
2023-08-17T10:51:09.3145910Z Module                  Size  Used by
2023-08-17T10:51:09.3146345Z btrfs                1536000  0
2023-08-17T10:51:09.3146740Z blake2b_generic        20480  0
2023-08-17T10:51:09.3147156Z xor                    24576  1 btrfs
2023-08-17T10:51:09.3147672Z zstd_compress         225280  1 btrfs
2023-08-17T10:51:09.3148127Z raid6_pq              122880  1 btrfs
2023-08-17T10:51:09.3148445Z ufs                   106496  0
2023-08-17T10:51:09.3148685Z msdos                  20480  0
2023-08-17T10:51:09.3148922Z xfs                  1753088  0
2023-08-17T10:51:09.3149163Z nvidia_modeset       1302528  0
2023-08-17T10:51:09.3149391Z veth                   32768  0
2023-08-17T10:51:09.3149635Z xt_conntrack           16384  1
2023-08-17T10:51:09.3149883Z xt_MASQUERADE          20480  1
2023-08-17T10:51:09.3150126Z xfrm_user              40960  1
2023-08-17T10:51:09.3150370Z xfrm_algo              16384  1 xfrm_user
2023-08-17T10:51:09.3150619Z xt_addrtype            16384  2
2023-08-17T10:51:09.3150867Z iptable_filter         16384  1
2023-08-17T10:51:09.3151115Z iptable_nat            16384  1
2023-08-17T10:51:09.3151389Z nf_nat                 49152  2 iptable_nat,xt_MASQUERADE
2023-08-17T10:51:09.3151668Z bpfilter               16384  0
2023-08-17T10:51:09.3152322Z br_netfilter           28672  0
2023-08-17T10:51:09.3152730Z bridge                307200  1 br_netfilter
2023-08-17T10:51:09.3153080Z stp                    16384  1 bridge
2023-08-17T10:51:09.3153537Z llc                    16384  2 bridge,stp
2023-08-17T10:51:09.3153970Z nvidia_uvm           1540096  0
2023-08-17T10:51:09.3154363Z aufs                  270336  0
2023-08-17T10:51:09.3154719Z overlay               151552  0
2023-08-17T10:51:09.3155128Z nls_iso8859_1          16384  1
2023-08-17T10:51:09.3155581Z dm_multipath           40960  0
2023-08-17T10:51:09.3156077Z scsi_dh_rdac           16384  0
2023-08-17T10:51:09.3156497Z scsi_dh_emc            16384  0
2023-08-17T10:51:09.3156923Z scsi_dh_alua           20480  0
2023-08-17T10:51:09.3157316Z nvidia              56455168  2 nvidia_uvm,nvidia_modeset
2023-08-17T10:51:09.3157597Z binfmt_misc            24576  1
2023-08-17T10:51:09.3157828Z crct10dif_pclmul       16384  1
2023-08-17T10:51:09.3158073Z crc32_pclmul           16384  0
2023-08-17T10:51:09.3158400Z ghash_clmulni_intel    16384  0
2023-08-17T10:51:09.3158653Z aesni_intel           376832  0
2023-08-17T10:51:09.3158879Z virtio_net             61440  0
2023-08-17T10:51:09.3159115Z psmouse               176128  0
2023-08-17T10:51:09.3159357Z input_leds             16384  0
2023-08-17T10:51:09.3159617Z crypto_simd            16384  1 aesni_intel
2023-08-17T10:51:09.3159882Z net_failover           20480  1 virtio_net
2023-08-17T10:51:09.3160194Z cryptd                 24576  2 crypto_simd,ghash_clmulni_intel
2023-08-17T10:51:09.3160476Z serio_raw              20480  0
2023-08-17T10:51:09.3160739Z failover               16384  1 net_failover
2023-08-17T10:51:09.3160981Z sch_fq_codel           24576  13
2023-08-17T10:51:09.3161230Z efi_pstore             16384  0
2023-08-17T10:51:09.3161474Z drm                   618496  1 nvidia
2023-08-17T10:51:09.3161717Z virtio_rng             16384  0
2023-08-17T10:51:09.3161993Z ip_tables              32768  2 iptable_filter,iptable_nat
2023-08-17T10:51:09.3162870Z x_tables               53248  6 xt_conntrack,iptable_filter,xt_addrtype,ip_tables,iptable_nat,xt_MASQUERADE
2023-08-17T10:51:09.3163340Z autofs4                49152  2
2023-08-17T10:51:09.3163577Z + modinfo nvidia
2023-08-17T10:51:09.3164614Z filename:       /lib/modules/5.15.0-1037-gcp/kernel/drivers/video/nvidia.ko
2023-08-17T10:51:09.3165213Z firmware:       nvidia/535.54.03/gsp_tu10x.bin
2023-08-17T10:51:09.3165783Z firmware:       nvidia/535.54.03/gsp_ga10x.bin
2023-08-17T10:51:09.3166385Z alias:          char-major-195-*
2023-08-17T10:51:09.3166783Z version:        535.54.03
2023-08-17T10:51:09.3167537Z supported:      external
2023-08-17T10:51:09.3167910Z license:        NVIDIA
2023-08-17T10:51:09.3168177Z srcversion:     415DC1001E7D8AD79B6207C
2023-08-17T10:51:09.3168520Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2023-08-17T10:51:09.3168819Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2023-08-17T10:51:09.3169117Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2023-08-17T10:51:09.3169388Z depends:        drm
2023-08-17T10:51:09.3169611Z retpoline:      Y
2023-08-17T10:51:09.3169839Z name:           nvidia
2023-08-17T10:51:09.3170234Z vermagic:       5.15.0-1037-gcp SMP mod_unload modversions 
2023-08-17T10:51:09.3170595Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2023-08-17T10:51:09.3170972Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2023-08-17T10:51:09.3171331Z parm:           NVreg_ResmanDebugLevel:int
2023-08-17T10:51:09.3171616Z parm:           NVreg_RmLogonRC:int
2023-08-17T10:51:09.3171916Z parm:           NVreg_ModifyDeviceFiles:int
2023-08-17T10:51:09.3172197Z parm:           NVreg_DeviceFileUID:int
2023-08-17T10:51:09.3172482Z parm:           NVreg_DeviceFileGID:int
2023-08-17T10:51:09.3172771Z parm:           NVreg_DeviceFileMode:int
2023-08-17T10:51:09.3173126Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2023-08-17T10:51:09.3173478Z parm:           NVreg_UsePageAttributeTable:int
2023-08-17T10:51:09.3173795Z parm:           NVreg_EnablePCIeGen3:int
2023-08-17T10:51:09.3174077Z parm:           NVreg_EnableMSI:int
2023-08-17T10:51:09.3174358Z parm:           NVreg_TCEBypassMode:int
2023-08-17T10:51:09.3174652Z parm:           NVreg_EnableStreamMemOPs:int
2023-08-17T10:51:09.3175009Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2023-08-17T10:51:09.3175417Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2023-08-17T10:51:09.3175784Z parm:           NVreg_EnableS0ixPowerManagement:int
2023-08-17T10:51:09.3176192Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2023-08-17T10:51:09.3176576Z parm:           NVreg_DynamicPowerManagement:int
2023-08-17T10:51:09.3176985Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2023-08-17T10:51:09.3177382Z parm:           NVreg_EnableGpuFirmware:int
2023-08-17T10:51:09.3177704Z parm:           NVreg_EnableGpuFirmwareLogs:int
2023-08-17T10:51:09.3178043Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2023-08-17T10:51:09.3178461Z parm:           NVreg_EnableUserNUMAManagement:int
2023-08-17T10:51:09.3178783Z parm:           NVreg_MemoryPoolSize:int
2023-08-17T10:51:09.3179090Z parm:           NVreg_KMallocHeapMaxSize:int
2023-08-17T10:51:09.3179389Z parm:           NVreg_VMallocHeapMaxSize:int
2023-08-17T10:51:09.3179697Z parm:           NVreg_IgnoreMMIOCheck:int
2023-08-17T10:51:09.3179986Z parm:           NVreg_NvLinkDisable:int
2023-08-17T10:51:09.3180319Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2023-08-17T10:51:09.3180651Z parm:           NVreg_RegisterPCIDriver:int
2023-08-17T10:51:09.3180966Z parm:           NVreg_EnableResizableBar:int
2023-08-17T10:51:09.3181282Z parm:           NVreg_EnableDbgBreakpoint:int
2023-08-17T10:51:09.3181590Z parm:           NVreg_RegistryDwords:charp
2023-08-17T10:51:09.3181903Z parm:           NVreg_RegistryDwordsPerDevice:charp
2023-08-17T10:51:09.3182206Z parm:           NVreg_RmMsg:charp
2023-08-17T10:51:09.3182480Z parm:           NVreg_GpuBlacklist:charp
2023-08-17T10:51:09.3182953Z parm:           NVreg_TemporaryFilePath:charp
2023-08-17T10:51:09.3183245Z parm:           NVreg_ExcludedGpus:charp
2023-08-17T10:51:09.3183550Z parm:           NVreg_DmaRemapPeerMmio:int
2023-08-17T10:51:09.3183861Z parm:           NVreg_RmNvlinkBandwidth:charp
2023-08-17T10:51:09.3184159Z parm:           rm_firmware_active:charp
2023-08-17T10:51:09.3184389Z + set +e
2023-08-17T10:51:09.3184685Z + nvidia-smi
2023-08-17T10:51:09.3347070Z Thu Aug 17 10:51:09 2023       
2023-08-17T10:51:09.3348980Z +---------------------------------------------------------------------------------------+
2023-08-17T10:51:09.3349617Z | NVIDIA-SMI 535.54.03              Driver Version: 535.54.03    CUDA Version: 12.2     |
2023-08-17T10:51:09.3350147Z |-----------------------------------------+----------------------+----------------------+
2023-08-17T10:51:09.3350835Z | GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
2023-08-17T10:51:09.3351368Z | Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
2023-08-17T10:51:09.3351739Z |                                         |                      |               MIG M. |
2023-08-17T10:51:09.3352016Z |=========================================+======================+======================|
2023-08-17T10:51:09.3456084Z |   0  NVIDIA A100-SXM4-40GB          On  | 00000000:00:04.0 Off |                    0 |
2023-08-17T10:51:09.3456900Z | N/A   34C    P0              43W / 400W |      4MiB / 40960MiB |      0%      Default |
2023-08-17T10:51:09.3457552Z |                                         |                      |             Disabled |
2023-08-17T10:51:09.3458201Z +-----------------------------------------+----------------------+----------------------+
2023-08-17T10:51:09.3458558Z                                                                                          
2023-08-17T10:51:09.3459028Z +---------------------------------------------------------------------------------------+
2023-08-17T10:51:09.3459431Z | Processes:                                                                            |
2023-08-17T10:51:09.3459887Z |  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
2023-08-17T10:51:09.3460604Z |        ID   ID                                                             Usage      |
2023-08-17T10:51:09.3461142Z |=======================================================================================|
2023-08-17T10:51:09.3461463Z |  No running processes found                                                           |
2023-08-17T10:51:09.3461984Z +---------------------------------------------------------------------------------------+
2023-08-17T10:51:09.7043532Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0
2023-08-17T10:51:09.7199363Z NVIDIA A100-SXM4-40GB
2023-08-17T10:51:09.7225210Z + NVIDIA_SMI_STATUS=0
2023-08-17T10:51:09.7225787Z + '[' 0 -eq 0 ']'
2023-08-17T10:51:09.7226463Z + echo 'INFO: Ignoring allowed status 0'
2023-08-17T10:51:09.7226843Z + set -e
2023-08-17T10:51:09.7227092Z INFO: Ignoring allowed status 0
2023-08-17T10:51:09.7234358Z == Installing nvidia container toolkit for ubuntu20.04 ==
2023-08-17T10:51:09.7238070Z + sudo apt-get install -y nvidia-docker2
2023-08-17T10:51:09.7958762Z Reading package lists...
2023-08-17T10:51:10.0175735Z Building dependency tree...
2023-08-17T10:51:10.0181156Z Reading state information...
2023-08-17T10:51:10.1978828Z nvidia-docker2 is already the newest version (2.13.0-1).
2023-08-17T10:51:10.1979326Z The following packages were automatically installed and are no longer required:
2023-08-17T10:51:10.1979903Z   libatasmart4 libblockdev-fs2 libblockdev-loop2 libblockdev-part-err2
2023-08-17T10:51:10.1980504Z   libblockdev-part2 libblockdev-swap2 libblockdev-utils2 libblockdev2
2023-08-17T10:51:10.1981070Z   libexpat1-dev libmm-glib0 libnspr4 libnss3 libnuma1 libparted-fs-resize0
2023-08-17T10:51:10.1985010Z   libpython3-dev libpython3.8-dev libudisks2-0 usb-modeswitch
2023-08-17T10:51:10.1985805Z   usb-modeswitch-data
2023-08-17T10:51:10.1986153Z Use 'sudo apt autoremove' to remove them.
2023-08-17T10:51:10.2315150Z 0 upgraded, 0 newly installed, 0 to remove and 71 not upgraded.
2023-08-17T10:51:10.2330391Z + sudo systemctl restart docker
2023-08-17T10:51:19.2612060Z Command completed after 1 attempt(s).
2023-08-17T10:51:19.2717981Z ##[group]Run sudo nvidia-smi -pm 1
2023-08-17T10:51:19.2718336Z sudo nvidia-smi -pm 1
2023-08-17T10:51:19.2718600Z sudo nvidia-smi -ac 1215,1410
2023-08-17T10:51:19.2718854Z nvidia-smi
2023-08-17T10:51:19.2738927Z shell: /usr/bin/bash -e {0}
2023-08-17T10:51:19.2739177Z env:
2023-08-17T10:51:19.2739396Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:51:19.2739715Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T10:51:19.2739991Z ##[endgroup]
2023-08-17T10:51:19.3072003Z Persistence mode is already Enabled for GPU 00000000:00:04.0.
2023-08-17T10:51:19.3072396Z All done.
2023-08-17T10:51:19.3503833Z Applications clocks set to "(MEM 1215, SM 1410)" for GPU 00000000:00:04.0
2023-08-17T10:51:19.3504180Z All done.
2023-08-17T10:51:19.3693118Z Thu Aug 17 10:51:19 2023       
2023-08-17T10:51:19.3694325Z +---------------------------------------------------------------------------------------+
2023-08-17T10:51:19.3695366Z | NVIDIA-SMI 535.54.03              Driver Version: 535.54.03    CUDA Version: 12.2     |
2023-08-17T10:51:19.3695888Z |-----------------------------------------+----------------------+----------------------+
2023-08-17T10:51:19.3696429Z | GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
2023-08-17T10:51:19.3696967Z | Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
2023-08-17T10:51:19.3697330Z |                                         |                      |               MIG M. |
2023-08-17T10:51:19.3697619Z |=========================================+======================+======================|
2023-08-17T10:51:19.3801190Z |   0  NVIDIA A100-SXM4-40GB          On  | 00000000:00:04.0 Off |                    0 |
2023-08-17T10:51:19.3801619Z | N/A   34C    P0              43W / 400W |      4MiB / 40960MiB |      0%      Default |
2023-08-17T10:51:19.3801942Z |                                         |                      |             Disabled |
2023-08-17T10:51:19.3802522Z +-----------------------------------------+----------------------+----------------------+
2023-08-17T10:51:19.3803232Z                                                                                          
2023-08-17T10:51:19.3803727Z +---------------------------------------------------------------------------------------+
2023-08-17T10:51:19.3804102Z | Processes:                                                                            |
2023-08-17T10:51:19.3804446Z |  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
2023-08-17T10:51:19.3804780Z |        ID   ID                                                             Usage      |
2023-08-17T10:51:19.3805164Z |=======================================================================================|
2023-08-17T10:51:19.3805985Z |  No running processes found                                                           |
2023-08-17T10:51:19.3806504Z +---------------------------------------------------------------------------------------+
2023-08-17T10:51:19.7499827Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84
2023-08-17T10:51:19.7500412Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84
2023-08-17T10:51:19.7500796Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2023-08-17T10:51:19.7501154Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}"
2023-08-17T10:51:19.7519937Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-08-17T10:51:19.7520227Z env:
2023-08-17T10:51:19.7520453Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:51:19.7520772Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T10:51:19.7521279Z ##[endgroup]
2023-08-17T10:51:20.8796681Z Requirement already satisfied: psutil==5.9.1 in /home/ubuntu/.local/lib/python3.8/site-packages (5.9.1)
2023-08-17T10:51:20.8882141Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ubuntu/.local/lib/python3.8/site-packages (11.525.84)
2023-08-17T10:51:21.1044303Z Prepare all required actions
2023-08-17T10:51:21.1044912Z Getting action download info
2023-08-17T10:51:21.2635458Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6)
2023-08-17T10:51:21.7944256Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a)
2023-08-17T10:51:22.1516188Z ##[group]Run ./.github/actions/download-build-artifacts
2023-08-17T10:51:22.1516476Z with:
2023-08-17T10:51:22.1516762Z   name: linux-focal-cuda12.1-py3.10-gcc9-sm80
2023-08-17T10:51:22.1517042Z env:
2023-08-17T10:51:22.1517250Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:51:22.1517571Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T10:51:22.1517854Z ##[endgroup]
2023-08-17T10:51:22.1546146Z ##[group]Run seemethere/download-artifact-s3@v4
2023-08-17T10:51:22.1546433Z with:
2023-08-17T10:51:22.1546712Z   name: linux-focal-cuda12.1-py3.10-gcc9-sm80
2023-08-17T10:51:22.1547055Z   s3-bucket: gha-artifacts
2023-08-17T10:51:22.1547291Z   region: us-east-1
2023-08-17T10:51:22.1547505Z env:
2023-08-17T10:51:22.1547725Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:51:22.1548033Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T10:51:22.1548309Z ##[endgroup]
2023-08-17T10:51:22.7445969Z (node:3517823) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2023-08-17T10:51:22.7446477Z 
2023-08-17T10:51:22.7446736Z Please migrate your code to use AWS SDK for JavaScript (v3).
2023-08-17T10:51:22.7447166Z For more information, check the migration guide at https://a.co/7PzMCcy
2023-08-17T10:51:22.7447802Z (Use `node --trace-warnings ...` to show where the warning was created)
2023-08-17T10:51:22.9216785Z Found 1 objects with prefix pytorch/pytorch/5887783871/linux-focal-cuda12.1-py3.10-gcc9-sm80/
2023-08-17T10:51:22.9219391Z Starting download (1/1): /home/weiwangmeta/actions-runner/_work/pytorch/pytorch/artifacts.zip
2023-08-17T10:51:38.3307889Z Finished download (1/1): /home/weiwangmeta/actions-runner/_work/pytorch/pytorch/artifacts.zip
2023-08-17T10:51:38.3318729Z Artifact download has finished successfully
2023-08-17T10:51:38.3603419Z ##[group]Run unzip -o artifacts.zip
2023-08-17T10:51:38.3603729Z unzip -o artifacts.zip
2023-08-17T10:51:38.3622248Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-08-17T10:51:38.3622543Z env:
2023-08-17T10:51:38.3622772Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:51:38.3623091Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T10:51:38.3623366Z ##[endgroup]
2023-08-17T10:51:38.3677306Z Archive:  artifacts.zip
2023-08-17T10:51:38.3679325Z    creating: dist/
2023-08-17T10:51:40.5229267Z   inflating: dist/torch-2.1.0a0+git384e0d1-cp310-cp310-linux_x86_64.whl  
2023-08-17T10:51:40.5229692Z    creating: build/custom_test_artifacts/
2023-08-17T10:51:40.5230163Z    creating: build/custom_test_artifacts/custom-op-build/
2023-08-17T10:51:40.5230669Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2023-08-17T10:51:40.5231213Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/
2023-08-17T10:51:40.5237294Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml  
2023-08-17T10:51:40.5237879Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/
2023-08-17T10:51:40.5238441Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2023-08-17T10:51:40.5239012Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/
2023-08-17T10:51:40.5239591Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2023-08-17T10:51:40.5240938Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2023-08-17T10:51:40.5242912Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2023-08-17T10:51:40.5243772Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/
2023-08-17T10:51:40.5244370Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2023-08-17T10:51:40.5246184Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2023-08-17T10:51:40.5247573Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2023-08-17T10:51:40.5249477Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2023-08-17T10:51:40.5250173Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2023-08-17T10:51:40.5251341Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2023-08-17T10:51:40.5252093Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2023-08-17T10:51:40.5252700Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2023-08-17T10:51:40.5253290Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2023-08-17T10:51:40.5334693Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2023-08-17T10:51:40.5379443Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2023-08-17T10:51:40.5380274Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2023-08-17T10:51:40.5433216Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2023-08-17T10:51:40.5433981Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2023-08-17T10:51:40.5434751Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2023-08-17T10:51:40.5435530Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2023-08-17T10:51:40.5436259Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2023-08-17T10:51:40.5436983Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2023-08-17T10:51:40.5437711Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2023-08-17T10:51:40.5438452Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2023-08-17T10:51:40.5439164Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2023-08-17T10:51:40.5439886Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2023-08-17T10:51:40.5440525Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2023-08-17T10:51:40.5441173Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2023-08-17T10:51:40.5441828Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2023-08-17T10:51:40.5442471Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2023-08-17T10:51:40.5443573Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2023-08-17T10:51:40.5513606Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2023-08-17T10:51:40.5585140Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2023-08-17T10:51:40.5585823Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2023-08-17T10:51:40.5586411Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/
2023-08-17T10:51:40.5586956Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2023-08-17T10:51:40.5587490Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2023-08-17T10:51:40.5588040Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2023-08-17T10:51:40.5588650Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2023-08-17T10:51:40.5589289Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2023-08-17T10:51:40.5589982Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2023-08-17T10:51:40.5590565Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2023-08-17T10:51:40.5591174Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2023-08-17T10:51:40.5591785Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2023-08-17T10:51:40.5592397Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2023-08-17T10:51:40.5592998Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2023-08-17T10:51:40.5593591Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2023-08-17T10:51:40.5614901Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2023-08-17T10:51:40.5728396Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2023-08-17T10:51:40.5728986Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2023-08-17T10:51:40.5729612Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2023-08-17T10:51:40.5730324Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2023-08-17T10:51:40.5730940Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2023-08-17T10:51:40.5731549Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2023-08-17T10:51:40.5732186Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2023-08-17T10:51:40.5732804Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2023-08-17T10:51:40.5733462Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2023-08-17T10:51:40.5734077Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2023-08-17T10:51:40.5734699Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2023-08-17T10:51:40.5756590Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2023-08-17T10:51:40.5841727Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2023-08-17T10:51:40.5842395Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2023-08-17T10:51:40.5843376Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2023-08-17T10:51:40.5843968Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2023-08-17T10:51:40.5844646Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2023-08-17T10:51:40.5845203Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2023-08-17T10:51:40.5845742Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2023-08-17T10:51:40.5848272Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2023-08-17T10:51:40.5848908Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2023-08-17T10:51:40.5850018Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2023-08-17T10:51:40.5943612Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2023-08-17T10:51:40.6008118Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2023-08-17T10:51:40.6008612Z    creating: build/custom_test_artifacts/jit-hook-build/
2023-08-17T10:51:40.6009086Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2023-08-17T10:51:40.6009620Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/
2023-08-17T10:51:40.6017489Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml  
2023-08-17T10:51:40.6018068Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/
2023-08-17T10:51:40.6018632Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2023-08-17T10:51:40.6019190Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/
2023-08-17T10:51:40.6019761Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2023-08-17T10:51:40.6020985Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2023-08-17T10:51:40.6022819Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2023-08-17T10:51:40.6023404Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/
2023-08-17T10:51:40.6023983Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2023-08-17T10:51:40.6025770Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2023-08-17T10:51:40.6027016Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2023-08-17T10:51:40.6028993Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2023-08-17T10:51:40.6029636Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2023-08-17T10:51:40.6031072Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2023-08-17T10:51:40.6032159Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2023-08-17T10:51:40.6032815Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2023-08-17T10:51:40.6033447Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2023-08-17T10:51:40.6079603Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2023-08-17T10:51:40.6125352Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2023-08-17T10:51:40.6126121Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2023-08-17T10:51:40.6179216Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2023-08-17T10:51:40.6180259Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2023-08-17T10:51:40.6181149Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2023-08-17T10:51:40.6181940Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2023-08-17T10:51:40.6182689Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2023-08-17T10:51:40.6183404Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2023-08-17T10:51:40.6184128Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2023-08-17T10:51:40.6184844Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2023-08-17T10:51:40.6185547Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2023-08-17T10:51:40.6186226Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2023-08-17T10:51:40.6186877Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2023-08-17T10:51:40.6187574Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2023-08-17T10:51:40.6188210Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2023-08-17T10:51:40.6188914Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2023-08-17T10:51:40.6189580Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2023-08-17T10:51:40.6259663Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2023-08-17T10:51:40.6331268Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2023-08-17T10:51:40.6331944Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2023-08-17T10:51:40.6332512Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/
2023-08-17T10:51:40.6333048Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2023-08-17T10:51:40.6333632Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2023-08-17T10:51:40.6334185Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2023-08-17T10:51:40.6334781Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2023-08-17T10:51:40.6335433Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2023-08-17T10:51:40.6336070Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2023-08-17T10:51:40.6336676Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2023-08-17T10:51:40.6337297Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2023-08-17T10:51:40.6337889Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2023-08-17T10:51:40.6338565Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2023-08-17T10:51:40.6339174Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2023-08-17T10:51:40.6340027Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2023-08-17T10:51:40.6360758Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2023-08-17T10:51:40.6431773Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2023-08-17T10:51:40.6432449Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2023-08-17T10:51:40.6433115Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2023-08-17T10:51:40.6433699Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2023-08-17T10:51:40.6434252Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2023-08-17T10:51:40.6434796Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2023-08-17T10:51:40.6435324Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2023-08-17T10:51:40.6437449Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2023-08-17T10:51:40.6437948Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2023-08-17T10:51:40.6438683Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2023-08-17T10:51:40.6494181Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2023-08-17T10:51:40.6494691Z    creating: build/custom_test_artifacts/custom-backend-build/
2023-08-17T10:51:40.6495203Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2023-08-17T10:51:40.6495771Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/
2023-08-17T10:51:40.6503414Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml  
2023-08-17T10:51:40.6504023Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/
2023-08-17T10:51:40.6504630Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2023-08-17T10:51:40.6505235Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/
2023-08-17T10:51:40.6505850Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2023-08-17T10:51:40.6506708Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2023-08-17T10:51:40.6508785Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2023-08-17T10:51:40.6509391Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/
2023-08-17T10:51:40.6510055Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2023-08-17T10:51:40.6511534Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2023-08-17T10:51:40.6513431Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2023-08-17T10:51:40.6514819Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2023-08-17T10:51:40.6515503Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2023-08-17T10:51:40.6517060Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2023-08-17T10:51:40.6517814Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2023-08-17T10:51:40.6518444Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2023-08-17T10:51:40.6519059Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2023-08-17T10:51:40.6567366Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2023-08-17T10:51:40.6613522Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2023-08-17T10:51:40.6614466Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2023-08-17T10:51:40.6667674Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2023-08-17T10:51:40.6668454Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2023-08-17T10:51:40.6669235Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2023-08-17T10:51:40.6670076Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2023-08-17T10:51:40.6670864Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2023-08-17T10:51:40.6671618Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2023-08-17T10:51:40.6672370Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2023-08-17T10:51:40.6673124Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2023-08-17T10:51:40.6673865Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2023-08-17T10:51:40.6674554Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2023-08-17T10:51:40.6675227Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2023-08-17T10:51:40.6675911Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2023-08-17T10:51:40.6676597Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2023-08-17T10:51:40.6677267Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2023-08-17T10:51:40.6677937Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2023-08-17T10:51:40.6751750Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2023-08-17T10:51:40.6823068Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2023-08-17T10:51:40.6823756Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2023-08-17T10:51:40.6824418Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/
2023-08-17T10:51:40.6824986Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2023-08-17T10:51:40.6825563Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2023-08-17T10:51:40.6826162Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2023-08-17T10:51:40.6826799Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2023-08-17T10:51:40.6827481Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2023-08-17T10:51:40.6828140Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2023-08-17T10:51:40.6828770Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2023-08-17T10:51:40.6829618Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2023-08-17T10:51:40.6830342Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2023-08-17T10:51:40.6831114Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2023-08-17T10:51:40.6831775Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2023-08-17T10:51:40.6832414Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2023-08-17T10:51:40.6834437Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2023-08-17T10:51:40.6977702Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2023-08-17T10:51:40.6978371Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2023-08-17T10:51:40.6979035Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2023-08-17T10:51:40.6979749Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2023-08-17T10:51:40.6980457Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2023-08-17T10:51:40.6981108Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2023-08-17T10:51:40.6981786Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2023-08-17T10:51:40.6982453Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2023-08-17T10:51:40.6983127Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2023-08-17T10:51:40.6983788Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2023-08-17T10:51:40.6984450Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2023-08-17T10:51:40.7006322Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2023-08-17T10:51:40.7067749Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2023-08-17T10:51:40.7068455Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2023-08-17T10:51:40.7069107Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2023-08-17T10:51:40.7069707Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2023-08-17T10:51:40.7070342Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2023-08-17T10:51:40.7070920Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2023-08-17T10:51:40.7071497Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2023-08-17T10:51:40.7073970Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2023-08-17T10:51:40.7074594Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2023-08-17T10:51:40.7075136Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2023-08-17T10:51:40.7192896Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2023-08-17T10:51:40.7241142Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2023-08-17T10:51:40.7241493Z    creating: build/lib/
2023-08-17T10:51:40.7242114Z   inflating: build/lib/libclog.a     
2023-08-17T10:51:40.7252086Z   inflating: build/lib/libpthreadpool.a  
2023-08-17T10:51:40.7329247Z   inflating: build/lib/libgtest.a    
2023-08-17T10:51:40.7434605Z   inflating: build/lib/libprotobuf-lite.a  
2023-08-17T10:51:40.7524624Z   inflating: build/lib/libbenchmark.a  
2023-08-17T10:51:40.7533722Z   inflating: build/lib/libittnotify.a  
2023-08-17T10:51:40.8073757Z   inflating: build/lib/libprotobuf.a  
2023-08-17T10:51:40.8106779Z   inflating: build/lib/libtensorpipe_uv.a  
2023-08-17T10:51:40.8182823Z   inflating: build/lib/libasmjit.a   
2023-08-17T10:51:40.8331944Z   inflating: build/lib/libgloo.a     
2023-08-17T10:51:40.8356863Z   inflating: build/lib/libfmt.a      
2023-08-17T10:51:40.8358665Z   inflating: build/lib/libcaffe2_nvrtc.so  
2023-08-17T10:51:40.8359373Z   inflating: build/lib/libfoxi_loader.a  
2023-08-17T10:51:40.8448095Z   inflating: build/lib/libc10.so     
2023-08-17T10:51:40.8449233Z   inflating: build/lib/libtorch_global_deps.so  
2023-08-17T10:51:40.8458571Z   inflating: build/lib/libcpuinfo.a  
2023-08-17T10:51:40.8467305Z   inflating: build/lib/libcpuinfo_internals.a  
2023-08-17T10:51:40.8469916Z   inflating: build/lib/libnnpack_reference_layers.a  
2023-08-17T10:51:40.9068050Z   inflating: build/lib/libprotoc.a   
2023-08-17T10:51:40.9089245Z   inflating: build/lib/libgmock.a    
2023-08-17T10:51:40.9089678Z   inflating: build/lib/libgtest_main.a  
2023-08-17T10:51:40.9090459Z   inflating: build/lib/libbenchmark_main.a  
2023-08-17T10:51:41.9077899Z   inflating: build/lib/libdnnl.a     
2023-08-17T10:51:41.9756594Z   inflating: build/lib/libtensorpipe.a  
2023-08-17T10:51:41.9817172Z   inflating: build/lib/libc10_cuda.so  
2023-08-17T10:51:41.9832935Z   inflating: build/lib/libqnnpack.a  
2023-08-17T10:51:41.9856060Z   inflating: build/lib/libpytorch_qnnpack.a  
2023-08-17T10:51:42.1344404Z   inflating: build/lib/libfbgemm.a   
2023-08-17T10:51:42.1344780Z   inflating: build/lib/libgmock_main.a  
2023-08-17T10:51:42.2510217Z   inflating: build/lib/libdnnl_graph.a  
2023-08-17T10:51:42.3103540Z   inflating: build/lib/libkineto.a   
2023-08-17T10:51:42.3407050Z   inflating: build/lib/libtensorpipe_cuda.a  
2023-08-17T10:51:42.3452683Z   inflating: build/lib/libcaffe2_protos.a  
2023-08-17T10:51:42.3474111Z   inflating: build/lib/libnnpack.a   
2023-08-17T10:51:42.3611985Z   inflating: build/lib/libXNNPACK.a  
2023-08-17T10:51:42.3660981Z   inflating: build/lib/libonnx_proto.a  
2023-08-17T10:51:42.4421836Z   inflating: build/lib/libonnx.a     
2023-08-17T10:51:42.4881710Z   inflating: build/lib/libgloo_cuda.a  
2023-08-17T10:51:45.2218010Z   inflating: build/lib/libtorch_cpu.so  
2023-08-17T10:51:45.2221821Z   inflating: build/lib/libunbox_lib.a  
2023-08-17T10:51:47.2004368Z   inflating: build/lib/libtorch_cuda.so  
2023-08-17T10:51:47.2004765Z   inflating: build/lib/libtorch.so   
2023-08-17T10:51:47.2006887Z   inflating: build/lib/libc10d_cuda_test.so  
2023-08-17T10:51:48.1911842Z   inflating: build/lib/libtorch_cuda_linalg.so  
2023-08-17T10:51:48.1935273Z   inflating: build/lib/libjitbackend_test.so  
2023-08-17T10:51:48.1996125Z   inflating: build/lib/libtorchbind_test.so  
2023-08-17T10:51:48.2025777Z   inflating: build/lib/libbackend_with_compiler.so  
2023-08-17T10:51:48.2030730Z   inflating: build/lib/libshm.so     
2023-08-17T10:51:48.2748544Z   inflating: build/lib/libnvfuser_codegen.so  
2023-08-17T10:51:48.4825587Z   inflating: build/lib/libtorch_python.so  
2023-08-17T10:51:48.4865939Z   inflating: build/lib/libnnapi_backend.so  
2023-08-17T10:51:48.4866224Z    creating: build/bin/
2023-08-17T10:51:48.4926326Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2023-08-17T10:51:48.4987685Z   inflating: build/bin/c10_DeviceGuard_test  
2023-08-17T10:51:48.5047717Z   inflating: build/bin/c10_Device_test  
2023-08-17T10:51:48.5106043Z   inflating: build/bin/c10_StreamGuard_test  
2023-08-17T10:51:48.5176804Z   inflating: build/bin/c10_DispatchKeySet_test  
2023-08-17T10:51:48.5236216Z   inflating: build/bin/c10_SymInt_test  
2023-08-17T10:51:48.5301697Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2023-08-17T10:51:48.5368272Z   inflating: build/bin/c10_InlineStreamGuard_test  
2023-08-17T10:51:48.5435292Z   inflating: build/bin/c10_SizesAndStrides_test  
2023-08-17T10:51:48.5498069Z   inflating: build/bin/c10_context_test  
2023-08-17T10:51:48.5561823Z   inflating: build/bin/c10_Bitset_test  
2023-08-17T10:51:48.5620165Z   inflating: build/bin/c10_ConstexprCrc_test  
2023-08-17T10:51:48.5679635Z   inflating: build/bin/c10_DeadlockDetection_test  
2023-08-17T10:51:48.5739180Z   inflating: build/bin/c10_Half_test  
2023-08-17T10:51:48.5804899Z   inflating: build/bin/c10_Metaprogramming_test  
2023-08-17T10:51:48.5872001Z   inflating: build/bin/c10_LeftRight_test  
2023-08-17T10:51:48.5931992Z   inflating: build/bin/c10_Synchronized_test  
2023-08-17T10:51:48.5998301Z   inflating: build/bin/c10_ThreadLocal_test  
2023-08-17T10:51:48.6060023Z   inflating: build/bin/c10_TypeIndex_test  
2023-08-17T10:51:48.6120764Z   inflating: build/bin/c10_TypeList_test  
2023-08-17T10:51:48.6179436Z   inflating: build/bin/c10_TypeTraits_test  
2023-08-17T10:51:48.6241188Z   inflating: build/bin/c10_accumulate_test  
2023-08-17T10:51:48.6307516Z   inflating: build/bin/c10_bfloat16_test  
2023-08-17T10:51:48.6367867Z   inflating: build/bin/c10_bit_cast_test  
2023-08-17T10:51:48.6434082Z   inflating: build/bin/c10_complex_test  
2023-08-17T10:51:48.6501292Z   inflating: build/bin/c10_complex_math_test  
2023-08-17T10:51:48.6632372Z   inflating: build/bin/c10_either_test  
2023-08-17T10:51:48.6695376Z   inflating: build/bin/c10_exception_test  
2023-08-17T10:51:48.6755719Z   inflating: build/bin/c10_flags_test  
2023-08-17T10:51:48.6952834Z   inflating: build/bin/c10_intrusive_ptr_test  
2023-08-17T10:51:48.7019278Z   inflating: build/bin/c10_logging_test  
2023-08-17T10:51:48.7079947Z   inflating: build/bin/c10_irange_test  
2023-08-17T10:51:48.7169130Z   inflating: build/bin/c10_optional_test  
2023-08-17T10:51:48.7244189Z   inflating: build/bin/c10_registry_test  
2023-08-17T10:51:48.7317760Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2023-08-17T10:51:48.7497934Z   inflating: build/bin/c10_small_vector_test  
2023-08-17T10:51:48.7559469Z   inflating: build/bin/c10_ssize_test  
2023-08-17T10:51:48.7620468Z   inflating: build/bin/c10_tempfile_test  
2023-08-17T10:51:48.7690515Z   inflating: build/bin/c10_string_view_test  
2023-08-17T10:51:48.7748282Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2023-08-17T10:51:48.7815176Z   inflating: build/bin/c10_typeid_test  
2023-08-17T10:51:48.8350094Z   inflating: build/bin/protoc-3.13.0.0  
2023-08-17T10:51:48.8882863Z   inflating: build/bin/protoc        
2023-08-17T10:51:48.8945980Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test  
2023-08-17T10:51:48.9008851Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream  
2023-08-17T10:51:48.9071650Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device  
2023-08-17T10:51:48.9134036Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes  
2023-08-17T10:51:48.9196895Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads  
2023-08-17T10:51:48.9260148Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks  
2023-08-17T10:51:48.9318728Z   inflating: build/bin/c10_cuda_CUDATest  
2023-08-17T10:51:48.9382380Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block  
2023-08-17T10:51:48.9743751Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2023-08-17T10:51:49.0124285Z   inflating: build/bin/vec_test_all_types_AVX512  
2023-08-17T10:51:49.0519528Z   inflating: build/bin/vec_test_all_types_AVX2  
2023-08-17T10:51:49.0582512Z   inflating: build/bin/FileStoreTest  
2023-08-17T10:51:49.0647336Z   inflating: build/bin/TCPStoreTest  
2023-08-17T10:51:49.0710683Z   inflating: build/bin/HashStoreTest  
2023-08-17T10:51:49.0726867Z   inflating: build/bin/ProcessGroupMPITest  
2023-08-17T10:51:49.0790894Z   inflating: build/bin/test_edge_op_registration  
2023-08-17T10:51:49.0793918Z   inflating: build/bin/example_allreduce  
2023-08-17T10:51:49.0855818Z   inflating: build/bin/Dimname_test  
2023-08-17T10:51:49.0943111Z   inflating: build/bin/Dict_test     
2023-08-17T10:51:49.1020294Z   inflating: build/bin/MaybeOwned_test  
2023-08-17T10:51:49.1088267Z   inflating: build/bin/NamedTensor_test  
2023-08-17T10:51:49.1158642Z   inflating: build/bin/apply_utils_test  
2023-08-17T10:51:49.1228131Z   inflating: build/bin/atest         
2023-08-17T10:51:49.1302458Z   inflating: build/bin/basic         
2023-08-17T10:51:49.1368237Z   inflating: build/bin/broadcast_test  
2023-08-17T10:51:49.1437417Z   inflating: build/bin/cpu_generator_test  
2023-08-17T10:51:49.1500480Z   inflating: build/bin/cpu_profiling_allocator_test  
2023-08-17T10:51:49.1610733Z   inflating: build/bin/cpu_rng_test  
2023-08-17T10:51:49.1669813Z   inflating: build/bin/dispatch_key_set_test  
2023-08-17T10:51:49.1729807Z   inflating: build/bin/dlconvertor_test  
2023-08-17T10:51:49.1798736Z   inflating: build/bin/extension_backend_test  
2023-08-17T10:51:49.1864832Z   inflating: build/bin/half_test     
2023-08-17T10:51:49.1977539Z   inflating: build/bin/ivalue_test   
2023-08-17T10:51:49.2037964Z   inflating: build/bin/lazy_tensor_test  
2023-08-17T10:51:49.2103048Z   inflating: build/bin/math_kernel_test  
2023-08-17T10:51:49.2167411Z   inflating: build/bin/memory_format_test  
2023-08-17T10:51:49.2230287Z   inflating: build/bin/memory_overlapping_test  
2023-08-17T10:51:49.2290470Z   inflating: build/bin/operator_name_test  
2023-08-17T10:51:49.2354017Z   inflating: build/bin/mobile_memory_cleanup  
2023-08-17T10:51:49.2420406Z   inflating: build/bin/native_test   
2023-08-17T10:51:49.2482316Z   inflating: build/bin/packedtensoraccessor_test  
2023-08-17T10:51:49.2543860Z   inflating: build/bin/operators_test  
2023-08-17T10:51:49.2622780Z   inflating: build/bin/pow_test      
2023-08-17T10:51:49.2690468Z   inflating: build/bin/quantized_test  
2023-08-17T10:51:49.2749745Z   inflating: build/bin/reduce_ops_test  
2023-08-17T10:51:49.2817280Z   inflating: build/bin/scalar_tensor_test  
2023-08-17T10:51:49.2878190Z   inflating: build/bin/reportMemoryUsage_test  
2023-08-17T10:51:49.2947275Z   inflating: build/bin/scalar_test   
2023-08-17T10:51:49.3008536Z   inflating: build/bin/StorageUtils_test  
2023-08-17T10:51:49.3071431Z   inflating: build/bin/stride_properties_test  
2023-08-17T10:51:49.3137593Z   inflating: build/bin/type_ptr_test  
2023-08-17T10:51:49.3140636Z   inflating: build/bin/thread_init_test  
2023-08-17T10:51:49.3233418Z   inflating: build/bin/tensor_iterator_test  
2023-08-17T10:51:49.3297675Z   inflating: build/bin/test_parallel  
2023-08-17T10:51:49.3370263Z   inflating: build/bin/type_test     
2023-08-17T10:51:49.3429358Z   inflating: build/bin/variant_test  
2023-08-17T10:51:49.3491691Z   inflating: build/bin/undefined_tensor_test  
2023-08-17T10:51:49.3493183Z   inflating: build/bin/verify_api_visibility  
2023-08-17T10:51:49.3554814Z   inflating: build/bin/weakref_test  
2023-08-17T10:51:49.3637253Z   inflating: build/bin/legacy_vmap_test  
2023-08-17T10:51:49.3698727Z   inflating: build/bin/wrapdim_test  
2023-08-17T10:51:49.3757676Z   inflating: build/bin/xla_tensor_test  
2023-08-17T10:51:49.3829205Z   inflating: build/bin/IListRef_test  
2023-08-17T10:51:49.3956194Z   inflating: build/bin/List_test     
2023-08-17T10:51:49.4099696Z   inflating: build/bin/kernel_function_legacy_test  
2023-08-17T10:51:49.4213659Z   inflating: build/bin/kernel_function_test  
2023-08-17T10:51:49.4364301Z   inflating: build/bin/kernel_lambda_legacy_test  
2023-08-17T10:51:49.4442257Z   inflating: build/bin/KernelFunction_test  
2023-08-17T10:51:49.4564348Z   inflating: build/bin/kernel_lambda_test  
2023-08-17T10:51:49.4635821Z   inflating: build/bin/kernel_stackbased_test  
2023-08-17T10:51:49.4749221Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2023-08-17T10:51:49.4810200Z   inflating: build/bin/CppSignature_test  
2023-08-17T10:51:49.4868475Z   inflating: build/bin/op_allowlist_test  
2023-08-17T10:51:49.4937749Z   inflating: build/bin/inline_container_test  
2023-08-17T10:51:49.5291456Z   inflating: build/bin/op_registration_test  
2023-08-17T10:51:49.5359252Z   inflating: build/bin/backend_fallback_test  
2023-08-17T10:51:49.5422504Z   inflating: build/bin/cuda_apply_test  
2023-08-17T10:51:49.5486749Z   inflating: build/bin/cuda_caching_host_allocator_test  
2023-08-17T10:51:49.5557231Z   inflating: build/bin/cuda_atomic_ops_test  
2023-08-17T10:51:49.5641354Z   inflating: build/bin/cuda_complex_math_test  
2023-08-17T10:51:49.5700486Z   inflating: build/bin/cuda_device_test  
2023-08-17T10:51:49.5771335Z   inflating: build/bin/cuda_complex_test  
2023-08-17T10:51:49.5839462Z   inflating: build/bin/cuda_cub_test  
2023-08-17T10:51:49.5900523Z   inflating: build/bin/cuda_dlconvertor_test  
2023-08-17T10:51:49.5961651Z   inflating: build/bin/cuda_integer_divider_test  
2023-08-17T10:51:49.6037945Z   inflating: build/bin/cuda_distributions_test  
2023-08-17T10:51:49.6105762Z   inflating: build/bin/cuda_generator_test  
2023-08-17T10:51:49.6165381Z   inflating: build/bin/cuda_half_test  
2023-08-17T10:51:49.6228167Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2023-08-17T10:51:49.6300516Z   inflating: build/bin/cuda_stream_test  
2023-08-17T10:51:49.6359171Z   inflating: build/bin/cuda_optional_test  
2023-08-17T10:51:49.6417884Z   inflating: build/bin/cuda_cudnn_test  
2023-08-17T10:51:49.6479690Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2023-08-17T10:51:49.6541455Z   inflating: build/bin/cuda_vectorized_test  
2023-08-17T10:51:49.6621862Z   inflating: build/bin/ProcessGroupGlooTest  
2023-08-17T10:51:49.6639794Z   inflating: build/bin/tutorial_tensorexpr  
2023-08-17T10:51:49.6708276Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2023-08-17T10:51:49.6782913Z   inflating: build/bin/ProcessGroupNCCLTest  
2023-08-17T10:51:49.6850949Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2023-08-17T10:51:49.6916223Z   inflating: build/bin/test_dist_autograd  
2023-08-17T10:51:49.6919085Z   inflating: build/bin/parallel_benchmark  
2023-08-17T10:51:49.7001808Z   inflating: build/bin/test_cpp_rpc  
2023-08-17T10:51:49.7082057Z   inflating: build/bin/test_mobile_nnc  
2023-08-17T10:51:49.7092363Z   inflating: build/bin/aot_model_compiler_test  
2023-08-17T10:51:49.7509192Z   inflating: build/bin/test_lazy     
2023-08-17T10:51:49.8495846Z   inflating: build/bin/test_tensorexpr  
2023-08-17T10:51:49.8500597Z   inflating: build/bin/torch_shm_manager  
2023-08-17T10:51:49.9871822Z   inflating: build/bin/test_api      
2023-08-17T10:51:50.0535492Z   inflating: build/bin/test_jit      
2023-08-17T10:51:50.1210795Z   inflating: build/bin/nvfuser_tests  
2023-08-17T10:51:50.1630616Z   inflating: .pytorch-test-file-ratings.json  
2023-08-17T10:51:50.1675006Z ##[group]Run df -H
2023-08-17T10:51:50.1675244Z df -H
2023-08-17T10:51:50.1695341Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-08-17T10:51:50.1695632Z env:
2023-08-17T10:51:50.1695859Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:51:50.1696171Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T10:51:50.1696456Z ##[endgroup]
2023-08-17T10:51:50.1753675Z Filesystem      Size  Used Avail Use% Mounted on
2023-08-17T10:51:50.1754191Z /dev/root       1.1T  228G  838G  22% /
2023-08-17T10:51:50.1754595Z devtmpfs         45G     0   45G   0% /dev
2023-08-17T10:51:50.1755039Z tmpfs            45G     0   45G   0% /dev/shm
2023-08-17T10:51:50.1755309Z tmpfs           9.0G  1.1M  9.0G   1% /run
2023-08-17T10:51:50.1755708Z tmpfs           5.3M     0  5.3M   0% /run/lock
2023-08-17T10:51:50.1756130Z tmpfs            45G     0   45G   0% /sys/fs/cgroup
2023-08-17T10:51:50.1756823Z /dev/loop0       16M   16M     0 100% /snap/aws-cli/130
2023-08-17T10:51:50.1759135Z /dev/loop1      124M  124M     0 100% /snap/core/15419
2023-08-17T10:51:50.1759750Z /dev/loop2      124M  124M     0 100% /snap/core/15511
2023-08-17T10:51:50.1760606Z /dev/loop3       59M   59M     0 100% /snap/core18/2751
2023-08-17T10:51:50.1760927Z /dev/loop5       67M   67M     0 100% /snap/core20/1950
2023-08-17T10:51:50.1761212Z /dev/loop4       59M   59M     0 100% /snap/core18/2785
2023-08-17T10:51:50.1761514Z /dev/loop6       67M   67M     0 100% /snap/core20/1974
2023-08-17T10:51:50.1761966Z /dev/loop7       97M   97M     0 100% /snap/lxd/23991
2023-08-17T10:51:50.1762270Z /dev/loop10      97M   97M     0 100% /snap/lxd/24061
2023-08-17T10:51:50.1762735Z /dev/loop12      56M   56M     0 100% /snap/snapd/19457
2023-08-17T10:51:50.1763313Z /dev/loop11      56M   56M     0 100% /snap/snapd/19361
2023-08-17T10:51:50.1763700Z /dev/sda15      110M  5.5M  104M   5% /boot/efi
2023-08-17T10:51:50.1764239Z /dev/loop13     366M  366M     0 100% /snap/google-cloud-sdk/372
2023-08-17T10:51:50.1764663Z /dev/loop8      366M  366M     0 100% /snap/google-cloud-sdk/374
2023-08-17T10:51:50.1789876Z ##[group]Run .github/scripts/parse_ref.py
2023-08-17T10:51:50.1790210Z .github/scripts/parse_ref.py
2023-08-17T10:51:50.1807474Z shell: /usr/bin/bash -e {0}
2023-08-17T10:51:50.1807714Z env:
2023-08-17T10:51:50.1807941Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:51:50.1808264Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T10:51:50.1808597Z ##[endgroup]
2023-08-17T10:51:50.2146677Z Prepare all required actions
2023-08-17T10:51:50.2147020Z Getting action download info
2023-08-17T10:51:50.3573840Z ##[group]Run ./.github/actions/filter-test-configs
2023-08-17T10:51:50.3574122Z with:
2023-08-17T10:51:50.3574651Z   github-token: ***
2023-08-17T10:51:50.3576249Z   test-matrix: {"include": [{"config": "inductor_huggingface_perf", "shard": 1, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_huggingface_perf", "shard": 2, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_huggingface_perf", "shard": 3, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 1, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 2, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 3, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 4, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 5, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 1, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 2, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 3, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 4, "num_shards": 4, "runner": "linux.gcp.a100.large"}]}
2023-08-17T10:51:50.3577933Z env:
2023-08-17T10:51:50.3578159Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:51:50.3578469Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T10:51:50.3578762Z ##[endgroup]
2023-08-17T10:51:50.3612813Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2023-08-17T10:51:50.3613122Z with:
2023-08-17T10:51:50.3613329Z   shell: bash
2023-08-17T10:51:50.3613540Z   timeout_minutes: 10
2023-08-17T10:51:50.3613771Z   max_attempts: 5
2023-08-17T10:51:50.3614019Z   retry_wait_seconds: 30
2023-08-17T10:51:50.3614348Z   command: set -eux
python3 -m pip install requests==2.26.0 pyyaml==6.0

2023-08-17T10:51:50.3614675Z   polling_interval_seconds: 1
2023-08-17T10:51:50.3614916Z   warning_on_retry: true
2023-08-17T10:51:50.3615160Z   continue_on_error: false
2023-08-17T10:51:50.3615385Z env:
2023-08-17T10:51:50.3615588Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:51:50.3615895Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T10:51:50.3616325Z   GITHUB_TOKEN: ***
2023-08-17T10:51:50.3616598Z ##[endgroup]
2023-08-17T10:51:50.4375841Z + python3 -m pip install requests==2.26.0 pyyaml==6.0
2023-08-17T10:51:51.5458231Z Requirement already satisfied: requests==2.26.0 in /home/ubuntu/.local/lib/python3.8/site-packages (2.26.0)
2023-08-17T10:51:51.5585857Z Requirement already satisfied: pyyaml==6.0 in /home/ubuntu/.local/lib/python3.8/site-packages (6.0)
2023-08-17T10:51:51.5601816Z Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests==2.26.0) (2019.11.28)
2023-08-17T10:51:51.5614776Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3/dist-packages (from requests==2.26.0) (1.25.8)
2023-08-17T10:51:51.5677643Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /usr/lib/python3/dist-packages (from requests==2.26.0) (2.8)
2023-08-17T10:51:51.5700409Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ubuntu/.local/lib/python3.8/site-packages (from requests==2.26.0) (2.0.12)
2023-08-17T10:51:52.4394971Z Command completed after 1 attempt(s).
2023-08-17T10:51:52.4479930Z ##[group]Run set -x
2023-08-17T10:51:52.4480210Z set -x
2023-08-17T10:51:52.4480430Z 
2023-08-17T10:51:52.4480763Z # Use relative path here as this could be checked out anywhere, not necessarily
2023-08-17T10:51:52.4481107Z # in runner workspace
2023-08-17T10:51:52.4481627Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py"
2023-08-17T10:51:52.4500304Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-08-17T10:51:52.4500594Z env:
2023-08-17T10:51:52.4500820Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:51:52.4501124Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T10:51:52.4501415Z ##[endgroup]
2023-08-17T10:51:52.4540198Z + python3 /home/weiwangmeta/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py
2023-08-17T10:51:52.4794700Z ##[group]Run set -x
2023-08-17T10:51:52.4795050Z set -x
2023-08-17T10:51:52.4795266Z 
2023-08-17T10:51:52.4795584Z # TODO: This is a very hacky way to get the job name. GitHub runner has the info
2023-08-17T10:51:52.4796010Z # but doesn't expose it in anyway. The job name is part of the job message the
2023-08-17T10:51:52.4796430Z # runner receives, so it's there and printed out to the diag log. Below is the
2023-08-17T10:51:52.4796866Z # code responsible for printing it. Need to check with GitHub to see if they can
2023-08-17T10:51:52.4797255Z # expose this variable as part of GitHub context.
2023-08-17T10:51:52.4797681Z # https://github.com/actions/runner/blob/main/src/Runner.Worker/JobExtension.cs#L345
2023-08-17T10:51:52.4798111Z pushd "/home/weiwangmeta/actions-runner/_work/pytorch/../../_diag"
2023-08-17T10:51:52.4798413Z pwd
2023-08-17T10:51:52.4798621Z 
2023-08-17T10:51:52.4798939Z LOG_FILE=$(grep -l -r "384e0d104fd077d31efafc564129660e9b7a0f25" *.log | tail -n 1)
2023-08-17T10:51:52.4799278Z if [ -n "${LOG_FILE}" ]; then
2023-08-17T10:51:52.4799628Z   # For some reasons, awk {print $2} on Linux and Windows (bash) work correctly while it
2023-08-17T10:51:52.4799995Z   # needs to be awk {print $3} on MacOS
2023-08-17T10:51:52.4800271Z   case ${RUNNER_OS} in
2023-08-17T10:51:52.4800507Z     macOS)
2023-08-17T10:51:52.4800857Z       JOB_NAME=$(grep -r "\"jobDisplayName\"" "${LOG_FILE}" | awk -F '[:]' '{print $3}' | sed 's/"//g' | xargs)
2023-08-17T10:51:52.4801196Z       ;;
2023-08-17T10:51:52.4801406Z     *)
2023-08-17T10:51:52.4801743Z       JOB_NAME=$(grep -r "\"jobDisplayName\"" "${LOG_FILE}" | awk -F '[:]' '{print $2}' | sed 's/"//g' | xargs)
2023-08-17T10:51:52.4802067Z       ;;
2023-08-17T10:51:52.4802262Z   esac
2023-08-17T10:51:52.4802535Z   echo "job-name=${JOB_NAME}" >> "${GITHUB_OUTPUT}"
2023-08-17T10:51:52.4803714Z fi
2023-08-17T10:51:52.4803933Z 
2023-08-17T10:51:52.4804150Z popd
2023-08-17T10:51:52.4821968Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-08-17T10:51:52.4822261Z env:
2023-08-17T10:51:52.4822488Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:51:52.4822800Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T10:51:52.4823144Z ##[endgroup]
2023-08-17T10:51:52.4859029Z + pushd /home/weiwangmeta/actions-runner/_work/pytorch/../../_diag
2023-08-17T10:51:52.4859620Z /home/weiwangmeta/actions-runner/_diag /home/weiwangmeta/actions-runner/_work/pytorch/pytorch
2023-08-17T10:51:52.4860082Z /home/weiwangmeta/actions-runner/_diag
2023-08-17T10:51:52.4860348Z + pwd
2023-08-17T10:51:52.4867094Z ++ tail -n 1
2023-08-17T10:51:52.4875894Z ++ grep -l -r 384e0d104fd077d31efafc564129660e9b7a0f25 Runner_20230715-185700-utc.log Runner_20230728-154314-utc.log Runner_20230816-123947-utc.log Worker_20230718-095005-utc.log Worker_20230718-170305-utc.log Worker_20230718-200321-utc.log Worker_20230718-231527-utc.log Worker_20230719-044108-utc.log Worker_20230719-075048-utc.log Worker_20230720-075156-utc.log Worker_20230720-210534-utc.log Worker_20230720-234035-utc.log Worker_20230721-015621-utc.log Worker_20230721-024206-utc.log Worker_20230721-075004-utc.log Worker_20230721-110021-utc.log Worker_20230721-195203-utc.log Worker_20230721-224945-utc.log Worker_20230722-004642-utc.log Worker_20230722-074912-utc.log Worker_20230723-075112-utc.log Worker_20230723-231540-utc.log Worker_20230724-074957-utc.log Worker_20230724-110047-utc.log Worker_20230725-074949-utc.log Worker_20230725-220014-utc.log Worker_20230726-065813-utc.log Worker_20230726-090227-utc.log Worker_20230726-231121-utc.log Worker_20230727-005120-utc.log Worker_20230727-075200-utc.log Worker_20230727-220153-utc.log Worker_20230728-023051-utc.log Worker_20230728-073633-utc.log Worker_20230728-113721-utc.log Worker_20230728-154317-utc.log Worker_20230729-021253-utc.log Worker_20230729-050446-utc.log Worker_20230729-075555-utc.log Worker_20230730-075352-utc.log Worker_20230731-075004-utc.log Worker_20230731-184152-utc.log Worker_20230731-233250-utc.log Worker_20230801-075217-utc.log Worker_20230801-150833-utc.log Worker_20230801-203540-utc.log Worker_20230801-224818-utc.log Worker_20230801-234205-utc.log Worker_20230802-004416-utc.log Worker_20230802-075259-utc.log Worker_20230802-160547-utc.log Worker_20230803-065217-utc.log Worker_20230803-095925-utc.log Worker_20230803-222351-utc.log Worker_20230804-032703-utc.log Worker_20230804-035500-utc.log Worker_20230804-075144-utc.log Worker_20230804-111746-utc.log Worker_20230804-142655-utc.log Worker_20230804-161219-utc.log Worker_20230805-075043-utc.log Worker_20230806-074942-utc.log Worker_20230806-111139-utc.log Worker_20230807-074928-utc.log Worker_20230808-001246-utc.log Worker_20230808-082237-utc.log Worker_20230808-180931-utc.log Worker_20230808-204125-utc.log Worker_20230808-223121-utc.log Worker_20230809-045247-utc.log Worker_20230809-060227-utc.log Worker_20230809-075416-utc.log Worker_20230809-104959-utc.log Worker_20230809-221034-utc.log Worker_20230810-000643-utc.log Worker_20230810-023636-utc.log Worker_20230810-072650-utc.log Worker_20230810-130633-utc.log Worker_20230810-233300-utc.log Worker_20230811-022019-utc.log Worker_20230811-031127-utc.log Worker_20230811-070837-utc.log Worker_20230811-090526-utc.log Worker_20230811-105241-utc.log Worker_20230811-141624-utc.log Worker_20230812-072652-utc.log Worker_20230812-151631-utc.log Worker_20230813-072704-utc.log Worker_20230813-104039-utc.log Worker_20230814-072734-utc.log Worker_20230814-105417-utc.log Worker_20230814-200922-utc.log Worker_20230815-072808-utc.log Worker_20230815-193204-utc.log Worker_20230815-203825-utc.log Worker_20230815-233204-utc.log Worker_20230816-004334-utc.log Worker_20230816-072743-utc.log Worker_20230816-192813-utc.log Worker_20230816-203428-utc.log Worker_20230816-215930-utc.log Worker_20230817-072804-utc.log Worker_20230817-104400-utc.log
2023-08-17T10:51:52.5422169Z + LOG_FILE=Worker_20230817-104400-utc.log
2023-08-17T10:51:52.5422847Z + '[' -n Worker_20230817-104400-utc.log ']'
2023-08-17T10:51:52.5423477Z + case ${RUNNER_OS} in
2023-08-17T10:51:52.5429782Z ++ grep -r '"jobDisplayName"' Worker_20230817-104400-utc.log
2023-08-17T10:51:52.5430178Z ++ awk -F '[:]' '{print $2}'
2023-08-17T10:51:52.5431714Z ++ sed 's/"//g'
2023-08-17T10:51:52.5432345Z ++ xargs
2023-08-17T10:51:52.5464103Z + JOB_NAME='cuda12.1-py3.10-gcc9-sm80 / test (inductor_huggingface_perf, 3, 3, linux.gcp.a100.large),'
2023-08-17T10:51:52.5464793Z + echo 'job-name=cuda12.1-py3.10-gcc9-sm80 / test (inductor_huggingface_perf, 3, 3, linux.gcp.a100.large),'
2023-08-17T10:51:52.5465472Z + popd
2023-08-17T10:51:52.5466052Z /home/weiwangmeta/actions-runner/_work/pytorch/pytorch
2023-08-17T10:51:52.5485815Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}"
2023-08-17T10:51:52.5486168Z echo "Workflow: ${GITHUB_WORKFLOW}"
2023-08-17T10:51:52.5486457Z echo "Job name: ${JOB_NAME}"
2023-08-17T10:51:52.5486698Z 
2023-08-17T10:51:52.5487023Z # Use relative path here as this could be checked out anywhere, not necessarily
2023-08-17T10:51:52.5487370Z # in runner workspace
2023-08-17T10:51:52.5487710Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \
2023-08-17T10:51:52.5488063Z   --workflow "${GITHUB_WORKFLOW}" \
2023-08-17T10:51:52.5488489Z   --job-name "${JOB_NAME}" \
2023-08-17T10:51:52.5490077Z   --test-matrix "{"include": [{"config": "inductor_huggingface_perf", "shard": 1, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_huggingface_perf", "shard": 2, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_huggingface_perf", "shard": 3, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 1, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 2, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 3, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 4, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 5, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 1, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 2, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 3, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 4, "num_shards": 4, "runner": "linux.gcp.a100.large"}]}" \
2023-08-17T10:51:52.5491817Z   --pr-number "" \
2023-08-17T10:51:52.5492057Z   --tag "" \
2023-08-17T10:51:52.5492297Z   --event-name "schedule" \
2023-08-17T10:51:52.5492564Z   --schedule "0 7 * * *" \
2023-08-17T10:51:52.5492808Z   --branch ""
2023-08-17T10:51:52.5509573Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-08-17T10:51:52.5509856Z env:
2023-08-17T10:51:52.5510079Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:51:52.5510391Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T10:51:52.5510873Z   GITHUB_TOKEN: ***
2023-08-17T10:51:52.5511244Z   JOB_NAME: cuda12.1-py3.10-gcc9-sm80 / test (inductor_huggingface_perf, 3, 3, linux.gcp.a100.large),
2023-08-17T10:51:52.5511592Z ##[endgroup]
2023-08-17T10:51:52.5546191Z Workflow: inductor-A100-perf-nightly
2023-08-17T10:51:52.5546781Z Job name: cuda12.1-py3.10-gcc9-sm80 / test (inductor_huggingface_perf, 3, 3, linux.gcp.a100.large),
2023-08-17T10:51:52.9773064Z fatal: unknown commit main
2023-08-17T10:51:52.9799222Z /home/weiwangmeta/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/filter_test_configs.py:471: UserWarning: failed to get commit messages: Command '['git', 'cherry', '-v', 'main']' returned non-zero exit status 128.
2023-08-17T10:51:52.9799897Z   warnings.warn(f"failed to get commit messages: {e}")
2023-08-17T10:51:52.9930632Z ##[group]Run echo "Filtered matrix:"
2023-08-17T10:51:52.9931190Z echo "Filtered matrix:"
2023-08-17T10:51:52.9932913Z echo "{"include": [{"config": "inductor_huggingface_perf", "shard": 1, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_huggingface_perf", "shard": 2, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_huggingface_perf", "shard": 3, "num_shards": 3, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 1, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 2, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 3, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 4, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_timm_perf", "shard": 5, "num_shards": 5, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 1, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 2, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 3, "num_shards": 4, "runner": "linux.gcp.a100.large"}, {"config": "inductor_torchbench_perf", "shard": 4, "num_shards": 4, "runner": "linux.gcp.a100.large"}]}"
2023-08-17T10:51:52.9934606Z 
2023-08-17T10:51:52.9934800Z echo
2023-08-17T10:51:52.9935072Z echo "Is the current job unstable? False"
2023-08-17T10:51:52.9935336Z 
2023-08-17T10:51:52.9935540Z echo
2023-08-17T10:51:52.9935786Z echo "Is keep-going label set? False"
2023-08-17T10:51:52.9936046Z 
2023-08-17T10:51:52.9936247Z echo
2023-08-17T10:51:52.9936483Z echo "Renabled issues? "
2023-08-17T10:51:52.9955117Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-08-17T10:51:52.9955410Z env:
2023-08-17T10:51:52.9955669Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:51:52.9956001Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T10:51:52.9956274Z ##[endgroup]
2023-08-17T10:51:52.9993719Z Filtered matrix:
2023-08-17T10:51:52.9996680Z {include: [{config: inductor_huggingface_perf, shard: 1, num_shards: 3, runner: linux.gcp.a100.large}, {config: inductor_huggingface_perf, shard: 2, num_shards: 3, runner: linux.gcp.a100.large}, {config: inductor_huggingface_perf, shard: 3, num_shards: 3, runner: linux.gcp.a100.large}, {config: inductor_timm_perf, shard: 1, num_shards: 5, runner: linux.gcp.a100.large}, {config: inductor_timm_perf, shard: 2, num_shards: 5, runner: linux.gcp.a100.large}, {config: inductor_timm_perf, shard: 3, num_shards: 5, runner: linux.gcp.a100.large}, {config: inductor_timm_perf, shard: 4, num_shards: 5, runner: linux.gcp.a100.large}, {config: inductor_timm_perf, shard: 5, num_shards: 5, runner: linux.gcp.a100.large}, {config: inductor_torchbench_perf, shard: 1, num_shards: 4, runner: linux.gcp.a100.large}, {config: inductor_torchbench_perf, shard: 2, num_shards: 4, runner: linux.gcp.a100.large}, {config: inductor_torchbench_perf, shard: 3, num_shards: 4, runner: linux.gcp.a100.large}, {config: inductor_torchbench_perf, shard: 4, num_shards: 4, runner: linux.gcp.a100.large}]}
2023-08-17T10:51:52.9999418Z 
2023-08-17T10:51:52.9999680Z Is the current job unstable? False
2023-08-17T10:51:52.9999986Z 
2023-08-17T10:51:53.0000447Z Is keep-going label set? False
2023-08-17T10:51:53.0000731Z 
2023-08-17T10:51:53.0000873Z Renabled issues? 
2023-08-17T10:51:53.0050707Z Prepare all required actions
2023-08-17T10:51:53.0051581Z Getting action download info
2023-08-17T10:51:53.2257710Z ##[group]Run ./.github/actions/pytest-cache-download
2023-08-17T10:51:53.2258025Z with:
2023-08-17T10:51:53.2258239Z   cache_dir: .pytest_cache
2023-08-17T10:51:53.2258688Z   job_identifier: inductor-A100-perf-nightly_linux-focal-cuda12.1-py3.10-gcc9-sm80_test_inductor_huggingface_perf
2023-08-17T10:51:53.2259087Z env:
2023-08-17T10:51:53.2259303Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:51:53.2259796Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T10:51:53.2260082Z ##[endgroup]
2023-08-17T10:51:53.2291129Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2023-08-17T10:51:53.2291426Z with:
2023-08-17T10:51:53.2291633Z   shell: bash
2023-08-17T10:51:53.2291863Z   timeout_minutes: 5
2023-08-17T10:51:53.2292081Z   max_attempts: 5
2023-08-17T10:51:53.2292316Z   retry_wait_seconds: 30
2023-08-17T10:51:53.2292679Z   command: set -eu
python3 -m pip install boto3==1.19.12

2023-08-17T10:51:53.2293040Z   polling_interval_seconds: 1
2023-08-17T10:51:53.2293295Z   warning_on_retry: true
2023-08-17T10:51:53.2293542Z   continue_on_error: false
2023-08-17T10:51:53.2293765Z env:
2023-08-17T10:51:53.2293968Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:51:53.2294275Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T10:51:53.2294562Z ##[endgroup]
2023-08-17T10:51:54.4221736Z Requirement already satisfied: boto3==1.19.12 in /home/ubuntu/.local/lib/python3.8/site-packages (1.19.12)
2023-08-17T10:51:54.4279953Z Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/ubuntu/.local/lib/python3.8/site-packages (from boto3==1.19.12) (0.10.0)
2023-08-17T10:51:54.4299928Z Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /home/ubuntu/.local/lib/python3.8/site-packages (from boto3==1.19.12) (0.5.2)
2023-08-17T10:51:54.4332071Z Requirement already satisfied: botocore<1.23.0,>=1.22.12 in /home/ubuntu/.local/lib/python3.8/site-packages (from boto3==1.19.12) (1.22.12)
2023-08-17T10:51:54.4379053Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3/dist-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.25.8)
2023-08-17T10:51:54.4437746Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ubuntu/.local/lib/python3.8/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (2.8.2)
2023-08-17T10:51:54.4460317Z Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.14.0)
2023-08-17T10:51:55.2974291Z Command completed after 1 attempt(s).
2023-08-17T10:51:55.3387314Z ##[group]Run python3 .github/scripts/pytest_cache.py \
2023-08-17T10:51:55.3387852Z python3 .github/scripts/pytest_cache.py \
2023-08-17T10:51:55.3388277Z   --download \
2023-08-17T10:51:55.3388701Z   --cache_dir $GITHUB_WORKSPACE/$CACHE_DIR \
2023-08-17T10:51:55.3389166Z   --pr_identifier $GITHUB_REF \
2023-08-17T10:51:55.3389584Z   --job_identifier $JOB_IDENTIFIER \
2023-08-17T10:51:55.3390002Z   --temp_dir $RUNNER_TEMP \
2023-08-17T10:51:55.3390367Z   --repo $REPO \
2023-08-17T10:51:55.3417778Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-08-17T10:51:55.3418256Z env:
2023-08-17T10:51:55.3418654Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:51:55.3419207Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T10:51:55.3419737Z   CACHE_DIR: .pytest_cache
2023-08-17T10:51:55.3420514Z   JOB_IDENTIFIER: inductor-A100-perf-nightly_linux-focal-cuda12.1-py3.10-gcc9-sm80_test_inductor_huggingface_perf
2023-08-17T10:51:55.3421251Z   REPO: pytorch/pytorch
2023-08-17T10:51:55.3421658Z ##[endgroup]
2023-08-17T10:51:55.8009695Z PR identifier for `refs/heads/main` is `96e092540d6b3c4076e3d2bc6f1f9013`
2023-08-17T10:51:55.8011768Z Downloading cache with args Namespace(bucket=None, cache_dir='/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/.pytest_cache', download=True, job_identifier='inductor-A100-perf-nightly_linux-focal-cuda12.1-py3.10-gcc9-sm80_test_inductor_huggingface_perf', pr_identifier='refs/heads/main', repo='pytorch/pytorch', shard=None, temp_dir='/home/weiwangmeta/actions-runner/_work/_temp', upload=False)
2023-08-17T10:51:55.8013283Z There were no files matching the prefix `pytest_cache/pytorch/pytorch/96e092540d6b3c4076e3d2bc6f1f9013/inductor-A100-perf-nightly_linux-focal-cuda12_1-py3_10-gcc9-sm80_test_inductor_huggingface_perf` in bucket `gha-artifacts`
2023-08-17T10:51:55.9058952Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2023-08-17T10:51:55.9059376Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2023-08-17T10:51:55.9077565Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-08-17T10:51:55.9077861Z env:
2023-08-17T10:51:55.9078073Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:51:55.9078446Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T10:51:55.9078739Z   JOB_TIMEOUT: 720
2023-08-17T10:51:55.9078964Z ##[endgroup]
2023-08-17T10:51:56.0194424Z ##[group]Run set -x
2023-08-17T10:51:56.0194774Z set -x
2023-08-17T10:51:56.0194991Z 
2023-08-17T10:51:56.0195249Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2023-08-17T10:51:56.0195565Z   TEST_COMMAND=.ci/pytorch/multigpu-test.sh
2023-08-17T10:51:56.0195906Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2023-08-17T10:51:56.0196206Z   TEST_COMMAND=.ci/onnx/test.sh
2023-08-17T10:51:56.0196472Z else
2023-08-17T10:51:56.0196715Z   TEST_COMMAND=.ci/pytorch/test.sh
2023-08-17T10:51:56.0196967Z fi
2023-08-17T10:51:56.0197167Z 
2023-08-17T10:51:56.0197466Z # detached container should get cleaned up by teardown_ec2_linux
2023-08-17T10:51:56.0197873Z # TODO: Stop building test binaries as part of the build phase
2023-08-17T10:51:56.0198304Z # Used for GPU_FLAG since that doesn't play nice
2023-08-17T10:51:56.0198694Z # shellcheck disable=SC2086,SC2090
2023-08-17T10:51:56.0198991Z container_name=$(docker run \
2023-08-17T10:51:56.0199243Z   ${GPU_FLAG:-} \
2023-08-17T10:51:56.0199505Z   -e BUILD_ENVIRONMENT \
2023-08-17T10:51:56.0199768Z   -e PR_NUMBER \
2023-08-17T10:51:56.0200052Z   -e GITHUB_ACTIONS \
2023-08-17T10:51:56.0200302Z   -e GITHUB_REPOSITORY \
2023-08-17T10:51:56.0200573Z   -e GITHUB_WORKFLOW \
2023-08-17T10:51:56.0200838Z   -e GITHUB_JOB \
2023-08-17T10:51:56.0201089Z   -e GITHUB_RUN_ID \
2023-08-17T10:51:56.0201333Z   -e GITHUB_RUN_NUMBER \
2023-08-17T10:51:56.0201599Z   -e GITHUB_RUN_ATTEMPT \
2023-08-17T10:51:56.0201860Z   -e BASE_SHA \
2023-08-17T10:51:56.0202097Z   -e BRANCH \
2023-08-17T10:51:56.0202310Z   -e SHA1 \
2023-08-17T10:51:56.0202777Z   -e AWS_DEFAULT_REGION \
2023-08-17T10:51:56.0203047Z   -e IN_WHEEL_TEST \
2023-08-17T10:51:56.0203291Z   -e SHARD_NUMBER \
2023-08-17T10:51:56.0203545Z   -e TEST_CONFIG \
2023-08-17T10:51:56.0203801Z   -e NUM_TEST_SHARDS \
2023-08-17T10:51:56.0204060Z   -e REENABLED_ISSUES \
2023-08-17T10:51:56.0204324Z   -e CONTINUE_THROUGH_ERROR \
2023-08-17T10:51:56.0204621Z   -e PYTORCH_RETRY_TEST_CASES \
2023-08-17T10:51:56.0204928Z   -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \
2023-08-17T10:51:56.0205213Z   -e PR_LABELS \
2023-08-17T10:51:56.0205493Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2023-08-17T10:51:56.0205757Z   -e SCCACHE_BUCKET \
2023-08-17T10:51:56.0206023Z   -e SCCACHE_S3_KEY_PREFIX \
2023-08-17T10:51:56.0206304Z   -e XLA_CUDA \
2023-08-17T10:51:56.0206579Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2023-08-17T10:51:56.0206873Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2023-08-17T10:51:56.0207189Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2023-08-17T10:51:56.0207514Z   -e SKIP_SCCACHE_INITIALIZATION=1 \
2023-08-17T10:51:56.0207820Z   -e HUGGING_FACE_HUB_TOKEN \
2023-08-17T10:51:56.0208085Z   -e DASHBOARD_TAG \
2023-08-17T10:51:56.0208464Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2023-08-17T10:51:56.0208781Z   --ulimit stack=10485760:83886080 \
2023-08-17T10:51:56.0209083Z   --security-opt seccomp=unconfined \
2023-08-17T10:51:56.0209364Z   --cap-add=SYS_PTRACE \
2023-08-17T10:51:56.0209952Z   --ipc=host \
2023-08-17T10:51:56.0210233Z   --shm-size="${SHM_SIZE}" \
2023-08-17T10:51:56.0210490Z   --tty \
2023-08-17T10:51:56.0210704Z   --detach \
2023-08-17T10:51:56.0210968Z   --name="${container_name}" \
2023-08-17T10:51:56.0211234Z   --user jenkins \
2023-08-17T10:51:56.0211542Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2023-08-17T10:51:56.0211880Z   -w /var/lib/jenkins/workspace \
2023-08-17T10:51:56.0212308Z   "${DOCKER_IMAGE}"
2023-08-17T10:51:56.0212546Z )
2023-08-17T10:51:56.0212844Z # Propagate download.pytorch.org IP to container
2023-08-17T10:51:56.0213285Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts"
2023-08-17T10:51:56.0213737Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}"
2023-08-17T10:51:56.0214190Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"
2023-08-17T10:51:56.0232672Z shell: /usr/bin/bash -e {0}
2023-08-17T10:51:56.0232922Z env:
2023-08-17T10:51:56.0233134Z   GIT_DEFAULT_BRANCH: main
2023-08-17T10:51:56.0233458Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T10:51:56.0233847Z   BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm80
2023-08-17T10:51:56.0234159Z   PR_NUMBER: 
2023-08-17T10:51:56.0234417Z   GITHUB_REPOSITORY: pytorch/pytorch
2023-08-17T10:51:56.0234741Z   GITHUB_WORKFLOW: inductor-A100-perf-nightly
2023-08-17T10:51:56.0235025Z   GITHUB_JOB: test
2023-08-17T10:51:56.0235260Z   GITHUB_RUN_ID: 5887783871
2023-08-17T10:51:56.0235508Z   GITHUB_RUN_NUMBER: 974
2023-08-17T10:51:56.0235739Z   GITHUB_RUN_ATTEMPT: 1
2023-08-17T10:51:56.0235971Z   BRANCH: main
2023-08-17T10:51:56.0236236Z   SHA1: 384e0d104fd077d31efafc564129660e9b7a0f25
2023-08-17T10:51:56.0236552Z   BASE_SHA: 384e0d104fd077d31efafc564129660e9b7a0f25
2023-08-17T10:51:56.0236837Z   PYTORCH_RETRY_TEST_CASES: 1
2023-08-17T10:51:56.0237113Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2023-08-17T10:51:56.0237409Z   TEST_CONFIG: inductor_huggingface_perf
2023-08-17T10:51:56.0237672Z   SHARD_NUMBER: 3
2023-08-17T10:51:56.0237888Z   NUM_TEST_SHARDS: 3
2023-08-17T10:51:56.0238121Z   REENABLED_ISSUES: 
2023-08-17T10:51:56.0238430Z   CONTINUE_THROUGH_ERROR: False
2023-08-17T10:51:56.0238747Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2023-08-17T10:51:56.0239128Z   SCCACHE_S3_KEY_PREFIX: inductor-A100-perf-nightly
2023-08-17T10:51:56.0239423Z   SHM_SIZE: 2g
2023-08-17T10:51:56.0239996Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:f89d62c8626a24aae4b1a64d42a3caf9cd95bddd
2023-08-17T10:51:56.0240551Z   XLA_CUDA: 
2023-08-17T10:51:56.0240882Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2023-08-17T10:51:56.0241268Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2023-08-17T10:51:56.0241564Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2023-08-17T10:51:56.0242257Z   DASHBOARD_TAG: training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true
2023-08-17T10:51:56.0243239Z   HUGGING_FACE_HUB_TOKEN: ***
2023-08-17T10:51:56.0243472Z ##[endgroup]
2023-08-17T10:51:56.0811671Z + [[ inductor_huggingface_perf == \m\u\l\t\i\g\p\u ]]
2023-08-17T10:51:56.0812581Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *onnx* ]]
2023-08-17T10:51:56.0813157Z + TEST_COMMAND=.ci/pytorch/test.sh
2023-08-17T10:51:56.0823338Z +++ nproc --ignore=2
2023-08-17T10:51:56.0838623Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=10 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_5887783871 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/weiwangmeta/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:f89d62c8626a24aae4b1a64d42a3caf9cd95bddd
2023-08-17T10:51:59.9425347Z + container_name=8fe9204520fb1e14a5fbb249a5da40cb451cc64386cf92c59420d7165072e296
2023-08-17T10:51:59.9427869Z + grep download.pytorch.org /etc/hosts
2023-08-17T10:51:59.9429962Z + docker exec -i 8fe9204520fb1e14a5fbb249a5da40cb451cc64386cf92c59420d7165072e296 sudo bash -c '/bin/cat >> /etc/hosts'
2023-08-17T10:52:00.0177094Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted
2023-08-17T10:52:00.0237637Z + echo DOCKER_CONTAINER_ID=8fe9204520fb1e14a5fbb249a5da40cb451cc64386cf92c59420d7165072e296
2023-08-17T10:52:00.0243879Z ++ echo dist/torch-2.1.0a0+git384e0d1-cp310-cp310-linux_x86_64.whl
2023-08-17T10:52:00.0246657Z + docker exec -t 8fe9204520fb1e14a5fbb249a5da40cb451cc64386cf92c59420d7165072e296 sh -c 'pip install dist/torch-2.1.0a0+git384e0d1-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh'
2023-08-17T10:52:00.5423049Z Processing ./dist/torch-2.1.0a0+git384e0d1-cp310-cp310-linux_x86_64.whl
2023-08-17T10:52:01.5041334Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.1.0a0+git384e0d1) (3.9.0)
2023-08-17T10:52:01.5043249Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.1.0a0+git384e0d1) (4.7.1)
2023-08-17T10:52:01.5047352Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.1.0a0+git384e0d1) (1.12)
2023-08-17T10:52:01.5053341Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.1.0a0+git384e0d1) (2.8.8)
2023-08-17T10:52:01.5056884Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.1.0a0+git384e0d1) (3.1.2)
2023-08-17T10:52:01.5060283Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.1.0a0+git384e0d1) (2023.4.0)
2023-08-17T10:52:01.5072894Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.1.0a0+git384e0d1) (3.3.0)
2023-08-17T10:52:01.5144815Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.1.0a0+git384e0d1) (1.21.2)
2023-08-17T10:52:01.5660086Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.1.0a0+git384e0d1) (2.1.3)
2023-08-17T10:52:01.5862388Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.1.0a0+git384e0d1) (1.3.0)
2023-08-17T10:52:03.0187234Z Installing collected packages: torch
2023-08-17T10:52:13.0370063Z Successfully installed torch-2.1.0a0+git384e0d1
2023-08-17T10:52:13.1303585Z + echo 'Environment variables:'
2023-08-17T10:52:13.1303922Z Environment variables:
2023-08-17T10:52:13.1304141Z + env
2023-08-17T10:52:13.1315093Z INSTALLED_DB=yes
2023-08-17T10:52:13.1315776Z NV_LIBCUBLAS_VERSION=12.1.0.26-1
2023-08-17T10:52:13.1316237Z NVIDIA_VISIBLE_DEVICES=all
2023-08-17T10:52:13.1317596Z NV_NVML_DEV_VERSION=12.1.55-1
2023-08-17T10:52:13.1318860Z GITHUB_WORKSPACE=/home/weiwangmeta/actions-runner/_work/pytorch/pytorch
2023-08-17T10:52:13.1320464Z CONTINUE_THROUGH_ERROR=False
2023-08-17T10:52:13.1322277Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1
2023-08-17T10:52:13.1323272Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1
2023-08-17T10:52:13.1324065Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm80
2023-08-17T10:52:13.1324604Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2023-08-17T10:52:13.1325077Z HOSTNAME=8fe9204520fb
2023-08-17T10:52:13.1326155Z GITHUB_PATH=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/add_path_c0f1af99-9d59-44d3-885d-e11f042906cf
2023-08-17T10:52:13.1327318Z GITHUB_ACTION=__self
2023-08-17T10:52:13.1327756Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2023-08-17T10:52:13.1331379Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=510,driver<511 brand=unknown,driver>=510,driver<511 brand=nvidia,driver>=510,driver<511 brand=nvidiartx,driver>=510,driver<511 brand=geforce,driver>=510,driver<511 brand=geforcertx,driver>=510,driver<511 brand=quadro,driver>=510,driver<511 brand=quadrortx,driver>=510,driver<511 brand=titan,driver>=510,driver<511 brand=titanrtx,driver>=510,driver<511 brand=tesla,driver>=515,driver<516 brand=unknown,driver>=515,driver<516 brand=nvidia,driver>=515,driver<516 brand=nvidiartx,driver>=515,driver<516 brand=geforce,driver>=515,driver<516 brand=geforcertx,driver>=515,driver<516 brand=quadro,driver>=515,driver<516 brand=quadrortx,driver>=515,driver<516 brand=titan,driver>=515,driver<516 brand=titanrtx,driver>=515,driver<516 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526
2023-08-17T10:52:13.1335976Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.0.26-1
2023-08-17T10:52:13.1336585Z NV_NVTX_VERSION=12.1.66-1
2023-08-17T10:52:13.1336930Z GITHUB_RUN_NUMBER=974
2023-08-17T10:52:13.1337426Z TEST_CONFIG=inductor_huggingface_perf
2023-08-17T10:52:13.1337904Z GITHUB_REPOSITORY_OWNER_ID=21003710
2023-08-17T10:52:13.1338535Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2023-08-17T10:52:13.1339098Z NV_CUDA_CUDART_DEV_VERSION=12.1.55-1
2023-08-17T10:52:13.1339606Z NV_LIBCUSPARSE_VERSION=12.0.2.55-1
2023-08-17T10:52:13.1340164Z NV_LIBNPP_VERSION=12.0.2.50-1
2023-08-17T10:52:13.1340562Z GITHUB_TRIGGERING_ACTOR=malfet
2023-08-17T10:52:13.1341129Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2023-08-17T10:52:13.1341574Z GITHUB_REF_TYPE=branch
2023-08-17T10:52:13.1342034Z TORCH_CUDA_ARCH_LIST=Maxwell
2023-08-17T10:52:13.1342478Z NCCL_VERSION=2.17.1-1
2023-08-17T10:52:13.1342972Z BASE_SHA=384e0d104fd077d31efafc564129660e9b7a0f25
2023-08-17T10:52:13.1343485Z XLA_CUDA=
2023-08-17T10:52:13.1344139Z HUGGING_FACE_HUB_TOKEN=***
2023-08-17T10:52:13.1347988Z ***
2023-08-17T10:52:13.1348411Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2023-08-17T10:52:13.1348845Z GITHUB_REPOSITORY_ID=65600975
2023-08-17T10:52:13.1349307Z GITHUB_ACTIONS=true
2023-08-17T10:52:13.1349713Z NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T10:52:13.1350416Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.55-1
2023-08-17T10:52:13.1351106Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.0.2.50-1
2023-08-17T10:52:13.1351645Z SHA1=384e0d104fd077d31efafc564129660e9b7a0f25
2023-08-17T10:52:13.1352259Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2023-08-17T10:52:13.1352836Z GITHUB_SHA=384e0d104fd077d31efafc564129660e9b7a0f25
2023-08-17T10:52:13.1353855Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-perf-test-nightly.yml@refs/heads/main
2023-08-17T10:52:13.1354984Z UCC_HOME=/usr
2023-08-17T10:52:13.1355505Z NV_LIBCUBLAS_DEV_VERSION=12.1.0.26-1
2023-08-17T10:52:13.1355947Z NVIDIA_PRODUCT_NAME=CUDA
2023-08-17T10:52:13.1356532Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1
2023-08-17T10:52:13.1356949Z GITHUB_REF=refs/heads/main
2023-08-17T10:52:13.1357484Z NV_CUDA_CUDART_VERSION=12.1.55-1
2023-08-17T10:52:13.1357743Z SHARD_NUMBER=3
2023-08-17T10:52:13.1357970Z GITHUB_REF_PROTECTED=true
2023-08-17T10:52:13.1358258Z HOME=/var/lib/jenkins
2023-08-17T10:52:13.1358963Z GITHUB_API_URL=https://api.github.com
2023-08-17T10:52:13.1359284Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2023-08-17T10:52:13.1359596Z UCX_COMMIT=00bcc6bb18fc282eb160623b4c0d300147f579af
2023-08-17T10:52:13.1360044Z SCCACHE_S3_KEY_PREFIX=inductor-A100-perf-nightly
2023-08-17T10:52:13.1360346Z NVIDIA_CUDA_END_OF_LIFE=1
2023-08-17T10:52:13.1360674Z CUDA_VERSION=12.1.0
2023-08-17T10:52:13.1361074Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.0.26-1
2023-08-17T10:52:13.1361344Z NUM_TEST_SHARDS=3
2023-08-17T10:52:13.1361577Z UCX_HOME=/usr
2023-08-17T10:52:13.1361991Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.0-1
2023-08-17T10:52:13.1362896Z GITHUB_STATE=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/save_state_c0f1af99-9d59-44d3-885d-e11f042906cf
2023-08-17T10:52:13.1363367Z PYTORCH_RETRY_TEST_CASES=1
2023-08-17T10:52:13.1363936Z GITHUB_ENV=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/set_env_c0f1af99-9d59-44d3-885d-e11f042906cf
2023-08-17T10:52:13.1364684Z GITHUB_EVENT_PATH=/home/weiwangmeta/actions-runner/_work/_temp/_github_workflow/event.json
2023-08-17T10:52:13.1365153Z GITHUB_EVENT_NAME=schedule
2023-08-17T10:52:13.1365958Z DASHBOARD_TAG=training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true
2023-08-17T10:52:13.1366613Z GITHUB_RUN_ID=5887783871
2023-08-17T10:52:13.1366985Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.0.2.50-1
2023-08-17T10:52:13.1367371Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1
2023-08-17T10:52:13.1367990Z GITHUB_STEP_SUMMARY=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/step_summary_c0f1af99-9d59-44d3-885d-e11f042906cf
2023-08-17T10:52:13.1368396Z GITHUB_ACTOR=malfet
2023-08-17T10:52:13.1368684Z NV_LIBNPP_DEV_VERSION=12.0.2.50-1
2023-08-17T10:52:13.1368928Z PR_NUMBER=
2023-08-17T10:52:13.1369140Z GITHUB_RUN_ATTEMPT=1
2023-08-17T10:52:13.1369397Z ANACONDA_PYTHON_VERSION=3.10
2023-08-17T10:52:13.1369719Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2023-08-17T10:52:13.1370000Z TERM=xterm
2023-08-17T10:52:13.1370285Z NV_LIBCUSPARSE_DEV_VERSION=12.0.2.55-1
2023-08-17T10:52:13.1370552Z INSTALLED_VISION=yes
2023-08-17T10:52:13.1370783Z BRANCH=main
2023-08-17T10:52:13.1371006Z OPENSSL_ROOT_DIR=/opt/openssl
2023-08-17T10:52:13.1371288Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2023-08-17T10:52:13.1371605Z CUDA_PATH=/usr/local/cuda
2023-08-17T10:52:13.1372252Z GITHUB_ACTION_PATH=/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2023-08-17T10:52:13.1372658Z GITHUB_SERVER_URL=https://github.com
2023-08-17T10:52:13.1373044Z UCC_COMMIT=7cb07a76ccedad7e56ceb136b865eb9319c258ea
2023-08-17T10:52:13.1373337Z REENABLED_ISSUES=
2023-08-17T10:52:13.1373559Z SHLVL=1
2023-08-17T10:52:13.1373749Z MAX_JOBS=10
2023-08-17T10:52:13.1374013Z NV_CUDA_LIB_VERSION=12.1.0-1
2023-08-17T10:52:13.1374259Z NVARCH=x86_64
2023-08-17T10:52:13.1374490Z GITHUB_ACTOR_ID=2453524
2023-08-17T10:52:13.1374780Z GITHUB_WORKFLOW_SHA=384e0d104fd077d31efafc564129660e9b7a0f25
2023-08-17T10:52:13.1375073Z GITHUB_REF_NAME=main
2023-08-17T10:52:13.1375410Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1
2023-08-17T10:52:13.1375896Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2023-08-17T10:52:13.1376225Z GITHUB_JOB=test
2023-08-17T10:52:13.1376559Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1
2023-08-17T10:52:13.1376892Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2023-08-17T10:52:13.1377249Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.0-1
2023-08-17T10:52:13.1377796Z GITHUB_REPOSITORY=pytorch/pytorch
2023-08-17T10:52:13.1378104Z NV_NVPROF_VERSION=12.1.55-1
2023-08-17T10:52:13.1378363Z GITHUB_RETENTION_DAYS=90
2023-08-17T10:52:13.1378615Z OPENSSL_DIR=/opt/openssl
2023-08-17T10:52:13.1378858Z GITHUB_ACTION_REPOSITORY=
2023-08-17T10:52:13.1379311Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-08-17T10:52:13.1379719Z GITHUB_BASE_REF=
2023-08-17T10:52:13.1380101Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2023-08-17T10:52:13.1380330Z CI=true
2023-08-17T10:52:13.1380605Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1
2023-08-17T10:52:13.1380889Z GITHUB_REPOSITORY_OWNER=pytorch
2023-08-17T10:52:13.1381153Z INSTALLED_PROTOBUF=yes
2023-08-17T10:52:13.1381380Z GITHUB_HEAD_REF=
2023-08-17T10:52:13.1381612Z GITHUB_ACTION_REF=
2023-08-17T10:52:13.1381988Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2023-08-17T10:52:13.1382415Z GITHUB_WORKFLOW=inductor-A100-perf-nightly
2023-08-17T10:52:13.1382715Z DEBIAN_FRONTEND=noninteractive
2023-08-17T10:52:13.1383348Z GITHUB_OUTPUT=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/set_output_c0f1af99-9d59-44d3-885d-e11f042906cf
2023-08-17T10:52:13.1383758Z SKIP_SCCACHE_INITIALIZATION=1
2023-08-17T10:52:13.1384008Z _=/usr/bin/env
2023-08-17T10:52:13.1384382Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2023-08-17T10:52:13.1526531Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch
2023-08-17T10:52:13.1527285Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2023-08-17T10:52:13.1527810Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib
2023-08-17T10:52:13.1528441Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test
2023-08-17T10:52:13.1528994Z + BUILD_DIR=build
2023-08-17T10:52:13.1529446Z + BUILD_RENAMED_DIR=build_renamed
2023-08-17T10:52:13.1529714Z + BUILD_BIN_DIR=build/bin
2023-08-17T10:52:13.1529955Z + export VALGRIND=ON
2023-08-17T10:52:13.1530189Z + VALGRIND=ON
2023-08-17T10:52:13.1530652Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *clang9* ]]
2023-08-17T10:52:13.1531139Z + [[ 0 == \1 ]]
2023-08-17T10:52:13.1531496Z + [[ False == \1 ]]
2023-08-17T10:52:13.1532048Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 != *bazel* ]]
2023-08-17T10:52:13.1537130Z ++ realpath build/custom_test_artifacts
2023-08-17T10:52:13.1548325Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2023-08-17T10:52:13.1552141Z ++ dirname .ci/pytorch/test.sh
2023-08-17T10:52:13.1561485Z + source .ci/pytorch/common.sh
2023-08-17T10:52:13.1566467Z +++ dirname .ci/pytorch/common.sh
2023-08-17T10:52:13.1575670Z ++ source .ci/pytorch/common_utils.sh
2023-08-17T10:52:13.1579098Z +++ declare -f -t trap_add
2023-08-17T10:52:13.1587547Z ++ set -ex
2023-08-17T10:52:13.1588109Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *rocm* ]]
2023-08-17T10:52:13.1588492Z ++ BUILD_TEST_LIBTORCH=0
2023-08-17T10:52:13.1588832Z + echo 'Environment variables'
2023-08-17T10:52:13.1589091Z Environment variables
2023-08-17T10:52:13.1589301Z + env
2023-08-17T10:52:13.1597993Z INSTALLED_DB=yes
2023-08-17T10:52:13.1598626Z NV_LIBCUBLAS_VERSION=12.1.0.26-1
2023-08-17T10:52:13.1599083Z NVIDIA_VISIBLE_DEVICES=all
2023-08-17T10:52:13.1599572Z NV_NVML_DEV_VERSION=12.1.55-1
2023-08-17T10:52:13.1600441Z GITHUB_WORKSPACE=/home/weiwangmeta/actions-runner/_work/pytorch/pytorch
2023-08-17T10:52:13.1601060Z CONTINUE_THROUGH_ERROR=False
2023-08-17T10:52:13.1601804Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1
2023-08-17T10:52:13.1602436Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1
2023-08-17T10:52:13.1603442Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm80
2023-08-17T10:52:13.1604091Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2023-08-17T10:52:13.1604439Z HOSTNAME=8fe9204520fb
2023-08-17T10:52:13.1605014Z GITHUB_PATH=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/add_path_c0f1af99-9d59-44d3-885d-e11f042906cf
2023-08-17T10:52:13.1605686Z GITHUB_ACTION=__self
2023-08-17T10:52:13.1605951Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2023-08-17T10:52:13.1608961Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=510,driver<511 brand=unknown,driver>=510,driver<511 brand=nvidia,driver>=510,driver<511 brand=nvidiartx,driver>=510,driver<511 brand=geforce,driver>=510,driver<511 brand=geforcertx,driver>=510,driver<511 brand=quadro,driver>=510,driver<511 brand=quadrortx,driver>=510,driver<511 brand=titan,driver>=510,driver<511 brand=titanrtx,driver>=510,driver<511 brand=tesla,driver>=515,driver<516 brand=unknown,driver>=515,driver<516 brand=nvidia,driver>=515,driver<516 brand=nvidiartx,driver>=515,driver<516 brand=geforce,driver>=515,driver<516 brand=geforcertx,driver>=515,driver<516 brand=quadro,driver>=515,driver<516 brand=quadrortx,driver>=515,driver<516 brand=titan,driver>=515,driver<516 brand=titanrtx,driver>=515,driver<516 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526
2023-08-17T10:52:13.1611946Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.0.26-1
2023-08-17T10:52:13.1612283Z NV_NVTX_VERSION=12.1.66-1
2023-08-17T10:52:13.1612530Z GITHUB_RUN_NUMBER=974
2023-08-17T10:52:13.1612787Z TEST_CONFIG=inductor_huggingface_perf
2023-08-17T10:52:13.1613078Z GITHUB_REPOSITORY_OWNER_ID=21003710
2023-08-17T10:52:13.1613440Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2023-08-17T10:52:13.1613763Z NV_CUDA_CUDART_DEV_VERSION=12.1.55-1
2023-08-17T10:52:13.1614061Z NV_LIBCUSPARSE_VERSION=12.0.2.55-1
2023-08-17T10:52:13.1614359Z NV_LIBNPP_VERSION=12.0.2.50-1
2023-08-17T10:52:13.1614622Z GITHUB_TRIGGERING_ACTOR=malfet
2023-08-17T10:52:13.1614931Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2023-08-17T10:52:13.1615206Z GITHUB_REF_TYPE=branch
2023-08-17T10:52:13.1615456Z TORCH_CUDA_ARCH_LIST=Maxwell
2023-08-17T10:52:13.1615727Z NCCL_VERSION=2.17.1-1
2023-08-17T10:52:13.1616002Z BASE_SHA=384e0d104fd077d31efafc564129660e9b7a0f25
2023-08-17T10:52:13.1616271Z XLA_CUDA=
2023-08-17T10:52:13.1616665Z HUGGING_FACE_HUB_TOKEN=***
2023-08-17T10:52:13.1616964Z ***
2023-08-17T10:52:13.1617196Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2023-08-17T10:52:13.1617449Z GITHUB_REPOSITORY_ID=65600975
2023-08-17T10:52:13.1617698Z GITHUB_ACTIONS=true
2023-08-17T10:52:13.1617952Z NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T10:52:13.1618339Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.55-1
2023-08-17T10:52:13.1618711Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.0.2.50-1
2023-08-17T10:52:13.1619021Z SHA1=384e0d104fd077d31efafc564129660e9b7a0f25
2023-08-17T10:52:13.1619382Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2023-08-17T10:52:13.1619752Z GITHUB_SHA=384e0d104fd077d31efafc564129660e9b7a0f25
2023-08-17T10:52:13.1620321Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-perf-test-nightly.yml@refs/heads/main
2023-08-17T10:52:13.1620696Z UCC_HOME=/usr
2023-08-17T10:52:13.1620999Z NV_LIBCUBLAS_DEV_VERSION=12.1.0.26-1
2023-08-17T10:52:13.1621427Z NVIDIA_PRODUCT_NAME=CUDA
2023-08-17T10:52:13.1621805Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1
2023-08-17T10:52:13.1622106Z GITHUB_REF=refs/heads/main
2023-08-17T10:52:13.1622401Z NV_CUDA_CUDART_VERSION=12.1.55-1
2023-08-17T10:52:13.1622648Z SHARD_NUMBER=3
2023-08-17T10:52:13.1622875Z GITHUB_REF_PROTECTED=true
2023-08-17T10:52:13.1623124Z HOME=/var/lib/jenkins
2023-08-17T10:52:13.1623409Z GITHUB_API_URL=https://api.github.com
2023-08-17T10:52:13.1623921Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2023-08-17T10:52:13.1624227Z UCX_COMMIT=00bcc6bb18fc282eb160623b4c0d300147f579af
2023-08-17T10:52:13.1624654Z SCCACHE_S3_KEY_PREFIX=inductor-A100-perf-nightly
2023-08-17T10:52:13.1624957Z NVIDIA_CUDA_END_OF_LIFE=1
2023-08-17T10:52:13.1625186Z CUDA_VERSION=12.1.0
2023-08-17T10:52:13.1625546Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.0.26-1
2023-08-17T10:52:13.1625822Z NUM_TEST_SHARDS=3
2023-08-17T10:52:13.1626045Z UCX_HOME=/usr
2023-08-17T10:52:13.1626544Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.0-1
2023-08-17T10:52:13.1627197Z GITHUB_STATE=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/save_state_c0f1af99-9d59-44d3-885d-e11f042906cf
2023-08-17T10:52:13.1627598Z PYTORCH_RETRY_TEST_CASES=1
2023-08-17T10:52:13.1628154Z GITHUB_ENV=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/set_env_c0f1af99-9d59-44d3-885d-e11f042906cf
2023-08-17T10:52:13.1628758Z GITHUB_EVENT_PATH=/home/weiwangmeta/actions-runner/_work/_temp/_github_workflow/event.json
2023-08-17T10:52:13.1629104Z GITHUB_EVENT_NAME=schedule
2023-08-17T10:52:13.1629971Z DASHBOARD_TAG=training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true
2023-08-17T10:52:13.1630623Z GITHUB_RUN_ID=5887783871
2023-08-17T10:52:13.1630994Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.0.2.50-1
2023-08-17T10:52:13.1631365Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1
2023-08-17T10:52:13.1631989Z GITHUB_STEP_SUMMARY=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/step_summary_c0f1af99-9d59-44d3-885d-e11f042906cf
2023-08-17T10:52:13.1632395Z GITHUB_ACTOR=malfet
2023-08-17T10:52:13.1632683Z NV_LIBNPP_DEV_VERSION=12.0.2.50-1
2023-08-17T10:52:13.1632916Z PR_NUMBER=
2023-08-17T10:52:13.1633135Z GITHUB_RUN_ATTEMPT=1
2023-08-17T10:52:13.1633362Z VALGRIND=ON
2023-08-17T10:52:13.1633600Z ANACONDA_PYTHON_VERSION=3.10
2023-08-17T10:52:13.1633904Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2023-08-17T10:52:13.1634191Z TERM=xterm
2023-08-17T10:52:13.1634466Z NV_LIBCUSPARSE_DEV_VERSION=12.0.2.55-1
2023-08-17T10:52:13.1634729Z INSTALLED_VISION=yes
2023-08-17T10:52:13.1634945Z BRANCH=main
2023-08-17T10:52:13.1635182Z OPENSSL_ROOT_DIR=/opt/openssl
2023-08-17T10:52:13.1635462Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2023-08-17T10:52:13.1635730Z CUDA_PATH=/usr/local/cuda
2023-08-17T10:52:13.1636232Z GITHUB_ACTION_PATH=/home/weiwangmeta/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2023-08-17T10:52:13.1636643Z GITHUB_SERVER_URL=https://github.com
2023-08-17T10:52:13.1636971Z UCC_COMMIT=7cb07a76ccedad7e56ceb136b865eb9319c258ea
2023-08-17T10:52:13.1637260Z REENABLED_ISSUES=
2023-08-17T10:52:13.1637465Z SHLVL=1
2023-08-17T10:52:13.1637661Z MAX_JOBS=10
2023-08-17T10:52:13.1637921Z NV_CUDA_LIB_VERSION=12.1.0-1
2023-08-17T10:52:13.1638145Z NVARCH=x86_64
2023-08-17T10:52:13.1638370Z GITHUB_ACTOR_ID=2453524
2023-08-17T10:52:13.1638673Z GITHUB_WORKFLOW_SHA=384e0d104fd077d31efafc564129660e9b7a0f25
2023-08-17T10:52:13.1638970Z GITHUB_REF_NAME=main
2023-08-17T10:52:13.1639291Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1
2023-08-17T10:52:13.1639822Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2023-08-17T10:52:13.1640164Z GITHUB_JOB=test
2023-08-17T10:52:13.1640502Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1
2023-08-17T10:52:13.1640823Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2023-08-17T10:52:13.1641178Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.0-1
2023-08-17T10:52:13.1641475Z GITHUB_REPOSITORY=pytorch/pytorch
2023-08-17T10:52:13.1641774Z NV_NVPROF_VERSION=12.1.55-1
2023-08-17T10:52:13.1642015Z GITHUB_RETENTION_DAYS=90
2023-08-17T10:52:13.1642266Z OPENSSL_DIR=/opt/openssl
2023-08-17T10:52:13.1642521Z GITHUB_ACTION_REPOSITORY=
2023-08-17T10:52:13.1643464Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-08-17T10:52:13.1644048Z GITHUB_BASE_REF=
2023-08-17T10:52:13.1644303Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2023-08-17T10:52:13.1644542Z CI=true
2023-08-17T10:52:13.1644838Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1
2023-08-17T10:52:13.1645105Z GITHUB_REPOSITORY_OWNER=pytorch
2023-08-17T10:52:13.1645367Z INSTALLED_PROTOBUF=yes
2023-08-17T10:52:13.1645607Z GITHUB_HEAD_REF=
2023-08-17T10:52:13.1645840Z GITHUB_ACTION_REF=
2023-08-17T10:52:13.1646205Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2023-08-17T10:52:13.1646631Z GITHUB_WORKFLOW=inductor-A100-perf-nightly
2023-08-17T10:52:13.1647035Z DEBIAN_FRONTEND=noninteractive
2023-08-17T10:52:13.1647633Z GITHUB_OUTPUT=/home/weiwangmeta/actions-runner/_work/_temp/_runner_file_commands/set_output_c0f1af99-9d59-44d3-885d-e11f042906cf
2023-08-17T10:52:13.1648037Z SKIP_SCCACHE_INITIALIZATION=1
2023-08-17T10:52:13.1648282Z _=/usr/bin/env
2023-08-17T10:52:13.1648561Z + echo 'Testing pytorch'
2023-08-17T10:52:13.1648800Z Testing pytorch
2023-08-17T10:52:13.1649062Z + export LANG=C.UTF-8
2023-08-17T10:52:13.1649321Z + LANG=C.UTF-8
2023-08-17T10:52:13.1649536Z + PR_NUMBER=
2023-08-17T10:52:13.1649867Z + [[ inductor_huggingface_perf == \d\e\f\a\u\l\t ]]
2023-08-17T10:52:13.1650193Z + [[ inductor_huggingface_perf == \d\i\s\t\r\i\b\u\t\e\d ]]
2023-08-17T10:52:13.1650518Z + [[ inductor_huggingface_perf == \s\l\o\w ]]
2023-08-17T10:52:13.1650977Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *slow-gradcheck* ]]
2023-08-17T10:52:13.1651459Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *cuda* ]]
2023-08-17T10:52:13.1651795Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2023-08-17T10:52:13.1652099Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2023-08-17T10:52:13.1652407Z + [[ inductor_huggingface_perf == *crossref* ]]
2023-08-17T10:52:13.1652827Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *rocm* ]]
2023-08-17T10:52:13.1681067Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 != *-bazel-* ]]
2023-08-17T10:52:13.1681493Z + pip_install --user ninja==1.10.2
2023-08-17T10:52:13.1681876Z + pip install --progress-bar off --user ninja==1.10.2
2023-08-17T10:52:13.7362065Z Collecting ninja==1.10.2
2023-08-17T10:52:13.8099800Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2023-08-17T10:52:15.2272039Z Installing collected packages: ninja
2023-08-17T10:52:15.2363763Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2023-08-17T10:52:15.2364478Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2023-08-17T10:52:15.2425980Z Successfully installed ninja-1.10.2
2023-08-17T10:52:15.3292119Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-08-17T10:52:15.3293389Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-08-17T10:52:15.3294806Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *asan* ]]
2023-08-17T10:52:15.3295550Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *-debug* ]]
2023-08-17T10:52:15.3296012Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 != *-bazel-* ]]
2023-08-17T10:52:15.3296606Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm80. Expect the assertion to pass'
2023-08-17T10:52:15.3297250Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm80. Expect the assertion to pass
2023-08-17T10:52:15.3299611Z + cd test
2023-08-17T10:52:15.3300933Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)'
2023-08-17T10:52:17.1591360Z + [[ inductor_huggingface_perf == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2023-08-17T10:52:17.1591990Z + [[ inductor_huggingface_perf == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2023-08-17T10:52:17.1592384Z + DYNAMO_BENCHMARK_FLAGS=()
2023-08-17T10:52:17.1593142Z + [[ inductor_huggingface_perf == *dynamo_eager* ]]
2023-08-17T10:52:17.1594033Z + [[ inductor_huggingface_perf == *aot_eager* ]]
2023-08-17T10:52:17.1594341Z + [[ inductor_huggingface_perf == *inductor* ]]
2023-08-17T10:52:17.1594651Z + [[ inductor_huggingface_perf != *perf* ]]
2023-08-17T10:52:17.1595053Z + [[ inductor_huggingface_perf == *dynamic* ]]
2023-08-17T10:52:17.1595377Z + [[ inductor_huggingface_perf == *cpu_accuracy* ]]
2023-08-17T10:52:17.1595961Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda)
2023-08-17T10:52:17.1606050Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *tbb* ]]
2023-08-17T10:52:17.1622902Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *libtorch* ]]
2023-08-17T10:52:17.1623427Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *-bazel-* ]]
2023-08-17T10:52:17.1626156Z + cd test
2023-08-17T10:52:17.1626998Z + python -c 'import torch; print(torch.__config__.show())'
2023-08-17T10:52:18.7348532Z PyTorch built with:
2023-08-17T10:52:18.7349229Z   - GCC 9.4
2023-08-17T10:52:18.7349543Z   - C++ Version: 201703
2023-08-17T10:52:18.7350152Z   - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2023-08-17T10:52:18.7350797Z   - Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)
2023-08-17T10:52:18.7351423Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2023-08-17T10:52:18.7351797Z   - LAPACK is enabled (usually provided by MKL)
2023-08-17T10:52:18.7352108Z   - NNPACK is enabled
2023-08-17T10:52:18.7352416Z   - CPU capability usage: AVX2
2023-08-17T10:52:18.7352721Z   - CUDA Runtime 12.1
2023-08-17T10:52:18.7353139Z   - NVCC architecture flags: -gencode;arch=compute_80,code=sm_80
2023-08-17T10:52:18.7353466Z   - CuDNN 8.9.2
2023-08-17T10:52:18.7353724Z   - Magma 2.6.1
2023-08-17T10:52:18.7357522Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.9.2, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-invalid-partial-specialization -Wno-unused-private-field -Wno-aligned-allocation-unavailable -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.1.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 
2023-08-17T10:52:18.7360041Z 
2023-08-17T10:52:19.0481807Z + cd test
2023-08-17T10:52:19.0483099Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2023-08-17T10:52:20.5543458Z ATen/Parallel:
2023-08-17T10:52:20.5544992Z 	at::get_num_threads() : 6
2023-08-17T10:52:20.5545323Z 	at::get_num_interop_threads() : 6
2023-08-17T10:52:20.5545619Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2023-08-17T10:52:20.5545883Z 	omp_get_max_threads() : 6
2023-08-17T10:52:20.5546639Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2023-08-17T10:52:20.5547026Z 	mkl_get_max_threads() : 6
2023-08-17T10:52:20.5547504Z Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)
2023-08-17T10:52:20.5547896Z std::thread::hardware_concurrency() : 12
2023-08-17T10:52:20.5548177Z Environment variables:
2023-08-17T10:52:20.5548422Z 	OMP_NUM_THREADS : [not set]
2023-08-17T10:52:20.5549697Z 	MKL_NUM_THREADS : [not set]
2023-08-17T10:52:20.5550173Z ATen parallel backend: OpenMP
2023-08-17T10:52:20.5550424Z 
2023-08-17T10:52:20.8161396Z + [[ inductor_huggingface_perf == *backward* ]]
2023-08-17T10:52:20.8162104Z + [[ inductor_huggingface_perf == *xla* ]]
2023-08-17T10:52:20.8162811Z + [[ inductor_huggingface_perf == \j\i\t\_\l\e\g\a\c\y ]]
2023-08-17T10:52:20.8164138Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm80 == *libtorch* ]]
2023-08-17T10:52:20.8165712Z + [[ inductor_huggingface_perf == distributed ]]
2023-08-17T10:52:20.8166542Z + [[ inductor_huggingface_perf == deploy ]]
2023-08-17T10:52:20.8166899Z + [[ inductor_huggingface_perf == *inductor_distributed* ]]
2023-08-17T10:52:20.8167248Z + [[ inductor_huggingface_perf == *huggingface* ]]
2023-08-17T10:52:20.8167536Z + install_torchvision
2023-08-17T10:52:20.8167765Z + local orig_preload
2023-08-17T10:52:20.8167994Z + local commit
2023-08-17T10:52:20.8168256Z ++ get_pinned_commit vision
2023-08-17T10:52:20.8168616Z ++ cat .github/ci_commit_pins/vision.txt
2023-08-17T10:52:20.8184839Z + commit=4cba51c5254407ec1e460a8cbb4ea06700e92637
2023-08-17T10:52:20.8185355Z + orig_preload=
2023-08-17T10:52:20.8185764Z + '[' -n '' ']'
2023-08-17T10:52:20.8186375Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@4cba51c5254407ec1e460a8cbb4ea06700e92637
2023-08-17T10:52:20.8187160Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@4cba51c5254407ec1e460a8cbb4ea06700e92637
2023-08-17T10:52:21.2417048Z Collecting git+https://github.com/pytorch/vision.git@4cba51c5254407ec1e460a8cbb4ea06700e92637
2023-08-17T10:52:21.2423932Z   Cloning https://github.com/pytorch/vision.git (to revision 4cba51c5254407ec1e460a8cbb4ea06700e92637) to /tmp/pip-req-build-2slmpte6
2023-08-17T10:52:21.2456345Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-2slmpte6
2023-08-17T10:52:23.4445724Z   Running command git rev-parse -q --verify 'sha^4cba51c5254407ec1e460a8cbb4ea06700e92637'
2023-08-17T10:52:23.4470542Z   Running command git fetch -q https://github.com/pytorch/vision.git 4cba51c5254407ec1e460a8cbb4ea06700e92637
2023-08-17T10:52:24.9223211Z   Running command git checkout -q 4cba51c5254407ec1e460a8cbb4ea06700e92637
2023-08-17T10:52:25.4098755Z   Resolved https://github.com/pytorch/vision.git to commit 4cba51c5254407ec1e460a8cbb4ea06700e92637
2023-08-17T10:52:27.8924431Z   Preparing metadata (setup.py) ... [?25l- \ done
2023-08-17T10:52:27.8986303Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.16.0a0+4cba51c) (1.21.2)
2023-08-17T10:52:27.8990898Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.16.0a0+4cba51c) (2.31.0)
2023-08-17T10:52:27.8995066Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.16.0a0+4cba51c) (2.1.0a0+git384e0d1)
2023-08-17T10:52:27.9003036Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.16.0a0+4cba51c) (9.5.0)
2023-08-17T10:52:27.9209048Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchvision==0.16.0a0+4cba51c) (3.2.0)
2023-08-17T10:52:27.9216269Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchvision==0.16.0a0+4cba51c) (3.4)
2023-08-17T10:52:27.9225353Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchvision==0.16.0a0+4cba51c) (1.26.16)
2023-08-17T10:52:27.9232527Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchvision==0.16.0a0+4cba51c) (2023.7.22)
2023-08-17T10:52:27.9300507Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.16.0a0+4cba51c) (3.9.0)
2023-08-17T10:52:27.9304500Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.16.0a0+4cba51c) (4.7.1)
2023-08-17T10:52:27.9308645Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.16.0a0+4cba51c) (1.12)
2023-08-17T10:52:27.9313314Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.16.0a0+4cba51c) (2.8.8)
2023-08-17T10:52:27.9319233Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.16.0a0+4cba51c) (3.1.2)
2023-08-17T10:52:27.9323558Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.16.0a0+4cba51c) (2023.4.0)
2023-08-17T10:52:27.9986494Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.16.0a0+4cba51c) (2.1.3)
2023-08-17T10:52:28.0181080Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.16.0a0+4cba51c) (1.3.0)
2023-08-17T10:52:28.0285896Z Building wheels for collected packages: torchvision
2023-08-17T10:54:05.1119500Z   Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done
2023-08-17T10:54:05.1187437Z [?25h  Created wheel for torchvision: filename=torchvision-0.16.0a0+4cba51c-cp310-cp310-linux_x86_64.whl size=1917847 sha256=cb665ae158cd9dfe80b3f61438d2e0535d0cad8ddd93fff10f3d33db44530171
2023-08-17T10:54:05.1190793Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/f7/da/a5/a343c0c91d6b67d190b3830517ef78cf410e34c95ff05ff013
2023-08-17T10:54:05.1235616Z Successfully built torchvision
2023-08-17T10:54:06.4088650Z Installing collected packages: torchvision
2023-08-17T10:54:06.8956450Z Successfully installed torchvision-0.16.0a0+4cba51c
2023-08-17T10:54:07.0277516Z + '[' -n '' ']'
2023-08-17T10:54:07.0277804Z + id=2
2023-08-17T10:54:07.0278068Z + test_dynamo_benchmark huggingface 2
2023-08-17T10:54:07.0284942Z ++ pwd
2023-08-17T10:54:07.0290614Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2023-08-17T10:54:07.0291190Z + local suite=huggingface
2023-08-17T10:54:07.0291551Z + shift
2023-08-17T10:54:07.0291834Z + local shard_id=2
2023-08-17T10:54:07.0292081Z + shift
2023-08-17T10:54:07.0292601Z + [[ inductor_huggingface_perf == *perf_compare* ]]
2023-08-17T10:54:07.0295187Z + [[ inductor_huggingface_perf == *perf* ]]
2023-08-17T10:54:07.0295903Z + test_single_dynamo_benchmark dashboard huggingface 2
2023-08-17T10:54:07.0297889Z ++ pwd
2023-08-17T10:54:07.0301019Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2023-08-17T10:54:07.0301513Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports
2023-08-17T10:54:07.0323747Z + local name=dashboard
2023-08-17T10:54:07.0324154Z + shift
2023-08-17T10:54:07.0324490Z + local suite=huggingface
2023-08-17T10:54:07.0324879Z + shift
2023-08-17T10:54:07.0325102Z + local shard_id=2
2023-08-17T10:54:07.0325323Z + shift
2023-08-17T10:54:07.0325524Z + partition_flags=()
2023-08-17T10:54:07.0325771Z + local partition_flags
2023-08-17T10:54:07.0330842Z + [[ -n 3 ]]
2023-08-17T10:54:07.0331324Z + [[ -n 2 ]]
2023-08-17T10:54:07.0332128Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id")
2023-08-17T10:54:07.0332878Z + [[ inductor_huggingface_perf == *perf_compare* ]]
2023-08-17T10:54:07.0333463Z + [[ inductor_huggingface_perf == *perf* ]]
2023-08-17T10:54:07.0334298Z + test_perf_for_dashboard huggingface --device cuda --total-partitions 3 --partition-id 2
2023-08-17T10:54:07.0334650Z ++ pwd
2023-08-17T10:54:07.0335424Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2023-08-17T10:54:07.0335873Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports
2023-08-17T10:54:07.0350542Z + local suite=huggingface
2023-08-17T10:54:07.0350977Z + shift
2023-08-17T10:54:07.0351318Z + local backend=inductor
2023-08-17T10:54:07.0351737Z + modes=()
2023-08-17T10:54:07.0352135Z + local modes
2023-08-17T10:54:07.0353667Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *training-true* ]]
2023-08-17T10:54:07.0354682Z + modes+=(training)
2023-08-17T10:54:07.0355574Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *inference-true* ]]
2023-08-17T10:54:07.0356238Z + modes+=(inference)
2023-08-17T10:54:07.0356484Z + targets=(accuracy performance)
2023-08-17T10:54:07.0356734Z + local targets
2023-08-17T10:54:07.0356969Z + for mode in "${modes[@]}"
2023-08-17T10:54:07.0357233Z + [[ training == \i\n\f\e\r\e\n\c\e ]]
2023-08-17T10:54:07.0357500Z + [[ training == \t\r\a\i\n\i\n\g ]]
2023-08-17T10:54:07.0357739Z + dtype=amp
2023-08-17T10:54:07.0357977Z + for target in "${targets[@]}"
2023-08-17T10:54:07.0358293Z + target_flag=("--${target}")
2023-08-17T10:54:07.0358523Z + local target_flag
2023-08-17T10:54:07.0358783Z + [[ accuracy == \p\e\r\f\o\r\m\a\n\c\e ]]
2023-08-17T10:54:07.0359054Z + [[ accuracy == \a\c\c\u\r\a\c\y ]]
2023-08-17T10:54:07.0359427Z + target_flag+=(--no-translation-validation)
2023-08-17T10:54:07.0360365Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *default-true* ]]
2023-08-17T10:54:07.0361838Z + python benchmarks/dynamo/huggingface.py --accuracy --no-translation-validation --training --amp --backend inductor --disable-cudagraphs --device cuda --total-partitions 3 --partition-id 2 --output /var/lib/jenkins/workspace/test/test-reports/inductor_no_cudagraphs_huggingface_amp_training_cuda_accuracy.csv
2023-08-17T10:54:10.0601345Z 
2023-08-17T10:54:10.0607714Z Downloading (…)lve/main/config.json:   0% 0.00/694 [00:00<?, ?B/s]
2023-08-17T10:54:10.0608910Z Downloading (…)lve/main/config.json: 100% 694/694 [00:00<00:00, 5.40MB/s]
2023-08-17T10:54:10.1634964Z 
2023-08-17T10:54:10.1635827Z Downloading (…)lve/main/config.json:   0% 0.00/1.21k [00:00<?, ?B/s]
2023-08-17T10:54:10.1636690Z Downloading (…)lve/main/config.json: 100% 1.21k/1.21k [00:00<00:00, 12.6MB/s]
2023-08-17T10:54:10.2580131Z 
2023-08-17T10:54:10.2581618Z Downloading (…)lve/main/config.json:   0% 0.00/762 [00:00<?, ?B/s]
2023-08-17T10:54:10.2582475Z Downloading (…)lve/main/config.json: 100% 762/762 [00:00<00:00, 8.39MB/s]
2023-08-17T10:54:10.3602310Z 
2023-08-17T10:54:10.3603699Z Downloading (…)lve/main/config.json:   0% 0.00/626 [00:00<?, ?B/s]
2023-08-17T10:54:10.3604541Z Downloading (…)lve/main/config.json: 100% 626/626 [00:00<00:00, 7.12MB/s]
2023-08-17T10:54:10.4568576Z 
2023-08-17T10:54:10.4569424Z Downloading (…)lve/main/config.json:   0% 0.00/674 [00:00<?, ?B/s]
2023-08-17T10:54:10.4570310Z Downloading (…)lve/main/config.json: 100% 674/674 [00:00<00:00, 7.54MB/s]
2023-08-17T10:54:10.5515600Z 
2023-08-17T10:54:10.5518454Z Downloading (…)lve/main/config.json:   0% 0.00/508 [00:00<?, ?B/s]
2023-08-17T10:54:10.5519293Z Downloading (…)lve/main/config.json: 100% 508/508 [00:00<00:00, 2.31MB/s]
2023-08-17T10:54:13.8423279Z 
2023-08-17T10:54:20.1162326Z loading model: 0it [00:00, ?it/s]
2023-08-17T10:54:20.1162972Z loading model: 0it [00:06, ?it/s]
2023-08-17T10:54:20.1163552Z cuda train MegatronBertForQuestionAnswering   
2023-08-17T10:55:41.2239699Z pass
2023-08-17T10:55:46.3825407Z 
2023-08-17T10:55:48.5383680Z loading model: 0it [00:00, ?it/s]
2023-08-17T10:55:48.5384569Z loading model: 0it [00:02, ?it/s]
2023-08-17T10:55:48.5384882Z cuda train MobileBertForMaskedLM              
2023-08-17T10:58:32.0337844Z pass
2023-08-17T10:58:38.6484878Z 
2023-08-17T10:58:40.6404169Z loading model: 0it [00:00, ?it/s]
2023-08-17T10:58:40.6404966Z loading model: 0it [00:01, ?it/s]
2023-08-17T10:58:40.6407015Z cuda train MobileBertForQuestionAnswering     
2023-08-17T11:01:23.0387112Z pass
2023-08-17T11:01:29.3096929Z 
2023-08-17T11:01:33.0168154Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:01:33.0168718Z loading model: 0it [00:03, ?it/s]
2023-08-17T11:01:33.0169024Z cuda train OPTForCausalLM                     
2023-08-17T11:02:14.8547581Z pass
2023-08-17T11:02:19.4799515Z 
2023-08-17T11:02:22.5066968Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:02:22.5067452Z loading model: 0it [00:03, ?it/s]
2023-08-17T11:02:22.5068565Z cuda train PLBartForCausalLM                  
2023-08-17T11:02:48.4020282Z pass
2023-08-17T11:02:52.8374046Z 
2023-08-17T11:02:57.8496018Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:02:57.8496622Z loading model: 0it [00:05, ?it/s]
2023-08-17T11:02:57.8502367Z cuda train PLBartForConditionalGeneration     
2023-08-17T11:03:50.4509525Z pass
2023-08-17T11:03:55.1602360Z 
2023-08-17T11:04:03.1891841Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:04:03.1892201Z loading model: 0it [00:08, ?it/s]
2023-08-17T11:04:03.1892499Z cuda train PegasusForCausalLM                 
2023-08-17T11:04:45.8587993Z pass
2023-08-17T11:04:50.5820744Z 
2023-08-17T11:05:03.6941367Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:05:03.6941700Z loading model: 0it [00:13, ?it/s]
2023-08-17T11:05:03.6942024Z cuda train PegasusForConditionalGeneration    
2023-08-17T11:06:36.6869043Z pass
2023-08-17T11:06:42.2978800Z 
2023-08-17T11:06:42.3022742Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`
2023-08-17T11:06:45.3143932Z 
2023-08-17T11:06:45.3144514Z loading model: 0it [00:03, ?it/s]
2023-08-17T11:06:45.3145048Z cuda train RobertaForCausalLM                 
2023-08-17T11:07:27.6395519Z pass
2023-08-17T11:07:32.3117665Z 
2023-08-17T11:07:35.0159334Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:07:35.0159726Z loading model: 0it [00:02, ?it/s]
2023-08-17T11:07:35.0160114Z cuda train RobertaForQuestionAnswering        
2023-08-17T11:08:15.7800009Z pass
2023-08-17T11:08:20.4251866Z 
2023-08-17T11:08:21.6002401Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:08:21.6006438Z loading model: 0it [00:01, ?it/s]
2023-08-17T11:08:21.6006802Z cuda train Speech2Text2ForCausalLM            
2023-08-17T11:08:45.0220368Z pass
2023-08-17T11:08:49.3668770Z 
2023-08-17T11:08:51.8819877Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:08:51.8820381Z loading model: 0it [00:02, ?it/s]
2023-08-17T11:08:51.8822144Z cuda train T5ForConditionalGeneration         
2023-08-17T11:09:43.5744314Z pass
2023-08-17T11:09:48.3269182Z 
2023-08-17T11:09:50.6173631Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:09:50.6174528Z loading model: 0it [00:02, ?it/s]
2023-08-17T11:09:50.6174992Z cuda train T5Small                            
2023-08-17T11:10:40.1632282Z pass
2023-08-17T11:10:44.7355220Z 
2023-08-17T11:10:50.4176821Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:10:50.4177360Z loading model: 0it [00:05, ?it/s]
2023-08-17T11:10:50.4177685Z cuda train TrOCRForCausalLM                   
2023-08-17T11:11:32.4077635Z pass
2023-08-17T11:11:37.0742414Z 
2023-08-17T11:11:50.8879228Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:11:50.8879830Z loading model: 0it [00:13, ?it/s]
2023-08-17T11:11:50.8880132Z cuda train XGLMForCausalLM                    
2023-08-17T11:13:10.4816087Z pass
2023-08-17T11:13:15.9374457Z 
2023-08-17T11:13:22.4433382Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:13:22.4433758Z loading model: 0it [00:06, ?it/s]
2023-08-17T11:13:22.4434253Z cuda train XLNetLMHeadModel                   
2023-08-17T11:14:23.3124577Z [2023-08-17 11:14:23,310] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-08-17T11:15:07.5379644Z pass
2023-08-17T11:15:12.8783055Z 
2023-08-17T11:15:16.1412128Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:15:16.1412831Z loading model: 0it [00:03, ?it/s]
2023-08-17T11:15:16.1413131Z cuda train YituTechConvBert                   
2023-08-17T11:16:17.9601864Z pass
2023-08-17T11:16:19.5221913Z accuracy            pass_rate=100.00%
2023-08-17T11:16:19.5223078Z calls_captured      gmean=640.38x mean=727.235x
2023-08-17T11:16:19.5223770Z unique_graphs       gmean=1.00x mean=1.000x
2023-08-17T11:16:19.5224859Z graph_breaks        gmean=5.59x mean=5.647x
2023-08-17T11:16:19.5226984Z unique_graph_breaks gmean=4.57x mean=4.647x
2023-08-17T11:16:19.9946398Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *cudagraphs-true* ]]
2023-08-17T11:16:19.9949350Z + python benchmarks/dynamo/huggingface.py --accuracy --no-translation-validation --training --amp --backend inductor --device cuda --total-partitions 3 --partition-id 2 --output /var/lib/jenkins/workspace/test/test-reports/inductor_with_cudagraphs_huggingface_amp_training_cuda_accuracy.csv
2023-08-17T11:16:26.6447560Z 
2023-08-17T11:16:32.9664535Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:16:32.9665033Z loading model: 0it [00:06, ?it/s]
2023-08-17T11:16:32.9665369Z cuda train MegatronBertForQuestionAnswering   
2023-08-17T11:17:50.8240852Z pass
2023-08-17T11:17:56.0190910Z 
2023-08-17T11:17:58.2193582Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:17:58.2194078Z loading model: 0it [00:02, ?it/s]
2023-08-17T11:17:58.2194374Z cuda train MobileBertForMaskedLM              
2023-08-17T11:20:39.0196152Z pass
2023-08-17T11:20:45.5013069Z 
2023-08-17T11:20:47.3072978Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:20:47.3073671Z loading model: 0it [00:01, ?it/s]
2023-08-17T11:20:47.3074679Z cuda train MobileBertForQuestionAnswering     
2023-08-17T11:23:27.9053418Z pass
2023-08-17T11:23:34.4832479Z 
2023-08-17T11:23:38.0613104Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:23:38.0613596Z loading model: 0it [00:03, ?it/s]
2023-08-17T11:23:38.0614057Z cuda train OPTForCausalLM                     
2023-08-17T11:24:18.1572019Z pass
2023-08-17T11:24:22.7559219Z 
2023-08-17T11:24:25.6824557Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:24:25.6824869Z loading model: 0it [00:02, ?it/s]
2023-08-17T11:24:25.6825209Z cuda train PLBartForCausalLM                  
2023-08-17T11:24:49.9480408Z pass
2023-08-17T11:24:54.2275246Z 
2023-08-17T11:24:59.1370103Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:24:59.1370431Z loading model: 0it [00:04, ?it/s]
2023-08-17T11:24:59.1370778Z cuda train PLBartForConditionalGeneration     
2023-08-17T11:25:51.7202128Z pass
2023-08-17T11:25:56.3248862Z 
2023-08-17T11:26:04.3915340Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:26:44.6792111Z loading model: 0it [00:08, ?it/s]
2023-08-17T11:26:44.6792687Z cuda train PegasusForCausalLM                 
2023-08-17T11:26:44.6792948Z pass
2023-08-17T11:26:49.3228698Z 
2023-08-17T11:27:02.2519325Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:27:02.2519987Z loading model: 0it [00:12, ?it/s]
2023-08-17T11:27:02.2520389Z cuda train PegasusForConditionalGeneration    
2023-08-17T11:28:35.3656055Z pass
2023-08-17T11:28:40.6052288Z 
2023-08-17T11:28:40.6084622Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`
2023-08-17T11:28:43.6840494Z 
2023-08-17T11:28:43.6841399Z loading model: 0it [00:03, ?it/s]
2023-08-17T11:28:43.6841925Z cuda train RobertaForCausalLM                 
2023-08-17T11:29:24.9621448Z pass
2023-08-17T11:29:30.0023150Z 
2023-08-17T11:29:32.6941763Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:29:32.6942722Z loading model: 0it [00:02, ?it/s]
2023-08-17T11:29:32.6943199Z cuda train RobertaForQuestionAnswering        
2023-08-17T11:30:13.7226447Z pass
2023-08-17T11:30:18.6542375Z 
2023-08-17T11:30:19.7893423Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:30:19.7894274Z loading model: 0it [00:01, ?it/s]
2023-08-17T11:30:19.7894821Z cuda train Speech2Text2ForCausalLM            
2023-08-17T11:30:42.5891809Z pass
2023-08-17T11:30:46.9711624Z 
2023-08-17T11:30:49.2557123Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:30:49.2557801Z loading model: 0it [00:02, ?it/s]
2023-08-17T11:30:49.2558104Z cuda train T5ForConditionalGeneration         
2023-08-17T11:31:38.9280090Z pass
2023-08-17T11:31:43.7961589Z 
2023-08-17T11:31:46.1141254Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:31:46.1141592Z loading model: 0it [00:02, ?it/s]
2023-08-17T11:31:46.1141875Z cuda train T5Small                            
2023-08-17T11:32:35.4429916Z pass
2023-08-17T11:32:40.1568499Z 
2023-08-17T11:32:45.8487095Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:32:45.8487457Z loading model: 0it [00:05, ?it/s]
2023-08-17T11:32:45.8487925Z cuda train TrOCRForCausalLM                   
2023-08-17T11:33:26.3034984Z pass
2023-08-17T11:33:31.2148035Z 
2023-08-17T11:33:45.0146095Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:33:45.0146597Z loading model: 0it [00:13, ?it/s]
2023-08-17T11:33:45.0146894Z cuda train XGLMForCausalLM                    
2023-08-17T11:35:03.3324199Z pass
2023-08-17T11:35:08.8430995Z 
2023-08-17T11:35:15.3086970Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:35:15.3087302Z loading model: 0it [00:06, ?it/s]
2023-08-17T11:35:15.3087843Z cuda train XLNetLMHeadModel                   
2023-08-17T11:36:16.5292259Z [2023-08-17 11:36:16,527] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-08-17T11:36:34.4387377Z skipping cudagraphs due to multiple devices
2023-08-17T11:36:59.2871798Z pass
2023-08-17T11:37:04.6187904Z 
2023-08-17T11:37:07.8468949Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:38:08.6696116Z loading model: 0it [00:03, ?it/s]
2023-08-17T11:38:08.6696632Z cuda train YituTechConvBert                   
2023-08-17T11:38:08.6696904Z pass
2023-08-17T11:38:10.2823541Z accuracy            pass_rate=100.00%
2023-08-17T11:38:10.2824224Z calls_captured      gmean=640.38x mean=727.235x
2023-08-17T11:38:10.2826458Z unique_graphs       gmean=1.00x mean=1.000x
2023-08-17T11:38:10.2828816Z graph_breaks        gmean=5.59x mean=5.647x
2023-08-17T11:38:10.2832097Z unique_graph_breaks gmean=4.57x mean=4.647x
2023-08-17T11:38:10.7615781Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *dynamic-true* ]]
2023-08-17T11:38:10.7618134Z + python benchmarks/dynamo/huggingface.py --accuracy --no-translation-validation --training --amp --backend inductor --dynamic-shapes --dynamic-batch-only --device cuda --total-partitions 3 --partition-id 2 --output /var/lib/jenkins/workspace/test/test-reports/inductor_dynamic_huggingface_amp_training_cuda_accuracy.csv
2023-08-17T11:38:17.3896888Z 
2023-08-17T11:38:23.8823403Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:38:23.8823731Z loading model: 0it [00:06, ?it/s]
2023-08-17T11:38:23.8824244Z cuda train MegatronBertForQuestionAnswering   
2023-08-17T11:39:41.9490873Z pass
2023-08-17T11:39:47.1581829Z 
2023-08-17T11:39:49.2194190Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:39:49.2194676Z loading model: 0it [00:02, ?it/s]
2023-08-17T11:39:49.2194983Z cuda train MobileBertForMaskedLM              
2023-08-17T11:42:30.1367421Z pass
2023-08-17T11:42:36.3434358Z 
2023-08-17T11:42:38.1395370Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:42:38.1395732Z loading model: 0it [00:01, ?it/s]
2023-08-17T11:42:38.1396064Z cuda train MobileBertForQuestionAnswering     
2023-08-17T11:45:17.9313903Z pass
2023-08-17T11:45:24.5542831Z 
2023-08-17T11:45:28.1366477Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:45:28.1366977Z loading model: 0it [00:03, ?it/s]
2023-08-17T11:45:28.1367286Z cuda train OPTForCausalLM                     
2023-08-17T11:46:08.3370461Z pass
2023-08-17T11:46:12.9535548Z 
2023-08-17T11:46:16.0079990Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:46:16.0080806Z loading model: 0it [00:03, ?it/s]
2023-08-17T11:46:16.0082120Z cuda train PLBartForCausalLM                  
2023-08-17T11:46:40.3125297Z pass
2023-08-17T11:46:44.6482858Z 
2023-08-17T11:46:49.4884480Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:46:49.4885682Z loading model: 0it [00:04, ?it/s]
2023-08-17T11:46:49.4886244Z cuda train PLBartForConditionalGeneration     
2023-08-17T11:47:42.2105781Z pass
2023-08-17T11:47:46.8677825Z 
2023-08-17T11:47:54.7966823Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:47:54.7967331Z loading model: 0it [00:07, ?it/s]
2023-08-17T11:47:54.7967629Z cuda train PegasusForCausalLM                 
2023-08-17T11:48:35.2670649Z pass
2023-08-17T11:48:40.0106053Z 
2023-08-17T11:48:53.1113536Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:48:53.1114552Z loading model: 0it [00:13, ?it/s]
2023-08-17T11:48:53.1115087Z cuda train PegasusForConditionalGeneration    
2023-08-17T11:50:26.4561694Z pass
2023-08-17T11:50:31.8213638Z 
2023-08-17T11:50:31.8245110Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`
2023-08-17T11:50:34.8595160Z 
2023-08-17T11:50:34.8596265Z loading model: 0it [00:03, ?it/s]
2023-08-17T11:50:34.8596879Z cuda train RobertaForCausalLM                 
2023-08-17T11:51:16.0847704Z pass
2023-08-17T11:51:20.7286420Z 
2023-08-17T11:51:23.5267853Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:51:23.5268368Z loading model: 0it [00:02, ?it/s]
2023-08-17T11:51:23.5268725Z cuda train RobertaForQuestionAnswering        
2023-08-17T11:52:04.5933047Z pass
2023-08-17T11:52:09.2197274Z 
2023-08-17T11:52:10.3496416Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:52:10.3496751Z loading model: 0it [00:01, ?it/s]
2023-08-17T11:52:10.3497070Z cuda train Speech2Text2ForCausalLM            
2023-08-17T11:52:33.1080338Z pass
2023-08-17T11:52:37.5727224Z 
2023-08-17T11:52:39.8365909Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:52:39.8369378Z loading model: 0it [00:02, ?it/s]
2023-08-17T11:52:39.8369784Z cuda train T5ForConditionalGeneration         
2023-08-17T11:53:29.4989339Z pass
2023-08-17T11:53:34.2339806Z 
2023-08-17T11:53:36.4453689Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:53:36.4454022Z loading model: 0it [00:02, ?it/s]
2023-08-17T11:53:36.4454292Z cuda train T5Small                            
2023-08-17T11:54:26.0314028Z pass
2023-08-17T11:54:30.6931565Z 
2023-08-17T11:54:36.3877493Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:54:36.3878025Z loading model: 0it [00:05, ?it/s]
2023-08-17T11:54:36.3878513Z cuda train TrOCRForCausalLM                   
2023-08-17T11:55:16.6955299Z pass
2023-08-17T11:55:21.3615981Z 
2023-08-17T11:55:35.0605521Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:55:35.0606123Z loading model: 0it [00:13, ?it/s]
2023-08-17T11:55:35.0606422Z cuda train XGLMForCausalLM                    
2023-08-17T11:56:53.7179182Z pass
2023-08-17T11:56:59.1090556Z 
2023-08-17T11:57:05.7946353Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:57:05.7946840Z loading model: 0it [00:06, ?it/s]
2023-08-17T11:57:05.7947159Z cuda train XLNetLMHeadModel                   
2023-08-17T11:58:06.8733168Z [2023-08-17 11:58:06,871] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-08-17T11:58:24.7257045Z skipping cudagraphs due to multiple devices
2023-08-17T11:58:49.5260924Z pass
2023-08-17T11:58:54.7799787Z 
2023-08-17T11:58:58.2055181Z loading model: 0it [00:00, ?it/s]
2023-08-17T11:58:58.2056425Z loading model: 0it [00:03, ?it/s]
2023-08-17T11:58:58.2057288Z cuda train YituTechConvBert                   
2023-08-17T11:59:59.0053033Z pass
2023-08-17T12:00:00.6118979Z accuracy            pass_rate=100.00%
2023-08-17T12:00:00.6120320Z calls_captured      gmean=640.38x mean=727.235x
2023-08-17T12:00:00.6123120Z unique_graphs       gmean=1.00x mean=1.000x
2023-08-17T12:00:00.6125261Z graph_breaks        gmean=5.59x mean=5.647x
2023-08-17T12:00:00.6127794Z unique_graph_breaks gmean=4.57x mean=4.647x
2023-08-17T12:00:01.1021354Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *cppwrapper-true* ]]
2023-08-17T12:00:01.1022729Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *freezing_cudagraphs-true* ]]
2023-08-17T12:00:01.1023455Z + [[ training == \i\n\f\e\r\e\n\c\e ]]
2023-08-17T12:00:01.1024309Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *aotinductor-true* ]]
2023-08-17T12:00:01.1026507Z + [[ training == \i\n\f\e\r\e\n\c\e ]]
2023-08-17T12:00:01.1028177Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *maxautotune-true* ]]
2023-08-17T12:00:01.1028985Z + for target in "${targets[@]}"
2023-08-17T12:00:01.1029318Z + target_flag=("--${target}")
2023-08-17T12:00:01.1029588Z + local target_flag
2023-08-17T12:00:01.1029864Z + [[ performance == \p\e\r\f\o\r\m\a\n\c\e ]]
2023-08-17T12:00:01.1030606Z + target_flag+=(--cold-start-latency)
2023-08-17T12:00:01.1031490Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *default-true* ]]
2023-08-17T12:00:01.1032955Z + python benchmarks/dynamo/huggingface.py --performance --cold-start-latency --training --amp --backend inductor --disable-cudagraphs --device cuda --total-partitions 3 --partition-id 2 --output /var/lib/jenkins/workspace/test/test-reports/inductor_no_cudagraphs_huggingface_amp_training_cuda_performance.csv
2023-08-17T12:00:07.7363304Z 
2023-08-17T12:00:14.4866776Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:00:14.4867145Z loading model: 0it [00:06, ?it/s]
2023-08-17T12:00:14.4867483Z cuda train MegatronBertForQuestionAnswering   
2023-08-17T12:01:41.8424896Z 
2023-08-17T12:01:42.1000593Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:01:42.3200094Z running benchmark:   3% 1/30 [00:00<00:07,  3.88it/s]
2023-08-17T12:01:42.5398645Z running benchmark:   7% 2/30 [00:00<00:06,  4.24it/s]
2023-08-17T12:01:42.7597335Z running benchmark:  10% 3/30 [00:00<00:06,  4.38it/s]
2023-08-17T12:01:42.9804043Z running benchmark:  13% 4/30 [00:00<00:05,  4.44it/s]
2023-08-17T12:01:43.2007582Z running benchmark:  17% 5/30 [00:01<00:05,  4.47it/s]
2023-08-17T12:01:43.4204668Z running benchmark:  20% 6/30 [00:01<00:05,  4.50it/s]
2023-08-17T12:01:43.6403973Z running benchmark:  23% 7/30 [00:01<00:05,  4.51it/s]
2023-08-17T12:01:43.8599950Z running benchmark:  27% 8/30 [00:01<00:04,  4.52it/s]
2023-08-17T12:01:44.0807812Z running benchmark:  30% 9/30 [00:02<00:04,  4.53it/s]
2023-08-17T12:01:44.3014982Z running benchmark:  33% 10/30 [00:02<00:04,  4.53it/s]
2023-08-17T12:01:44.5217080Z running benchmark:  37% 11/30 [00:02<00:04,  4.53it/s]
2023-08-17T12:01:44.7415973Z running benchmark:  40% 12/30 [00:02<00:03,  4.53it/s]
2023-08-17T12:01:44.9612365Z running benchmark:  43% 13/30 [00:02<00:03,  4.54it/s]
2023-08-17T12:01:45.1819781Z running benchmark:  47% 14/30 [00:03<00:03,  4.54it/s]
2023-08-17T12:01:45.4024997Z running benchmark:  50% 15/30 [00:03<00:03,  4.54it/s]
2023-08-17T12:01:45.6223143Z running benchmark:  53% 16/30 [00:03<00:03,  4.54it/s]
2023-08-17T12:01:45.8419453Z running benchmark:  57% 17/30 [00:03<00:02,  4.54it/s]
2023-08-17T12:01:46.0616962Z running benchmark:  60% 18/30 [00:04<00:02,  4.54it/s]
2023-08-17T12:01:46.2816727Z running benchmark:  63% 19/30 [00:04<00:02,  4.55it/s]
2023-08-17T12:01:46.5022733Z running benchmark:  67% 20/30 [00:04<00:02,  4.55it/s]
2023-08-17T12:01:46.7221511Z running benchmark:  70% 21/30 [00:04<00:01,  4.54it/s]
2023-08-17T12:01:46.9417999Z running benchmark:  73% 22/30 [00:04<00:01,  4.54it/s]
2023-08-17T12:01:47.1616159Z running benchmark:  77% 23/30 [00:05<00:01,  4.55it/s]
2023-08-17T12:01:47.3812231Z running benchmark:  80% 24/30 [00:05<00:01,  4.55it/s]
2023-08-17T12:01:47.6017002Z running benchmark:  83% 25/30 [00:05<00:01,  4.55it/s]
2023-08-17T12:01:47.8216924Z running benchmark:  87% 26/30 [00:05<00:00,  4.55it/s]
2023-08-17T12:01:48.0416435Z running benchmark:  90% 27/30 [00:05<00:00,  4.55it/s]
2023-08-17T12:01:48.2619897Z running benchmark:  93% 28/30 [00:06<00:00,  4.55it/s]
2023-08-17T12:01:48.4817751Z running benchmark:  97% 29/30 [00:06<00:00,  4.54it/s]
2023-08-17T12:01:48.4818092Z running benchmark: 100% 30/30 [00:06<00:00,  4.55it/s]
2023-08-17T12:01:48.4818378Z running benchmark: 100% 30/30 [00:06<00:00,  4.52it/s]
2023-08-17T12:01:48.4845931Z 1.784x
2023-08-17T12:01:53.8613234Z 
2023-08-17T12:01:55.9385518Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:01:55.9385986Z loading model: 0it [00:02, ?it/s]
2023-08-17T12:01:55.9386520Z cuda train MobileBertForMaskedLM              
2023-08-17T12:04:55.4127693Z 
2023-08-17T12:04:55.7274996Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:04:56.0308428Z running benchmark:   3% 1/30 [00:00<00:09,  3.17it/s]
2023-08-17T12:04:56.3268093Z running benchmark:   7% 2/30 [00:00<00:08,  3.24it/s]
2023-08-17T12:04:56.6333855Z running benchmark:  10% 3/30 [00:00<00:08,  3.30it/s]
2023-08-17T12:04:56.9323191Z running benchmark:  13% 4/30 [00:01<00:07,  3.29it/s]
2023-08-17T12:04:57.2328514Z running benchmark:  17% 5/30 [00:01<00:07,  3.31it/s]
2023-08-17T12:04:57.5280158Z running benchmark:  20% 6/30 [00:01<00:07,  3.31it/s]
2023-08-17T12:04:57.8239702Z running benchmark:  23% 7/30 [00:02<00:06,  3.34it/s]
2023-08-17T12:04:58.1204622Z running benchmark:  27% 8/30 [00:02<00:06,  3.35it/s]
2023-08-17T12:04:58.4197330Z running benchmark:  30% 9/30 [00:02<00:06,  3.36it/s]
2023-08-17T12:04:58.7230175Z running benchmark:  33% 10/30 [00:03<00:05,  3.35it/s]
2023-08-17T12:04:59.0194637Z running benchmark:  37% 11/30 [00:03<00:05,  3.34it/s]
2023-08-17T12:04:59.3195237Z running benchmark:  40% 12/30 [00:03<00:05,  3.35it/s]
2023-08-17T12:04:59.6190847Z running benchmark:  43% 13/30 [00:03<00:05,  3.34it/s]
2023-08-17T12:04:59.9212975Z running benchmark:  47% 14/30 [00:04<00:04,  3.34it/s]
2023-08-17T12:05:00.2165291Z running benchmark:  50% 15/30 [00:04<00:04,  3.33it/s]
2023-08-17T12:05:00.5154417Z running benchmark:  53% 16/30 [00:04<00:04,  3.35it/s]
2023-08-17T12:05:00.8163593Z running benchmark:  57% 17/30 [00:05<00:03,  3.35it/s]
2023-08-17T12:05:01.1176115Z running benchmark:  60% 18/30 [00:05<00:03,  3.34it/s]
2023-08-17T12:05:01.4199154Z running benchmark:  63% 19/30 [00:05<00:03,  3.33it/s]
2023-08-17T12:05:01.7388079Z running benchmark:  67% 20/30 [00:06<00:03,  3.33it/s]
2023-08-17T12:05:02.0408815Z running benchmark:  70% 21/30 [00:06<00:02,  3.27it/s]
2023-08-17T12:05:02.3420774Z running benchmark:  73% 22/30 [00:06<00:02,  3.28it/s]
2023-08-17T12:05:02.6422329Z running benchmark:  77% 23/30 [00:06<00:02,  3.29it/s]
2023-08-17T12:05:02.9558367Z running benchmark:  80% 24/30 [00:07<00:01,  3.30it/s]
2023-08-17T12:05:03.2564710Z running benchmark:  83% 25/30 [00:07<00:01,  3.27it/s]
2023-08-17T12:05:03.5539178Z running benchmark:  87% 26/30 [00:07<00:01,  3.29it/s]
2023-08-17T12:05:03.8594693Z running benchmark:  90% 27/30 [00:08<00:00,  3.31it/s]
2023-08-17T12:05:04.1596543Z running benchmark:  93% 28/30 [00:08<00:00,  3.30it/s]
2023-08-17T12:05:04.4585175Z running benchmark:  97% 29/30 [00:08<00:00,  3.31it/s]
2023-08-17T12:05:04.4585739Z running benchmark: 100% 30/30 [00:09<00:00,  3.32it/s]
2023-08-17T12:05:04.4587092Z running benchmark: 100% 30/30 [00:09<00:00,  3.32it/s]
2023-08-17T12:05:04.4623825Z 1.482x
2023-08-17T12:05:11.3962015Z 
2023-08-17T12:05:13.2791550Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:05:13.2792055Z loading model: 0it [00:01, ?it/s]
2023-08-17T12:05:13.2792568Z cuda train MobileBertForQuestionAnswering     
2023-08-17T12:08:08.5014210Z 
2023-08-17T12:08:08.8069349Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:08:09.0996938Z running benchmark:   3% 1/30 [00:00<00:08,  3.27it/s]
2023-08-17T12:08:09.3806577Z running benchmark:   7% 2/30 [00:00<00:08,  3.35it/s]
2023-08-17T12:08:09.6569975Z running benchmark:  10% 3/30 [00:00<00:07,  3.44it/s]
2023-08-17T12:08:09.9412436Z running benchmark:  13% 4/30 [00:01<00:07,  3.51it/s]
2023-08-17T12:08:10.2378473Z running benchmark:  17% 5/30 [00:01<00:07,  3.51it/s]
2023-08-17T12:08:10.5151552Z running benchmark:  20% 6/30 [00:01<00:06,  3.46it/s]
2023-08-17T12:08:10.7904260Z running benchmark:  23% 7/30 [00:02<00:06,  3.51it/s]
2023-08-17T12:08:11.0764465Z running benchmark:  27% 8/30 [00:02<00:06,  3.55it/s]
2023-08-17T12:08:11.3683903Z running benchmark:  30% 9/30 [00:02<00:05,  3.53it/s]
2023-08-17T12:08:11.6491490Z running benchmark:  33% 10/30 [00:02<00:05,  3.50it/s]
2023-08-17T12:08:11.9290516Z running benchmark:  37% 11/30 [00:03<00:05,  3.52it/s]
2023-08-17T12:08:12.2303753Z running benchmark:  40% 12/30 [00:03<00:05,  3.53it/s]
2023-08-17T12:08:12.5123788Z running benchmark:  43% 13/30 [00:03<00:04,  3.47it/s]
2023-08-17T12:08:12.7891306Z running benchmark:  47% 14/30 [00:04<00:04,  3.49it/s]
2023-08-17T12:08:13.0648579Z running benchmark:  50% 15/30 [00:04<00:04,  3.53it/s]
2023-08-17T12:08:13.3579766Z running benchmark:  53% 16/30 [00:04<00:03,  3.56it/s]
2023-08-17T12:08:13.6426646Z running benchmark:  57% 17/30 [00:04<00:03,  3.51it/s]
2023-08-17T12:08:13.9196466Z running benchmark:  60% 18/30 [00:05<00:03,  3.51it/s]
2023-08-17T12:08:14.2114085Z running benchmark:  63% 19/30 [00:05<00:03,  3.54it/s]
2023-08-17T12:08:14.4950860Z running benchmark:  67% 20/30 [00:05<00:02,  3.51it/s]
2023-08-17T12:08:14.7811743Z running benchmark:  70% 21/30 [00:05<00:02,  3.51it/s]
2023-08-17T12:08:15.0612644Z running benchmark:  73% 22/30 [00:06<00:02,  3.51it/s]
2023-08-17T12:08:15.3486891Z running benchmark:  77% 23/30 [00:06<00:01,  3.53it/s]
2023-08-17T12:08:15.6267001Z running benchmark:  80% 24/30 [00:06<00:01,  3.51it/s]
2023-08-17T12:08:15.9131451Z running benchmark:  83% 25/30 [00:07<00:01,  3.54it/s]
2023-08-17T12:08:16.1923896Z running benchmark:  87% 26/30 [00:07<00:01,  3.52it/s]
2023-08-17T12:08:16.4801575Z running benchmark:  90% 27/30 [00:07<00:00,  3.54it/s]
2023-08-17T12:08:16.7531522Z running benchmark:  93% 28/30 [00:07<00:00,  3.52it/s]
2023-08-17T12:08:17.0350560Z running benchmark:  97% 29/30 [00:08<00:00,  3.56it/s]
2023-08-17T12:08:17.0350992Z running benchmark: 100% 30/30 [00:08<00:00,  3.56it/s]
2023-08-17T12:08:17.0351294Z running benchmark: 100% 30/30 [00:08<00:00,  3.52it/s]
2023-08-17T12:08:17.0389817Z 1.375x
2023-08-17T12:08:23.8878449Z 
2023-08-17T12:08:27.7160505Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:08:27.7160832Z loading model: 0it [00:03, ?it/s]
2023-08-17T12:08:27.7161137Z cuda train OPTForCausalLM                     
2023-08-17T12:09:17.5460932Z 
2023-08-17T12:09:17.8438071Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:09:18.0784900Z running benchmark:   3% 1/30 [00:00<00:08,  3.35it/s]
2023-08-17T12:09:18.3162041Z running benchmark:   7% 2/30 [00:00<00:07,  3.83it/s]
2023-08-17T12:09:18.5517561Z running benchmark:  10% 3/30 [00:00<00:06,  3.99it/s]
2023-08-17T12:09:18.7879471Z running benchmark:  13% 4/30 [00:01<00:06,  4.09it/s]
2023-08-17T12:09:19.0239691Z running benchmark:  17% 5/30 [00:01<00:06,  4.14it/s]
2023-08-17T12:09:19.2591707Z running benchmark:  20% 6/30 [00:01<00:05,  4.17it/s]
2023-08-17T12:09:19.4978284Z running benchmark:  23% 7/30 [00:01<00:05,  4.20it/s]
2023-08-17T12:09:19.7336201Z running benchmark:  27% 8/30 [00:01<00:05,  4.20it/s]
2023-08-17T12:09:19.9686821Z running benchmark:  30% 9/30 [00:02<00:04,  4.21it/s]
2023-08-17T12:09:20.2043243Z running benchmark:  33% 10/30 [00:02<00:04,  4.22it/s]
2023-08-17T12:09:20.4404834Z running benchmark:  37% 11/30 [00:02<00:04,  4.23it/s]
2023-08-17T12:09:20.6779000Z running benchmark:  40% 12/30 [00:02<00:04,  4.23it/s]
2023-08-17T12:09:20.9140745Z running benchmark:  43% 13/30 [00:03<00:04,  4.23it/s]
2023-08-17T12:09:21.1499589Z running benchmark:  47% 14/30 [00:03<00:03,  4.23it/s]
2023-08-17T12:09:21.3852444Z running benchmark:  50% 15/30 [00:03<00:03,  4.23it/s]
2023-08-17T12:09:21.6201402Z running benchmark:  53% 16/30 [00:03<00:03,  4.24it/s]
2023-08-17T12:09:21.8583057Z running benchmark:  57% 17/30 [00:04<00:03,  4.24it/s]
2023-08-17T12:09:22.0941723Z running benchmark:  60% 18/30 [00:04<00:02,  4.23it/s]
2023-08-17T12:09:22.3293668Z running benchmark:  63% 19/30 [00:04<00:02,  4.23it/s]
2023-08-17T12:09:22.5648664Z running benchmark:  67% 20/30 [00:04<00:02,  4.24it/s]
2023-08-17T12:09:22.8131917Z running benchmark:  70% 21/30 [00:05<00:02,  4.24it/s]
2023-08-17T12:09:23.0508102Z running benchmark:  73% 22/30 [00:05<00:01,  4.25it/s]
2023-08-17T12:09:23.2861682Z running benchmark:  77% 23/30 [00:05<00:01,  4.16it/s]
2023-08-17T12:09:23.5212849Z running benchmark:  80% 24/30 [00:05<00:01,  4.19it/s]
2023-08-17T12:09:23.7571026Z running benchmark:  83% 25/30 [00:05<00:01,  4.21it/s]
2023-08-17T12:09:23.9925466Z running benchmark:  87% 26/30 [00:06<00:00,  4.22it/s]
2023-08-17T12:09:24.2302153Z running benchmark:  90% 27/30 [00:06<00:00,  4.23it/s]
2023-08-17T12:09:24.4660424Z running benchmark:  93% 28/30 [00:06<00:00,  4.22it/s]
2023-08-17T12:09:24.7012947Z running benchmark:  97% 29/30 [00:06<00:00,  4.23it/s]
2023-08-17T12:09:24.7013564Z running benchmark: 100% 30/30 [00:07<00:00,  4.23it/s]
2023-08-17T12:09:24.7014592Z running benchmark: 100% 30/30 [00:07<00:00,  4.19it/s]
2023-08-17T12:09:24.7037744Z 2.659x
2023-08-17T12:09:29.8045265Z 
2023-08-17T12:09:33.2461044Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:09:33.2461362Z loading model: 0it [00:03, ?it/s]
2023-08-17T12:09:33.2461674Z cuda train PLBartForCausalLM                  
2023-08-17T12:10:11.8500263Z 
2023-08-17T12:10:12.1411850Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:10:12.3201103Z running benchmark:   3% 1/30 [00:00<00:08,  3.43it/s]
2023-08-17T12:10:12.4984719Z running benchmark:   7% 2/30 [00:00<00:06,  4.44it/s]
2023-08-17T12:10:12.6768819Z running benchmark:  10% 3/30 [00:00<00:05,  4.90it/s]
2023-08-17T12:10:12.8554264Z running benchmark:  13% 4/30 [00:00<00:05,  5.16it/s]
2023-08-17T12:10:13.0524600Z running benchmark:  17% 5/30 [00:01<00:04,  5.31it/s]
2023-08-17T12:10:13.2323218Z running benchmark:  20% 6/30 [00:01<00:04,  5.23it/s]
2023-08-17T12:10:13.4114926Z running benchmark:  23% 7/30 [00:01<00:04,  5.33it/s]
2023-08-17T12:10:13.5896731Z running benchmark:  27% 8/30 [00:01<00:04,  5.41it/s]
2023-08-17T12:10:13.7677418Z running benchmark:  30% 9/30 [00:01<00:03,  5.47it/s]
2023-08-17T12:10:13.9458245Z running benchmark:  33% 10/30 [00:01<00:03,  5.51it/s]
2023-08-17T12:10:14.1249237Z running benchmark:  37% 11/30 [00:02<00:03,  5.55it/s]
2023-08-17T12:10:14.3030800Z running benchmark:  40% 12/30 [00:02<00:03,  5.56it/s]
2023-08-17T12:10:14.4823454Z running benchmark:  43% 13/30 [00:02<00:03,  5.57it/s]
2023-08-17T12:10:14.6608579Z running benchmark:  47% 14/30 [00:02<00:02,  5.57it/s]
2023-08-17T12:10:14.8391806Z running benchmark:  50% 15/30 [00:02<00:02,  5.58it/s]
2023-08-17T12:10:15.0174255Z running benchmark:  53% 16/30 [00:02<00:02,  5.59it/s]
2023-08-17T12:10:15.2109786Z running benchmark:  57% 17/30 [00:03<00:02,  5.60it/s]
2023-08-17T12:10:15.3878161Z running benchmark:  60% 18/30 [00:03<00:02,  5.46it/s]
2023-08-17T12:10:15.5673062Z running benchmark:  63% 19/30 [00:03<00:01,  5.52it/s]
2023-08-17T12:10:15.7461061Z running benchmark:  67% 20/30 [00:03<00:01,  5.53it/s]
2023-08-17T12:10:15.9244158Z running benchmark:  70% 21/30 [00:03<00:01,  5.55it/s]
2023-08-17T12:10:16.1027160Z running benchmark:  73% 22/30 [00:04<00:01,  5.57it/s]
2023-08-17T12:10:16.2808939Z running benchmark:  77% 23/30 [00:04<00:01,  5.58it/s]
2023-08-17T12:10:16.4593747Z running benchmark:  80% 24/30 [00:04<00:01,  5.59it/s]
2023-08-17T12:10:16.6382416Z running benchmark:  83% 25/30 [00:04<00:00,  5.59it/s]
2023-08-17T12:10:16.8172115Z running benchmark:  87% 26/30 [00:04<00:00,  5.59it/s]
2023-08-17T12:10:16.9957369Z running benchmark:  90% 27/30 [00:04<00:00,  5.59it/s]
2023-08-17T12:10:17.1746902Z running benchmark:  93% 28/30 [00:05<00:00,  5.59it/s]
2023-08-17T12:10:17.3531559Z running benchmark:  97% 29/30 [00:05<00:00,  5.59it/s]
2023-08-17T12:10:17.3532062Z running benchmark: 100% 30/30 [00:05<00:00,  5.60it/s]
2023-08-17T12:10:17.3532362Z running benchmark: 100% 30/30 [00:05<00:00,  5.45it/s]
2023-08-17T12:10:17.3553565Z 1.972x
2023-08-17T12:10:21.7680189Z 
2023-08-17T12:10:27.0660968Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:10:27.0661280Z loading model: 0it [00:05, ?it/s]
2023-08-17T12:10:27.0661629Z cuda train PLBartForConditionalGeneration     
2023-08-17T12:11:32.9287107Z 
2023-08-17T12:11:33.1604879Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:11:33.3447317Z running benchmark:   3% 1/30 [00:00<00:06,  4.31it/s]
2023-08-17T12:11:33.5287151Z running benchmark:   7% 2/30 [00:00<00:05,  4.90it/s]
2023-08-17T12:11:33.7124803Z running benchmark:  10% 3/30 [00:00<00:05,  5.13it/s]
2023-08-17T12:11:33.8965381Z running benchmark:  13% 4/30 [00:00<00:04,  5.25it/s]
2023-08-17T12:11:34.0811004Z running benchmark:  17% 5/30 [00:00<00:04,  5.32it/s]
2023-08-17T12:11:34.2650476Z running benchmark:  20% 6/30 [00:01<00:04,  5.35it/s]
2023-08-17T12:11:34.4498211Z running benchmark:  23% 7/30 [00:01<00:04,  5.38it/s]
2023-08-17T12:11:34.6339756Z running benchmark:  27% 8/30 [00:01<00:04,  5.39it/s]
2023-08-17T12:11:34.8188694Z running benchmark:  30% 9/30 [00:01<00:03,  5.40it/s]
2023-08-17T12:11:35.0028777Z running benchmark:  33% 10/30 [00:01<00:03,  5.40it/s]
2023-08-17T12:11:35.1864657Z running benchmark:  37% 11/30 [00:02<00:03,  5.41it/s]
2023-08-17T12:11:35.3701061Z running benchmark:  40% 12/30 [00:02<00:03,  5.42it/s]
2023-08-17T12:11:35.5545070Z running benchmark:  43% 13/30 [00:02<00:03,  5.43it/s]
2023-08-17T12:11:35.7389216Z running benchmark:  47% 14/30 [00:02<00:02,  5.43it/s]
2023-08-17T12:11:35.9288565Z running benchmark:  50% 15/30 [00:02<00:02,  5.43it/s]
2023-08-17T12:11:36.1130057Z running benchmark:  53% 16/30 [00:02<00:02,  5.43it/s]
2023-08-17T12:11:36.2966119Z running benchmark:  57% 17/30 [00:03<00:02,  5.38it/s]
2023-08-17T12:11:36.4802133Z running benchmark:  60% 18/30 [00:03<00:02,  5.40it/s]
2023-08-17T12:11:36.6639160Z running benchmark:  63% 19/30 [00:03<00:02,  5.41it/s]
2023-08-17T12:11:36.8482817Z running benchmark:  67% 20/30 [00:03<00:01,  5.42it/s]
2023-08-17T12:11:37.0324502Z running benchmark:  70% 21/30 [00:03<00:01,  5.42it/s]
2023-08-17T12:11:37.2162845Z running benchmark:  73% 22/30 [00:04<00:01,  5.43it/s]
2023-08-17T12:11:37.4000334Z running benchmark:  77% 23/30 [00:04<00:01,  5.43it/s]
2023-08-17T12:11:37.5836351Z running benchmark:  80% 24/30 [00:04<00:01,  5.43it/s]
2023-08-17T12:11:37.7671277Z running benchmark:  83% 25/30 [00:04<00:00,  5.44it/s]
2023-08-17T12:11:37.9514381Z running benchmark:  87% 26/30 [00:04<00:00,  5.44it/s]
2023-08-17T12:11:38.1487418Z running benchmark:  90% 27/30 [00:05<00:00,  5.44it/s]
2023-08-17T12:11:38.3321920Z running benchmark:  93% 28/30 [00:05<00:00,  5.32it/s]
2023-08-17T12:11:38.5157531Z running benchmark:  97% 29/30 [00:05<00:00,  5.36it/s]
2023-08-17T12:11:38.5158054Z running benchmark: 100% 30/30 [00:05<00:00,  5.39it/s]
2023-08-17T12:11:38.5158454Z running benchmark: 100% 30/30 [00:05<00:00,  5.37it/s]
2023-08-17T12:11:38.5183311Z 1.763x
2023-08-17T12:11:43.5692851Z 
2023-08-17T12:11:52.1227818Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:11:52.1228180Z loading model: 0it [00:08, ?it/s]
2023-08-17T12:11:52.1228498Z cuda train PegasusForCausalLM                 
2023-08-17T12:12:43.0822209Z 
2023-08-17T12:12:43.2465735Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:12:43.3666795Z running benchmark:   3% 1/30 [00:00<00:04,  6.06it/s]
2023-08-17T12:12:43.4866834Z running benchmark:   7% 2/30 [00:00<00:03,  7.21it/s]
2023-08-17T12:12:43.6091957Z running benchmark:  10% 3/30 [00:00<00:03,  7.68it/s]
2023-08-17T12:12:43.7292979Z running benchmark:  13% 4/30 [00:00<00:03,  7.87it/s]
2023-08-17T12:12:43.8493200Z running benchmark:  17% 5/30 [00:00<00:03,  8.03it/s]
2023-08-17T12:12:43.9696672Z running benchmark:  20% 6/30 [00:00<00:02,  8.13it/s]
2023-08-17T12:12:44.0899186Z running benchmark:  23% 7/30 [00:00<00:02,  8.19it/s]
2023-08-17T12:12:44.2101843Z running benchmark:  27% 8/30 [00:01<00:02,  8.23it/s]
2023-08-17T12:12:44.3315519Z running benchmark:  30% 9/30 [00:01<00:02,  8.26it/s]
2023-08-17T12:12:44.4525826Z running benchmark:  33% 10/30 [00:01<00:02,  8.25it/s]
2023-08-17T12:12:44.5930214Z running benchmark:  37% 11/30 [00:01<00:02,  8.25it/s]
2023-08-17T12:12:44.7133339Z running benchmark:  40% 12/30 [00:01<00:02,  7.87it/s]
2023-08-17T12:12:44.8334401Z running benchmark:  43% 13/30 [00:01<00:02,  8.00it/s]
2023-08-17T12:12:44.9536861Z running benchmark:  47% 14/30 [00:01<00:01,  8.09it/s]
2023-08-17T12:12:45.0738735Z running benchmark:  50% 15/30 [00:01<00:01,  8.16it/s]
2023-08-17T12:12:45.1940804Z running benchmark:  53% 16/30 [00:01<00:01,  8.21it/s]
2023-08-17T12:12:45.3142355Z running benchmark:  57% 17/30 [00:02<00:01,  8.24it/s]
2023-08-17T12:12:45.4346556Z running benchmark:  60% 18/30 [00:02<00:01,  8.27it/s]
2023-08-17T12:12:45.5621689Z running benchmark:  63% 19/30 [00:02<00:01,  8.28it/s]
2023-08-17T12:12:45.6830233Z running benchmark:  67% 20/30 [00:02<00:01,  8.15it/s]
2023-08-17T12:12:45.8031354Z running benchmark:  70% 21/30 [00:02<00:01,  8.18it/s]
2023-08-17T12:12:45.9232101Z running benchmark:  73% 22/30 [00:02<00:00,  8.22it/s]
2023-08-17T12:12:46.0431627Z running benchmark:  77% 23/30 [00:02<00:00,  8.26it/s]
2023-08-17T12:12:46.1631542Z running benchmark:  80% 24/30 [00:02<00:00,  8.28it/s]
2023-08-17T12:12:46.2833234Z running benchmark:  83% 25/30 [00:03<00:00,  8.29it/s]
2023-08-17T12:12:46.4033729Z running benchmark:  87% 26/30 [00:03<00:00,  8.30it/s]
2023-08-17T12:12:46.5234320Z running benchmark:  90% 27/30 [00:03<00:00,  8.31it/s]
2023-08-17T12:12:46.6442958Z running benchmark:  93% 28/30 [00:03<00:00,  8.32it/s]
2023-08-17T12:12:46.7653582Z running benchmark:  97% 29/30 [00:03<00:00,  8.31it/s]
2023-08-17T12:12:46.7654125Z running benchmark: 100% 30/30 [00:03<00:00,  8.29it/s]
2023-08-17T12:12:46.7654611Z running benchmark: 100% 30/30 [00:03<00:00,  8.14it/s]
2023-08-17T12:12:46.7682473Z 1.358x
2023-08-17T12:12:51.7330902Z 
2023-08-17T12:13:05.3153234Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:13:05.3153608Z loading model: 0it [00:13, ?it/s]
2023-08-17T12:13:05.3153990Z cuda train PegasusForConditionalGeneration    
2023-08-17T12:14:52.2783451Z 
2023-08-17T12:14:52.5405997Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:14:52.7789692Z running benchmark:   3% 1/30 [00:00<00:07,  3.81it/s]
2023-08-17T12:14:53.0160843Z running benchmark:   7% 2/30 [00:00<00:06,  4.02it/s]
2023-08-17T12:14:53.2529176Z running benchmark:  10% 3/30 [00:00<00:06,  4.11it/s]
2023-08-17T12:14:53.4893645Z running benchmark:  13% 4/30 [00:00<00:06,  4.15it/s]
2023-08-17T12:14:53.7256926Z running benchmark:  17% 5/30 [00:01<00:05,  4.18it/s]
2023-08-17T12:14:53.9638414Z running benchmark:  20% 6/30 [00:01<00:05,  4.20it/s]
2023-08-17T12:14:54.2012793Z running benchmark:  23% 7/30 [00:01<00:05,  4.20it/s]
2023-08-17T12:14:54.4381931Z running benchmark:  27% 8/30 [00:01<00:05,  4.20it/s]
2023-08-17T12:14:54.6839006Z running benchmark:  30% 9/30 [00:02<00:04,  4.21it/s]
2023-08-17T12:14:54.9201144Z running benchmark:  33% 10/30 [00:02<00:04,  4.17it/s]
2023-08-17T12:14:55.1581072Z running benchmark:  37% 11/30 [00:02<00:04,  4.18it/s]
2023-08-17T12:14:55.3954009Z running benchmark:  40% 12/30 [00:02<00:04,  4.19it/s]
2023-08-17T12:14:55.6318731Z running benchmark:  43% 13/30 [00:03<00:04,  4.20it/s]
2023-08-17T12:14:55.8681666Z running benchmark:  47% 14/30 [00:03<00:03,  4.21it/s]
2023-08-17T12:14:56.1049290Z running benchmark:  50% 15/30 [00:03<00:03,  4.21it/s]
2023-08-17T12:14:56.3428561Z running benchmark:  53% 16/30 [00:03<00:03,  4.22it/s]
2023-08-17T12:14:56.5806738Z running benchmark:  57% 17/30 [00:04<00:03,  4.21it/s]
2023-08-17T12:14:56.8219966Z running benchmark:  60% 18/30 [00:04<00:02,  4.21it/s]
2023-08-17T12:14:57.0582455Z running benchmark:  63% 19/30 [00:04<00:02,  4.19it/s]
2023-08-17T12:14:57.2945691Z running benchmark:  67% 20/30 [00:04<00:02,  4.20it/s]
2023-08-17T12:14:57.5320505Z running benchmark:  70% 21/30 [00:05<00:02,  4.21it/s]
2023-08-17T12:14:57.7709256Z running benchmark:  73% 22/30 [00:05<00:01,  4.21it/s]
2023-08-17T12:14:58.0074432Z running benchmark:  77% 23/30 [00:05<00:01,  4.20it/s]
2023-08-17T12:14:58.2436814Z running benchmark:  80% 24/30 [00:05<00:01,  4.21it/s]
2023-08-17T12:14:58.4797085Z running benchmark:  83% 25/30 [00:05<00:01,  4.22it/s]
2023-08-17T12:14:58.7165067Z running benchmark:  87% 26/30 [00:06<00:00,  4.22it/s]
2023-08-17T12:14:58.9546036Z running benchmark:  90% 27/30 [00:06<00:00,  4.22it/s]
2023-08-17T12:14:59.1913454Z running benchmark:  93% 28/30 [00:06<00:00,  4.22it/s]
2023-08-17T12:14:59.4280973Z running benchmark:  97% 29/30 [00:06<00:00,  4.22it/s]
2023-08-17T12:14:59.4282058Z running benchmark: 100% 30/30 [00:07<00:00,  4.22it/s]
2023-08-17T12:14:59.4285834Z running benchmark: 100% 30/30 [00:07<00:00,  4.20it/s]
2023-08-17T12:14:59.4309155Z 1.386x
2023-08-17T12:15:04.9869457Z 
2023-08-17T12:15:04.9898680Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`
2023-08-17T12:15:08.2237210Z 
2023-08-17T12:15:08.2238209Z loading model: 0it [00:03, ?it/s]
2023-08-17T12:15:08.2239085Z cuda train RobertaForCausalLM                 
2023-08-17T12:15:59.8616531Z 
2023-08-17T12:16:00.1020082Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:16:00.2871915Z running benchmark:   3% 1/30 [00:00<00:06,  4.15it/s]
2023-08-17T12:16:00.4640650Z running benchmark:   7% 2/30 [00:00<00:05,  4.80it/s]
2023-08-17T12:16:00.6412889Z running benchmark:  10% 3/30 [00:00<00:05,  5.15it/s]
2023-08-17T12:16:00.8189328Z running benchmark:  13% 4/30 [00:00<00:04,  5.34it/s]
2023-08-17T12:16:00.9963439Z running benchmark:  17% 5/30 [00:00<00:04,  5.44it/s]
2023-08-17T12:16:01.1745685Z running benchmark:  20% 6/30 [00:01<00:04,  5.51it/s]
2023-08-17T12:16:01.3521926Z running benchmark:  23% 7/30 [00:01<00:04,  5.54it/s]
2023-08-17T12:16:01.5293872Z running benchmark:  27% 8/30 [00:01<00:03,  5.57it/s]
2023-08-17T12:16:01.7074644Z running benchmark:  30% 9/30 [00:01<00:03,  5.59it/s]
2023-08-17T12:16:01.8855910Z running benchmark:  33% 10/30 [00:01<00:03,  5.60it/s]
2023-08-17T12:16:02.0634266Z running benchmark:  37% 11/30 [00:02<00:03,  5.60it/s]
2023-08-17T12:16:02.2423566Z running benchmark:  40% 12/30 [00:02<00:03,  5.61it/s]
2023-08-17T12:16:02.4208470Z running benchmark:  43% 13/30 [00:02<00:03,  5.61it/s]
2023-08-17T12:16:02.5991235Z running benchmark:  47% 14/30 [00:02<00:02,  5.60it/s]
2023-08-17T12:16:02.7774081Z running benchmark:  50% 15/30 [00:02<00:02,  5.60it/s]
2023-08-17T12:16:02.9551446Z running benchmark:  53% 16/30 [00:02<00:02,  5.61it/s]
2023-08-17T12:16:03.1331533Z running benchmark:  57% 17/30 [00:03<00:02,  5.61it/s]
2023-08-17T12:16:03.3115114Z running benchmark:  60% 18/30 [00:03<00:02,  5.61it/s]
2023-08-17T12:16:03.4902093Z running benchmark:  63% 19/30 [00:03<00:01,  5.61it/s]
2023-08-17T12:16:03.6683760Z running benchmark:  67% 20/30 [00:03<00:01,  5.61it/s]
2023-08-17T12:16:03.8465856Z running benchmark:  70% 21/30 [00:03<00:01,  5.61it/s]
2023-08-17T12:16:04.0242002Z running benchmark:  73% 22/30 [00:03<00:01,  5.61it/s]
2023-08-17T12:16:04.2015822Z running benchmark:  77% 23/30 [00:04<00:01,  5.62it/s]
2023-08-17T12:16:04.3786824Z running benchmark:  80% 24/30 [00:04<00:01,  5.62it/s]
2023-08-17T12:16:04.5562778Z running benchmark:  83% 25/30 [00:04<00:00,  5.63it/s]
2023-08-17T12:16:04.7339890Z running benchmark:  87% 26/30 [00:04<00:00,  5.63it/s]
2023-08-17T12:16:04.9111050Z running benchmark:  90% 27/30 [00:04<00:00,  5.63it/s]
2023-08-17T12:16:05.0882307Z running benchmark:  93% 28/30 [00:05<00:00,  5.63it/s]
2023-08-17T12:16:05.2650405Z running benchmark:  97% 29/30 [00:05<00:00,  5.64it/s]
2023-08-17T12:16:05.2650875Z running benchmark: 100% 30/30 [00:05<00:00,  5.64it/s]
2023-08-17T12:16:05.2651171Z running benchmark: 100% 30/30 [00:05<00:00,  5.55it/s]
2023-08-17T12:16:05.2675604Z 1.837x
2023-08-17T12:16:09.9963801Z 
2023-08-17T12:16:12.9194845Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:16:12.9195308Z loading model: 0it [00:02, ?it/s]
2023-08-17T12:16:12.9195821Z cuda train RobertaForQuestionAnswering        
2023-08-17T12:16:59.2611843Z 
2023-08-17T12:16:59.4539237Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:16:59.5966471Z running benchmark:   3% 1/30 [00:00<00:05,  5.18it/s]
2023-08-17T12:16:59.7395973Z running benchmark:   7% 2/30 [00:00<00:04,  6.11it/s]
2023-08-17T12:16:59.8824282Z running benchmark:  10% 3/30 [00:00<00:04,  6.48it/s]
2023-08-17T12:17:00.0252813Z running benchmark:  13% 4/30 [00:00<00:03,  6.68it/s]
2023-08-17T12:17:00.1680667Z running benchmark:  17% 5/30 [00:00<00:03,  6.79it/s]
2023-08-17T12:17:00.3118129Z running benchmark:  20% 6/30 [00:00<00:03,  6.86it/s]
2023-08-17T12:17:00.4547512Z running benchmark:  23% 7/30 [00:01<00:03,  6.89it/s]
2023-08-17T12:17:00.5981160Z running benchmark:  27% 8/30 [00:01<00:03,  6.92it/s]
2023-08-17T12:17:00.7407396Z running benchmark:  30% 9/30 [00:01<00:03,  6.94it/s]
2023-08-17T12:17:00.8836479Z running benchmark:  33% 10/30 [00:01<00:02,  6.96it/s]
2023-08-17T12:17:01.0271280Z running benchmark:  37% 11/30 [00:01<00:02,  6.97it/s]
2023-08-17T12:17:01.1699222Z running benchmark:  40% 12/30 [00:01<00:02,  6.97it/s]
2023-08-17T12:17:01.3128586Z running benchmark:  43% 13/30 [00:01<00:02,  6.98it/s]
2023-08-17T12:17:01.4562339Z running benchmark:  47% 14/30 [00:02<00:02,  6.99it/s]
2023-08-17T12:17:01.6080072Z running benchmark:  50% 15/30 [00:02<00:02,  6.98it/s]
2023-08-17T12:17:01.7511200Z running benchmark:  53% 16/30 [00:02<00:02,  6.86it/s]
2023-08-17T12:17:01.8950290Z running benchmark:  57% 17/30 [00:02<00:01,  6.90it/s]
2023-08-17T12:17:02.0385001Z running benchmark:  60% 18/30 [00:02<00:01,  6.91it/s]
2023-08-17T12:17:02.1817380Z running benchmark:  63% 19/30 [00:02<00:01,  6.93it/s]
2023-08-17T12:17:02.3248688Z running benchmark:  67% 20/30 [00:02<00:01,  6.94it/s]
2023-08-17T12:17:02.4682414Z running benchmark:  70% 21/30 [00:03<00:01,  6.96it/s]
2023-08-17T12:17:02.6168578Z running benchmark:  73% 22/30 [00:03<00:01,  6.96it/s]
2023-08-17T12:17:02.7612844Z running benchmark:  77% 23/30 [00:03<00:01,  6.89it/s]
2023-08-17T12:17:02.9047852Z running benchmark:  80% 24/30 [00:03<00:00,  6.90it/s]
2023-08-17T12:17:03.0481276Z running benchmark:  83% 25/30 [00:03<00:00,  6.92it/s]
2023-08-17T12:17:03.1912754Z running benchmark:  87% 26/30 [00:03<00:00,  6.94it/s]
2023-08-17T12:17:03.3345472Z running benchmark:  90% 27/30 [00:03<00:00,  6.95it/s]
2023-08-17T12:17:03.4779192Z running benchmark:  93% 28/30 [00:04<00:00,  6.96it/s]
2023-08-17T12:17:03.6210371Z running benchmark:  97% 29/30 [00:04<00:00,  6.96it/s]
2023-08-17T12:17:03.6210862Z running benchmark: 100% 30/30 [00:04<00:00,  6.97it/s]
2023-08-17T12:17:03.6211186Z running benchmark: 100% 30/30 [00:04<00:00,  6.88it/s]
2023-08-17T12:17:03.6237156Z 1.997x
2023-08-17T12:17:08.4191181Z 
2023-08-17T12:17:09.6598211Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:17:09.6598914Z loading model: 0it [00:01, ?it/s]
2023-08-17T12:17:09.6599233Z cuda train Speech2Text2ForCausalLM            
2023-08-17T12:17:41.1472847Z 
2023-08-17T12:17:41.2989803Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:17:41.4745496Z running benchmark:   3% 1/30 [00:00<00:04,  6.56it/s]
2023-08-17T12:17:41.6533951Z running benchmark:  10% 3/30 [00:00<00:02,  9.56it/s]
2023-08-17T12:17:41.8268747Z running benchmark:  17% 5/30 [00:00<00:02, 10.33it/s]
2023-08-17T12:17:41.9998113Z running benchmark:  23% 7/30 [00:00<00:02, 10.81it/s]
2023-08-17T12:17:42.1731126Z running benchmark:  30% 9/30 [00:00<00:01, 11.08it/s]
2023-08-17T12:17:42.3459822Z running benchmark:  37% 11/30 [00:01<00:01, 11.24it/s]
2023-08-17T12:17:42.5191248Z running benchmark:  43% 13/30 [00:01<00:01, 11.34it/s]
2023-08-17T12:17:42.6931956Z running benchmark:  50% 15/30 [00:01<00:01, 11.41it/s]
2023-08-17T12:17:42.8661231Z running benchmark:  57% 17/30 [00:01<00:01, 11.43it/s]
2023-08-17T12:17:43.0389349Z running benchmark:  63% 19/30 [00:01<00:00, 11.47it/s]
2023-08-17T12:17:43.2118181Z running benchmark:  70% 21/30 [00:01<00:00, 11.51it/s]
2023-08-17T12:17:43.3845940Z running benchmark:  77% 23/30 [00:02<00:00, 11.52it/s]
2023-08-17T12:17:43.5573157Z running benchmark:  83% 25/30 [00:02<00:00, 11.54it/s]
2023-08-17T12:17:43.7310306Z running benchmark:  90% 27/30 [00:02<00:00, 11.55it/s]
2023-08-17T12:17:43.8180790Z running benchmark:  97% 29/30 [00:02<00:00, 11.54it/s]
2023-08-17T12:17:43.8181285Z running benchmark: 100% 30/30 [00:02<00:00, 11.23it/s]
2023-08-17T12:17:43.8203039Z 1.613x
2023-08-17T12:17:48.2674797Z 
2023-08-17T12:17:50.6457195Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:17:50.6457893Z loading model: 0it [00:02, ?it/s]
2023-08-17T12:17:50.6458702Z cuda train T5ForConditionalGeneration         
2023-08-17T12:18:53.9423050Z 
2023-08-17T12:18:54.1632297Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:18:54.3294616Z running benchmark:   3% 1/30 [00:00<00:06,  4.51it/s]
2023-08-17T12:18:54.4964726Z running benchmark:   7% 2/30 [00:00<00:05,  5.29it/s]
2023-08-17T12:18:54.6620792Z running benchmark:  10% 3/30 [00:00<00:04,  5.59it/s]
2023-08-17T12:18:54.8272929Z running benchmark:  13% 4/30 [00:00<00:04,  5.75it/s]
2023-08-17T12:18:54.9928722Z running benchmark:  17% 5/30 [00:00<00:04,  5.86it/s]
2023-08-17T12:18:55.1587754Z running benchmark:  20% 6/30 [00:01<00:04,  5.92it/s]
2023-08-17T12:18:55.3243016Z running benchmark:  23% 7/30 [00:01<00:03,  5.96it/s]
2023-08-17T12:18:55.4901610Z running benchmark:  27% 8/30 [00:01<00:03,  5.98it/s]
2023-08-17T12:18:55.6566509Z running benchmark:  30% 9/30 [00:01<00:03,  6.00it/s]
2023-08-17T12:18:55.8225705Z running benchmark:  33% 10/30 [00:01<00:03,  6.00it/s]
2023-08-17T12:18:55.9881802Z running benchmark:  37% 11/30 [00:01<00:03,  6.01it/s]
2023-08-17T12:18:56.1535641Z running benchmark:  40% 12/30 [00:02<00:02,  6.02it/s]
2023-08-17T12:18:56.3195727Z running benchmark:  43% 13/30 [00:02<00:02,  6.03it/s]
2023-08-17T12:18:56.4856271Z running benchmark:  47% 14/30 [00:02<00:02,  6.03it/s]
2023-08-17T12:18:56.6512651Z running benchmark:  50% 15/30 [00:02<00:02,  6.02it/s]
2023-08-17T12:18:56.8176881Z running benchmark:  53% 16/30 [00:02<00:02,  6.03it/s]
2023-08-17T12:18:56.9832004Z running benchmark:  57% 17/30 [00:02<00:02,  6.02it/s]
2023-08-17T12:18:57.1488165Z running benchmark:  60% 18/30 [00:03<00:01,  6.03it/s]
2023-08-17T12:18:57.3195537Z running benchmark:  63% 19/30 [00:03<00:01,  6.03it/s]
2023-08-17T12:18:57.4852909Z running benchmark:  67% 20/30 [00:03<00:01,  5.98it/s]
2023-08-17T12:18:57.6508546Z running benchmark:  70% 21/30 [00:03<00:01,  5.99it/s]
2023-08-17T12:18:57.8164248Z running benchmark:  73% 22/30 [00:03<00:01,  6.01it/s]
2023-08-17T12:18:57.9824393Z running benchmark:  77% 23/30 [00:03<00:01,  6.02it/s]
2023-08-17T12:18:58.1481639Z running benchmark:  80% 24/30 [00:04<00:00,  6.02it/s]
2023-08-17T12:18:58.3137659Z running benchmark:  83% 25/30 [00:04<00:00,  6.02it/s]
2023-08-17T12:18:58.4793118Z running benchmark:  87% 26/30 [00:04<00:00,  6.03it/s]
2023-08-17T12:18:58.6448142Z running benchmark:  90% 27/30 [00:04<00:00,  6.03it/s]
2023-08-17T12:18:58.8101493Z running benchmark:  93% 28/30 [00:04<00:00,  6.03it/s]
2023-08-17T12:18:58.9753944Z running benchmark:  97% 29/30 [00:04<00:00,  6.04it/s]
2023-08-17T12:18:58.9754326Z running benchmark: 100% 30/30 [00:05<00:00,  6.04it/s]
2023-08-17T12:18:58.9754634Z running benchmark: 100% 30/30 [00:05<00:00,  5.96it/s]
2023-08-17T12:18:58.9782089Z 1.690x
2023-08-17T12:19:04.1368243Z 
2023-08-17T12:19:06.6566303Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:19:06.6566633Z loading model: 0it [00:02, ?it/s]
2023-08-17T12:19:06.6566921Z cuda train T5Small                            
2023-08-17T12:20:10.1128633Z 
2023-08-17T12:20:10.3347653Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:20:10.5007386Z running benchmark:   3% 1/30 [00:00<00:06,  4.50it/s]
2023-08-17T12:20:10.6671250Z running benchmark:   7% 2/30 [00:00<00:05,  5.28it/s]
2023-08-17T12:20:10.8330911Z running benchmark:  10% 3/30 [00:00<00:04,  5.59it/s]
2023-08-17T12:20:11.0132329Z running benchmark:  13% 4/30 [00:00<00:04,  5.75it/s]
2023-08-17T12:20:11.1791927Z running benchmark:  17% 5/30 [00:00<00:04,  5.68it/s]
2023-08-17T12:20:11.3449845Z running benchmark:  20% 6/30 [00:01<00:04,  5.79it/s]
2023-08-17T12:20:11.5104881Z running benchmark:  23% 7/30 [00:01<00:03,  5.87it/s]
2023-08-17T12:20:11.6759610Z running benchmark:  27% 8/30 [00:01<00:03,  5.92it/s]
2023-08-17T12:20:11.8432818Z running benchmark:  30% 9/30 [00:01<00:03,  5.96it/s]
2023-08-17T12:20:12.0096617Z running benchmark:  33% 10/30 [00:01<00:03,  5.97it/s]
2023-08-17T12:20:12.1833594Z running benchmark:  37% 11/30 [00:01<00:03,  5.98it/s]
2023-08-17T12:20:12.3491762Z running benchmark:  40% 12/30 [00:02<00:03,  5.91it/s]
2023-08-17T12:20:12.5150925Z running benchmark:  43% 13/30 [00:02<00:02,  5.95it/s]
2023-08-17T12:20:12.6808226Z running benchmark:  47% 14/30 [00:02<00:02,  5.97it/s]
2023-08-17T12:20:12.8463018Z running benchmark:  50% 15/30 [00:02<00:02,  5.99it/s]
2023-08-17T12:20:13.0130593Z running benchmark:  53% 16/30 [00:02<00:02,  6.01it/s]
2023-08-17T12:20:13.1806398Z running benchmark:  57% 17/30 [00:02<00:02,  6.00it/s]
2023-08-17T12:20:13.3462743Z running benchmark:  60% 18/30 [00:03<00:02,  5.99it/s]
2023-08-17T12:20:13.5118709Z running benchmark:  63% 19/30 [00:03<00:01,  6.01it/s]
2023-08-17T12:20:13.6774342Z running benchmark:  67% 20/30 [00:03<00:01,  6.02it/s]
2023-08-17T12:20:13.8427545Z running benchmark:  70% 21/30 [00:03<00:01,  6.02it/s]
2023-08-17T12:20:14.0089802Z running benchmark:  73% 22/30 [00:03<00:01,  6.03it/s]
2023-08-17T12:20:14.1759032Z running benchmark:  77% 23/30 [00:03<00:01,  6.03it/s]
2023-08-17T12:20:14.3420832Z running benchmark:  80% 24/30 [00:04<00:00,  6.02it/s]
2023-08-17T12:20:14.5080031Z running benchmark:  83% 25/30 [00:04<00:00,  6.02it/s]
2023-08-17T12:20:14.6738663Z running benchmark:  87% 26/30 [00:04<00:00,  6.02it/s]
2023-08-17T12:20:14.8394799Z running benchmark:  90% 27/30 [00:04<00:00,  6.02it/s]
2023-08-17T12:20:15.0051551Z running benchmark:  93% 28/30 [00:04<00:00,  6.03it/s]
2023-08-17T12:20:15.1713141Z running benchmark:  97% 29/30 [00:04<00:00,  6.03it/s]
2023-08-17T12:20:15.1714048Z running benchmark: 100% 30/30 [00:05<00:00,  6.03it/s]
2023-08-17T12:20:15.1717804Z running benchmark: 100% 30/30 [00:05<00:00,  5.93it/s]
2023-08-17T12:20:15.1741777Z 1.694x
2023-08-17T12:20:19.9995384Z 
2023-08-17T12:20:26.2131532Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:20:26.2132032Z loading model: 0it [00:06, ?it/s]
2023-08-17T12:20:26.2132333Z cuda train TrOCRForCausalLM                   
2023-08-17T12:21:21.4687974Z 
2023-08-17T12:21:21.7823952Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:21:22.0118811Z running benchmark:   3% 1/30 [00:00<00:09,  3.18it/s]
2023-08-17T12:21:22.2445279Z running benchmark:   7% 2/30 [00:00<00:07,  3.78it/s]
2023-08-17T12:21:22.4737243Z running benchmark:  10% 3/30 [00:00<00:06,  4.00it/s]
2023-08-17T12:21:22.7036423Z running benchmark:  13% 4/30 [00:01<00:06,  4.14it/s]
2023-08-17T12:21:22.9333969Z running benchmark:  17% 5/30 [00:01<00:05,  4.21it/s]
2023-08-17T12:21:23.1629599Z running benchmark:  20% 6/30 [00:01<00:05,  4.26it/s]
2023-08-17T12:21:23.3920590Z running benchmark:  23% 7/30 [00:01<00:05,  4.29it/s]
2023-08-17T12:21:23.6208827Z running benchmark:  27% 8/30 [00:01<00:05,  4.31it/s]
2023-08-17T12:21:23.8513935Z running benchmark:  30% 9/30 [00:02<00:04,  4.33it/s]
2023-08-17T12:21:24.0811117Z running benchmark:  33% 10/30 [00:02<00:04,  4.33it/s]
2023-08-17T12:21:24.3101888Z running benchmark:  37% 11/30 [00:02<00:04,  4.34it/s]
2023-08-17T12:21:24.5395374Z running benchmark:  40% 12/30 [00:02<00:04,  4.35it/s]
2023-08-17T12:21:24.7688604Z running benchmark:  43% 13/30 [00:03<00:03,  4.35it/s]
2023-08-17T12:21:24.9992590Z running benchmark:  47% 14/30 [00:03<00:03,  4.35it/s]
2023-08-17T12:21:25.2286492Z running benchmark:  50% 15/30 [00:03<00:03,  4.35it/s]
2023-08-17T12:21:25.4577428Z running benchmark:  53% 16/30 [00:03<00:03,  4.35it/s]
2023-08-17T12:21:25.6866429Z running benchmark:  57% 17/30 [00:03<00:02,  4.36it/s]
2023-08-17T12:21:25.9155338Z running benchmark:  60% 18/30 [00:04<00:02,  4.36it/s]
2023-08-17T12:21:26.1455443Z running benchmark:  63% 19/30 [00:04<00:02,  4.36it/s]
2023-08-17T12:21:26.3751660Z running benchmark:  67% 20/30 [00:04<00:02,  4.36it/s]
2023-08-17T12:21:26.6042999Z running benchmark:  70% 21/30 [00:04<00:02,  4.36it/s]
2023-08-17T12:21:26.8344340Z running benchmark:  73% 22/30 [00:05<00:01,  4.36it/s]
2023-08-17T12:21:27.0633423Z running benchmark:  77% 23/30 [00:05<00:01,  4.36it/s]
2023-08-17T12:21:27.2928667Z running benchmark:  80% 24/30 [00:05<00:01,  4.36it/s]
2023-08-17T12:21:27.5221133Z running benchmark:  83% 25/30 [00:05<00:01,  4.36it/s]
2023-08-17T12:21:27.7509960Z running benchmark:  87% 26/30 [00:06<00:00,  4.36it/s]
2023-08-17T12:21:27.9803751Z running benchmark:  90% 27/30 [00:06<00:00,  4.36it/s]
2023-08-17T12:21:28.2093881Z running benchmark:  93% 28/30 [00:06<00:00,  4.36it/s]
2023-08-17T12:21:28.4390861Z running benchmark:  97% 29/30 [00:06<00:00,  4.36it/s]
2023-08-17T12:21:28.4391215Z running benchmark: 100% 30/30 [00:06<00:00,  4.36it/s]
2023-08-17T12:21:28.4391515Z running benchmark: 100% 30/30 [00:06<00:00,  4.30it/s]
2023-08-17T12:21:28.4419321Z 1.502x
2023-08-17T12:21:33.2615823Z 
2023-08-17T12:21:48.0404921Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:21:48.0405374Z loading model: 0it [00:14, ?it/s]
2023-08-17T12:21:48.0405678Z cuda train XGLMForCausalLM                    
2023-08-17T12:23:20.1806243Z 
2023-08-17T12:23:20.3342418Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:23:20.4868884Z running benchmark:   3% 1/30 [00:00<00:04,  6.49it/s]
2023-08-17T12:23:20.6317335Z running benchmark:   7% 2/30 [00:00<00:04,  6.52it/s]
2023-08-17T12:23:20.7769659Z running benchmark:  10% 3/30 [00:00<00:04,  6.69it/s]
2023-08-17T12:23:20.9205618Z running benchmark:  13% 4/30 [00:00<00:03,  6.77it/s]
2023-08-17T12:23:21.0655297Z running benchmark:  17% 5/30 [00:00<00:03,  6.84it/s]
2023-08-17T12:23:21.2174293Z running benchmark:  20% 6/30 [00:00<00:03,  6.86it/s]
2023-08-17T12:23:21.3657627Z running benchmark:  23% 7/30 [00:01<00:03,  6.77it/s]
2023-08-17T12:23:21.5246107Z running benchmark:  27% 8/30 [00:01<00:03,  6.76it/s]
2023-08-17T12:23:21.6697255Z running benchmark:  30% 9/30 [00:01<00:03,  6.61it/s]
2023-08-17T12:23:21.8137081Z running benchmark:  33% 10/30 [00:01<00:02,  6.69it/s]
2023-08-17T12:23:21.9579301Z running benchmark:  37% 11/30 [00:01<00:02,  6.77it/s]
2023-08-17T12:23:22.1013701Z running benchmark:  40% 12/30 [00:01<00:02,  6.82it/s]
2023-08-17T12:23:22.2443893Z running benchmark:  43% 13/30 [00:01<00:02,  6.86it/s]
2023-08-17T12:23:22.3923334Z running benchmark:  47% 14/30 [00:02<00:02,  6.90it/s]
2023-08-17T12:23:22.5522881Z running benchmark:  50% 15/30 [00:02<00:02,  6.86it/s]
2023-08-17T12:23:22.6968850Z running benchmark:  53% 16/30 [00:02<00:02,  6.66it/s]
2023-08-17T12:23:22.8408203Z running benchmark:  57% 17/30 [00:02<00:01,  6.74it/s]
2023-08-17T12:23:22.9850634Z running benchmark:  60% 18/30 [00:02<00:01,  6.80it/s]
2023-08-17T12:23:23.1283785Z running benchmark:  63% 19/30 [00:02<00:01,  6.84it/s]
2023-08-17T12:23:23.2717522Z running benchmark:  67% 20/30 [00:02<00:01,  6.88it/s]
2023-08-17T12:23:23.4159644Z running benchmark:  70% 21/30 [00:03<00:01,  6.91it/s]
2023-08-17T12:23:23.5657440Z running benchmark:  73% 22/30 [00:03<00:01,  6.92it/s]
2023-08-17T12:23:23.7129830Z running benchmark:  77% 23/30 [00:03<00:01,  6.84it/s]
2023-08-17T12:23:23.8568138Z running benchmark:  80% 24/30 [00:03<00:00,  6.83it/s]
2023-08-17T12:23:24.0013655Z running benchmark:  83% 25/30 [00:03<00:00,  6.86it/s]
2023-08-17T12:23:24.1445786Z running benchmark:  87% 26/30 [00:03<00:00,  6.88it/s]
2023-08-17T12:23:24.2882726Z running benchmark:  90% 27/30 [00:03<00:00,  6.91it/s]
2023-08-17T12:23:24.4324728Z running benchmark:  93% 28/30 [00:04<00:00,  6.93it/s]
2023-08-17T12:23:24.5838078Z running benchmark:  97% 29/30 [00:04<00:00,  6.93it/s]
2023-08-17T12:23:24.5838421Z running benchmark: 100% 30/30 [00:04<00:00,  6.83it/s]
2023-08-17T12:23:24.5838764Z running benchmark: 100% 30/30 [00:04<00:00,  6.81it/s]
2023-08-17T12:23:24.5866434Z 1.370x
2023-08-17T12:23:29.8425770Z 
2023-08-17T12:23:36.8531663Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:23:36.8532573Z loading model: 0it [00:07, ?it/s]
2023-08-17T12:23:36.8532878Z cuda train XLNetLMHeadModel                   
2023-08-17T12:24:39.7475986Z [2023-08-17 12:24:39,745] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-08-17T12:25:39.7227912Z 
2023-08-17T12:25:40.2061970Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:25:40.6386553Z running benchmark:   3% 1/30 [00:00<00:14,  2.07it/s]
2023-08-17T12:25:41.0686364Z running benchmark:   7% 2/30 [00:00<00:12,  2.20it/s]
2023-08-17T12:25:41.4974636Z running benchmark:  10% 3/30 [00:01<00:11,  2.26it/s]
2023-08-17T12:25:41.9278554Z running benchmark:  13% 4/30 [00:01<00:11,  2.29it/s]
2023-08-17T12:25:42.3570140Z running benchmark:  17% 5/30 [00:02<00:10,  2.30it/s]
2023-08-17T12:25:42.7868501Z running benchmark:  20% 6/30 [00:02<00:10,  2.31it/s]
2023-08-17T12:25:43.2205251Z running benchmark:  23% 7/30 [00:03<00:09,  2.32it/s]
2023-08-17T12:25:43.6507143Z running benchmark:  27% 8/30 [00:03<00:09,  2.31it/s]
2023-08-17T12:25:44.0833695Z running benchmark:  30% 9/30 [00:03<00:09,  2.32it/s]
2023-08-17T12:25:44.5180398Z running benchmark:  33% 10/30 [00:04<00:08,  2.31it/s]
2023-08-17T12:25:44.9542206Z running benchmark:  37% 11/30 [00:04<00:08,  2.31it/s]
2023-08-17T12:25:45.3853919Z running benchmark:  40% 12/30 [00:05<00:07,  2.30it/s]
2023-08-17T12:25:45.8145447Z running benchmark:  43% 13/30 [00:05<00:07,  2.31it/s]
2023-08-17T12:25:46.2465804Z running benchmark:  47% 14/30 [00:06<00:06,  2.32it/s]
2023-08-17T12:25:46.6771383Z running benchmark:  50% 15/30 [00:06<00:06,  2.32it/s]
2023-08-17T12:25:47.1069070Z running benchmark:  53% 16/30 [00:06<00:06,  2.32it/s]
2023-08-17T12:25:47.5468978Z running benchmark:  57% 17/30 [00:07<00:05,  2.32it/s]
2023-08-17T12:25:47.9769166Z running benchmark:  60% 18/30 [00:07<00:05,  2.31it/s]
2023-08-17T12:25:48.4074927Z running benchmark:  63% 19/30 [00:08<00:04,  2.31it/s]
2023-08-17T12:25:48.8377935Z running benchmark:  67% 20/30 [00:08<00:04,  2.31it/s]
2023-08-17T12:25:49.2674693Z running benchmark:  70% 21/30 [00:09<00:03,  2.32it/s]
2023-08-17T12:25:49.6966545Z running benchmark:  73% 22/30 [00:09<00:03,  2.32it/s]
2023-08-17T12:25:50.1367150Z running benchmark:  77% 23/30 [00:09<00:03,  2.32it/s]
2023-08-17T12:25:50.5699247Z running benchmark:  80% 24/30 [00:10<00:02,  2.31it/s]
2023-08-17T12:25:51.0071090Z running benchmark:  83% 25/30 [00:10<00:02,  2.31it/s]
2023-08-17T12:25:51.4416502Z running benchmark:  87% 26/30 [00:11<00:01,  2.30it/s]
2023-08-17T12:25:51.8719336Z running benchmark:  90% 27/30 [00:11<00:01,  2.30it/s]
2023-08-17T12:25:52.3045808Z running benchmark:  93% 28/30 [00:12<00:00,  2.31it/s]
2023-08-17T12:25:52.7340713Z running benchmark:  97% 29/30 [00:12<00:00,  2.31it/s]
2023-08-17T12:25:52.7341303Z running benchmark: 100% 30/30 [00:13<00:00,  2.31it/s]
2023-08-17T12:25:52.7341788Z running benchmark: 100% 30/30 [00:13<00:00,  2.31it/s]
2023-08-17T12:25:52.7369916Z 1.861x
2023-08-17T12:25:58.5429373Z 
2023-08-17T12:26:02.0215118Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:26:02.0215528Z loading model: 0it [00:03, ?it/s]
2023-08-17T12:26:02.0215827Z cuda train YituTechConvBert                   
2023-08-17T12:27:16.6691791Z 
2023-08-17T12:27:16.9101710Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:27:17.1121944Z running benchmark:   3% 1/30 [00:00<00:07,  4.14it/s]
2023-08-17T12:27:17.3147003Z running benchmark:   7% 2/30 [00:00<00:06,  4.58it/s]
2023-08-17T12:27:17.5178381Z running benchmark:  10% 3/30 [00:00<00:05,  4.74it/s]
2023-08-17T12:27:17.7198003Z running benchmark:  13% 4/30 [00:00<00:05,  4.81it/s]
2023-08-17T12:27:17.9216710Z running benchmark:  17% 5/30 [00:01<00:05,  4.86it/s]
2023-08-17T12:27:18.1239248Z running benchmark:  20% 6/30 [00:01<00:04,  4.89it/s]
2023-08-17T12:27:18.3258242Z running benchmark:  23% 7/30 [00:01<00:04,  4.91it/s]
2023-08-17T12:27:18.5282391Z running benchmark:  27% 8/30 [00:01<00:04,  4.92it/s]
2023-08-17T12:27:18.7310247Z running benchmark:  30% 9/30 [00:01<00:04,  4.93it/s]
2023-08-17T12:27:18.9338272Z running benchmark:  33% 10/30 [00:02<00:04,  4.93it/s]
2023-08-17T12:27:19.1361405Z running benchmark:  37% 11/30 [00:02<00:03,  4.93it/s]
2023-08-17T12:27:19.3381139Z running benchmark:  40% 12/30 [00:02<00:03,  4.93it/s]
2023-08-17T12:27:19.5400062Z running benchmark:  43% 13/30 [00:02<00:03,  4.94it/s]
2023-08-17T12:27:19.7427797Z running benchmark:  47% 14/30 [00:02<00:03,  4.94it/s]
2023-08-17T12:27:19.9454345Z running benchmark:  50% 15/30 [00:03<00:03,  4.94it/s]
2023-08-17T12:27:20.1477086Z running benchmark:  53% 16/30 [00:03<00:02,  4.94it/s]
2023-08-17T12:27:20.3496972Z running benchmark:  57% 17/30 [00:03<00:02,  4.94it/s]
2023-08-17T12:27:20.5514966Z running benchmark:  60% 18/30 [00:03<00:02,  4.94it/s]
2023-08-17T12:27:20.7532233Z running benchmark:  63% 19/30 [00:03<00:02,  4.95it/s]
2023-08-17T12:27:20.9565527Z running benchmark:  67% 20/30 [00:04<00:02,  4.95it/s]
2023-08-17T12:27:21.1625477Z running benchmark:  70% 21/30 [00:04<00:01,  4.94it/s]
2023-08-17T12:27:21.3646716Z running benchmark:  73% 22/30 [00:04<00:01,  4.92it/s]
2023-08-17T12:27:21.5667084Z running benchmark:  77% 23/30 [00:04<00:01,  4.92it/s]
2023-08-17T12:27:21.7685548Z running benchmark:  80% 24/30 [00:04<00:01,  4.93it/s]
2023-08-17T12:27:21.9709454Z running benchmark:  83% 25/30 [00:05<00:01,  4.94it/s]
2023-08-17T12:27:22.1796261Z running benchmark:  87% 26/30 [00:05<00:00,  4.94it/s]
2023-08-17T12:27:22.3818849Z running benchmark:  90% 27/30 [00:05<00:00,  4.89it/s]
2023-08-17T12:27:22.5837616Z running benchmark:  93% 28/30 [00:05<00:00,  4.91it/s]
2023-08-17T12:27:22.7854275Z running benchmark:  97% 29/30 [00:05<00:00,  4.92it/s]
2023-08-17T12:27:22.7854621Z running benchmark: 100% 30/30 [00:06<00:00,  4.93it/s]
2023-08-17T12:27:22.7854909Z running benchmark: 100% 30/30 [00:06<00:00,  4.90it/s]
2023-08-17T12:27:22.7880776Z 1.616x
2023-08-17T12:27:24.5079524Z speedup             gmean=1.68x mean=1.703x
2023-08-17T12:27:24.5080001Z abs_latency         gmean=71.61x mean=76.655x
2023-08-17T12:27:24.5080758Z compilation_latency mean=78.317 seconds
2023-08-17T12:27:24.5082890Z compression_ratio   mean=1.095x
2023-08-17T12:27:24.5084939Z eager_peak_mem      gmean=11.07x mean=11.485x
2023-08-17T12:27:24.5087296Z dynamo_peak_mem     gmean=10.46x mean=11.193x
2023-08-17T12:27:24.5089695Z calls_captured      gmean=640.38x mean=727.235x
2023-08-17T12:27:24.5091787Z unique_graphs       gmean=1.00x mean=1.000x
2023-08-17T12:27:24.5093786Z graph_breaks        gmean=6.60x mean=6.647x
2023-08-17T12:27:24.5096278Z unique_graph_breaks gmean=5.59x mean=5.647x
2023-08-17T12:27:25.0018950Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *cudagraphs-true* ]]
2023-08-17T12:27:25.0022494Z + python benchmarks/dynamo/huggingface.py --performance --cold-start-latency --training --amp --backend inductor --device cuda --total-partitions 3 --partition-id 2 --output /var/lib/jenkins/workspace/test/test-reports/inductor_with_cudagraphs_huggingface_amp_training_cuda_performance.csv
2023-08-17T12:27:31.5330195Z 
2023-08-17T12:27:38.4061228Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:27:38.4061895Z loading model: 0it [00:06, ?it/s]
2023-08-17T12:27:38.4062227Z cuda train MegatronBertForQuestionAnswering   
2023-08-17T12:29:06.6437203Z 
2023-08-17T12:29:06.9977707Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:29:07.2161963Z running benchmark:   3% 1/30 [00:00<00:10,  2.82it/s]
2023-08-17T12:29:07.4340618Z running benchmark:   7% 2/30 [00:00<00:07,  3.64it/s]
2023-08-17T12:29:07.6522990Z running benchmark:  10% 3/30 [00:00<00:06,  4.02it/s]
2023-08-17T12:29:07.8708909Z running benchmark:  13% 4/30 [00:01<00:06,  4.23it/s]
2023-08-17T12:29:08.0890601Z running benchmark:  17% 5/30 [00:01<00:05,  4.35it/s]
2023-08-17T12:29:08.3067836Z running benchmark:  20% 6/30 [00:01<00:05,  4.42it/s]
2023-08-17T12:29:08.5242232Z running benchmark:  23% 7/30 [00:01<00:05,  4.48it/s]
2023-08-17T12:29:08.7418463Z running benchmark:  27% 8/30 [00:01<00:04,  4.51it/s]
2023-08-17T12:29:08.9600312Z running benchmark:  30% 9/30 [00:02<00:04,  4.54it/s]
2023-08-17T12:29:09.1778503Z running benchmark:  33% 10/30 [00:02<00:04,  4.55it/s]
2023-08-17T12:29:09.3953115Z running benchmark:  37% 11/30 [00:02<00:04,  4.56it/s]
2023-08-17T12:29:09.6127666Z running benchmark:  40% 12/30 [00:02<00:03,  4.57it/s]
2023-08-17T12:29:09.8302623Z running benchmark:  43% 13/30 [00:02<00:03,  4.58it/s]
2023-08-17T12:29:10.0492117Z running benchmark:  47% 14/30 [00:03<00:03,  4.59it/s]
2023-08-17T12:29:10.2669117Z running benchmark:  50% 15/30 [00:03<00:03,  4.58it/s]
2023-08-17T12:29:10.4843234Z running benchmark:  53% 16/30 [00:03<00:03,  4.58it/s]
2023-08-17T12:29:10.7016800Z running benchmark:  57% 17/30 [00:03<00:02,  4.59it/s]
2023-08-17T12:29:10.9214379Z running benchmark:  60% 18/30 [00:04<00:02,  4.59it/s]
2023-08-17T12:29:11.1396982Z running benchmark:  63% 19/30 [00:04<00:02,  4.58it/s]
2023-08-17T12:29:11.3578893Z running benchmark:  67% 20/30 [00:04<00:02,  4.58it/s]
2023-08-17T12:29:11.5756869Z running benchmark:  70% 21/30 [00:04<00:01,  4.58it/s]
2023-08-17T12:29:11.7933327Z running benchmark:  73% 22/30 [00:04<00:01,  4.58it/s]
2023-08-17T12:29:12.0113581Z running benchmark:  77% 23/30 [00:05<00:01,  4.59it/s]
2023-08-17T12:29:12.2294083Z running benchmark:  80% 24/30 [00:05<00:01,  4.59it/s]
2023-08-17T12:29:12.4476856Z running benchmark:  83% 25/30 [00:05<00:01,  4.59it/s]
2023-08-17T12:29:12.6652096Z running benchmark:  87% 26/30 [00:05<00:00,  4.59it/s]
2023-08-17T12:29:12.8910733Z running benchmark:  90% 27/30 [00:06<00:00,  4.59it/s]
2023-08-17T12:29:13.1088072Z running benchmark:  93% 28/30 [00:06<00:00,  4.54it/s]
2023-08-17T12:29:13.3263500Z running benchmark:  97% 29/30 [00:06<00:00,  4.56it/s]
2023-08-17T12:29:13.3263864Z running benchmark: 100% 30/30 [00:06<00:00,  4.57it/s]
2023-08-17T12:29:13.3268305Z running benchmark: 100% 30/30 [00:06<00:00,  4.49it/s]
2023-08-17T12:29:13.3291045Z 1.824x
2023-08-17T12:29:18.4893005Z 
2023-08-17T12:29:20.6726081Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:29:20.6726630Z loading model: 0it [00:02, ?it/s]
2023-08-17T12:29:20.6726946Z cuda train MobileBertForMaskedLM              
2023-08-17T12:32:24.3856079Z 
2023-08-17T12:32:24.8509650Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:32:25.1161474Z running benchmark:   3% 1/30 [00:00<00:13,  2.15it/s]
2023-08-17T12:32:25.3834385Z running benchmark:   7% 2/30 [00:00<00:09,  2.87it/s]
2023-08-17T12:32:25.6638109Z running benchmark:  10% 3/30 [00:00<00:08,  3.21it/s]
2023-08-17T12:32:25.9288188Z running benchmark:  13% 4/30 [00:01<00:07,  3.34it/s]
2023-08-17T12:32:26.1938217Z running benchmark:  17% 5/30 [00:01<00:07,  3.49it/s]
2023-08-17T12:32:26.4589947Z running benchmark:  20% 6/30 [00:01<00:06,  3.58it/s]
2023-08-17T12:32:26.7290773Z running benchmark:  23% 7/30 [00:02<00:06,  3.64it/s]
2023-08-17T12:32:26.9956226Z running benchmark:  27% 8/30 [00:02<00:06,  3.66it/s]
2023-08-17T12:32:27.2640944Z running benchmark:  30% 9/30 [00:02<00:05,  3.69it/s]
2023-08-17T12:32:27.5401476Z running benchmark:  33% 10/30 [00:02<00:05,  3.70it/s]
2023-08-17T12:32:27.8065864Z running benchmark:  37% 11/30 [00:03<00:05,  3.68it/s]
2023-08-17T12:32:28.0758924Z running benchmark:  40% 12/30 [00:03<00:04,  3.70it/s]
2023-08-17T12:32:28.3414062Z running benchmark:  43% 13/30 [00:03<00:04,  3.70it/s]
2023-08-17T12:32:28.6069253Z running benchmark:  47% 14/30 [00:03<00:04,  3.72it/s]
2023-08-17T12:32:28.8721928Z running benchmark:  50% 15/30 [00:04<00:04,  3.74it/s]
2023-08-17T12:32:29.1415879Z running benchmark:  53% 16/30 [00:04<00:03,  3.75it/s]
2023-08-17T12:32:29.4068424Z running benchmark:  57% 17/30 [00:04<00:03,  3.74it/s]
2023-08-17T12:32:29.6717531Z running benchmark:  60% 18/30 [00:05<00:03,  3.75it/s]
2023-08-17T12:32:29.9359625Z running benchmark:  63% 19/30 [00:05<00:02,  3.75it/s]
2023-08-17T12:32:30.2030137Z running benchmark:  67% 20/30 [00:05<00:02,  3.76it/s]
2023-08-17T12:32:30.4699424Z running benchmark:  70% 21/30 [00:05<00:02,  3.76it/s]
2023-08-17T12:32:30.7432783Z running benchmark:  73% 22/30 [00:06<00:02,  3.75it/s]
2023-08-17T12:32:31.0075199Z running benchmark:  77% 23/30 [00:06<00:01,  3.72it/s]
2023-08-17T12:32:31.2733243Z running benchmark:  80% 24/30 [00:06<00:01,  3.74it/s]
2023-08-17T12:32:31.5486996Z running benchmark:  83% 25/30 [00:06<00:01,  3.75it/s]
2023-08-17T12:32:31.8158412Z running benchmark:  87% 26/30 [00:07<00:01,  3.71it/s]
2023-08-17T12:32:32.0842366Z running benchmark:  90% 27/30 [00:07<00:00,  3.72it/s]
2023-08-17T12:32:32.3507581Z running benchmark:  93% 28/30 [00:07<00:00,  3.72it/s]
2023-08-17T12:32:32.6190390Z running benchmark:  97% 29/30 [00:07<00:00,  3.73it/s]
2023-08-17T12:32:32.6190775Z running benchmark: 100% 30/30 [00:08<00:00,  3.73it/s]
2023-08-17T12:32:32.6191088Z running benchmark: 100% 30/30 [00:08<00:00,  3.64it/s]
2023-08-17T12:32:32.6229245Z 2.000x
2023-08-17T12:32:39.1502513Z 
2023-08-17T12:32:40.9590272Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:32:40.9591285Z loading model: 0it [00:01, ?it/s]
2023-08-17T12:32:40.9595583Z cuda train MobileBertForQuestionAnswering     
2023-08-17T12:35:40.9276726Z 
2023-08-17T12:35:41.3105908Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:35:41.5353895Z running benchmark:   3% 1/30 [00:00<00:11,  2.61it/s]
2023-08-17T12:35:41.7662072Z running benchmark:   7% 2/30 [00:00<00:08,  3.45it/s]
2023-08-17T12:35:41.9996619Z running benchmark:  10% 3/30 [00:00<00:07,  3.80it/s]
2023-08-17T12:35:42.2241603Z running benchmark:  13% 4/30 [00:01<00:06,  3.98it/s]
2023-08-17T12:35:42.4492448Z running benchmark:  17% 5/30 [00:01<00:06,  4.14it/s]
2023-08-17T12:35:42.6728385Z running benchmark:  20% 6/30 [00:01<00:05,  4.23it/s]
2023-08-17T12:35:42.9008963Z running benchmark:  23% 7/30 [00:01<00:05,  4.31it/s]
2023-08-17T12:35:43.1476495Z running benchmark:  27% 8/30 [00:01<00:05,  4.33it/s]
2023-08-17T12:35:43.3749280Z running benchmark:  30% 9/30 [00:02<00:04,  4.24it/s]
2023-08-17T12:35:43.5996283Z running benchmark:  33% 10/30 [00:02<00:04,  4.29it/s]
2023-08-17T12:35:43.8240916Z running benchmark:  37% 11/30 [00:02<00:04,  4.34it/s]
2023-08-17T12:35:44.0581956Z running benchmark:  40% 12/30 [00:02<00:04,  4.37it/s]
2023-08-17T12:35:44.2921759Z running benchmark:  43% 13/30 [00:03<00:03,  4.34it/s]
2023-08-17T12:35:44.5196917Z running benchmark:  47% 14/30 [00:03<00:03,  4.32it/s]
2023-08-17T12:35:44.7449363Z running benchmark:  50% 15/30 [00:03<00:03,  4.34it/s]
2023-08-17T12:35:44.9686696Z running benchmark:  53% 16/30 [00:03<00:03,  4.37it/s]
2023-08-17T12:35:45.2076420Z running benchmark:  57% 17/30 [00:04<00:02,  4.40it/s]
2023-08-17T12:35:45.4410937Z running benchmark:  60% 18/30 [00:04<00:02,  4.33it/s]
2023-08-17T12:35:45.6669675Z running benchmark:  63% 19/30 [00:04<00:02,  4.32it/s]
2023-08-17T12:35:45.8900550Z running benchmark:  67% 20/30 [00:04<00:02,  4.35it/s]
2023-08-17T12:35:46.1136381Z running benchmark:  70% 21/30 [00:04<00:02,  4.39it/s]
2023-08-17T12:35:46.3402167Z running benchmark:  73% 22/30 [00:05<00:01,  4.41it/s]
2023-08-17T12:35:46.5706392Z running benchmark:  77% 23/30 [00:05<00:01,  4.41it/s]
2023-08-17T12:35:46.8027335Z running benchmark:  80% 24/30 [00:05<00:01,  4.39it/s]
2023-08-17T12:35:47.0324020Z running benchmark:  83% 25/30 [00:05<00:01,  4.37it/s]
2023-08-17T12:35:47.2647154Z running benchmark:  87% 26/30 [00:06<00:00,  4.36it/s]
2023-08-17T12:35:47.4880388Z running benchmark:  90% 27/30 [00:06<00:00,  4.34it/s]
2023-08-17T12:35:47.7166464Z running benchmark:  93% 28/30 [00:06<00:00,  4.38it/s]
2023-08-17T12:35:47.9489052Z running benchmark:  97% 29/30 [00:06<00:00,  4.38it/s]
2023-08-17T12:35:47.9489431Z running benchmark: 100% 30/30 [00:07<00:00,  4.36it/s]
2023-08-17T12:35:47.9492096Z running benchmark: 100% 30/30 [00:07<00:00,  4.27it/s]
2023-08-17T12:35:47.9525365Z 2.472x
2023-08-17T12:35:54.5641348Z 
2023-08-17T12:35:58.4784912Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:35:58.4785579Z loading model: 0it [00:03, ?it/s]
2023-08-17T12:35:58.4786061Z cuda train OPTForCausalLM                     
2023-08-17T12:36:49.2127931Z 
2023-08-17T12:36:49.6178200Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:36:49.8549900Z running benchmark:   3% 1/30 [00:00<00:11,  2.46it/s]
2023-08-17T12:36:50.0922778Z running benchmark:   7% 2/30 [00:00<00:08,  3.26it/s]
2023-08-17T12:36:50.3302818Z running benchmark:  10% 3/30 [00:00<00:07,  3.64it/s]
2023-08-17T12:36:50.5680617Z running benchmark:  13% 4/30 [00:01<00:06,  3.84it/s]
2023-08-17T12:36:50.8052424Z running benchmark:  17% 5/30 [00:01<00:06,  3.96it/s]
2023-08-17T12:36:51.0423868Z running benchmark:  20% 6/30 [00:01<00:05,  4.05it/s]
2023-08-17T12:36:51.2796255Z running benchmark:  23% 7/30 [00:01<00:05,  4.10it/s]
2023-08-17T12:36:51.5171276Z running benchmark:  27% 8/30 [00:02<00:05,  4.14it/s]
2023-08-17T12:36:51.7546780Z running benchmark:  30% 9/30 [00:02<00:05,  4.16it/s]
2023-08-17T12:36:51.9922762Z running benchmark:  33% 10/30 [00:02<00:04,  4.17it/s]
2023-08-17T12:36:52.2294987Z running benchmark:  37% 11/30 [00:02<00:04,  4.18it/s]
2023-08-17T12:36:52.4664663Z running benchmark:  40% 12/30 [00:03<00:04,  4.19it/s]
2023-08-17T12:36:52.7041295Z running benchmark:  43% 13/30 [00:03<00:04,  4.20it/s]
2023-08-17T12:36:52.9420118Z running benchmark:  47% 14/30 [00:03<00:03,  4.20it/s]
2023-08-17T12:36:53.1795703Z running benchmark:  50% 15/30 [00:03<00:03,  4.20it/s]
2023-08-17T12:36:53.4169441Z running benchmark:  53% 16/30 [00:03<00:03,  4.21it/s]
2023-08-17T12:36:53.6540943Z running benchmark:  57% 17/30 [00:04<00:03,  4.21it/s]
2023-08-17T12:36:53.9068447Z running benchmark:  60% 18/30 [00:04<00:02,  4.21it/s]
2023-08-17T12:36:54.1448687Z running benchmark:  63% 19/30 [00:04<00:02,  4.13it/s]
2023-08-17T12:36:54.3821880Z running benchmark:  67% 20/30 [00:04<00:02,  4.15it/s]
2023-08-17T12:36:54.6201550Z running benchmark:  70% 21/30 [00:05<00:02,  4.17it/s]
2023-08-17T12:36:54.8576187Z running benchmark:  73% 22/30 [00:05<00:01,  4.18it/s]
2023-08-17T12:36:55.0981178Z running benchmark:  77% 23/30 [00:05<00:01,  4.19it/s]
2023-08-17T12:36:55.3357919Z running benchmark:  80% 24/30 [00:05<00:01,  4.18it/s]
2023-08-17T12:36:55.5733281Z running benchmark:  83% 25/30 [00:06<00:01,  4.19it/s]
2023-08-17T12:36:55.8095030Z running benchmark:  87% 26/30 [00:06<00:00,  4.19it/s]
2023-08-17T12:36:56.0449185Z running benchmark:  90% 27/30 [00:06<00:00,  4.21it/s]
2023-08-17T12:36:56.2829353Z running benchmark:  93% 28/30 [00:06<00:00,  4.22it/s]
2023-08-17T12:36:56.5180999Z running benchmark:  97% 29/30 [00:07<00:00,  4.21it/s]
2023-08-17T12:36:56.5181608Z running benchmark: 100% 30/30 [00:07<00:00,  4.23it/s]
2023-08-17T12:36:56.5185077Z running benchmark: 100% 30/30 [00:07<00:00,  4.11it/s]
2023-08-17T12:36:56.5205342Z 2.697x
2023-08-17T12:37:01.1898187Z 
2023-08-17T12:37:04.4913424Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:37:04.4913940Z loading model: 0it [00:03, ?it/s]
2023-08-17T12:37:04.4914257Z cuda train PLBartForCausalLM                  
2023-08-17T12:37:43.4355919Z 
2023-08-17T12:37:43.7777357Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:37:43.9560838Z running benchmark:   3% 1/30 [00:00<00:09,  2.92it/s]
2023-08-17T12:37:44.1373405Z running benchmark:   7% 2/30 [00:00<00:06,  4.06it/s]
2023-08-17T12:37:44.3171856Z running benchmark:  10% 3/30 [00:00<00:05,  4.62it/s]
2023-08-17T12:37:44.4971142Z running benchmark:  13% 4/30 [00:00<00:05,  4.95it/s]
2023-08-17T12:37:44.6766173Z running benchmark:  17% 5/30 [00:01<00:04,  5.15it/s]
2023-08-17T12:37:44.8555461Z running benchmark:  20% 6/30 [00:01<00:04,  5.29it/s]
2023-08-17T12:37:45.0349370Z running benchmark:  23% 7/30 [00:01<00:04,  5.38it/s]
2023-08-17T12:37:45.2154269Z running benchmark:  27% 8/30 [00:01<00:04,  5.44it/s]
2023-08-17T12:37:45.3954932Z running benchmark:  30% 9/30 [00:01<00:03,  5.47it/s]
2023-08-17T12:37:45.5753586Z running benchmark:  33% 10/30 [00:01<00:03,  5.50it/s]
2023-08-17T12:37:45.7546830Z running benchmark:  37% 11/30 [00:02<00:03,  5.52it/s]
2023-08-17T12:37:45.9345406Z running benchmark:  40% 12/30 [00:02<00:03,  5.53it/s]
2023-08-17T12:37:46.1140190Z running benchmark:  43% 13/30 [00:02<00:03,  5.54it/s]
2023-08-17T12:37:46.2947301Z running benchmark:  47% 14/30 [00:02<00:02,  5.55it/s]
2023-08-17T12:37:46.4751039Z running benchmark:  50% 15/30 [00:02<00:02,  5.55it/s]
2023-08-17T12:37:46.6550159Z running benchmark:  53% 16/30 [00:03<00:02,  5.55it/s]
2023-08-17T12:37:46.8339270Z running benchmark:  57% 17/30 [00:03<00:02,  5.55it/s]
2023-08-17T12:37:47.0138668Z running benchmark:  60% 18/30 [00:03<00:02,  5.56it/s]
2023-08-17T12:37:47.1932701Z running benchmark:  63% 19/30 [00:03<00:01,  5.56it/s]
2023-08-17T12:37:47.3728053Z running benchmark:  67% 20/30 [00:03<00:01,  5.56it/s]
2023-08-17T12:37:47.5526623Z running benchmark:  70% 21/30 [00:03<00:01,  5.57it/s]
2023-08-17T12:37:47.7325352Z running benchmark:  73% 22/30 [00:04<00:01,  5.56it/s]
2023-08-17T12:37:47.9118875Z running benchmark:  77% 23/30 [00:04<00:01,  5.56it/s]
2023-08-17T12:37:48.0916303Z running benchmark:  80% 24/30 [00:04<00:01,  5.57it/s]
2023-08-17T12:37:48.2708714Z running benchmark:  83% 25/30 [00:04<00:00,  5.57it/s]
2023-08-17T12:37:48.4501065Z running benchmark:  87% 26/30 [00:04<00:00,  5.57it/s]
2023-08-17T12:37:48.6297402Z running benchmark:  90% 27/30 [00:05<00:00,  5.57it/s]
2023-08-17T12:37:48.8100484Z running benchmark:  93% 28/30 [00:05<00:00,  5.57it/s]
2023-08-17T12:37:48.9899024Z running benchmark:  97% 29/30 [00:05<00:00,  5.56it/s]
2023-08-17T12:37:48.9899419Z running benchmark: 100% 30/30 [00:05<00:00,  5.56it/s]
2023-08-17T12:37:48.9899713Z running benchmark: 100% 30/30 [00:05<00:00,  5.40it/s]
2023-08-17T12:37:48.9922133Z 1.945x
2023-08-17T12:37:53.4086199Z 
2023-08-17T12:37:58.7453342Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:37:58.7453659Z loading model: 0it [00:05, ?it/s]
2023-08-17T12:37:58.7454002Z cuda train PLBartForConditionalGeneration     
2023-08-17T12:39:05.5946720Z 
2023-08-17T12:39:05.9585766Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:39:06.1415563Z running benchmark:   3% 1/30 [00:00<00:10,  2.74it/s]
2023-08-17T12:39:06.3248817Z running benchmark:   7% 2/30 [00:00<00:07,  3.88it/s]
2023-08-17T12:39:06.5089786Z running benchmark:  10% 3/30 [00:00<00:06,  4.47it/s]
2023-08-17T12:39:06.6925515Z running benchmark:  13% 4/30 [00:00<00:05,  4.80it/s]
2023-08-17T12:39:06.8764842Z running benchmark:  17% 5/30 [00:01<00:04,  5.02it/s]
2023-08-17T12:39:07.0598674Z running benchmark:  20% 6/30 [00:01<00:04,  5.15it/s]
2023-08-17T12:39:07.2431570Z running benchmark:  23% 7/30 [00:01<00:04,  5.25it/s]
2023-08-17T12:39:07.4263921Z running benchmark:  27% 8/30 [00:01<00:04,  5.31it/s]
2023-08-17T12:39:07.6102125Z running benchmark:  30% 9/30 [00:01<00:03,  5.36it/s]
2023-08-17T12:39:07.7939675Z running benchmark:  33% 10/30 [00:02<00:03,  5.38it/s]
2023-08-17T12:39:07.9771196Z running benchmark:  37% 11/30 [00:02<00:03,  5.40it/s]
2023-08-17T12:39:08.1601617Z running benchmark:  40% 12/30 [00:02<00:03,  5.42it/s]
2023-08-17T12:39:08.3432580Z running benchmark:  43% 13/30 [00:02<00:03,  5.43it/s]
2023-08-17T12:39:08.5262922Z running benchmark:  47% 14/30 [00:02<00:02,  5.44it/s]
2023-08-17T12:39:08.7099162Z running benchmark:  50% 15/30 [00:02<00:02,  5.45it/s]
2023-08-17T12:39:08.8940529Z running benchmark:  53% 16/30 [00:03<00:02,  5.45it/s]
2023-08-17T12:39:09.0777723Z running benchmark:  57% 17/30 [00:03<00:02,  5.44it/s]
2023-08-17T12:39:09.2608289Z running benchmark:  60% 18/30 [00:03<00:02,  5.44it/s]
2023-08-17T12:39:09.4448319Z running benchmark:  63% 19/30 [00:03<00:02,  5.45it/s]
2023-08-17T12:39:09.6286935Z running benchmark:  67% 20/30 [00:03<00:01,  5.45it/s]
2023-08-17T12:39:09.8118761Z running benchmark:  70% 21/30 [00:04<00:01,  5.44it/s]
2023-08-17T12:39:09.9961218Z running benchmark:  73% 22/30 [00:04<00:01,  5.45it/s]
2023-08-17T12:39:10.1797870Z running benchmark:  77% 23/30 [00:04<00:01,  5.44it/s]
2023-08-17T12:39:10.3628024Z running benchmark:  80% 24/30 [00:04<00:01,  5.44it/s]
2023-08-17T12:39:10.5459053Z running benchmark:  83% 25/30 [00:04<00:00,  5.45it/s]
2023-08-17T12:39:10.7288682Z running benchmark:  87% 26/30 [00:04<00:00,  5.45it/s]
2023-08-17T12:39:10.9124332Z running benchmark:  90% 27/30 [00:05<00:00,  5.46it/s]
2023-08-17T12:39:11.0969121Z running benchmark:  93% 28/30 [00:05<00:00,  5.45it/s]
2023-08-17T12:39:11.2813548Z running benchmark:  97% 29/30 [00:05<00:00,  5.44it/s]
2023-08-17T12:39:11.2813886Z running benchmark: 100% 30/30 [00:05<00:00,  5.44it/s]
2023-08-17T12:39:11.2814187Z running benchmark: 100% 30/30 [00:05<00:00,  5.27it/s]
2023-08-17T12:39:11.2837289Z 1.766x
2023-08-17T12:39:16.2055766Z 
2023-08-17T12:39:24.7241949Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:39:24.7243311Z loading model: 0it [00:08, ?it/s]
2023-08-17T12:39:24.7243609Z cuda train PegasusForCausalLM                 
2023-08-17T12:40:16.1161726Z 
2023-08-17T12:40:16.3629183Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:40:16.4886590Z running benchmark:   3% 1/30 [00:00<00:07,  4.04it/s]
2023-08-17T12:40:16.6098257Z running benchmark:   7% 2/30 [00:00<00:04,  5.69it/s]
2023-08-17T12:40:16.7310312Z running benchmark:  10% 3/30 [00:00<00:04,  6.63it/s]
2023-08-17T12:40:16.8510562Z running benchmark:  13% 4/30 [00:00<00:03,  7.18it/s]
2023-08-17T12:40:16.9710678Z running benchmark:  17% 5/30 [00:00<00:03,  7.56it/s]
2023-08-17T12:40:17.0909637Z running benchmark:  20% 6/30 [00:00<00:03,  7.81it/s]
2023-08-17T12:40:17.2106220Z running benchmark:  23% 7/30 [00:00<00:02,  7.97it/s]
2023-08-17T12:40:17.3303397Z running benchmark:  27% 8/30 [00:01<00:02,  8.09it/s]
2023-08-17T12:40:17.4502065Z running benchmark:  30% 9/30 [00:01<00:02,  8.17it/s]
2023-08-17T12:40:17.5784677Z running benchmark:  33% 10/30 [00:01<00:02,  8.22it/s]
2023-08-17T12:40:17.6980301Z running benchmark:  37% 11/30 [00:01<00:02,  8.09it/s]
2023-08-17T12:40:17.8183056Z running benchmark:  40% 12/30 [00:01<00:02,  8.17it/s]
2023-08-17T12:40:17.9388377Z running benchmark:  43% 13/30 [00:01<00:02,  8.21it/s]
2023-08-17T12:40:18.0587394Z running benchmark:  47% 14/30 [00:01<00:01,  8.24it/s]
2023-08-17T12:40:18.1786963Z running benchmark:  50% 15/30 [00:01<00:01,  8.27it/s]
2023-08-17T12:40:18.2983756Z running benchmark:  53% 16/30 [00:02<00:01,  8.29it/s]
2023-08-17T12:40:18.4183469Z running benchmark:  57% 17/30 [00:02<00:01,  8.31it/s]
2023-08-17T12:40:18.5439317Z running benchmark:  60% 18/30 [00:02<00:01,  8.32it/s]
2023-08-17T12:40:18.6637426Z running benchmark:  63% 19/30 [00:02<00:01,  8.21it/s]
2023-08-17T12:40:18.7833969Z running benchmark:  67% 20/30 [00:02<00:01,  8.25it/s]
2023-08-17T12:40:18.9032522Z running benchmark:  70% 21/30 [00:02<00:01,  8.28it/s]
2023-08-17T12:40:19.0238437Z running benchmark:  73% 22/30 [00:02<00:00,  8.30it/s]
2023-08-17T12:40:19.1444288Z running benchmark:  77% 23/30 [00:02<00:00,  8.30it/s]
2023-08-17T12:40:19.2643999Z running benchmark:  80% 24/30 [00:03<00:00,  8.30it/s]
2023-08-17T12:40:19.3842741Z running benchmark:  83% 25/30 [00:03<00:00,  8.31it/s]
2023-08-17T12:40:19.5088037Z running benchmark:  87% 26/30 [00:03<00:00,  8.32it/s]
2023-08-17T12:40:19.6287643Z running benchmark:  90% 27/30 [00:03<00:00,  8.33it/s]
2023-08-17T12:40:19.7486684Z running benchmark:  93% 28/30 [00:03<00:00,  8.23it/s]
2023-08-17T12:40:19.8683317Z running benchmark:  97% 29/30 [00:03<00:00,  8.26it/s]
2023-08-17T12:40:19.8683846Z running benchmark: 100% 30/30 [00:03<00:00,  8.29it/s]
2023-08-17T12:40:19.8684232Z running benchmark: 100% 30/30 [00:03<00:00,  7.99it/s]
2023-08-17T12:40:19.8708530Z 1.358x
2023-08-17T12:40:24.4538399Z 
2023-08-17T12:40:38.1002225Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:40:38.1003057Z loading model: 0it [00:13, ?it/s]
2023-08-17T12:40:38.1003403Z cuda train PegasusForConditionalGeneration    
2023-08-17T12:42:27.2463998Z 
2023-08-17T12:42:27.7091462Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:42:27.9442473Z running benchmark:   3% 1/30 [00:00<00:13,  2.16it/s]
2023-08-17T12:42:28.1793046Z running benchmark:   7% 2/30 [00:00<00:09,  3.04it/s]
2023-08-17T12:42:28.4145713Z running benchmark:  10% 3/30 [00:00<00:07,  3.49it/s]
2023-08-17T12:42:28.6517532Z running benchmark:  13% 4/30 [00:01<00:06,  3.76it/s]
2023-08-17T12:42:28.8872304Z running benchmark:  17% 5/30 [00:01<00:06,  3.91it/s]
2023-08-17T12:42:29.1224773Z running benchmark:  20% 6/30 [00:01<00:05,  4.02it/s]
2023-08-17T12:42:29.3575558Z running benchmark:  23% 7/30 [00:01<00:05,  4.09it/s]
2023-08-17T12:42:29.5928070Z running benchmark:  27% 8/30 [00:02<00:05,  4.14it/s]
2023-08-17T12:42:29.8298584Z running benchmark:  30% 9/30 [00:02<00:05,  4.18it/s]
2023-08-17T12:42:30.0652122Z running benchmark:  33% 10/30 [00:02<00:04,  4.19it/s]
2023-08-17T12:42:30.3005981Z running benchmark:  37% 11/30 [00:02<00:04,  4.21it/s]
2023-08-17T12:42:30.5351603Z running benchmark:  40% 12/30 [00:03<00:04,  4.22it/s]
2023-08-17T12:42:30.7742114Z running benchmark:  43% 13/30 [00:03<00:04,  4.23it/s]
2023-08-17T12:42:31.0103374Z running benchmark:  47% 14/30 [00:03<00:03,  4.22it/s]
2023-08-17T12:42:31.2464858Z running benchmark:  50% 15/30 [00:03<00:03,  4.22it/s]
2023-08-17T12:42:31.4815254Z running benchmark:  53% 16/30 [00:04<00:03,  4.23it/s]
2023-08-17T12:42:31.7163660Z running benchmark:  57% 17/30 [00:04<00:03,  4.23it/s]
2023-08-17T12:42:31.9517303Z running benchmark:  60% 18/30 [00:04<00:02,  4.24it/s]
2023-08-17T12:42:32.1883462Z running benchmark:  63% 19/30 [00:04<00:02,  4.24it/s]
2023-08-17T12:42:32.4247780Z running benchmark:  67% 20/30 [00:04<00:02,  4.24it/s]
2023-08-17T12:42:32.6598962Z running benchmark:  70% 21/30 [00:05<00:02,  4.24it/s]
2023-08-17T12:42:32.8949434Z running benchmark:  73% 22/30 [00:05<00:01,  4.24it/s]
2023-08-17T12:42:33.1297070Z running benchmark:  77% 23/30 [00:05<00:01,  4.24it/s]
2023-08-17T12:42:33.3653860Z running benchmark:  80% 24/30 [00:05<00:01,  4.25it/s]
2023-08-17T12:42:33.6017103Z running benchmark:  83% 25/30 [00:06<00:01,  4.25it/s]
2023-08-17T12:42:33.8365902Z running benchmark:  87% 26/30 [00:06<00:00,  4.24it/s]
2023-08-17T12:42:34.0713954Z running benchmark:  90% 27/30 [00:06<00:00,  4.25it/s]
2023-08-17T12:42:34.3072961Z running benchmark:  93% 28/30 [00:06<00:00,  4.25it/s]
2023-08-17T12:42:34.5432121Z running benchmark:  97% 29/30 [00:07<00:00,  4.25it/s]
2023-08-17T12:42:34.5432491Z running benchmark: 100% 30/30 [00:07<00:00,  4.24it/s]
2023-08-17T12:42:34.5432885Z running benchmark: 100% 30/30 [00:07<00:00,  4.11it/s]
2023-08-17T12:42:34.5460910Z 1.403x
2023-08-17T12:42:40.0663323Z 
2023-08-17T12:42:40.0696211Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`
2023-08-17T12:42:43.3037299Z 
2023-08-17T12:42:43.3038276Z loading model: 0it [00:03, ?it/s]
2023-08-17T12:42:43.3038805Z cuda train RobertaForCausalLM                 
2023-08-17T12:43:35.8372796Z 
2023-08-17T12:43:36.1857693Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:43:36.3625865Z running benchmark:   3% 1/30 [00:00<00:10,  2.86it/s]
2023-08-17T12:43:36.5392792Z running benchmark:   7% 2/30 [00:00<00:06,  4.03it/s]
2023-08-17T12:43:36.7156767Z running benchmark:  10% 3/30 [00:00<00:05,  4.64it/s]
2023-08-17T12:43:36.8917146Z running benchmark:  13% 4/30 [00:00<00:05,  5.00it/s]
2023-08-17T12:43:37.0680826Z running benchmark:  17% 5/30 [00:01<00:04,  5.23it/s]
2023-08-17T12:43:37.2454618Z running benchmark:  20% 6/30 [00:01<00:04,  5.37it/s]
2023-08-17T12:43:37.4220478Z running benchmark:  23% 7/30 [00:01<00:04,  5.46it/s]
2023-08-17T12:43:37.5981035Z running benchmark:  27% 8/30 [00:01<00:03,  5.52it/s]
2023-08-17T12:43:37.7740812Z running benchmark:  30% 9/30 [00:01<00:03,  5.57it/s]
2023-08-17T12:43:37.9501987Z running benchmark:  33% 10/30 [00:01<00:03,  5.60it/s]
2023-08-17T12:43:38.1262943Z running benchmark:  37% 11/30 [00:02<00:03,  5.63it/s]
2023-08-17T12:43:38.3034592Z running benchmark:  40% 12/30 [00:02<00:03,  5.64it/s]
2023-08-17T12:43:38.4800904Z running benchmark:  43% 13/30 [00:02<00:03,  5.65it/s]
2023-08-17T12:43:38.6563260Z running benchmark:  47% 14/30 [00:02<00:02,  5.65it/s]
2023-08-17T12:43:38.8323945Z running benchmark:  50% 15/30 [00:02<00:02,  5.66it/s]
2023-08-17T12:43:39.0084484Z running benchmark:  53% 16/30 [00:02<00:02,  5.66it/s]
2023-08-17T12:43:39.1856037Z running benchmark:  57% 17/30 [00:03<00:02,  5.67it/s]
2023-08-17T12:43:39.3619787Z running benchmark:  60% 18/30 [00:03<00:02,  5.66it/s]
2023-08-17T12:43:39.5389138Z running benchmark:  63% 19/30 [00:03<00:01,  5.66it/s]
2023-08-17T12:43:39.7151751Z running benchmark:  67% 20/30 [00:03<00:01,  5.66it/s]
2023-08-17T12:43:39.8911008Z running benchmark:  70% 21/30 [00:03<00:01,  5.66it/s]
2023-08-17T12:43:40.0670873Z running benchmark:  73% 22/30 [00:04<00:01,  5.67it/s]
2023-08-17T12:43:40.2431904Z running benchmark:  77% 23/30 [00:04<00:01,  5.67it/s]
2023-08-17T12:43:40.4193602Z running benchmark:  80% 24/30 [00:04<00:01,  5.68it/s]
2023-08-17T12:43:40.5966251Z running benchmark:  83% 25/30 [00:04<00:00,  5.67it/s]
2023-08-17T12:43:40.7734476Z running benchmark:  87% 26/30 [00:04<00:00,  5.67it/s]
2023-08-17T12:43:40.9496176Z running benchmark:  90% 27/30 [00:04<00:00,  5.66it/s]
2023-08-17T12:43:41.1256592Z running benchmark:  93% 28/30 [00:05<00:00,  5.67it/s]
2023-08-17T12:43:41.3048417Z running benchmark:  97% 29/30 [00:05<00:00,  5.67it/s]
2023-08-17T12:43:41.3049478Z running benchmark: 100% 30/30 [00:05<00:00,  5.65it/s]
2023-08-17T12:43:41.3049789Z running benchmark: 100% 30/30 [00:05<00:00,  5.49it/s]
2023-08-17T12:43:41.3071735Z 1.858x
2023-08-17T12:43:46.0259926Z 
2023-08-17T12:43:48.9116031Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:43:48.9116617Z loading model: 0it [00:02, ?it/s]
2023-08-17T12:43:48.9117109Z cuda train RobertaForQuestionAnswering        
2023-08-17T12:44:36.1790460Z 
2023-08-17T12:44:36.4373886Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:44:36.5859405Z running benchmark:   3% 1/30 [00:00<00:07,  3.86it/s]
2023-08-17T12:44:36.7274532Z running benchmark:   7% 2/30 [00:00<00:05,  5.15it/s]
2023-08-17T12:44:36.8687389Z running benchmark:  10% 3/30 [00:00<00:04,  5.88it/s]
2023-08-17T12:44:37.0100811Z running benchmark:  13% 4/30 [00:00<00:04,  6.30it/s]
2023-08-17T12:44:37.1513166Z running benchmark:  17% 5/30 [00:00<00:03,  6.56it/s]
2023-08-17T12:44:37.2926785Z running benchmark:  20% 6/30 [00:00<00:03,  6.73it/s]
2023-08-17T12:44:37.4345861Z running benchmark:  23% 7/30 [00:01<00:03,  6.84it/s]
2023-08-17T12:44:37.5772033Z running benchmark:  27% 8/30 [00:01<00:03,  6.90it/s]
2023-08-17T12:44:37.7192769Z running benchmark:  30% 9/30 [00:01<00:03,  6.94it/s]
2023-08-17T12:44:37.8607345Z running benchmark:  33% 10/30 [00:01<00:02,  6.97it/s]
2023-08-17T12:44:38.0021733Z running benchmark:  37% 11/30 [00:01<00:02,  7.00it/s]
2023-08-17T12:44:38.1435239Z running benchmark:  40% 12/30 [00:01<00:02,  7.02it/s]
2023-08-17T12:44:38.2848196Z running benchmark:  43% 13/30 [00:01<00:02,  7.04it/s]
2023-08-17T12:44:38.4262037Z running benchmark:  47% 14/30 [00:02<00:02,  7.05it/s]
2023-08-17T12:44:38.5676939Z running benchmark:  50% 15/30 [00:02<00:02,  7.06it/s]
2023-08-17T12:44:38.7188642Z running benchmark:  53% 16/30 [00:02<00:01,  7.06it/s]
2023-08-17T12:44:38.8604366Z running benchmark:  57% 17/30 [00:02<00:01,  6.92it/s]
2023-08-17T12:44:39.0018376Z running benchmark:  60% 18/30 [00:02<00:01,  6.96it/s]
2023-08-17T12:44:39.1431209Z running benchmark:  63% 19/30 [00:02<00:01,  6.99it/s]
2023-08-17T12:44:39.2852862Z running benchmark:  67% 20/30 [00:02<00:01,  7.02it/s]
2023-08-17T12:44:39.4272835Z running benchmark:  70% 21/30 [00:03<00:01,  7.02it/s]
2023-08-17T12:44:39.5688395Z running benchmark:  73% 22/30 [00:03<00:01,  7.03it/s]
2023-08-17T12:44:39.7103944Z running benchmark:  77% 23/30 [00:03<00:00,  7.04it/s]
2023-08-17T12:44:39.8525620Z running benchmark:  80% 24/30 [00:03<00:00,  7.05it/s]
2023-08-17T12:44:39.9944813Z running benchmark:  83% 25/30 [00:03<00:00,  7.04it/s]
2023-08-17T12:44:40.1359405Z running benchmark:  87% 26/30 [00:03<00:00,  7.04it/s]
2023-08-17T12:44:40.2774138Z running benchmark:  90% 27/30 [00:03<00:00,  7.05it/s]
2023-08-17T12:44:40.4188390Z running benchmark:  93% 28/30 [00:04<00:00,  7.06it/s]
2023-08-17T12:44:40.5600885Z running benchmark:  97% 29/30 [00:04<00:00,  7.06it/s]
2023-08-17T12:44:40.5601220Z running benchmark: 100% 30/30 [00:04<00:00,  7.07it/s]
2023-08-17T12:44:40.5601531Z running benchmark: 100% 30/30 [00:04<00:00,  6.85it/s]
2023-08-17T12:44:40.5627457Z 2.042x
2023-08-17T12:44:45.2948329Z 
2023-08-17T12:44:46.4507342Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:44:46.4508048Z loading model: 0it [00:01, ?it/s]
2023-08-17T12:44:46.4508843Z cuda train Speech2Text2ForCausalLM            
2023-08-17T12:45:18.4162859Z 
2023-08-17T12:45:18.5994873Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:45:18.7737849Z running benchmark:   3% 1/30 [00:00<00:05,  5.45it/s]
2023-08-17T12:45:18.9491517Z running benchmark:  10% 3/30 [00:00<00:03,  8.91it/s]
2023-08-17T12:45:19.1235915Z running benchmark:  17% 5/30 [00:00<00:02, 10.04it/s]
2023-08-17T12:45:19.2980010Z running benchmark:  23% 7/30 [00:00<00:02, 10.60it/s]
2023-08-17T12:45:19.4723561Z running benchmark:  30% 9/30 [00:00<00:01, 10.91it/s]
2023-08-17T12:45:19.6483467Z running benchmark:  37% 11/30 [00:01<00:01, 11.10it/s]
2023-08-17T12:45:19.8228294Z running benchmark:  43% 13/30 [00:01<00:01, 11.19it/s]
2023-08-17T12:45:19.9972096Z running benchmark:  50% 15/30 [00:01<00:01, 11.27it/s]
2023-08-17T12:45:20.1715628Z running benchmark:  57% 17/30 [00:01<00:01, 11.33it/s]
2023-08-17T12:45:20.3455986Z running benchmark:  63% 19/30 [00:01<00:00, 11.38it/s]
2023-08-17T12:45:20.5194104Z running benchmark:  70% 21/30 [00:01<00:00, 11.41it/s]
2023-08-17T12:45:20.6968203Z running benchmark:  77% 23/30 [00:02<00:00, 11.44it/s]
2023-08-17T12:45:20.8737268Z running benchmark:  83% 25/30 [00:02<00:00, 11.39it/s]
2023-08-17T12:45:21.0498096Z running benchmark:  90% 27/30 [00:02<00:00, 11.36it/s]
2023-08-17T12:45:21.1374291Z running benchmark:  97% 29/30 [00:02<00:00, 11.36it/s]
2023-08-17T12:45:21.1374632Z running benchmark: 100% 30/30 [00:02<00:00, 11.02it/s]
2023-08-17T12:45:21.1396967Z 1.582x
2023-08-17T12:45:25.5688198Z 
2023-08-17T12:45:28.0407439Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:45:28.0408206Z loading model: 0it [00:02, ?it/s]
2023-08-17T12:45:28.0408774Z cuda train T5ForConditionalGeneration         
2023-08-17T12:46:32.3398977Z 
2023-08-17T12:46:32.6841168Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:46:32.8536908Z running benchmark:   3% 1/30 [00:00<00:10,  2.90it/s]
2023-08-17T12:46:33.0225690Z running benchmark:   7% 2/30 [00:00<00:06,  4.13it/s]
2023-08-17T12:46:33.1910985Z running benchmark:  10% 3/30 [00:00<00:05,  4.79it/s]
2023-08-17T12:46:33.3602431Z running benchmark:  13% 4/30 [00:00<00:05,  5.19it/s]
2023-08-17T12:46:33.5292383Z running benchmark:  17% 5/30 [00:01<00:04,  5.43it/s]
2023-08-17T12:46:33.6990999Z running benchmark:  20% 6/30 [00:01<00:04,  5.59it/s]
2023-08-17T12:46:33.8681549Z running benchmark:  23% 7/30 [00:01<00:04,  5.68it/s]
2023-08-17T12:46:34.0371864Z running benchmark:  27% 8/30 [00:01<00:03,  5.75it/s]
2023-08-17T12:46:34.2060234Z running benchmark:  30% 9/30 [00:01<00:03,  5.80it/s]
2023-08-17T12:46:34.3752229Z running benchmark:  33% 10/30 [00:01<00:03,  5.84it/s]
2023-08-17T12:46:34.5444234Z running benchmark:  37% 11/30 [00:02<00:03,  5.86it/s]
2023-08-17T12:46:34.7133227Z running benchmark:  40% 12/30 [00:02<00:03,  5.88it/s]
2023-08-17T12:46:34.8831844Z running benchmark:  43% 13/30 [00:02<00:02,  5.89it/s]
2023-08-17T12:46:35.0523773Z running benchmark:  47% 14/30 [00:02<00:02,  5.89it/s]
2023-08-17T12:46:35.2211692Z running benchmark:  50% 15/30 [00:02<00:02,  5.90it/s]
2023-08-17T12:46:35.3921076Z running benchmark:  53% 16/30 [00:02<00:02,  5.90it/s]
2023-08-17T12:46:35.5609824Z running benchmark:  57% 17/30 [00:03<00:02,  5.89it/s]
2023-08-17T12:46:35.7296746Z running benchmark:  60% 18/30 [00:03<00:02,  5.90it/s]
2023-08-17T12:46:35.8984854Z running benchmark:  63% 19/30 [00:03<00:01,  5.91it/s]
2023-08-17T12:46:36.0681987Z running benchmark:  67% 20/30 [00:03<00:01,  5.91it/s]
2023-08-17T12:46:36.2373014Z running benchmark:  70% 21/30 [00:03<00:01,  5.91it/s]
2023-08-17T12:46:36.4136927Z running benchmark:  73% 22/30 [00:03<00:01,  5.91it/s]
2023-08-17T12:46:36.5825621Z running benchmark:  77% 23/30 [00:04<00:01,  5.83it/s]
2023-08-17T12:46:36.7512852Z running benchmark:  80% 24/30 [00:04<00:01,  5.86it/s]
2023-08-17T12:46:36.9202132Z running benchmark:  83% 25/30 [00:04<00:00,  5.88it/s]
2023-08-17T12:46:37.0896906Z running benchmark:  87% 26/30 [00:04<00:00,  5.89it/s]
2023-08-17T12:46:37.2599638Z running benchmark:  90% 27/30 [00:04<00:00,  5.90it/s]
2023-08-17T12:46:37.4376737Z running benchmark:  93% 28/30 [00:04<00:00,  5.89it/s]
2023-08-17T12:46:37.6065652Z running benchmark:  97% 29/30 [00:05<00:00,  5.81it/s]
2023-08-17T12:46:37.6066012Z running benchmark: 100% 30/30 [00:05<00:00,  5.84it/s]
2023-08-17T12:46:37.6066759Z running benchmark: 100% 30/30 [00:05<00:00,  5.69it/s]
2023-08-17T12:46:37.6093900Z 1.614x
2023-08-17T12:46:42.2656092Z 
2023-08-17T12:46:44.7735143Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:46:44.7735489Z loading model: 0it [00:02, ?it/s]
2023-08-17T12:46:44.7735771Z cuda train T5Small                            
2023-08-17T12:47:49.4436761Z 
2023-08-17T12:47:49.7873047Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:47:49.9558458Z running benchmark:   3% 1/30 [00:00<00:09,  2.91it/s]
2023-08-17T12:47:50.1294621Z running benchmark:   7% 2/30 [00:00<00:06,  4.15it/s]
2023-08-17T12:47:50.2990211Z running benchmark:  10% 3/30 [00:00<00:05,  4.76it/s]
2023-08-17T12:47:50.4678319Z running benchmark:  13% 4/30 [00:00<00:05,  5.15it/s]
2023-08-17T12:47:50.6363493Z running benchmark:  17% 5/30 [00:01<00:04,  5.40it/s]
2023-08-17T12:47:50.8047770Z running benchmark:  20% 6/30 [00:01<00:04,  5.57it/s]
2023-08-17T12:47:50.9731315Z running benchmark:  23% 7/30 [00:01<00:04,  5.69it/s]
2023-08-17T12:47:51.1417026Z running benchmark:  27% 8/30 [00:01<00:03,  5.76it/s]
2023-08-17T12:47:51.3111046Z running benchmark:  30% 9/30 [00:01<00:03,  5.82it/s]
2023-08-17T12:47:51.4804779Z running benchmark:  33% 10/30 [00:01<00:03,  5.84it/s]
2023-08-17T12:47:51.6492516Z running benchmark:  37% 11/30 [00:02<00:03,  5.86it/s]
2023-08-17T12:47:51.8184227Z running benchmark:  40% 12/30 [00:02<00:03,  5.88it/s]
2023-08-17T12:47:51.9877341Z running benchmark:  43% 13/30 [00:02<00:02,  5.89it/s]
2023-08-17T12:47:52.1566893Z running benchmark:  47% 14/30 [00:02<00:02,  5.90it/s]
2023-08-17T12:47:52.3257434Z running benchmark:  50% 15/30 [00:02<00:02,  5.90it/s]
2023-08-17T12:47:52.4948641Z running benchmark:  53% 16/30 [00:02<00:02,  5.91it/s]
2023-08-17T12:47:52.6642732Z running benchmark:  57% 17/30 [00:03<00:02,  5.91it/s]
2023-08-17T12:47:52.8331706Z running benchmark:  60% 18/30 [00:03<00:02,  5.91it/s]
2023-08-17T12:47:53.0015608Z running benchmark:  63% 19/30 [00:03<00:01,  5.91it/s]
2023-08-17T12:47:53.1703102Z running benchmark:  67% 20/30 [00:03<00:01,  5.92it/s]
2023-08-17T12:47:53.3391465Z running benchmark:  70% 21/30 [00:03<00:01,  5.92it/s]
2023-08-17T12:47:53.5076678Z running benchmark:  73% 22/30 [00:03<00:01,  5.92it/s]
2023-08-17T12:47:53.6768163Z running benchmark:  77% 23/30 [00:04<00:01,  5.92it/s]
2023-08-17T12:47:53.8460547Z running benchmark:  80% 24/30 [00:04<00:01,  5.92it/s]
2023-08-17T12:47:54.0148491Z running benchmark:  83% 25/30 [00:04<00:00,  5.92it/s]
2023-08-17T12:47:54.1837594Z running benchmark:  87% 26/30 [00:04<00:00,  5.92it/s]
2023-08-17T12:47:54.3529410Z running benchmark:  90% 27/30 [00:04<00:00,  5.92it/s]
2023-08-17T12:47:54.5217635Z running benchmark:  93% 28/30 [00:04<00:00,  5.92it/s]
2023-08-17T12:47:54.6903602Z running benchmark:  97% 29/30 [00:05<00:00,  5.92it/s]
2023-08-17T12:47:54.6903959Z running benchmark: 100% 30/30 [00:05<00:00,  5.92it/s]
2023-08-17T12:47:54.6904253Z running benchmark: 100% 30/30 [00:05<00:00,  5.72it/s]
2023-08-17T12:47:54.6933373Z 1.609x
2023-08-17T12:47:59.3751966Z 
2023-08-17T12:48:05.5147444Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:48:05.5148188Z loading model: 0it [00:06, ?it/s]
2023-08-17T12:48:05.5148724Z cuda train TrOCRForCausalLM                   
2023-08-17T12:49:01.2796106Z 
2023-08-17T12:49:01.7321870Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:49:01.9689595Z running benchmark:   3% 1/30 [00:00<00:13,  2.21it/s]
2023-08-17T12:49:02.2005374Z running benchmark:   7% 2/30 [00:00<00:09,  3.07it/s]
2023-08-17T12:49:02.4320552Z running benchmark:  10% 3/30 [00:00<00:07,  3.54it/s]
2023-08-17T12:49:02.6627318Z running benchmark:  13% 4/30 [00:01<00:06,  3.81it/s]
2023-08-17T12:49:02.8936900Z running benchmark:  17% 5/30 [00:01<00:06,  3.98it/s]
2023-08-17T12:49:03.1244591Z running benchmark:  20% 6/30 [00:01<00:05,  4.09it/s]
2023-08-17T12:49:03.3555278Z running benchmark:  23% 7/30 [00:01<00:05,  4.17it/s]
2023-08-17T12:49:03.5859578Z running benchmark:  27% 8/30 [00:02<00:05,  4.22it/s]
2023-08-17T12:49:03.8165560Z running benchmark:  30% 9/30 [00:02<00:04,  4.26it/s]
2023-08-17T12:49:04.0468672Z running benchmark:  33% 10/30 [00:02<00:04,  4.28it/s]
2023-08-17T12:49:04.2773996Z running benchmark:  37% 11/30 [00:02<00:04,  4.30it/s]
2023-08-17T12:49:04.5075634Z running benchmark:  40% 12/30 [00:02<00:04,  4.31it/s]
2023-08-17T12:49:04.7385781Z running benchmark:  43% 13/30 [00:03<00:03,  4.32it/s]
2023-08-17T12:49:04.9691467Z running benchmark:  47% 14/30 [00:03<00:03,  4.32it/s]
2023-08-17T12:49:05.1988491Z running benchmark:  50% 15/30 [00:03<00:03,  4.33it/s]
2023-08-17T12:49:05.4289173Z running benchmark:  53% 16/30 [00:03<00:03,  4.34it/s]
2023-08-17T12:49:05.6582547Z running benchmark:  57% 17/30 [00:04<00:02,  4.34it/s]
2023-08-17T12:49:05.8894028Z running benchmark:  60% 18/30 [00:04<00:02,  4.34it/s]
2023-08-17T12:49:06.1202281Z running benchmark:  63% 19/30 [00:04<00:02,  4.34it/s]
2023-08-17T12:49:06.3513211Z running benchmark:  67% 20/30 [00:04<00:02,  4.34it/s]
2023-08-17T12:49:06.5816046Z running benchmark:  70% 21/30 [00:05<00:02,  4.33it/s]
2023-08-17T12:49:06.8116985Z running benchmark:  73% 22/30 [00:05<00:01,  4.34it/s]
2023-08-17T12:49:07.0427058Z running benchmark:  77% 23/30 [00:05<00:01,  4.34it/s]
2023-08-17T12:49:07.2731830Z running benchmark:  80% 24/30 [00:05<00:01,  4.34it/s]
2023-08-17T12:49:07.5036524Z running benchmark:  83% 25/30 [00:05<00:01,  4.34it/s]
2023-08-17T12:49:07.7334372Z running benchmark:  87% 26/30 [00:06<00:00,  4.34it/s]
2023-08-17T12:49:07.9639417Z running benchmark:  90% 27/30 [00:06<00:00,  4.34it/s]
2023-08-17T12:49:08.1953397Z running benchmark:  93% 28/30 [00:06<00:00,  4.34it/s]
2023-08-17T12:49:08.4261597Z running benchmark:  97% 29/30 [00:06<00:00,  4.34it/s]
2023-08-17T12:49:08.4262248Z running benchmark: 100% 30/30 [00:07<00:00,  4.33it/s]
2023-08-17T12:49:08.4262709Z running benchmark: 100% 30/30 [00:07<00:00,  4.20it/s]
2023-08-17T12:49:08.4288772Z 1.480x
2023-08-17T12:49:13.0743860Z 
2023-08-17T12:49:28.0701868Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:49:28.0702183Z loading model: 0it [00:14, ?it/s]
2023-08-17T12:49:28.0702488Z cuda train XGLMForCausalLM                    
2023-08-17T12:51:01.5119543Z 
2023-08-17T12:51:01.7754983Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:51:01.9111929Z running benchmark:   3% 1/30 [00:00<00:07,  3.78it/s]
2023-08-17T12:51:02.0463521Z running benchmark:   7% 2/30 [00:00<00:05,  5.29it/s]
2023-08-17T12:51:02.1797745Z running benchmark:  10% 3/30 [00:00<00:04,  6.08it/s]
2023-08-17T12:51:02.3149277Z running benchmark:  13% 4/30 [00:00<00:03,  6.57it/s]
2023-08-17T12:51:02.4515915Z running benchmark:  17% 5/30 [00:00<00:03,  6.85it/s]
2023-08-17T12:51:02.5887795Z running benchmark:  20% 6/30 [00:00<00:03,  7.00it/s]
2023-08-17T12:51:02.7268649Z running benchmark:  23% 7/30 [00:01<00:03,  7.10it/s]
2023-08-17T12:51:02.8642331Z running benchmark:  27% 8/30 [00:01<00:03,  7.14it/s]
2023-08-17T12:51:02.9990481Z running benchmark:  30% 9/30 [00:01<00:02,  7.18it/s]
2023-08-17T12:51:03.1346389Z running benchmark:  33% 10/30 [00:01<00:02,  7.25it/s]
2023-08-17T12:51:03.2688846Z running benchmark:  37% 11/30 [00:01<00:02,  7.29it/s]
2023-08-17T12:51:03.4033616Z running benchmark:  40% 12/30 [00:01<00:02,  7.34it/s]
2023-08-17T12:51:03.5371793Z running benchmark:  43% 13/30 [00:01<00:02,  7.37it/s]
2023-08-17T12:51:03.6888105Z running benchmark:  47% 14/30 [00:02<00:02,  7.40it/s]
2023-08-17T12:51:03.8250916Z running benchmark:  50% 15/30 [00:02<00:02,  7.14it/s]
2023-08-17T12:51:03.9585414Z running benchmark:  53% 16/30 [00:02<00:01,  7.19it/s]
2023-08-17T12:51:04.0910560Z running benchmark:  57% 17/30 [00:02<00:01,  7.28it/s]
2023-08-17T12:51:04.2226460Z running benchmark:  60% 18/30 [00:02<00:01,  7.36it/s]
2023-08-17T12:51:04.3548521Z running benchmark:  63% 19/30 [00:02<00:01,  7.43it/s]
2023-08-17T12:51:04.4875798Z running benchmark:  67% 20/30 [00:02<00:01,  7.47it/s]
2023-08-17T12:51:04.6193579Z running benchmark:  70% 21/30 [00:02<00:01,  7.49it/s]
2023-08-17T12:51:04.7651818Z running benchmark:  73% 22/30 [00:03<00:01,  7.52it/s]
2023-08-17T12:51:04.9005123Z running benchmark:  77% 23/30 [00:03<00:00,  7.31it/s]
2023-08-17T12:51:05.0347365Z running benchmark:  80% 24/30 [00:03<00:00,  7.33it/s]
2023-08-17T12:51:05.1670034Z running benchmark:  83% 25/30 [00:03<00:00,  7.37it/s]
2023-08-17T12:51:05.3010740Z running benchmark:  87% 26/30 [00:03<00:00,  7.42it/s]
2023-08-17T12:51:05.4323238Z running benchmark:  90% 27/30 [00:03<00:00,  7.43it/s]
2023-08-17T12:51:05.5659037Z running benchmark:  93% 28/30 [00:03<00:00,  7.49it/s]
2023-08-17T12:51:05.7054117Z running benchmark:  97% 29/30 [00:04<00:00,  7.49it/s]
2023-08-17T12:51:05.7054612Z running benchmark: 100% 30/30 [00:04<00:00,  7.39it/s]
2023-08-17T12:51:05.7055075Z running benchmark: 100% 30/30 [00:04<00:00,  7.15it/s]
2023-08-17T12:51:05.7081558Z 1.737x
2023-08-17T12:51:11.0451046Z 
2023-08-17T12:51:18.0175360Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:51:18.0175927Z loading model: 0it [00:06, ?it/s]
2023-08-17T12:51:18.0176420Z cuda train XLNetLMHeadModel                   
2023-08-17T12:52:21.2047146Z [2023-08-17 12:52:21,202] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-08-17T12:52:47.0715860Z skipping cudagraphs due to multiple devices
2023-08-17T12:53:21.3345353Z 
2023-08-17T12:53:21.8229228Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:53:22.2671064Z running benchmark:   3% 1/30 [00:00<00:14,  2.05it/s]
2023-08-17T12:53:22.6976508Z running benchmark:   7% 2/30 [00:00<00:12,  2.16it/s]
2023-08-17T12:53:23.1332011Z running benchmark:  10% 3/30 [00:01<00:12,  2.23it/s]
2023-08-17T12:53:23.5670776Z running benchmark:  13% 4/30 [00:01<00:11,  2.26it/s]
2023-08-17T12:53:24.0024847Z running benchmark:  17% 5/30 [00:02<00:10,  2.27it/s]
2023-08-17T12:53:24.4371391Z running benchmark:  20% 6/30 [00:02<00:10,  2.28it/s]
2023-08-17T12:53:24.8696410Z running benchmark:  23% 7/30 [00:03<00:10,  2.29it/s]
2023-08-17T12:53:25.2997582Z running benchmark:  27% 8/30 [00:03<00:09,  2.30it/s]
2023-08-17T12:53:25.7299678Z running benchmark:  30% 9/30 [00:03<00:09,  2.30it/s]
2023-08-17T12:53:26.1604011Z running benchmark:  33% 10/30 [00:04<00:08,  2.31it/s]
2023-08-17T12:53:26.5965152Z running benchmark:  37% 11/30 [00:04<00:08,  2.31it/s]
2023-08-17T12:53:27.0314554Z running benchmark:  40% 12/30 [00:05<00:07,  2.31it/s]
2023-08-17T12:53:27.4671095Z running benchmark:  43% 13/30 [00:05<00:07,  2.31it/s]
2023-08-17T12:53:27.9018639Z running benchmark:  47% 14/30 [00:06<00:06,  2.30it/s]
2023-08-17T12:53:28.3358940Z running benchmark:  50% 15/30 [00:06<00:06,  2.30it/s]
2023-08-17T12:53:28.7705909Z running benchmark:  53% 16/30 [00:07<00:06,  2.30it/s]
2023-08-17T12:53:29.2056742Z running benchmark:  57% 17/30 [00:07<00:05,  2.30it/s]
2023-08-17T12:53:29.6395852Z running benchmark:  60% 18/30 [00:07<00:05,  2.30it/s]
2023-08-17T12:53:30.0758917Z running benchmark:  63% 19/30 [00:08<00:04,  2.30it/s]
2023-08-17T12:53:30.5109462Z running benchmark:  67% 20/30 [00:08<00:04,  2.30it/s]
2023-08-17T12:53:30.9463506Z running benchmark:  70% 21/30 [00:09<00:03,  2.30it/s]
2023-08-17T12:53:31.3817671Z running benchmark:  73% 22/30 [00:09<00:03,  2.30it/s]
2023-08-17T12:53:31.8115640Z running benchmark:  77% 23/30 [00:10<00:03,  2.30it/s]
2023-08-17T12:53:32.2434421Z running benchmark:  80% 24/30 [00:10<00:02,  2.31it/s]
2023-08-17T12:53:32.6740440Z running benchmark:  83% 25/30 [00:10<00:02,  2.31it/s]
2023-08-17T12:53:33.1090955Z running benchmark:  87% 26/30 [00:11<00:01,  2.31it/s]
2023-08-17T12:53:33.5453671Z running benchmark:  90% 27/30 [00:11<00:01,  2.32it/s]
2023-08-17T12:53:33.9798261Z running benchmark:  93% 28/30 [00:12<00:00,  2.30it/s]
2023-08-17T12:53:34.4156763Z running benchmark:  97% 29/30 [00:12<00:00,  2.30it/s]
2023-08-17T12:53:34.4157120Z running benchmark: 100% 30/30 [00:13<00:00,  2.30it/s]
2023-08-17T12:53:34.4188832Z running benchmark: 100% 30/30 [00:13<00:00,  2.29it/s]
2023-08-17T12:53:34.4189110Z 1.838x
2023-08-17T12:53:40.3495729Z 
2023-08-17T12:53:43.7750713Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:53:43.7751043Z loading model: 0it [00:03, ?it/s]
2023-08-17T12:53:43.7751345Z cuda train YituTechConvBert                   
2023-08-17T12:54:59.3640856Z 
2023-08-17T12:54:59.7766535Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:54:59.9762821Z running benchmark:   3% 1/30 [00:00<00:11,  2.42it/s]
2023-08-17T12:55:00.1767714Z running benchmark:   7% 2/30 [00:00<00:08,  3.47it/s]
2023-08-17T12:55:00.3771286Z running benchmark:  10% 3/30 [00:00<00:06,  4.03it/s]
2023-08-17T12:55:00.5773989Z running benchmark:  13% 4/30 [00:01<00:05,  4.36it/s]
2023-08-17T12:55:00.7777143Z running benchmark:  17% 5/30 [00:01<00:05,  4.57it/s]
2023-08-17T12:55:00.9777573Z running benchmark:  20% 6/30 [00:01<00:05,  4.71it/s]
2023-08-17T12:55:01.1780818Z running benchmark:  23% 7/30 [00:01<00:04,  4.80it/s]
2023-08-17T12:55:01.3781700Z running benchmark:  27% 8/30 [00:01<00:04,  4.86it/s]
2023-08-17T12:55:01.5786018Z running benchmark:  30% 9/30 [00:02<00:04,  4.90it/s]
2023-08-17T12:55:01.7794279Z running benchmark:  33% 10/30 [00:02<00:04,  4.93it/s]
2023-08-17T12:55:01.9813686Z running benchmark:  37% 11/30 [00:02<00:03,  4.94it/s]
2023-08-17T12:55:02.1823324Z running benchmark:  40% 12/30 [00:02<00:03,  4.95it/s]
2023-08-17T12:55:02.3880840Z running benchmark:  43% 13/30 [00:02<00:03,  4.96it/s]
2023-08-17T12:55:02.5898762Z running benchmark:  47% 14/30 [00:03<00:03,  4.93it/s]
2023-08-17T12:55:02.7908476Z running benchmark:  50% 15/30 [00:03<00:03,  4.93it/s]
2023-08-17T12:55:02.9917657Z running benchmark:  53% 16/30 [00:03<00:02,  4.95it/s]
2023-08-17T12:55:03.1934038Z running benchmark:  57% 17/30 [00:03<00:02,  4.96it/s]
2023-08-17T12:55:03.4022970Z running benchmark:  60% 18/30 [00:03<00:02,  4.96it/s]
2023-08-17T12:55:03.6030609Z running benchmark:  63% 19/30 [00:04<00:02,  4.91it/s]
2023-08-17T12:55:03.8030887Z running benchmark:  67% 20/30 [00:04<00:02,  4.93it/s]
2023-08-17T12:55:04.0032461Z running benchmark:  70% 21/30 [00:04<00:01,  4.95it/s]
2023-08-17T12:55:04.2036588Z running benchmark:  73% 22/30 [00:04<00:01,  4.96it/s]
2023-08-17T12:55:04.4051738Z running benchmark:  77% 23/30 [00:04<00:01,  4.97it/s]
2023-08-17T12:55:04.6057396Z running benchmark:  80% 24/30 [00:05<00:01,  4.97it/s]
2023-08-17T12:55:04.8055626Z running benchmark:  83% 25/30 [00:05<00:01,  4.97it/s]
2023-08-17T12:55:05.0053614Z running benchmark:  87% 26/30 [00:05<00:00,  4.98it/s]
2023-08-17T12:55:05.2050819Z running benchmark:  90% 27/30 [00:05<00:00,  4.99it/s]
2023-08-17T12:55:05.4055887Z running benchmark:  93% 28/30 [00:05<00:00,  4.99it/s]
2023-08-17T12:55:05.6064520Z running benchmark:  97% 29/30 [00:06<00:00,  4.99it/s]
2023-08-17T12:55:05.6064921Z running benchmark: 100% 30/30 [00:06<00:00,  4.99it/s]
2023-08-17T12:55:05.6065220Z running benchmark: 100% 30/30 [00:06<00:00,  4.80it/s]
2023-08-17T12:55:05.6094011Z 1.646x
2023-08-17T12:55:07.3372754Z speedup             gmean=1.79x mean=1.816x
2023-08-17T12:55:07.3373916Z abs_latency         gmean=67.27x mean=71.203x
2023-08-17T12:55:07.3374493Z compilation_latency mean=79.637 seconds
2023-08-17T12:55:07.3375040Z compression_ratio   mean=1.085x
2023-08-17T12:55:07.3375486Z eager_peak_mem      gmean=11.07x mean=11.485x
2023-08-17T12:55:07.3380541Z dynamo_peak_mem     gmean=10.59x mean=11.372x
2023-08-17T12:55:07.3382892Z calls_captured      gmean=640.38x mean=727.235x
2023-08-17T12:55:07.3384525Z unique_graphs       gmean=1.00x mean=1.000x
2023-08-17T12:55:07.3387583Z graph_breaks        gmean=6.60x mean=6.647x
2023-08-17T12:55:07.3390172Z unique_graph_breaks gmean=5.59x mean=5.647x
2023-08-17T12:55:07.8295777Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *dynamic-true* ]]
2023-08-17T12:55:07.8299509Z + python benchmarks/dynamo/huggingface.py --performance --cold-start-latency --training --amp --backend inductor --dynamic-shapes --dynamic-batch-only --device cuda --total-partitions 3 --partition-id 2 --output /var/lib/jenkins/workspace/test/test-reports/inductor_dynamic_huggingface_amp_training_cuda_performance.csv
2023-08-17T12:55:14.3906097Z 
2023-08-17T12:55:21.1361835Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:55:21.1362321Z loading model: 0it [00:06, ?it/s]
2023-08-17T12:55:21.1362818Z cuda train MegatronBertForQuestionAnswering   
2023-08-17T12:57:46.2170736Z 
2023-08-17T12:57:46.6065514Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T12:57:46.8276344Z running benchmark:   3% 1/30 [00:00<00:11,  2.56it/s]
2023-08-17T12:57:47.0493359Z running benchmark:   7% 2/30 [00:00<00:08,  3.44it/s]
2023-08-17T12:57:47.2828074Z running benchmark:  10% 3/30 [00:00<00:06,  3.86it/s]
2023-08-17T12:57:47.5046367Z running benchmark:  13% 4/30 [00:01<00:06,  4.02it/s]
2023-08-17T12:57:47.7258921Z running benchmark:  17% 5/30 [00:01<00:05,  4.18it/s]
2023-08-17T12:57:47.9471977Z running benchmark:  20% 6/30 [00:01<00:05,  4.29it/s]
2023-08-17T12:57:48.1692257Z running benchmark:  23% 7/30 [00:01<00:05,  4.36it/s]
2023-08-17T12:57:48.3913159Z running benchmark:  27% 8/30 [00:01<00:04,  4.41it/s]
2023-08-17T12:57:48.6132120Z running benchmark:  30% 9/30 [00:02<00:04,  4.44it/s]
2023-08-17T12:57:48.8343596Z running benchmark:  33% 10/30 [00:02<00:04,  4.46it/s]
2023-08-17T12:57:49.0553716Z running benchmark:  37% 11/30 [00:02<00:04,  4.48it/s]
2023-08-17T12:57:49.2768097Z running benchmark:  40% 12/30 [00:02<00:04,  4.49it/s]
2023-08-17T12:57:49.4981216Z running benchmark:  43% 13/30 [00:03<00:03,  4.50it/s]
2023-08-17T12:57:49.7201920Z running benchmark:  47% 14/30 [00:03<00:03,  4.50it/s]
2023-08-17T12:57:49.9414147Z running benchmark:  50% 15/30 [00:03<00:03,  4.50it/s]
2023-08-17T12:57:50.1627365Z running benchmark:  53% 16/30 [00:03<00:03,  4.51it/s]
2023-08-17T12:57:50.3840719Z running benchmark:  57% 17/30 [00:03<00:02,  4.51it/s]
2023-08-17T12:57:50.6051942Z running benchmark:  60% 18/30 [00:04<00:02,  4.51it/s]
2023-08-17T12:57:50.8278148Z running benchmark:  63% 19/30 [00:04<00:02,  4.52it/s]
2023-08-17T12:57:51.0493102Z running benchmark:  67% 20/30 [00:04<00:02,  4.51it/s]
2023-08-17T12:57:51.2711193Z running benchmark:  70% 21/30 [00:04<00:01,  4.51it/s]
2023-08-17T12:57:51.4931299Z running benchmark:  73% 22/30 [00:05<00:01,  4.51it/s]
2023-08-17T12:57:51.7141102Z running benchmark:  77% 23/30 [00:05<00:01,  4.51it/s]
2023-08-17T12:57:51.9365902Z running benchmark:  80% 24/30 [00:05<00:01,  4.51it/s]
2023-08-17T12:57:52.1583999Z running benchmark:  83% 25/30 [00:05<00:01,  4.51it/s]
2023-08-17T12:57:52.3833499Z running benchmark:  87% 26/30 [00:05<00:00,  4.51it/s]
2023-08-17T12:57:52.6049273Z running benchmark:  90% 27/30 [00:06<00:00,  4.49it/s]
2023-08-17T12:57:52.8259426Z running benchmark:  93% 28/30 [00:06<00:00,  4.50it/s]
2023-08-17T12:57:53.0479317Z running benchmark:  97% 29/30 [00:06<00:00,  4.50it/s]
2023-08-17T12:57:53.0479681Z running benchmark: 100% 30/30 [00:06<00:00,  4.51it/s]
2023-08-17T12:57:53.0479983Z running benchmark: 100% 30/30 [00:06<00:00,  4.39it/s]
2023-08-17T12:57:53.0513016Z 1.743x
2023-08-17T12:57:58.5146139Z 
2023-08-17T12:58:00.5693297Z loading model: 0it [00:00, ?it/s]
2023-08-17T12:58:00.5694492Z loading model: 0it [00:02, ?it/s]
2023-08-17T12:58:00.5694967Z cuda train MobileBertForMaskedLM              
2023-08-17T13:02:23.5670086Z 
2023-08-17T13:02:24.0931203Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T13:02:24.3716358Z running benchmark:   3% 1/30 [00:00<00:15,  1.90it/s]
2023-08-17T13:02:24.6610899Z running benchmark:   7% 2/30 [00:00<00:10,  2.63it/s]
2023-08-17T13:02:24.9349323Z running benchmark:  10% 3/30 [00:01<00:09,  2.95it/s]
2023-08-17T13:02:25.2121236Z running benchmark:  13% 4/30 [00:01<00:08,  3.19it/s]
2023-08-17T13:02:25.4874700Z running benchmark:  17% 5/30 [00:01<00:07,  3.33it/s]
2023-08-17T13:02:25.7741262Z running benchmark:  20% 6/30 [00:01<00:07,  3.43it/s]
2023-08-17T13:02:26.0450288Z running benchmark:  23% 7/30 [00:02<00:06,  3.45it/s]
2023-08-17T13:02:26.3164547Z running benchmark:  27% 8/30 [00:02<00:06,  3.52it/s]
2023-08-17T13:02:26.5946191Z running benchmark:  30% 9/30 [00:02<00:05,  3.57it/s]
2023-08-17T13:02:26.8822551Z running benchmark:  33% 10/30 [00:03<00:05,  3.58it/s]
2023-08-17T13:02:27.1540617Z running benchmark:  37% 11/30 [00:03<00:05,  3.55it/s]
2023-08-17T13:02:27.4261421Z running benchmark:  40% 12/30 [00:03<00:05,  3.59it/s]
2023-08-17T13:02:27.7032821Z running benchmark:  43% 13/30 [00:03<00:04,  3.61it/s]
2023-08-17T13:02:27.9765045Z running benchmark:  47% 14/30 [00:04<00:04,  3.61it/s]
2023-08-17T13:02:28.2487846Z running benchmark:  50% 15/30 [00:04<00:04,  3.63it/s]
2023-08-17T13:02:28.5210558Z running benchmark:  53% 16/30 [00:04<00:03,  3.64it/s]
2023-08-17T13:02:28.8099503Z running benchmark:  57% 17/30 [00:04<00:03,  3.65it/s]
2023-08-17T13:02:29.0846490Z running benchmark:  60% 18/30 [00:05<00:03,  3.59it/s]
2023-08-17T13:02:29.3565450Z running benchmark:  63% 19/30 [00:05<00:03,  3.61it/s]
2023-08-17T13:02:29.6282389Z running benchmark:  67% 20/30 [00:05<00:02,  3.63it/s]
2023-08-17T13:02:29.9128206Z running benchmark:  70% 21/30 [00:06<00:02,  3.64it/s]
2023-08-17T13:02:30.1918249Z running benchmark:  73% 22/30 [00:06<00:02,  3.60it/s]
2023-08-17T13:02:30.4640169Z running benchmark:  77% 23/30 [00:06<00:01,  3.60it/s]
2023-08-17T13:02:30.7347068Z running benchmark:  80% 24/30 [00:06<00:01,  3.62it/s]
2023-08-17T13:02:31.0056655Z running benchmark:  83% 25/30 [00:07<00:01,  3.64it/s]
2023-08-17T13:02:31.2831310Z running benchmark:  87% 26/30 [00:07<00:01,  3.66it/s]
2023-08-17T13:02:31.5588052Z running benchmark:  90% 27/30 [00:07<00:00,  3.64it/s]
2023-08-17T13:02:31.8442059Z running benchmark:  93% 28/30 [00:07<00:00,  3.64it/s]
2023-08-17T13:02:32.1184943Z running benchmark:  97% 29/30 [00:08<00:00,  3.60it/s]
2023-08-17T13:02:32.1185298Z running benchmark: 100% 30/30 [00:08<00:00,  3.61it/s]
2023-08-17T13:02:32.1185600Z running benchmark: 100% 30/30 [00:08<00:00,  3.51it/s]
2023-08-17T13:02:32.1222668Z 1.880x
2023-08-17T13:02:39.1343662Z 
2023-08-17T13:02:40.9158444Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:02:40.9158945Z loading model: 0it [00:01, ?it/s]
2023-08-17T13:02:40.9159307Z cuda train MobileBertForQuestionAnswering     
2023-08-17T13:07:02.9748992Z 
2023-08-17T13:07:03.3703517Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T13:07:03.6135990Z running benchmark:   3% 1/30 [00:00<00:11,  2.53it/s]
2023-08-17T13:07:03.8576190Z running benchmark:   7% 2/30 [00:00<00:08,  3.27it/s]
2023-08-17T13:07:04.1068151Z running benchmark:  10% 3/30 [00:00<00:07,  3.60it/s]
2023-08-17T13:07:04.3585748Z running benchmark:  13% 4/30 [00:01<00:06,  3.75it/s]
2023-08-17T13:07:04.5984917Z running benchmark:  17% 5/30 [00:01<00:06,  3.83it/s]
2023-08-17T13:07:04.8365525Z running benchmark:  20% 6/30 [00:01<00:06,  3.94it/s]
2023-08-17T13:07:05.0755639Z running benchmark:  23% 7/30 [00:01<00:05,  4.02it/s]
2023-08-17T13:07:05.3305039Z running benchmark:  27% 8/30 [00:02<00:05,  4.07it/s]
2023-08-17T13:07:05.5670672Z running benchmark:  30% 9/30 [00:02<00:05,  4.02it/s]
2023-08-17T13:07:05.8012423Z running benchmark:  33% 10/30 [00:02<00:04,  4.08it/s]
2023-08-17T13:07:06.0357741Z running benchmark:  37% 11/30 [00:02<00:04,  4.14it/s]
2023-08-17T13:07:06.2851168Z running benchmark:  40% 12/30 [00:03<00:04,  4.18it/s]
2023-08-17T13:07:06.5319321Z running benchmark:  43% 13/30 [00:03<00:04,  4.13it/s]
2023-08-17T13:07:06.7715214Z running benchmark:  47% 14/30 [00:03<00:03,  4.10it/s]
2023-08-17T13:07:07.0091264Z running benchmark:  50% 15/30 [00:03<00:03,  4.12it/s]
2023-08-17T13:07:07.2601691Z running benchmark:  53% 16/30 [00:04<00:03,  4.15it/s]
2023-08-17T13:07:07.5003945Z running benchmark:  57% 17/30 [00:04<00:03,  4.10it/s]
2023-08-17T13:07:07.7463784Z running benchmark:  60% 18/30 [00:04<00:02,  4.12it/s]
2023-08-17T13:07:07.9843844Z running benchmark:  63% 19/30 [00:04<00:02,  4.10it/s]
2023-08-17T13:07:08.2211723Z running benchmark:  67% 20/30 [00:05<00:02,  4.13it/s]
2023-08-17T13:07:08.4610076Z running benchmark:  70% 21/30 [00:05<00:02,  4.16it/s]
2023-08-17T13:07:08.7027000Z running benchmark:  73% 22/30 [00:05<00:01,  4.16it/s]
2023-08-17T13:07:08.9462259Z running benchmark:  77% 23/30 [00:05<00:01,  4.15it/s]
2023-08-17T13:07:09.1836880Z running benchmark:  80% 24/30 [00:05<00:01,  4.14it/s]
2023-08-17T13:07:09.4371804Z running benchmark:  83% 25/30 [00:06<00:01,  4.16it/s]
2023-08-17T13:07:09.6739448Z running benchmark:  87% 26/30 [00:06<00:00,  4.09it/s]
2023-08-17T13:07:09.9198887Z running benchmark:  90% 27/30 [00:06<00:00,  4.13it/s]
2023-08-17T13:07:10.1624165Z running benchmark:  93% 28/30 [00:06<00:00,  4.11it/s]
2023-08-17T13:07:10.4052170Z running benchmark:  97% 29/30 [00:07<00:00,  4.11it/s]
2023-08-17T13:07:10.4052923Z running benchmark: 100% 30/30 [00:07<00:00,  4.12it/s]
2023-08-17T13:07:10.4053240Z running benchmark: 100% 30/30 [00:07<00:00,  4.04it/s]
2023-08-17T13:07:10.4089486Z 2.576x
2023-08-17T13:07:17.2727475Z 
2023-08-17T13:07:21.1628061Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:07:21.1628424Z loading model: 0it [00:03, ?it/s]
2023-08-17T13:08:38.0888351Z cuda train OPTForCausalLM                     
2023-08-17T13:08:38.0888960Z 
2023-08-17T13:08:38.5260281Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T13:08:38.7703675Z running benchmark:   3% 1/30 [00:00<00:12,  2.28it/s]
2023-08-17T13:08:39.0181806Z running benchmark:   7% 2/30 [00:00<00:09,  3.09it/s]
2023-08-17T13:08:39.2627684Z running benchmark:  10% 3/30 [00:00<00:07,  3.46it/s]
2023-08-17T13:08:39.5077582Z running benchmark:  13% 4/30 [00:01<00:07,  3.68it/s]
2023-08-17T13:08:39.7519321Z running benchmark:  17% 5/30 [00:01<00:06,  3.82it/s]
2023-08-17T13:08:39.9964752Z running benchmark:  20% 6/30 [00:01<00:06,  3.91it/s]
2023-08-17T13:08:40.2416047Z running benchmark:  23% 7/30 [00:01<00:05,  3.96it/s]
2023-08-17T13:08:40.4863798Z running benchmark:  27% 8/30 [00:02<00:05,  4.00it/s]
2023-08-17T13:08:40.7319532Z running benchmark:  30% 9/30 [00:02<00:05,  4.03it/s]
2023-08-17T13:08:40.9757190Z running benchmark:  33% 10/30 [00:02<00:04,  4.04it/s]
2023-08-17T13:08:41.2194601Z running benchmark:  37% 11/30 [00:02<00:04,  4.06it/s]
2023-08-17T13:08:41.4770648Z running benchmark:  40% 12/30 [00:03<00:04,  4.07it/s]
2023-08-17T13:08:41.7211166Z running benchmark:  43% 13/30 [00:03<00:04,  4.01it/s]
2023-08-17T13:08:41.9655222Z running benchmark:  47% 14/30 [00:03<00:03,  4.04it/s]
2023-08-17T13:08:42.2099548Z running benchmark:  50% 15/30 [00:03<00:03,  4.05it/s]
2023-08-17T13:08:42.4540687Z running benchmark:  53% 16/30 [00:04<00:03,  4.06it/s]
2023-08-17T13:08:42.6987367Z running benchmark:  57% 17/30 [00:04<00:03,  4.08it/s]
2023-08-17T13:08:42.9429310Z running benchmark:  60% 18/30 [00:04<00:02,  4.08it/s]
2023-08-17T13:08:43.1869575Z running benchmark:  63% 19/30 [00:04<00:02,  4.08it/s]
2023-08-17T13:08:43.4311671Z running benchmark:  67% 20/30 [00:05<00:02,  4.09it/s]
2023-08-17T13:08:43.6823463Z running benchmark:  70% 21/30 [00:05<00:02,  4.09it/s]
2023-08-17T13:08:43.9270348Z running benchmark:  73% 22/30 [00:05<00:01,  4.06it/s]
2023-08-17T13:08:44.1704313Z running benchmark:  77% 23/30 [00:05<00:01,  4.07it/s]
2023-08-17T13:08:44.4126053Z running benchmark:  80% 24/30 [00:06<00:01,  4.08it/s]
2023-08-17T13:08:44.6560835Z running benchmark:  83% 25/30 [00:06<00:01,  4.09it/s]
2023-08-17T13:08:44.8997849Z running benchmark:  87% 26/30 [00:06<00:00,  4.10it/s]
2023-08-17T13:08:45.1425454Z running benchmark:  90% 27/30 [00:06<00:00,  4.10it/s]
2023-08-17T13:08:45.3844079Z running benchmark:  93% 28/30 [00:07<00:00,  4.10it/s]
2023-08-17T13:08:45.6270972Z running benchmark:  97% 29/30 [00:07<00:00,  4.11it/s]
2023-08-17T13:08:45.6316510Z running benchmark: 100% 30/30 [00:07<00:00,  4.12it/s]
2023-08-17T13:08:45.6317034Z running benchmark: 100% 30/30 [00:07<00:00,  3.98it/s]
2023-08-17T13:08:45.6317382Z 2.422x
2023-08-17T13:08:50.4225678Z 
2023-08-17T13:08:53.7027022Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:08:53.7027675Z loading model: 0it [00:03, ?it/s]
2023-08-17T13:08:53.7028023Z cuda train PLBartForCausalLM                  
2023-08-17T13:09:43.7466444Z 
2023-08-17T13:09:44.1242335Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T13:09:44.3123251Z running benchmark:   3% 1/30 [00:00<00:10,  2.65it/s]
2023-08-17T13:09:44.5003159Z running benchmark:   7% 2/30 [00:00<00:07,  3.75it/s]
2023-08-17T13:09:44.6881424Z running benchmark:  10% 3/30 [00:00<00:06,  4.34it/s]
2023-08-17T13:09:44.8753455Z running benchmark:  13% 4/30 [00:00<00:05,  4.68it/s]
2023-08-17T13:09:45.0703925Z running benchmark:  17% 5/30 [00:01<00:05,  4.90it/s]
2023-08-17T13:09:45.2576910Z running benchmark:  20% 6/30 [00:01<00:04,  4.97it/s]
2023-08-17T13:09:45.4437785Z running benchmark:  23% 7/30 [00:01<00:04,  5.09it/s]
2023-08-17T13:09:45.6305526Z running benchmark:  27% 8/30 [00:01<00:04,  5.18it/s]
2023-08-17T13:09:45.8163085Z running benchmark:  30% 9/30 [00:01<00:04,  5.23it/s]
2023-08-17T13:09:46.0020914Z running benchmark:  33% 10/30 [00:02<00:03,  5.28it/s]
2023-08-17T13:09:46.1894745Z running benchmark:  37% 11/30 [00:02<00:03,  5.31it/s]
2023-08-17T13:09:46.3758025Z running benchmark:  40% 12/30 [00:02<00:03,  5.32it/s]
2023-08-17T13:09:46.5624172Z running benchmark:  43% 13/30 [00:02<00:03,  5.33it/s]
2023-08-17T13:09:46.7486325Z running benchmark:  47% 14/30 [00:02<00:02,  5.34it/s]
2023-08-17T13:09:46.9403312Z running benchmark:  50% 15/30 [00:03<00:02,  5.35it/s]
2023-08-17T13:09:47.1271631Z running benchmark:  53% 16/30 [00:03<00:02,  5.31it/s]
2023-08-17T13:09:47.3154837Z running benchmark:  57% 17/30 [00:03<00:02,  5.32it/s]
2023-08-17T13:09:47.5020526Z running benchmark:  60% 18/30 [00:03<00:02,  5.32it/s]
2023-08-17T13:09:47.6879700Z running benchmark:  63% 19/30 [00:03<00:02,  5.33it/s]
2023-08-17T13:09:47.8743141Z running benchmark:  67% 20/30 [00:03<00:01,  5.34it/s]
2023-08-17T13:09:48.0604282Z running benchmark:  70% 21/30 [00:04<00:01,  5.35it/s]
2023-08-17T13:09:48.2476715Z running benchmark:  73% 22/30 [00:04<00:01,  5.36it/s]
2023-08-17T13:09:48.4342409Z running benchmark:  77% 23/30 [00:04<00:01,  5.35it/s]
2023-08-17T13:09:48.6213596Z running benchmark:  80% 24/30 [00:04<00:01,  5.36it/s]
2023-08-17T13:09:48.8076041Z running benchmark:  83% 25/30 [00:04<00:00,  5.35it/s]
2023-08-17T13:09:48.9947125Z running benchmark:  87% 26/30 [00:05<00:00,  5.36it/s]
2023-08-17T13:09:49.1815171Z running benchmark:  90% 27/30 [00:05<00:00,  5.35it/s]
2023-08-17T13:09:49.3681568Z running benchmark:  93% 28/30 [00:05<00:00,  5.35it/s]
2023-08-17T13:09:49.5542302Z running benchmark:  97% 29/30 [00:05<00:00,  5.35it/s]
2023-08-17T13:09:49.5542774Z running benchmark: 100% 30/30 [00:05<00:00,  5.36it/s]
2023-08-17T13:09:49.5543249Z running benchmark: 100% 30/30 [00:05<00:00,  5.16it/s]
2023-08-17T13:09:49.5569173Z 1.690x
2023-08-17T13:09:54.0037563Z 
2023-08-17T13:09:59.3074251Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:09:59.3074917Z loading model: 0it [00:05, ?it/s]
2023-08-17T13:09:59.3075245Z cuda train PLBartForConditionalGeneration     
2023-08-17T13:11:40.1630976Z 
2023-08-17T13:11:40.5316763Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T13:11:40.7196883Z running benchmark:   3% 1/30 [00:00<00:10,  2.71it/s]
2023-08-17T13:11:40.9080317Z running benchmark:   7% 2/30 [00:00<00:07,  3.80it/s]
2023-08-17T13:11:41.0955546Z running benchmark:  10% 3/30 [00:00<00:06,  4.37it/s]
2023-08-17T13:11:41.2829749Z running benchmark:  13% 4/30 [00:00<00:05,  4.71it/s]
2023-08-17T13:11:41.4702432Z running benchmark:  17% 5/30 [00:01<00:05,  4.91it/s]
2023-08-17T13:11:41.6578504Z running benchmark:  20% 6/30 [00:01<00:04,  5.05it/s]
2023-08-17T13:11:41.8452875Z running benchmark:  23% 7/30 [00:01<00:04,  5.14it/s]
2023-08-17T13:11:42.0338758Z running benchmark:  27% 8/30 [00:01<00:04,  5.20it/s]
2023-08-17T13:11:42.2220116Z running benchmark:  30% 9/30 [00:01<00:04,  5.23it/s]
2023-08-17T13:11:42.4095501Z running benchmark:  33% 10/30 [00:02<00:03,  5.26it/s]
2023-08-17T13:11:42.5966634Z running benchmark:  37% 11/30 [00:02<00:03,  5.28it/s]
2023-08-17T13:11:42.7837451Z running benchmark:  40% 12/30 [00:02<00:03,  5.30it/s]
2023-08-17T13:11:42.9709077Z running benchmark:  43% 13/30 [00:02<00:03,  5.31it/s]
2023-08-17T13:11:43.1590503Z running benchmark:  47% 14/30 [00:02<00:03,  5.32it/s]
2023-08-17T13:11:43.3470156Z running benchmark:  50% 15/30 [00:02<00:02,  5.32it/s]
2023-08-17T13:11:43.5344293Z running benchmark:  53% 16/30 [00:03<00:02,  5.32it/s]
2023-08-17T13:11:43.7221101Z running benchmark:  57% 17/30 [00:03<00:02,  5.33it/s]
2023-08-17T13:11:43.9093769Z running benchmark:  60% 18/30 [00:03<00:02,  5.33it/s]
2023-08-17T13:11:44.0969374Z running benchmark:  63% 19/30 [00:03<00:02,  5.33it/s]
2023-08-17T13:11:44.2850985Z running benchmark:  67% 20/30 [00:03<00:01,  5.33it/s]
2023-08-17T13:11:44.4735334Z running benchmark:  70% 21/30 [00:04<00:01,  5.33it/s]
2023-08-17T13:11:44.6610073Z running benchmark:  73% 22/30 [00:04<00:01,  5.32it/s]
2023-08-17T13:11:44.8484888Z running benchmark:  77% 23/30 [00:04<00:01,  5.32it/s]
2023-08-17T13:11:45.0356062Z running benchmark:  80% 24/30 [00:04<00:01,  5.33it/s]
2023-08-17T13:11:45.2229188Z running benchmark:  83% 25/30 [00:04<00:00,  5.33it/s]
2023-08-17T13:11:45.4103878Z running benchmark:  87% 26/30 [00:05<00:00,  5.33it/s]
2023-08-17T13:11:45.5988648Z running benchmark:  90% 27/30 [00:05<00:00,  5.33it/s]
2023-08-17T13:11:45.7905154Z running benchmark:  93% 28/30 [00:05<00:00,  5.33it/s]
2023-08-17T13:11:45.9778576Z running benchmark:  97% 29/30 [00:05<00:00,  5.29it/s]
2023-08-17T13:11:45.9779131Z running benchmark: 100% 30/30 [00:05<00:00,  5.31it/s]
2023-08-17T13:11:45.9779705Z running benchmark: 100% 30/30 [00:05<00:00,  5.16it/s]
2023-08-17T13:11:45.9803075Z 1.667x
2023-08-17T13:11:51.1647898Z 
2023-08-17T13:11:59.6643929Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:11:59.6644642Z loading model: 0it [00:08, ?it/s]
2023-08-17T13:11:59.6645011Z cuda train PegasusForCausalLM                 
2023-08-17T13:13:15.9319843Z 
2023-08-17T13:13:16.1947685Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T13:13:16.3199445Z running benchmark:   3% 1/30 [00:00<00:07,  3.80it/s]
2023-08-17T13:13:16.4462939Z running benchmark:   7% 2/30 [00:00<00:05,  5.49it/s]
2023-08-17T13:13:16.5720137Z running benchmark:  10% 3/30 [00:00<00:04,  6.38it/s]
2023-08-17T13:13:16.6972126Z running benchmark:  13% 4/30 [00:00<00:03,  6.92it/s]
2023-08-17T13:13:16.8221464Z running benchmark:  17% 5/30 [00:00<00:03,  7.27it/s]
2023-08-17T13:13:16.9472326Z running benchmark:  20% 6/30 [00:00<00:03,  7.50it/s]
2023-08-17T13:13:17.0723990Z running benchmark:  23% 7/30 [00:01<00:03,  7.66it/s]
2023-08-17T13:13:17.1975273Z running benchmark:  27% 8/30 [00:01<00:02,  7.76it/s]
2023-08-17T13:13:17.3224763Z running benchmark:  30% 9/30 [00:01<00:02,  7.83it/s]
2023-08-17T13:13:17.4474780Z running benchmark:  33% 10/30 [00:01<00:02,  7.88it/s]
2023-08-17T13:13:17.5732362Z running benchmark:  37% 11/30 [00:01<00:02,  7.92it/s]
2023-08-17T13:13:17.6992512Z running benchmark:  40% 12/30 [00:01<00:02,  7.93it/s]
2023-08-17T13:13:17.8243727Z running benchmark:  43% 13/30 [00:01<00:02,  7.93it/s]
2023-08-17T13:13:17.9497848Z running benchmark:  47% 14/30 [00:01<00:02,  7.95it/s]
2023-08-17T13:13:18.0765740Z running benchmark:  50% 15/30 [00:02<00:01,  7.96it/s]
2023-08-17T13:13:18.2023070Z running benchmark:  53% 16/30 [00:02<00:01,  7.94it/s]
2023-08-17T13:13:18.3274814Z running benchmark:  57% 17/30 [00:02<00:01,  7.94it/s]
2023-08-17T13:13:18.4524809Z running benchmark:  60% 18/30 [00:02<00:01,  7.95it/s]
2023-08-17T13:13:18.5774339Z running benchmark:  63% 19/30 [00:02<00:01,  7.97it/s]
2023-08-17T13:13:18.7024682Z running benchmark:  67% 20/30 [00:02<00:01,  7.98it/s]
2023-08-17T13:13:18.8284625Z running benchmark:  70% 21/30 [00:02<00:01,  7.99it/s]
2023-08-17T13:13:18.9607784Z running benchmark:  73% 22/30 [00:02<00:01,  7.97it/s]
2023-08-17T13:13:19.0861968Z running benchmark:  77% 23/30 [00:03<00:00,  7.84it/s]
2023-08-17T13:13:19.2111341Z running benchmark:  80% 24/30 [00:03<00:00,  7.88it/s]
2023-08-17T13:13:19.3363307Z running benchmark:  83% 25/30 [00:03<00:00,  7.92it/s]
2023-08-17T13:13:19.4612630Z running benchmark:  87% 26/30 [00:03<00:00,  7.94it/s]
2023-08-17T13:13:19.5863360Z running benchmark:  90% 27/30 [00:03<00:00,  7.96it/s]
2023-08-17T13:13:19.7116059Z running benchmark:  93% 28/30 [00:03<00:00,  7.97it/s]
2023-08-17T13:13:19.8366893Z running benchmark:  97% 29/30 [00:03<00:00,  7.97it/s]
2023-08-17T13:13:19.8367490Z running benchmark: 100% 30/30 [00:03<00:00,  7.98it/s]
2023-08-17T13:13:19.8370184Z running benchmark: 100% 30/30 [00:03<00:00,  7.68it/s]
2023-08-17T13:13:19.8393600Z 1.231x
2023-08-17T13:13:24.9486105Z 
2023-08-17T13:13:38.6070409Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:13:38.6070894Z loading model: 0it [00:13, ?it/s]
2023-08-17T13:13:38.6071731Z cuda train PegasusForConditionalGeneration    
2023-08-17T13:16:31.4873765Z 
2023-08-17T13:16:31.9549851Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T13:16:32.1975128Z running benchmark:   3% 1/30 [00:00<00:13,  2.14it/s]
2023-08-17T13:16:32.4426722Z running benchmark:   7% 2/30 [00:00<00:09,  2.98it/s]
2023-08-17T13:16:32.6833054Z running benchmark:  10% 3/30 [00:00<00:07,  3.40it/s]
2023-08-17T13:16:32.9249183Z running benchmark:  13% 4/30 [00:01<00:07,  3.66it/s]
2023-08-17T13:16:33.1670864Z running benchmark:  17% 5/30 [00:01<00:06,  3.82it/s]
2023-08-17T13:16:33.4074878Z running benchmark:  20% 6/30 [00:01<00:06,  3.92it/s]
2023-08-17T13:16:33.6477615Z running benchmark:  23% 7/30 [00:01<00:05,  4.00it/s]
2023-08-17T13:16:33.8878431Z running benchmark:  27% 8/30 [00:02<00:05,  4.05it/s]
2023-08-17T13:16:34.1293064Z running benchmark:  30% 9/30 [00:02<00:05,  4.08it/s]
2023-08-17T13:16:34.3726433Z running benchmark:  33% 10/30 [00:02<00:04,  4.10it/s]
2023-08-17T13:16:34.6140930Z running benchmark:  37% 11/30 [00:02<00:04,  4.10it/s]
2023-08-17T13:16:34.8546526Z running benchmark:  40% 12/30 [00:03<00:04,  4.12it/s]
2023-08-17T13:16:35.0950452Z running benchmark:  43% 13/30 [00:03<00:04,  4.13it/s]
2023-08-17T13:16:35.3359162Z running benchmark:  47% 14/30 [00:03<00:03,  4.14it/s]
2023-08-17T13:16:35.5795273Z running benchmark:  50% 15/30 [00:03<00:03,  4.14it/s]
2023-08-17T13:16:35.8201710Z running benchmark:  53% 16/30 [00:04<00:03,  4.13it/s]
2023-08-17T13:16:36.0607314Z running benchmark:  57% 17/30 [00:04<00:03,  4.14it/s]
2023-08-17T13:16:36.3008363Z running benchmark:  60% 18/30 [00:04<00:02,  4.14it/s]
2023-08-17T13:16:36.5419813Z running benchmark:  63% 19/30 [00:04<00:02,  4.15it/s]
2023-08-17T13:16:36.7849698Z running benchmark:  67% 20/30 [00:05<00:02,  4.15it/s]
2023-08-17T13:16:37.0258639Z running benchmark:  70% 21/30 [00:05<00:02,  4.14it/s]
2023-08-17T13:16:37.2663112Z running benchmark:  73% 22/30 [00:05<00:01,  4.14it/s]
2023-08-17T13:16:37.5065504Z running benchmark:  77% 23/30 [00:05<00:01,  4.15it/s]
2023-08-17T13:16:37.7471561Z running benchmark:  80% 24/30 [00:06<00:01,  4.15it/s]
2023-08-17T13:16:37.9894214Z running benchmark:  83% 25/30 [00:06<00:01,  4.15it/s]
2023-08-17T13:16:38.2306471Z running benchmark:  87% 26/30 [00:06<00:00,  4.15it/s]
2023-08-17T13:16:38.4711915Z running benchmark:  90% 27/30 [00:06<00:00,  4.15it/s]
2023-08-17T13:16:38.7114599Z running benchmark:  93% 28/30 [00:06<00:00,  4.15it/s]
2023-08-17T13:16:38.9519395Z running benchmark:  97% 29/30 [00:07<00:00,  4.15it/s]
2023-08-17T13:16:38.9519816Z running benchmark: 100% 30/30 [00:07<00:00,  4.16it/s]
2023-08-17T13:16:38.9520120Z running benchmark: 100% 30/30 [00:07<00:00,  4.02it/s]
2023-08-17T13:16:38.9546935Z 1.327x
2023-08-17T13:16:44.7492393Z 
2023-08-17T13:16:44.7523076Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`
2023-08-17T13:16:48.0037614Z 
2023-08-17T13:16:48.0038325Z loading model: 0it [00:03, ?it/s]
2023-08-17T13:16:48.0038709Z cuda train RobertaForCausalLM                 
2023-08-17T13:18:08.2447419Z 
2023-08-17T13:18:08.5933637Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T13:18:08.7732771Z running benchmark:   3% 1/30 [00:00<00:10,  2.86it/s]
2023-08-17T13:18:08.9538971Z running benchmark:   7% 2/30 [00:00<00:06,  4.00it/s]
2023-08-17T13:18:09.1347243Z running benchmark:  10% 3/30 [00:00<00:05,  4.58it/s]
2023-08-17T13:18:09.3152602Z running benchmark:  13% 4/30 [00:00<00:05,  4.92it/s]
2023-08-17T13:18:09.4954053Z running benchmark:  17% 5/30 [00:01<00:04,  5.12it/s]
2023-08-17T13:18:09.6752770Z running benchmark:  20% 6/30 [00:01<00:04,  5.26it/s]
2023-08-17T13:18:09.8553734Z running benchmark:  23% 7/30 [00:01<00:04,  5.35it/s]
2023-08-17T13:18:10.0349187Z running benchmark:  27% 8/30 [00:01<00:04,  5.42it/s]
2023-08-17T13:18:10.2155952Z running benchmark:  30% 9/30 [00:01<00:03,  5.46it/s]
2023-08-17T13:18:10.3956658Z running benchmark:  33% 10/30 [00:01<00:03,  5.49it/s]
2023-08-17T13:18:10.5756516Z running benchmark:  37% 11/30 [00:02<00:03,  5.50it/s]
2023-08-17T13:18:10.7555949Z running benchmark:  40% 12/30 [00:02<00:03,  5.52it/s]
2023-08-17T13:18:10.9354855Z running benchmark:  43% 13/30 [00:02<00:03,  5.53it/s]
2023-08-17T13:18:11.1153083Z running benchmark:  47% 14/30 [00:02<00:02,  5.54it/s]
2023-08-17T13:18:11.2958552Z running benchmark:  50% 15/30 [00:02<00:02,  5.55it/s]
2023-08-17T13:18:11.4766448Z running benchmark:  53% 16/30 [00:03<00:02,  5.54it/s]
2023-08-17T13:18:11.6566687Z running benchmark:  57% 17/30 [00:03<00:02,  5.54it/s]
2023-08-17T13:18:11.8397600Z running benchmark:  60% 18/30 [00:03<00:02,  5.54it/s]
2023-08-17T13:18:12.0200568Z running benchmark:  63% 19/30 [00:03<00:01,  5.52it/s]
2023-08-17T13:18:12.1999184Z running benchmark:  67% 20/30 [00:03<00:01,  5.53it/s]
2023-08-17T13:18:12.3799703Z running benchmark:  70% 21/30 [00:03<00:01,  5.54it/s]
2023-08-17T13:18:12.5607266Z running benchmark:  73% 22/30 [00:04<00:01,  5.54it/s]
2023-08-17T13:18:12.7408480Z running benchmark:  77% 23/30 [00:04<00:01,  5.54it/s]
2023-08-17T13:18:12.9210218Z running benchmark:  80% 24/30 [00:04<00:01,  5.54it/s]
2023-08-17T13:18:13.1009221Z running benchmark:  83% 25/30 [00:04<00:00,  5.55it/s]
2023-08-17T13:18:13.2807612Z running benchmark:  87% 26/30 [00:04<00:00,  5.55it/s]
2023-08-17T13:18:13.4605700Z running benchmark:  90% 27/30 [00:05<00:00,  5.55it/s]
2023-08-17T13:18:13.6411518Z running benchmark:  93% 28/30 [00:05<00:00,  5.55it/s]
2023-08-17T13:18:13.8222545Z running benchmark:  97% 29/30 [00:05<00:00,  5.55it/s]
2023-08-17T13:18:13.8222913Z running benchmark: 100% 30/30 [00:05<00:00,  5.54it/s]
2023-08-17T13:18:13.8223218Z running benchmark: 100% 30/30 [00:05<00:00,  5.38it/s]
2023-08-17T13:18:13.8247436Z 1.751x
2023-08-17T13:18:18.6003234Z 
2023-08-17T13:18:21.4837545Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:18:21.4838369Z loading model: 0it [00:02, ?it/s]
2023-08-17T13:18:21.4838680Z cuda train RobertaForQuestionAnswering        
2023-08-17T13:19:38.3826581Z 
2023-08-17T13:19:38.6526402Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T13:19:38.7973145Z running benchmark:   3% 1/30 [00:00<00:07,  3.70it/s]
2023-08-17T13:19:38.9603975Z running benchmark:   7% 2/30 [00:00<00:05,  5.09it/s]
2023-08-17T13:19:39.1051461Z running benchmark:  10% 3/30 [00:00<00:04,  5.52it/s]
2023-08-17T13:19:39.2509314Z running benchmark:  13% 4/30 [00:00<00:04,  5.99it/s]
2023-08-17T13:19:39.3967322Z running benchmark:  17% 5/30 [00:00<00:03,  6.28it/s]
2023-08-17T13:19:39.5420580Z running benchmark:  20% 6/30 [00:01<00:03,  6.47it/s]
2023-08-17T13:19:39.6867500Z running benchmark:  23% 7/30 [00:01<00:03,  6.59it/s]
2023-08-17T13:19:39.8360858Z running benchmark:  27% 8/30 [00:01<00:03,  6.69it/s]
2023-08-17T13:19:39.9811918Z running benchmark:  30% 9/30 [00:01<00:03,  6.76it/s]
2023-08-17T13:19:40.1258631Z running benchmark:  33% 10/30 [00:01<00:02,  6.73it/s]
2023-08-17T13:19:40.2705654Z running benchmark:  37% 11/30 [00:01<00:02,  6.79it/s]
2023-08-17T13:19:40.4152144Z running benchmark:  40% 12/30 [00:01<00:02,  6.82it/s]
2023-08-17T13:19:40.5605248Z running benchmark:  43% 13/30 [00:02<00:02,  6.85it/s]
2023-08-17T13:19:40.7058930Z running benchmark:  47% 14/30 [00:02<00:02,  6.86it/s]
2023-08-17T13:19:40.8600002Z running benchmark:  50% 15/30 [00:02<00:02,  6.87it/s]
2023-08-17T13:19:41.0050388Z running benchmark:  53% 16/30 [00:02<00:02,  6.75it/s]
2023-08-17T13:19:41.1497909Z running benchmark:  57% 17/30 [00:02<00:01,  6.79it/s]
2023-08-17T13:19:41.2946104Z running benchmark:  60% 18/30 [00:02<00:01,  6.82it/s]
2023-08-17T13:19:41.4393023Z running benchmark:  63% 19/30 [00:02<00:01,  6.85it/s]
2023-08-17T13:19:41.5840665Z running benchmark:  67% 20/30 [00:03<00:01,  6.87it/s]
2023-08-17T13:19:41.7294561Z running benchmark:  70% 21/30 [00:03<00:01,  6.88it/s]
2023-08-17T13:19:41.8750187Z running benchmark:  73% 22/30 [00:03<00:01,  6.88it/s]
2023-08-17T13:19:42.0201138Z running benchmark:  77% 23/30 [00:03<00:01,  6.88it/s]
2023-08-17T13:19:42.1649443Z running benchmark:  80% 24/30 [00:03<00:00,  6.88it/s]
2023-08-17T13:19:42.3097132Z running benchmark:  83% 25/30 [00:03<00:00,  6.89it/s]
2023-08-17T13:19:42.4543896Z running benchmark:  87% 26/30 [00:03<00:00,  6.89it/s]
2023-08-17T13:19:42.5990289Z running benchmark:  90% 27/30 [00:04<00:00,  6.90it/s]
2023-08-17T13:19:42.7436308Z running benchmark:  93% 28/30 [00:04<00:00,  6.90it/s]
2023-08-17T13:19:42.8891032Z running benchmark:  97% 29/30 [00:04<00:00,  6.91it/s]
2023-08-17T13:19:42.8891417Z running benchmark: 100% 30/30 [00:04<00:00,  6.90it/s]
2023-08-17T13:19:42.8891742Z running benchmark: 100% 30/30 [00:04<00:00,  6.66it/s]
2023-08-17T13:19:42.8926481Z 1.903x
2023-08-17T13:19:47.6479486Z 
2023-08-17T13:19:48.9731434Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:19:48.9732114Z loading model: 0it [00:01, ?it/s]
2023-08-17T13:19:48.9732437Z cuda train Speech2Text2ForCausalLM            
2023-08-17T13:20:34.2907225Z 
2023-08-17T13:20:34.4674416Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T13:20:34.6421736Z running benchmark:   3% 1/30 [00:00<00:05,  5.64it/s]
2023-08-17T13:20:34.8167769Z running benchmark:  10% 3/30 [00:00<00:02,  9.03it/s]
2023-08-17T13:20:34.9913586Z running benchmark:  17% 5/30 [00:00<00:02, 10.13it/s]
2023-08-17T13:20:35.1665108Z running benchmark:  23% 7/30 [00:00<00:02, 10.65it/s]
2023-08-17T13:20:35.3418458Z running benchmark:  30% 9/30 [00:00<00:01, 10.93it/s]
2023-08-17T13:20:35.5176575Z running benchmark:  37% 11/30 [00:01<00:01, 11.09it/s]
2023-08-17T13:20:35.6927364Z running benchmark:  43% 13/30 [00:01<00:01, 11.18it/s]
2023-08-17T13:20:35.8667797Z running benchmark:  50% 15/30 [00:01<00:01, 11.26it/s]
2023-08-17T13:20:36.0410316Z running benchmark:  57% 17/30 [00:01<00:01, 11.33it/s]
2023-08-17T13:20:36.2156228Z running benchmark:  63% 19/30 [00:01<00:00, 11.38it/s]
2023-08-17T13:20:36.3906448Z running benchmark:  70% 21/30 [00:01<00:00, 11.40it/s]
2023-08-17T13:20:36.5665033Z running benchmark:  77% 23/30 [00:02<00:00, 11.41it/s]
2023-08-17T13:20:36.7422865Z running benchmark:  83% 25/30 [00:02<00:00, 11.40it/s]
2023-08-17T13:20:36.9168645Z running benchmark:  90% 27/30 [00:02<00:00, 11.39it/s]
2023-08-17T13:20:37.0046505Z running benchmark:  97% 29/30 [00:02<00:00, 11.41it/s]
2023-08-17T13:20:37.0046905Z running benchmark: 100% 30/30 [00:02<00:00, 11.05it/s]
2023-08-17T13:20:37.0069920Z 1.578x
2023-08-17T13:20:41.4661389Z 
2023-08-17T13:20:43.8524871Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:20:43.8525202Z loading model: 0it [00:02, ?it/s]
2023-08-17T13:20:43.8525546Z cuda train T5ForConditionalGeneration         
2023-08-17T13:22:21.4579931Z 
2023-08-17T13:22:21.8035249Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T13:22:21.9713861Z running benchmark:   3% 1/30 [00:00<00:10,  2.89it/s]
2023-08-17T13:22:22.1397188Z running benchmark:   7% 2/30 [00:00<00:06,  4.14it/s]
2023-08-17T13:22:22.3081192Z running benchmark:  10% 3/30 [00:00<00:05,  4.81it/s]
2023-08-17T13:22:22.4756066Z running benchmark:  13% 4/30 [00:00<00:05,  5.20it/s]
2023-08-17T13:22:22.6446710Z running benchmark:  17% 5/30 [00:01<00:04,  5.45it/s]
2023-08-17T13:22:22.8120588Z running benchmark:  20% 6/30 [00:01<00:04,  5.60it/s]
2023-08-17T13:22:22.9793605Z running benchmark:  23% 7/30 [00:01<00:04,  5.72it/s]
2023-08-17T13:22:23.1468037Z running benchmark:  27% 8/30 [00:01<00:03,  5.80it/s]
2023-08-17T13:22:23.3150347Z running benchmark:  30% 9/30 [00:01<00:03,  5.85it/s]
2023-08-17T13:22:23.4834115Z running benchmark:  33% 10/30 [00:01<00:03,  5.88it/s]
2023-08-17T13:22:23.6510496Z running benchmark:  37% 11/30 [00:02<00:03,  5.90it/s]
2023-08-17T13:22:23.8217924Z running benchmark:  40% 12/30 [00:02<00:03,  5.92it/s]
2023-08-17T13:22:23.9899770Z running benchmark:  43% 13/30 [00:02<00:02,  5.90it/s]
2023-08-17T13:22:24.1579593Z running benchmark:  47% 14/30 [00:02<00:02,  5.91it/s]
2023-08-17T13:22:24.3256543Z running benchmark:  50% 15/30 [00:02<00:02,  5.93it/s]
2023-08-17T13:22:24.4942791Z running benchmark:  53% 16/30 [00:02<00:02,  5.94it/s]
2023-08-17T13:22:24.6626010Z running benchmark:  57% 17/30 [00:03<00:02,  5.94it/s]
2023-08-17T13:22:24.8317279Z running benchmark:  60% 18/30 [00:03<00:02,  5.94it/s]
2023-08-17T13:22:24.9992123Z running benchmark:  63% 19/30 [00:03<00:01,  5.93it/s]
2023-08-17T13:22:25.1665838Z running benchmark:  67% 20/30 [00:03<00:01,  5.94it/s]
2023-08-17T13:22:25.3338633Z running benchmark:  70% 21/30 [00:03<00:01,  5.95it/s]
2023-08-17T13:22:25.5011563Z running benchmark:  73% 22/30 [00:03<00:01,  5.96it/s]
2023-08-17T13:22:25.6691509Z running benchmark:  77% 23/30 [00:04<00:01,  5.96it/s]
2023-08-17T13:22:25.8374305Z running benchmark:  80% 24/30 [00:04<00:01,  5.96it/s]
2023-08-17T13:22:26.0049471Z running benchmark:  83% 25/30 [00:04<00:00,  5.96it/s]
2023-08-17T13:22:26.1722512Z running benchmark:  87% 26/30 [00:04<00:00,  5.96it/s]
2023-08-17T13:22:26.3396914Z running benchmark:  90% 27/30 [00:04<00:00,  5.96it/s]
2023-08-17T13:22:26.5069715Z running benchmark:  93% 28/30 [00:04<00:00,  5.97it/s]
2023-08-17T13:22:26.6743204Z running benchmark:  97% 29/30 [00:05<00:00,  5.97it/s]
2023-08-17T13:22:26.6743737Z running benchmark: 100% 30/30 [00:05<00:00,  5.97it/s]
2023-08-17T13:22:26.6744041Z running benchmark: 100% 30/30 [00:05<00:00,  5.75it/s]
2023-08-17T13:22:26.6773687Z 1.628x
2023-08-17T13:22:31.5897430Z 
2023-08-17T13:22:33.9349509Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:22:33.9349837Z loading model: 0it [00:02, ?it/s]
2023-08-17T13:22:33.9350298Z cuda train T5Small                            
2023-08-17T13:24:12.0427780Z 
2023-08-17T13:24:12.3962812Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T13:24:12.5650439Z running benchmark:   3% 1/30 [00:00<00:10,  2.83it/s]
2023-08-17T13:24:12.7331942Z running benchmark:   7% 2/30 [00:00<00:06,  4.08it/s]
2023-08-17T13:24:12.9011676Z running benchmark:  10% 3/30 [00:00<00:05,  4.76it/s]
2023-08-17T13:24:13.0690369Z running benchmark:  13% 4/30 [00:00<00:05,  5.17it/s]
2023-08-17T13:24:13.2438212Z running benchmark:  17% 5/30 [00:01<00:04,  5.43it/s]
2023-08-17T13:24:13.4118546Z running benchmark:  20% 6/30 [00:01<00:04,  5.53it/s]
2023-08-17T13:24:13.5800739Z running benchmark:  23% 7/30 [00:01<00:04,  5.66it/s]
2023-08-17T13:24:13.7492809Z running benchmark:  27% 8/30 [00:01<00:03,  5.75it/s]
2023-08-17T13:24:13.9178504Z running benchmark:  30% 9/30 [00:01<00:03,  5.80it/s]
2023-08-17T13:24:14.0860979Z running benchmark:  33% 10/30 [00:01<00:03,  5.84it/s]
2023-08-17T13:24:14.2618429Z running benchmark:  37% 11/30 [00:02<00:03,  5.87it/s]
2023-08-17T13:24:14.4305115Z running benchmark:  40% 12/30 [00:02<00:03,  5.89it/s]
2023-08-17T13:24:14.5986110Z running benchmark:  43% 13/30 [00:02<00:02,  5.83it/s]
2023-08-17T13:24:14.7666149Z running benchmark:  47% 14/30 [00:02<00:02,  5.86it/s]
2023-08-17T13:24:14.9354149Z running benchmark:  50% 15/30 [00:02<00:02,  5.89it/s]
2023-08-17T13:24:15.1036216Z running benchmark:  53% 16/30 [00:02<00:02,  5.90it/s]
2023-08-17T13:24:15.2714120Z running benchmark:  57% 17/30 [00:03<00:02,  5.91it/s]
2023-08-17T13:24:15.4491584Z running benchmark:  60% 18/30 [00:03<00:02,  5.93it/s]
2023-08-17T13:24:15.6167980Z running benchmark:  63% 19/30 [00:03<00:01,  5.83it/s]
2023-08-17T13:24:15.7845619Z running benchmark:  67% 20/30 [00:03<00:01,  5.87it/s]
2023-08-17T13:24:15.9528124Z running benchmark:  70% 21/30 [00:03<00:01,  5.90it/s]
2023-08-17T13:24:16.1216243Z running benchmark:  73% 22/30 [00:03<00:01,  5.91it/s]
2023-08-17T13:24:16.2897113Z running benchmark:  77% 23/30 [00:04<00:01,  5.92it/s]
2023-08-17T13:24:16.4638034Z running benchmark:  80% 24/30 [00:04<00:01,  5.93it/s]
2023-08-17T13:24:16.6316845Z running benchmark:  83% 25/30 [00:04<00:00,  5.87it/s]
2023-08-17T13:24:16.7994481Z running benchmark:  87% 26/30 [00:04<00:00,  5.90it/s]
2023-08-17T13:24:16.9672230Z running benchmark:  90% 27/30 [00:04<00:00,  5.91it/s]
2023-08-17T13:24:17.1354913Z running benchmark:  93% 28/30 [00:04<00:00,  5.93it/s]
2023-08-17T13:24:17.3044154Z running benchmark:  97% 29/30 [00:05<00:00,  5.93it/s]
2023-08-17T13:24:17.3044729Z running benchmark: 100% 30/30 [00:05<00:00,  5.93it/s]
2023-08-17T13:24:17.3045688Z running benchmark: 100% 30/30 [00:05<00:00,  5.70it/s]
2023-08-17T13:24:17.3073458Z 1.637x
2023-08-17T13:24:22.1358512Z 
2023-08-17T13:24:28.4896792Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:24:28.4897113Z loading model: 0it [00:06, ?it/s]
2023-08-17T13:24:28.4897410Z cuda train TrOCRForCausalLM                   
2023-08-17T13:25:47.8070291Z 
2023-08-17T13:25:48.2931947Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T13:25:48.5351889Z running benchmark:   3% 1/30 [00:00<00:14,  2.05it/s]
2023-08-17T13:25:48.7777129Z running benchmark:   7% 2/30 [00:00<00:09,  2.92it/s]
2023-08-17T13:25:49.0196402Z running benchmark:  10% 3/30 [00:00<00:08,  3.37it/s]
2023-08-17T13:25:49.2614660Z running benchmark:  13% 4/30 [00:01<00:07,  3.63it/s]
2023-08-17T13:25:49.5035635Z running benchmark:  17% 5/30 [00:01<00:06,  3.80it/s]
2023-08-17T13:25:49.7457709Z running benchmark:  20% 6/30 [00:01<00:06,  3.91it/s]
2023-08-17T13:25:49.9880365Z running benchmark:  23% 7/30 [00:01<00:05,  3.98it/s]
2023-08-17T13:25:50.2303827Z running benchmark:  27% 8/30 [00:02<00:05,  4.02it/s]
2023-08-17T13:25:50.4723975Z running benchmark:  30% 9/30 [00:02<00:05,  4.05it/s]
2023-08-17T13:25:50.7145462Z running benchmark:  33% 10/30 [00:02<00:04,  4.08it/s]
2023-08-17T13:25:50.9569823Z running benchmark:  37% 11/30 [00:02<00:04,  4.09it/s]
2023-08-17T13:25:51.2005107Z running benchmark:  40% 12/30 [00:03<00:04,  4.10it/s]
2023-08-17T13:25:51.4427955Z running benchmark:  43% 13/30 [00:03<00:04,  4.10it/s]
2023-08-17T13:25:51.6848284Z running benchmark:  47% 14/30 [00:03<00:03,  4.11it/s]
2023-08-17T13:25:51.9271767Z running benchmark:  50% 15/30 [00:03<00:03,  4.12it/s]
2023-08-17T13:25:52.1702618Z running benchmark:  53% 16/30 [00:04<00:03,  4.12it/s]
2023-08-17T13:25:52.4128130Z running benchmark:  57% 17/30 [00:04<00:03,  4.12it/s]
2023-08-17T13:25:52.6547357Z running benchmark:  60% 18/30 [00:04<00:02,  4.12it/s]
2023-08-17T13:25:52.8968531Z running benchmark:  63% 19/30 [00:04<00:02,  4.12it/s]
2023-08-17T13:25:53.1388099Z running benchmark:  67% 20/30 [00:05<00:02,  4.13it/s]
2023-08-17T13:25:53.3819482Z running benchmark:  70% 21/30 [00:05<00:02,  4.13it/s]
2023-08-17T13:25:53.6244968Z running benchmark:  73% 22/30 [00:05<00:01,  4.12it/s]
2023-08-17T13:25:53.8673124Z running benchmark:  77% 23/30 [00:05<00:01,  4.12it/s]
2023-08-17T13:25:54.1097808Z running benchmark:  80% 24/30 [00:06<00:01,  4.12it/s]
2023-08-17T13:25:54.3525676Z running benchmark:  83% 25/30 [00:06<00:01,  4.12it/s]
2023-08-17T13:25:54.5953372Z running benchmark:  87% 26/30 [00:06<00:00,  4.12it/s]
2023-08-17T13:25:54.8412030Z running benchmark:  90% 27/30 [00:06<00:00,  4.12it/s]
2023-08-17T13:25:55.0832791Z running benchmark:  93% 28/30 [00:07<00:00,  4.10it/s]
2023-08-17T13:25:55.3253471Z running benchmark:  97% 29/30 [00:07<00:00,  4.11it/s]
2023-08-17T13:25:55.3258135Z running benchmark: 100% 30/30 [00:07<00:00,  4.12it/s]
2023-08-17T13:25:55.3258472Z running benchmark: 100% 30/30 [00:07<00:00,  3.99it/s]
2023-08-17T13:25:55.3281074Z 1.318x
2023-08-17T13:26:00.1280920Z 
2023-08-17T13:26:14.9465136Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:26:14.9465879Z loading model: 0it [00:14, ?it/s]
2023-08-17T13:26:14.9466416Z cuda train XGLMForCausalLM                    
2023-08-17T13:28:43.9002071Z 
2023-08-17T13:28:44.1587068Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T13:28:44.2978555Z running benchmark:   3% 1/30 [00:00<00:07,  3.86it/s]
2023-08-17T13:28:44.4366747Z running benchmark:   7% 2/30 [00:00<00:05,  5.30it/s]
2023-08-17T13:28:44.5754545Z running benchmark:  10% 3/30 [00:00<00:04,  6.03it/s]
2023-08-17T13:28:44.7264932Z running benchmark:  13% 4/30 [00:00<00:04,  6.44it/s]
2023-08-17T13:28:44.8636699Z running benchmark:  17% 5/30 [00:00<00:03,  6.51it/s]
2023-08-17T13:28:45.0022575Z running benchmark:  20% 6/30 [00:00<00:03,  6.75it/s]
2023-08-17T13:28:45.1426444Z running benchmark:  23% 7/30 [00:01<00:03,  6.90it/s]
2023-08-17T13:28:45.2799044Z running benchmark:  27% 8/30 [00:01<00:03,  6.97it/s]
2023-08-17T13:28:45.4163417Z running benchmark:  30% 9/30 [00:01<00:02,  7.06it/s]
2023-08-17T13:28:45.5524378Z running benchmark:  33% 10/30 [00:01<00:02,  7.14it/s]
2023-08-17T13:28:45.7013896Z running benchmark:  37% 11/30 [00:01<00:02,  7.20it/s]
2023-08-17T13:28:45.8375123Z running benchmark:  40% 12/30 [00:01<00:02,  7.05it/s]
2023-08-17T13:28:45.9734625Z running benchmark:  43% 13/30 [00:01<00:02,  7.13it/s]
2023-08-17T13:28:46.1113246Z running benchmark:  47% 14/30 [00:02<00:02,  7.20it/s]
2023-08-17T13:28:46.2509782Z running benchmark:  50% 15/30 [00:02<00:02,  7.22it/s]
2023-08-17T13:28:46.3896569Z running benchmark:  53% 16/30 [00:02<00:01,  7.20it/s]
2023-08-17T13:28:46.5258454Z running benchmark:  57% 17/30 [00:02<00:01,  7.20it/s]
2023-08-17T13:28:46.6618399Z running benchmark:  60% 18/30 [00:02<00:01,  7.24it/s]
2023-08-17T13:28:46.7979571Z running benchmark:  63% 19/30 [00:02<00:01,  7.28it/s]
2023-08-17T13:28:46.9335005Z running benchmark:  67% 20/30 [00:02<00:01,  7.30it/s]
2023-08-17T13:28:47.0710630Z running benchmark:  70% 21/30 [00:03<00:01,  7.32it/s]
2023-08-17T13:28:47.2073307Z running benchmark:  73% 22/30 [00:03<00:01,  7.31it/s]
2023-08-17T13:28:47.3447795Z running benchmark:  77% 23/30 [00:03<00:00,  7.31it/s]
2023-08-17T13:28:47.4853935Z running benchmark:  80% 24/30 [00:03<00:00,  7.31it/s]
2023-08-17T13:28:47.6234219Z running benchmark:  83% 25/30 [00:03<00:00,  7.24it/s]
2023-08-17T13:28:47.7742128Z running benchmark:  87% 26/30 [00:03<00:00,  7.25it/s]
2023-08-17T13:28:47.9106312Z running benchmark:  90% 27/30 [00:03<00:00,  7.05it/s]
2023-08-17T13:28:48.0474088Z running benchmark:  93% 28/30 [00:04<00:00,  7.13it/s]
2023-08-17T13:28:48.1851180Z running benchmark:  97% 29/30 [00:04<00:00,  7.18it/s]
2023-08-17T13:28:48.1851523Z running benchmark: 100% 30/30 [00:04<00:00,  7.21it/s]
2023-08-17T13:28:48.1851810Z running benchmark: 100% 30/30 [00:04<00:00,  7.00it/s]
2023-08-17T13:28:48.1878937Z 1.807x
2023-08-17T13:28:53.7176501Z 
2023-08-17T13:29:00.7566456Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:29:00.7566972Z loading model: 0it [00:07, ?it/s]
2023-08-17T13:29:00.7567254Z cuda train XLNetLMHeadModel                   
2023-08-17T13:31:20.7915021Z [2023-08-17 13:31:20,789] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-08-17T13:31:46.3129222Z skipping cudagraphs due to multiple devices
2023-08-17T13:32:43.2017699Z 
2023-08-17T13:32:43.6906970Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T13:32:44.1220220Z running benchmark:   3% 1/30 [00:00<00:14,  2.04it/s]
2023-08-17T13:32:44.5592491Z running benchmark:   7% 2/30 [00:00<00:12,  2.20it/s]
2023-08-17T13:32:44.9945384Z running benchmark:  10% 3/30 [00:01<00:12,  2.24it/s]
2023-08-17T13:32:45.4315661Z running benchmark:  13% 4/30 [00:01<00:11,  2.26it/s]
2023-08-17T13:32:45.8662182Z running benchmark:  17% 5/30 [00:02<00:11,  2.27it/s]
2023-08-17T13:32:46.3012748Z running benchmark:  20% 6/30 [00:02<00:10,  2.28it/s]
2023-08-17T13:32:46.7366028Z running benchmark:  23% 7/30 [00:03<00:10,  2.29it/s]
2023-08-17T13:32:47.1714328Z running benchmark:  27% 8/30 [00:03<00:09,  2.29it/s]
2023-08-17T13:32:47.6065829Z running benchmark:  30% 9/30 [00:03<00:09,  2.29it/s]
2023-08-17T13:32:48.0422433Z running benchmark:  33% 10/30 [00:04<00:08,  2.29it/s]
2023-08-17T13:32:48.4778367Z running benchmark:  37% 11/30 [00:04<00:08,  2.29it/s]
2023-08-17T13:32:48.9135828Z running benchmark:  40% 12/30 [00:05<00:07,  2.30it/s]
2023-08-17T13:32:49.3494320Z running benchmark:  43% 13/30 [00:05<00:07,  2.29it/s]
2023-08-17T13:32:49.7850733Z running benchmark:  47% 14/30 [00:06<00:06,  2.29it/s]
2023-08-17T13:32:50.2205116Z running benchmark:  50% 15/30 [00:06<00:06,  2.29it/s]
2023-08-17T13:32:50.6546767Z running benchmark:  53% 16/30 [00:07<00:06,  2.30it/s]
2023-08-17T13:32:51.0900205Z running benchmark:  57% 17/30 [00:07<00:05,  2.30it/s]
2023-08-17T13:32:51.5278791Z running benchmark:  60% 18/30 [00:07<00:05,  2.30it/s]
2023-08-17T13:32:51.9631407Z running benchmark:  63% 19/30 [00:08<00:04,  2.29it/s]
2023-08-17T13:32:52.4071631Z running benchmark:  67% 20/30 [00:08<00:04,  2.29it/s]
2023-08-17T13:32:52.8440481Z running benchmark:  70% 21/30 [00:09<00:03,  2.29it/s]
2023-08-17T13:32:53.2783940Z running benchmark:  73% 22/30 [00:09<00:03,  2.28it/s]
2023-08-17T13:32:53.7100595Z running benchmark:  77% 23/30 [00:10<00:03,  2.29it/s]
2023-08-17T13:32:54.1417800Z running benchmark:  80% 24/30 [00:10<00:02,  2.30it/s]
2023-08-17T13:32:54.5834167Z running benchmark:  83% 25/30 [00:10<00:02,  2.30it/s]
2023-08-17T13:32:55.0197511Z running benchmark:  87% 26/30 [00:11<00:01,  2.29it/s]
2023-08-17T13:32:55.4547227Z running benchmark:  90% 27/30 [00:11<00:01,  2.29it/s]
2023-08-17T13:32:55.8999769Z running benchmark:  93% 28/30 [00:12<00:00,  2.29it/s]
2023-08-17T13:32:56.3315749Z running benchmark:  97% 29/30 [00:12<00:00,  2.28it/s]
2023-08-17T13:32:56.3316109Z running benchmark: 100% 30/30 [00:13<00:00,  2.29it/s]
2023-08-17T13:32:56.3316411Z running benchmark: 100% 30/30 [00:13<00:00,  2.28it/s]
2023-08-17T13:32:56.3352761Z 1.831x
2023-08-17T13:33:01.9914263Z 
2023-08-17T13:33:05.4596351Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:33:05.4596916Z loading model: 0it [00:03, ?it/s]
2023-08-17T13:33:05.4597354Z cuda train YituTechConvBert                   
2023-08-17T13:35:02.3099508Z 
2023-08-17T13:35:02.7294220Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T13:35:02.9316791Z running benchmark:   3% 1/30 [00:00<00:12,  2.38it/s]
2023-08-17T13:35:03.1343732Z running benchmark:   7% 2/30 [00:00<00:08,  3.42it/s]
2023-08-17T13:35:03.3369011Z running benchmark:  10% 3/30 [00:00<00:06,  3.98it/s]
2023-08-17T13:35:03.5403402Z running benchmark:  13% 4/30 [00:01<00:06,  4.31it/s]
2023-08-17T13:35:03.7433696Z running benchmark:  17% 5/30 [00:01<00:05,  4.51it/s]
2023-08-17T13:35:03.9449993Z running benchmark:  20% 6/30 [00:01<00:05,  4.64it/s]
2023-08-17T13:35:04.1470885Z running benchmark:  23% 7/30 [00:01<00:04,  4.74it/s]
2023-08-17T13:35:04.3487520Z running benchmark:  27% 8/30 [00:01<00:04,  4.81it/s]
2023-08-17T13:35:04.5506142Z running benchmark:  30% 9/30 [00:02<00:04,  4.85it/s]
2023-08-17T13:35:04.7532627Z running benchmark:  33% 10/30 [00:02<00:04,  4.88it/s]
2023-08-17T13:35:04.9554575Z running benchmark:  37% 11/30 [00:02<00:03,  4.90it/s]
2023-08-17T13:35:05.1574074Z running benchmark:  40% 12/30 [00:02<00:03,  4.91it/s]
2023-08-17T13:35:05.3591372Z running benchmark:  43% 13/30 [00:02<00:03,  4.93it/s]
2023-08-17T13:35:05.5607845Z running benchmark:  47% 14/30 [00:03<00:03,  4.93it/s]
2023-08-17T13:35:05.7623574Z running benchmark:  50% 15/30 [00:03<00:03,  4.94it/s]
2023-08-17T13:35:05.9646156Z running benchmark:  53% 16/30 [00:03<00:02,  4.95it/s]
2023-08-17T13:35:06.1670410Z running benchmark:  57% 17/30 [00:03<00:02,  4.95it/s]
2023-08-17T13:35:06.3687921Z running benchmark:  60% 18/30 [00:03<00:02,  4.94it/s]
2023-08-17T13:35:06.5704547Z running benchmark:  63% 19/30 [00:04<00:02,  4.95it/s]
2023-08-17T13:35:06.7723411Z running benchmark:  67% 20/30 [00:04<00:02,  4.95it/s]
2023-08-17T13:35:06.9738189Z running benchmark:  70% 21/30 [00:04<00:01,  4.95it/s]
2023-08-17T13:35:07.1763757Z running benchmark:  73% 22/30 [00:04<00:01,  4.96it/s]
2023-08-17T13:35:07.3782755Z running benchmark:  77% 23/30 [00:04<00:01,  4.95it/s]
2023-08-17T13:35:07.5799933Z running benchmark:  80% 24/30 [00:05<00:01,  4.95it/s]
2023-08-17T13:35:07.7815918Z running benchmark:  83% 25/30 [00:05<00:01,  4.95it/s]
2023-08-17T13:35:07.9830094Z running benchmark:  87% 26/30 [00:05<00:00,  4.95it/s]
2023-08-17T13:35:08.1846957Z running benchmark:  90% 27/30 [00:05<00:00,  4.96it/s]
2023-08-17T13:35:08.3873259Z running benchmark:  93% 28/30 [00:05<00:00,  4.96it/s]
2023-08-17T13:35:08.5893012Z running benchmark:  97% 29/30 [00:06<00:00,  4.95it/s]
2023-08-17T13:35:08.5893516Z running benchmark: 100% 30/30 [00:06<00:00,  4.95it/s]
2023-08-17T13:35:08.5893816Z running benchmark: 100% 30/30 [00:06<00:00,  4.78it/s]
2023-08-17T13:35:08.6011774Z 1.613x
2023-08-17T13:35:10.4918558Z speedup             gmean=1.71x mean=1.741x
2023-08-17T13:35:10.4918921Z abs_latency         gmean=70.66x mean=74.821x
2023-08-17T13:35:10.4921353Z compilation_latency mean=122.871 seconds
2023-08-17T13:35:10.4922256Z compression_ratio   mean=0.951x
2023-08-17T13:35:10.4924514Z eager_peak_mem      gmean=11.07x mean=11.485x
2023-08-17T13:35:10.4926790Z dynamo_peak_mem     gmean=11.68x mean=12.081x
2023-08-17T13:35:10.4929190Z calls_captured      gmean=862.98x mean=962.294x
2023-08-17T13:35:10.4931037Z unique_graphs       gmean=1.00x mean=1.000x
2023-08-17T13:35:10.4933067Z graph_breaks        gmean=6.60x mean=6.647x
2023-08-17T13:35:10.4935299Z unique_graph_breaks gmean=5.59x mean=5.647x
2023-08-17T13:35:11.0314673Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *cppwrapper-true* ]]
2023-08-17T13:35:11.0316202Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *freezing_cudagraphs-true* ]]
2023-08-17T13:35:11.0317167Z + [[ training == \i\n\f\e\r\e\n\c\e ]]
2023-08-17T13:35:11.0318223Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *aotinductor-true* ]]
2023-08-17T13:35:11.0319253Z + [[ training == \i\n\f\e\r\e\n\c\e ]]
2023-08-17T13:35:11.0320745Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *maxautotune-true* ]]
2023-08-17T13:35:11.0321982Z + for mode in "${modes[@]}"
2023-08-17T13:35:11.0322440Z + [[ inference == \i\n\f\e\r\e\n\c\e ]]
2023-08-17T13:35:11.0323149Z + dtype=bfloat16
2023-08-17T13:35:11.0323647Z + for target in "${targets[@]}"
2023-08-17T13:35:11.0324161Z + target_flag=("--${target}")
2023-08-17T13:35:11.0324410Z + local target_flag
2023-08-17T13:35:11.0324662Z + [[ accuracy == \p\e\r\f\o\r\m\a\n\c\e ]]
2023-08-17T13:35:11.0324940Z + [[ accuracy == \a\c\c\u\r\a\c\y ]]
2023-08-17T13:35:11.0325320Z + target_flag+=(--no-translation-validation)
2023-08-17T13:35:11.0326226Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *default-true* ]]
2023-08-17T13:35:11.0327736Z + python benchmarks/dynamo/huggingface.py --accuracy --no-translation-validation --inference --bfloat16 --backend inductor --disable-cudagraphs --device cuda --total-partitions 3 --partition-id 2 --output /var/lib/jenkins/workspace/test/test-reports/inductor_no_cudagraphs_huggingface_bfloat16_inference_cuda_accuracy.csv
2023-08-17T13:35:17.5538621Z 
2023-08-17T13:35:23.6881463Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:35:23.6881958Z loading model: 0it [00:06, ?it/s]
2023-08-17T13:35:23.6987578Z cuda eval  MegatronBertForQuestionAnswering   
2023-08-17T13:35:56.2166467Z pass
2023-08-17T13:36:00.8510069Z 
2023-08-17T13:36:02.6750001Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:36:02.7053628Z loading model: 0it [00:01, ?it/s]
2023-08-17T13:36:02.7053980Z cuda eval  MobileBertForMaskedLM              
2023-08-17T13:37:21.6116560Z pass
2023-08-17T13:37:26.5535654Z 
2023-08-17T13:37:28.1395846Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:37:28.1396399Z loading model: 0it [00:01, ?it/s]
2023-08-17T13:37:28.1694365Z cuda eval  MobileBertForQuestionAnswering     
2023-08-17T13:38:46.4739715Z pass
2023-08-17T13:38:51.4097051Z 
2023-08-17T13:38:54.9218803Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:38:54.9219628Z loading model: 0it [00:03, ?it/s]
2023-08-17T13:38:54.9267511Z cuda eval  OPTForCausalLM                     
2023-08-17T13:39:14.0302202Z pass
2023-08-17T13:39:18.4971363Z 
2023-08-17T13:39:21.4547111Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:39:21.4547592Z loading model: 0it [00:02, ?it/s]
2023-08-17T13:39:21.4585550Z cuda eval  PLBartForCausalLM                  
2023-08-17T13:39:34.4619528Z pass
2023-08-17T13:39:39.5824682Z 
2023-08-17T13:39:44.3790537Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:39:44.3791140Z loading model: 0it [00:04, ?it/s]
2023-08-17T13:39:44.3858879Z cuda eval  PLBartForConditionalGeneration     
2023-08-17T13:40:07.4525881Z pass
2023-08-17T13:40:11.8323125Z 
2023-08-17T13:40:19.9197497Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:40:19.9198650Z loading model: 0it [00:08, ?it/s]
2023-08-17T13:40:19.9283345Z cuda eval  PegasusForCausalLM                 
2023-08-17T13:40:39.2298101Z pass
2023-08-17T13:40:43.5529143Z 
2023-08-17T13:40:56.4315838Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:40:56.4316604Z loading model: 0it [00:12, ?it/s]
2023-08-17T13:40:56.4449289Z cuda eval  PegasusForConditionalGeneration    
2023-08-17T13:41:35.3753109Z pass
2023-08-17T13:41:40.1196165Z 
2023-08-17T13:41:40.1228729Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`
2023-08-17T13:41:43.1131608Z 
2023-08-17T13:41:43.1132057Z loading model: 0it [00:02, ?it/s]
2023-08-17T13:41:43.1186020Z cuda eval  RobertaForCausalLM                 
2023-08-17T13:42:02.1716629Z pass
2023-08-17T13:42:06.5847763Z 
2023-08-17T13:42:09.1909460Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:42:09.1910333Z loading model: 0it [00:02, ?it/s]
2023-08-17T13:42:09.1964547Z cuda eval  RobertaForQuestionAnswering        
2023-08-17T13:42:27.6606843Z pass
2023-08-17T13:42:31.9588931Z 
2023-08-17T13:42:33.1910200Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:42:33.1910527Z loading model: 0it [00:01, ?it/s]
2023-08-17T13:42:33.1950989Z cuda eval  Speech2Text2ForCausalLM            
2023-08-17T13:42:45.2758395Z pass
2023-08-17T13:42:49.4348334Z 
2023-08-17T13:42:51.5942126Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:42:51.5942829Z loading model: 0it [00:02, ?it/s]
2023-08-17T13:42:51.5982014Z cuda eval  T5ForConditionalGeneration         
2023-08-17T13:43:14.8994512Z pass
2023-08-17T13:43:19.2175843Z 
2023-08-17T13:43:21.3477731Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:43:21.3478458Z loading model: 0it [00:02, ?it/s]
2023-08-17T13:43:21.3518253Z cuda eval  T5Small                            
2023-08-17T13:43:43.4026798Z pass
2023-08-17T13:43:47.7287364Z 
2023-08-17T13:43:53.3186338Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:43:53.3187837Z loading model: 0it [00:05, ?it/s]
2023-08-17T13:43:53.3265041Z cuda eval  TrOCRForCausalLM                   
2023-08-17T13:44:12.4450845Z pass
2023-08-17T13:44:16.6686207Z 
2023-08-17T13:44:30.2686926Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:44:30.2687897Z loading model: 0it [00:13, ?it/s]
2023-08-17T13:44:30.2783223Z cuda eval  XGLMForCausalLM                    
2023-08-17T13:45:03.1125228Z pass
2023-08-17T13:45:08.0136707Z 
2023-08-17T13:45:14.2403979Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:45:14.2404501Z loading model: 0it [00:06, ?it/s]
2023-08-17T13:45:14.2508940Z cuda eval  XLNetLMHeadModel                   
2023-08-17T13:45:44.4624647Z [2023-08-17 13:45:44,460] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-08-17T13:45:58.6624896Z pass
2023-08-17T13:46:03.4293649Z 
2023-08-17T13:46:06.5656260Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:46:06.5656570Z loading model: 0it [00:03, ?it/s]
2023-08-17T13:46:06.5737789Z cuda eval  YituTechConvBert                   
2023-08-17T13:46:33.3203164Z pass
2023-08-17T13:46:34.5039529Z accuracy            pass_rate=100.00%
2023-08-17T13:46:34.5041853Z calls_captured      gmean=642.21x mean=730.412x
2023-08-17T13:46:34.5044200Z unique_graphs       gmean=1.00x mean=1.000x
2023-08-17T13:46:34.5047239Z graph_breaks        gmean=0.00x mean=0.000x
2023-08-17T13:46:34.5049939Z unique_graph_breaks gmean=0.00x mean=0.000x
2023-08-17T13:46:35.0256303Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *cudagraphs-true* ]]
2023-08-17T13:46:35.0258369Z + python benchmarks/dynamo/huggingface.py --accuracy --no-translation-validation --inference --bfloat16 --backend inductor --device cuda --total-partitions 3 --partition-id 2 --output /var/lib/jenkins/workspace/test/test-reports/inductor_with_cudagraphs_huggingface_bfloat16_inference_cuda_accuracy.csv
2023-08-17T13:46:41.6176040Z 
2023-08-17T13:46:47.9257944Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:46:47.9259294Z loading model: 0it [00:06, ?it/s]
2023-08-17T13:46:47.9362723Z cuda eval  MegatronBertForQuestionAnswering   
2023-08-17T13:47:20.5340704Z pass
2023-08-17T13:47:25.0109907Z 
2023-08-17T13:47:26.8116374Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:47:26.8116897Z loading model: 0it [00:01, ?it/s]
2023-08-17T13:47:26.8417094Z cuda eval  MobileBertForMaskedLM              
2023-08-17T13:48:45.6714774Z pass
2023-08-17T13:48:50.5097251Z 
2023-08-17T13:48:52.2292899Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:48:52.2293254Z loading model: 0it [00:01, ?it/s]
2023-08-17T13:48:52.2593227Z cuda eval  MobileBertForQuestionAnswering     
2023-08-17T13:50:11.1545773Z pass
2023-08-17T13:50:15.9890466Z 
2023-08-17T13:50:19.4979140Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:50:19.4979622Z loading model: 0it [00:03, ?it/s]
2023-08-17T13:50:19.5028341Z cuda eval  OPTForCausalLM                     
2023-08-17T13:50:38.0164231Z pass
2023-08-17T13:50:42.3054903Z 
2023-08-17T13:50:45.1722170Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:50:45.1764073Z loading model: 0it [00:02, ?it/s]
2023-08-17T13:50:45.1764437Z cuda eval  PLBartForCausalLM                  
2023-08-17T13:50:57.4686078Z pass
2023-08-17T13:51:01.5620810Z 
2023-08-17T13:51:06.3348706Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:51:06.3349033Z loading model: 0it [00:04, ?it/s]
2023-08-17T13:51:06.3416607Z cuda eval  PLBartForConditionalGeneration     
2023-08-17T13:51:29.1967053Z pass
2023-08-17T13:51:33.5380403Z 
2023-08-17T13:51:41.4692277Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:51:41.4693103Z loading model: 0it [00:07, ?it/s]
2023-08-17T13:51:41.4777297Z cuda eval  PegasusForCausalLM                 
2023-08-17T13:52:00.2890038Z pass
2023-08-17T13:52:04.5839907Z 
2023-08-17T13:52:17.2747324Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:52:17.2747830Z loading model: 0it [00:12, ?it/s]
2023-08-17T13:52:17.2881935Z cuda eval  PegasusForConditionalGeneration    
2023-08-17T13:52:56.3116223Z pass
2023-08-17T13:53:00.9148722Z 
2023-08-17T13:53:00.9177943Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`
2023-08-17T13:53:03.8355147Z 
2023-08-17T13:53:03.8356062Z loading model: 0it [00:02, ?it/s]
2023-08-17T13:53:03.8415363Z cuda eval  RobertaForCausalLM                 
2023-08-17T13:53:22.6626228Z pass
2023-08-17T13:53:27.3237749Z 
2023-08-17T13:53:29.8903223Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:53:29.8903921Z loading model: 0it [00:02, ?it/s]
2023-08-17T13:53:29.8955863Z cuda eval  RobertaForQuestionAnswering        
2023-08-17T13:53:48.6028206Z pass
2023-08-17T13:53:52.9027917Z 
2023-08-17T13:53:54.0708864Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:53:54.0709195Z loading model: 0it [00:01, ?it/s]
2023-08-17T13:53:54.0750431Z cuda eval  Speech2Text2ForCausalLM            
2023-08-17T13:54:06.0563193Z pass
2023-08-17T13:54:10.2156057Z 
2023-08-17T13:54:12.3035615Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:54:12.3035978Z loading model: 0it [00:02, ?it/s]
2023-08-17T13:54:12.3076666Z cuda eval  T5ForConditionalGeneration         
2023-08-17T13:54:34.6211012Z pass
2023-08-17T13:54:38.8655372Z 
2023-08-17T13:54:41.0069042Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:54:41.0069699Z loading model: 0it [00:02, ?it/s]
2023-08-17T13:54:41.0108785Z cuda eval  T5Small                            
2023-08-17T13:55:03.4781106Z pass
2023-08-17T13:55:07.8363278Z 
2023-08-17T13:55:13.4337246Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:55:13.4338203Z loading model: 0it [00:05, ?it/s]
2023-08-17T13:55:13.4417995Z cuda eval  TrOCRForCausalLM                   
2023-08-17T13:55:32.0286195Z pass
2023-08-17T13:55:36.2488178Z 
2023-08-17T13:55:49.7418579Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:55:49.7419724Z loading model: 0it [00:13, ?it/s]
2023-08-17T13:55:49.7522367Z cuda eval  XGLMForCausalLM                    
2023-08-17T13:56:22.1434277Z pass
2023-08-17T13:56:26.6135577Z 
2023-08-17T13:56:32.8077058Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:56:32.8078014Z loading model: 0it [00:06, ?it/s]
2023-08-17T13:56:32.8185956Z cuda eval  XLNetLMHeadModel                   
2023-08-17T13:57:02.7506323Z [2023-08-17 13:57:02,748] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-08-17T13:57:16.1150631Z skipping cudagraphs due to multiple devices
2023-08-17T13:57:16.4885216Z pass
2023-08-17T13:57:21.1089857Z 
2023-08-17T13:57:24.3368484Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:57:24.3369529Z loading model: 0it [00:03, ?it/s]
2023-08-17T13:57:24.3445231Z cuda eval  YituTechConvBert                   
2023-08-17T13:57:50.9149813Z pass
2023-08-17T13:57:52.1278393Z accuracy            pass_rate=100.00%
2023-08-17T13:57:52.1280465Z calls_captured      gmean=642.21x mean=730.412x
2023-08-17T13:57:52.1282149Z unique_graphs       gmean=1.00x mean=1.000x
2023-08-17T13:57:52.1284519Z graph_breaks        gmean=0.00x mean=0.000x
2023-08-17T13:57:52.1286900Z unique_graph_breaks gmean=0.00x mean=0.000x
2023-08-17T13:57:52.6166121Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *dynamic-true* ]]
2023-08-17T13:57:52.6167748Z + python benchmarks/dynamo/huggingface.py --accuracy --no-translation-validation --inference --bfloat16 --backend inductor --dynamic-shapes --dynamic-batch-only --device cuda --total-partitions 3 --partition-id 2 --output /var/lib/jenkins/workspace/test/test-reports/inductor_dynamic_huggingface_bfloat16_inference_cuda_accuracy.csv
2023-08-17T13:57:59.8086475Z 
2023-08-17T13:58:06.0889394Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:58:06.0889728Z loading model: 0it [00:06, ?it/s]
2023-08-17T13:58:06.0994574Z cuda eval  MegatronBertForQuestionAnswering   
2023-08-17T13:58:38.9116513Z pass
2023-08-17T13:58:43.3652831Z 
2023-08-17T13:58:45.1907881Z loading model: 0it [00:00, ?it/s]
2023-08-17T13:58:45.1908220Z loading model: 0it [00:01, ?it/s]
2023-08-17T13:58:45.2239909Z cuda eval  MobileBertForMaskedLM              
2023-08-17T14:00:03.8848460Z pass
2023-08-17T14:00:08.7305641Z 
2023-08-17T14:00:10.3436873Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:00:10.3437369Z loading model: 0it [00:01, ?it/s]
2023-08-17T14:00:10.3739500Z cuda eval  MobileBertForQuestionAnswering     
2023-08-17T14:01:29.3785566Z pass
2023-08-17T14:01:34.5811341Z 
2023-08-17T14:01:37.9729517Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:01:37.9729841Z loading model: 0it [00:03, ?it/s]
2023-08-17T14:01:37.9778798Z cuda eval  OPTForCausalLM                     
2023-08-17T14:01:56.3388766Z pass
2023-08-17T14:02:00.5230829Z 
2023-08-17T14:02:03.4737199Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:02:03.4737521Z loading model: 0it [00:02, ?it/s]
2023-08-17T14:02:03.4778675Z cuda eval  PLBartForCausalLM                  
2023-08-17T14:02:15.7534279Z pass
2023-08-17T14:02:19.8509162Z 
2023-08-17T14:02:24.7597352Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:02:24.7597660Z loading model: 0it [00:04, ?it/s]
2023-08-17T14:02:24.7664882Z cuda eval  PLBartForConditionalGeneration     
2023-08-17T14:02:47.7646642Z pass
2023-08-17T14:02:52.1085402Z 
2023-08-17T14:03:00.0560245Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:03:00.0560577Z loading model: 0it [00:07, ?it/s]
2023-08-17T14:03:00.0645730Z cuda eval  PegasusForCausalLM                 
2023-08-17T14:03:18.9507764Z pass
2023-08-17T14:03:23.2079079Z 
2023-08-17T14:03:35.9655313Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:03:35.9655638Z loading model: 0it [00:12, ?it/s]
2023-08-17T14:03:35.9788792Z cuda eval  PegasusForConditionalGeneration    
2023-08-17T14:04:15.4626430Z pass
2023-08-17T14:04:20.1408117Z 
2023-08-17T14:04:20.1441145Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`
2023-08-17T14:04:23.0400828Z 
2023-08-17T14:04:23.0402079Z loading model: 0it [00:02, ?it/s]
2023-08-17T14:04:23.0456996Z cuda eval  RobertaForCausalLM                 
2023-08-17T14:04:41.8990112Z pass
2023-08-17T14:04:46.1751012Z 
2023-08-17T14:04:48.7125816Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:04:48.7126181Z loading model: 0it [00:02, ?it/s]
2023-08-17T14:04:48.7181203Z cuda eval  RobertaForQuestionAnswering        
2023-08-17T14:05:07.4928417Z pass
2023-08-17T14:05:11.7295714Z 
2023-08-17T14:05:12.8442986Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:05:12.8443494Z loading model: 0it [00:01, ?it/s]
2023-08-17T14:05:12.8484713Z cuda eval  Speech2Text2ForCausalLM            
2023-08-17T14:05:24.8163368Z pass
2023-08-17T14:05:28.9820503Z 
2023-08-17T14:05:31.2560929Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:05:31.2561287Z loading model: 0it [00:02, ?it/s]
2023-08-17T14:05:31.2600880Z cuda eval  T5ForConditionalGeneration         
2023-08-17T14:05:53.6706989Z pass
2023-08-17T14:05:57.8830425Z 
2023-08-17T14:06:00.0844786Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:06:00.0845127Z loading model: 0it [00:02, ?it/s]
2023-08-17T14:06:00.0889191Z cuda eval  T5Small                            
2023-08-17T14:06:22.4622507Z pass
2023-08-17T14:06:26.7426422Z 
2023-08-17T14:06:32.2493971Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:06:32.2497453Z loading model: 0it [00:05, ?it/s]
2023-08-17T14:06:32.2578981Z cuda eval  TrOCRForCausalLM                   
2023-08-17T14:06:50.8388980Z pass
2023-08-17T14:06:55.0306569Z 
2023-08-17T14:07:08.7201331Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:07:08.7201672Z loading model: 0it [00:13, ?it/s]
2023-08-17T14:07:08.7301643Z cuda eval  XGLMForCausalLM                    
2023-08-17T14:07:41.2808145Z pass
2023-08-17T14:07:45.8312695Z 
2023-08-17T14:07:52.0368687Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:07:52.0369024Z loading model: 0it [00:06, ?it/s]
2023-08-17T14:07:52.0474678Z cuda eval  XLNetLMHeadModel                   
2023-08-17T14:08:21.9479872Z [2023-08-17 14:08:21,946] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-08-17T14:08:35.3637253Z skipping cudagraphs due to multiple devices
2023-08-17T14:08:35.7284519Z pass
2023-08-17T14:08:40.8889222Z 
2023-08-17T14:08:44.0288874Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:08:44.0289194Z loading model: 0it [00:03, ?it/s]
2023-08-17T14:08:44.0369804Z cuda eval  YituTechConvBert                   
2023-08-17T14:09:10.6994340Z pass
2023-08-17T14:09:11.8996921Z accuracy            pass_rate=100.00%
2023-08-17T14:09:11.8997718Z calls_captured      gmean=642.21x mean=730.412x
2023-08-17T14:09:11.8998966Z unique_graphs       gmean=1.00x mean=1.000x
2023-08-17T14:09:11.9001133Z graph_breaks        gmean=0.00x mean=0.000x
2023-08-17T14:09:11.9002435Z unique_graph_breaks gmean=0.00x mean=0.000x
2023-08-17T14:09:12.3761431Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *cppwrapper-true* ]]
2023-08-17T14:09:12.3763071Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *freezing_cudagraphs-true* ]]
2023-08-17T14:09:12.3768798Z + [[ inference == \i\n\f\e\r\e\n\c\e ]]
2023-08-17T14:09:12.3770237Z + python benchmarks/dynamo/huggingface.py --accuracy --no-translation-validation --inference --bfloat16 --backend inductor --device cuda --total-partitions 3 --partition-id 2 --freezing --output /var/lib/jenkins/workspace/test/test-reports/inductor_with_cudagraphs_freezing_huggingface_bfloat16_inference_cuda_accuracy.csv
2023-08-17T14:09:19.0688585Z 
2023-08-17T14:09:25.2053603Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:09:25.2054084Z loading model: 0it [00:06, ?it/s]
2023-08-17T14:09:25.2162257Z cuda eval  MegatronBertForQuestionAnswering   
2023-08-17T14:10:12.6060381Z pass
2023-08-17T14:10:17.3378761Z 
2023-08-17T14:10:19.1578287Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:10:19.1579047Z loading model: 0it [00:01, ?it/s]
2023-08-17T14:10:19.1887482Z cuda eval  MobileBertForMaskedLM              
2023-08-17T14:12:03.9743470Z pass
2023-08-17T14:12:08.9406051Z 
2023-08-17T14:12:10.5041344Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:12:10.5041836Z loading model: 0it [00:01, ?it/s]
2023-08-17T14:12:10.5346331Z cuda eval  MobileBertForQuestionAnswering     
2023-08-17T14:13:54.6786779Z pass
2023-08-17T14:13:59.5594304Z 
2023-08-17T14:14:03.0784264Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:14:03.0785161Z loading model: 0it [00:03, ?it/s]
2023-08-17T14:14:03.0856951Z cuda eval  OPTForCausalLM                     
2023-08-17T14:14:28.5027433Z pass
2023-08-17T14:14:32.8021068Z 
2023-08-17T14:14:35.8499220Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:14:35.8499996Z loading model: 0it [00:03, ?it/s]
2023-08-17T14:14:35.8541462Z cuda eval  PLBartForCausalLM                  
2023-08-17T14:14:52.0983667Z pass
2023-08-17T14:14:56.2289880Z 
2023-08-17T14:15:01.1008323Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:15:01.1009116Z loading model: 0it [00:04, ?it/s]
2023-08-17T14:15:01.1078807Z cuda eval  PLBartForConditionalGeneration     
2023-08-17T14:15:33.9161200Z pass
2023-08-17T14:15:38.4390663Z 
2023-08-17T14:15:46.4360271Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:15:46.4360782Z loading model: 0it [00:07, ?it/s]
2023-08-17T14:15:46.4442058Z cuda eval  PegasusForCausalLM                 
2023-08-17T14:16:12.7460709Z pass
2023-08-17T14:16:17.1721349Z 
2023-08-17T14:16:30.2724715Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:16:30.2726102Z loading model: 0it [00:13, ?it/s]
2023-08-17T14:16:30.2869375Z cuda eval  PegasusForConditionalGeneration    
2023-08-17T14:17:29.1886766Z pass
2023-08-17T14:17:33.9794281Z 
2023-08-17T14:17:33.9826392Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`
2023-08-17T14:17:36.9920047Z 
2023-08-17T14:17:36.9920917Z loading model: 0it [00:03, ?it/s]
2023-08-17T14:17:36.9977415Z cuda eval  RobertaForCausalLM                 
2023-08-17T14:18:03.8388056Z pass
2023-08-17T14:18:08.2282992Z 
2023-08-17T14:18:10.8365211Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:18:10.8365570Z loading model: 0it [00:02, ?it/s]
2023-08-17T14:18:10.8436349Z cuda eval  RobertaForQuestionAnswering        
2023-08-17T14:18:37.0504281Z pass
2023-08-17T14:18:41.3237050Z 
2023-08-17T14:18:42.4582713Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:18:42.4583358Z loading model: 0it [00:01, ?it/s]
2023-08-17T14:18:42.4626296Z cuda eval  Speech2Text2ForCausalLM            
2023-08-17T14:18:58.0409360Z pass
2023-08-17T14:19:02.2840138Z 
2023-08-17T14:19:04.4369327Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:19:04.4370134Z loading model: 0it [00:02, ?it/s]
2023-08-17T14:19:04.4413518Z cuda eval  T5ForConditionalGeneration         
2023-08-17T14:19:34.1575842Z pass
2023-08-17T14:19:38.5259096Z 
2023-08-17T14:19:40.6454746Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:19:40.6455256Z loading model: 0it [00:02, ?it/s]
2023-08-17T14:19:40.6498568Z cuda eval  T5Small                            
2023-08-17T14:20:09.9382333Z pass
2023-08-17T14:20:14.3190057Z 
2023-08-17T14:20:19.8783678Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:20:19.8787044Z loading model: 0it [00:05, ?it/s]
2023-08-17T14:20:19.8864404Z cuda eval  TrOCRForCausalLM                   
2023-08-17T14:20:45.4352102Z pass
2023-08-17T14:20:49.6967917Z 
2023-08-17T14:21:03.3559355Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:21:03.3559697Z loading model: 0it [00:13, ?it/s]
2023-08-17T14:21:03.3658823Z cuda eval  XGLMForCausalLM                    
2023-08-17T14:21:49.9135984Z pass
2023-08-17T14:21:54.8600418Z 
2023-08-17T14:22:01.1019700Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:22:01.1020025Z loading model: 0it [00:06, ?it/s]
2023-08-17T14:22:01.1127437Z cuda eval  XLNetLMHeadModel                   
2023-08-17T14:22:47.1769185Z [2023-08-17 14:22:47,174] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-08-17T14:23:01.4767992Z skipping cudagraphs due to multiple devices
2023-08-17T14:23:01.8301694Z pass
2023-08-17T14:23:06.5930020Z 
2023-08-17T14:23:09.6743953Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:23:09.6744532Z loading model: 0it [00:03, ?it/s]
2023-08-17T14:23:09.6822449Z cuda eval  YituTechConvBert                   
2023-08-17T14:23:52.6068014Z pass
2023-08-17T14:23:53.8063674Z accuracy            pass_rate=100.00%
2023-08-17T14:23:53.8064205Z calls_captured      gmean=642.21x mean=730.412x
2023-08-17T14:23:53.8067171Z unique_graphs       gmean=1.00x mean=1.000x
2023-08-17T14:23:53.8069491Z graph_breaks        gmean=0.00x mean=0.000x
2023-08-17T14:23:53.8071858Z unique_graph_breaks gmean=0.00x mean=0.000x
2023-08-17T14:23:54.3215249Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *aotinductor-true* ]]
2023-08-17T14:23:54.3216062Z + [[ inference == \i\n\f\e\r\e\n\c\e ]]
2023-08-17T14:23:54.3217478Z + python benchmarks/dynamo/huggingface.py --accuracy --no-translation-validation --inference --bfloat16 --export-aot-inductor --disable-cudagraphs --device cuda --total-partitions 3 --partition-id 2 --output /var/lib/jenkins/workspace/test/test-reports/inductor_aot_inductor_huggingface_bfloat16_inference_cuda_accuracy.csv
2023-08-17T14:24:00.9646450Z 
2023-08-17T14:24:07.1392826Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:24:07.1393291Z loading model: 0it [00:06, ?it/s]
2023-08-17T14:24:07.1502547Z cuda eval  MegatronBertForQuestionAnswering   
2023-08-17T14:25:40.8323112Z pass
2023-08-17T14:25:45.2801125Z 
2023-08-17T14:25:47.1485645Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:25:47.1486118Z loading model: 0it [00:01, ?it/s]
2023-08-17T14:25:47.1784216Z cuda eval  MobileBertForMaskedLM              
2023-08-17T14:29:02.7827735Z pass
2023-08-17T14:29:07.4721528Z 
2023-08-17T14:29:09.0235938Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:29:09.0236618Z loading model: 0it [00:01, ?it/s]
2023-08-17T14:29:09.0536242Z cuda eval  MobileBertForQuestionAnswering     
2023-08-17T14:32:22.8667288Z pass
2023-08-17T14:32:27.5428830Z 
2023-08-17T14:32:30.9687305Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:32:30.9688068Z loading model: 0it [00:03, ?it/s]
2023-08-17T14:32:30.9738899Z cuda eval  OPTForCausalLM                     
2023-08-17T14:32:33.0209225Z ERROR:common:Dynamo attempts to add additional input during export: value=0.832506537437439, source=RandomValueSource(random_call_index=0)
2023-08-17T14:32:33.0209927Z 
2023-08-17T14:32:33.0210084Z from user code:
2023-08-17T14:32:33.0214911Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 930, in forward
2023-08-17T14:32:33.0215388Z     outputs = self.model.decoder(
2023-08-17T14:32:33.0215955Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:32:33.0216356Z     return forward_call(*args, **kwargs)
2023-08-17T14:32:33.0216932Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 673, in forward
2023-08-17T14:32:33.0217362Z     dropout_probability = random.uniform(0, 1)
2023-08-17T14:32:33.0217702Z 
2023-08-17T14:32:33.0217897Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-08-17T14:32:33.0218471Z Traceback (most recent call last):
2023-08-17T14:32:33.0219099Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2125, in check_accuracy
2023-08-17T14:32:33.0219873Z     new_result = optimized_model_iter_fn(model_copy, example_inputs)
2023-08-17T14:32:33.0220667Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1186, in opt_aot_inductor
2023-08-17T14:32:33.0221413Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-08-17T14:32:33.0222235Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1131, in load
2023-08-17T14:32:33.0222600Z     so_path, exported = torch._export.aot_compile(
2023-08-17T14:32:33.0223222Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 460, in aot_compile
2023-08-17T14:32:33.0223624Z     ep = export(f, args, kwargs, constraints)
2023-08-17T14:32:33.0224159Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 222, in export
2023-08-17T14:32:33.0224730Z     gm_torch_level, _ = torch._dynamo.export(
2023-08-17T14:32:33.0225259Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1201, in inner
2023-08-17T14:32:33.0225646Z     result_traced = opt_f(*args, **kwargs)
2023-08-17T14:32:33.0226202Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T14:32:33.0226612Z     return self._call_impl(*args, **kwargs)
2023-08-17T14:32:33.0227147Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:32:33.0227537Z     return forward_call(*args, **kwargs)
2023-08-17T14:32:33.0228108Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 333, in _fn
2023-08-17T14:32:33.0228469Z     return fn(*args, **kwargs)
2023-08-17T14:32:33.0229012Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T14:32:33.0229405Z     return self._call_impl(*args, **kwargs)
2023-08-17T14:32:33.0229939Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:32:33.0230325Z     return forward_call(*args, **kwargs)
2023-08-17T14:32:33.0230859Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 493, in catch_errors
2023-08-17T14:32:33.0231279Z     return callback(frame, cache_size, hooks, frame_state)
2023-08-17T14:32:33.0231827Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 132, in _fn
2023-08-17T14:32:33.0232186Z     return fn(*args, **kwargs)
2023-08-17T14:32:33.0232743Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 370, in _convert_frame_assert
2023-08-17T14:32:33.0233124Z     return _compile(
2023-08-17T14:32:33.0233626Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 554, in _compile
2023-08-17T14:32:33.0234067Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-08-17T14:32:33.0234638Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 180, in time_wrapper
2023-08-17T14:32:33.0234997Z     r = func(*args, **kwargs)
2023-08-17T14:32:33.0235514Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 465, in compile_inner
2023-08-17T14:32:33.0235939Z     out_code = transform_code_object(code, transform)
2023-08-17T14:32:33.0236566Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-08-17T14:32:33.0237021Z     transformations(instructions, code_options)
2023-08-17T14:32:33.0237580Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 432, in transform
2023-08-17T14:32:33.0237926Z     tracer.run()
2023-08-17T14:32:33.0238469Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2071, in run
2023-08-17T14:32:33.0238825Z     super().run()
2023-08-17T14:32:33.0239326Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:32:33.0239670Z     and self.step()
2023-08-17T14:32:33.0240174Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:32:33.0240685Z     getattr(self, inst.opname)(inst)
2023-08-17T14:32:33.0241234Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:32:33.0241599Z     return inner_fn(self, inst)
2023-08-17T14:32:33.0242151Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1167, in CALL_FUNCTION_KW
2023-08-17T14:32:33.0242879Z     self.call_function(fn, args, kwargs)
2023-08-17T14:32:33.0243455Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:32:33.0243885Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:32:33.0244451Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 331, in call_function
2023-08-17T14:32:33.0244860Z     return tx.inline_user_function_return(
2023-08-17T14:32:33.0245464Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T14:32:33.0245981Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T14:32:33.0246607Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T14:32:33.0247072Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T14:32:33.0247654Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T14:32:33.0248070Z     tracer.run()
2023-08-17T14:32:33.0248583Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:32:33.0248946Z     and self.step()
2023-08-17T14:32:33.0249438Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:32:33.0249827Z     getattr(self, inst.opname)(inst)
2023-08-17T14:32:33.0250368Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:32:33.0250746Z     return inner_fn(self, inst)
2023-08-17T14:32:33.0251286Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1155, in CALL_FUNCTION_EX
2023-08-17T14:32:33.0251747Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-08-17T14:32:33.0252343Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:32:33.0252765Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:32:33.0253341Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-08-17T14:32:33.0253747Z     return super().call_function(tx, args, kwargs)
2023-08-17T14:32:33.0254327Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-08-17T14:32:33.0254748Z     return super().call_function(tx, args, kwargs)
2023-08-17T14:32:33.0255320Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-08-17T14:32:33.0255715Z     return tx.inline_user_function_return(
2023-08-17T14:32:33.0256313Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T14:32:33.0256826Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T14:32:33.0257466Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T14:32:33.0257890Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T14:32:33.0258639Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T14:32:33.0259005Z     tracer.run()
2023-08-17T14:32:33.0259510Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:32:33.0259868Z     and self.step()
2023-08-17T14:32:33.0260357Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:32:33.0260844Z     getattr(self, inst.opname)(inst)
2023-08-17T14:32:33.0261394Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:32:33.0261772Z     return inner_fn(self, inst)
2023-08-17T14:32:33.0262331Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1115, in CALL_FUNCTION
2023-08-17T14:32:33.0262712Z     self.call_function(fn, args, {})
2023-08-17T14:32:33.0263270Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:32:33.0263697Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:32:33.0264286Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 328, in call_function
2023-08-17T14:32:33.0264746Z     return VariableBuilder(tx, source).wrap_unspecialized_primitive(
2023-08-17T14:32:33.0265412Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1119, in wrap_unspecialized_primitive
2023-08-17T14:32:33.0265824Z     raise AssertionError(
2023-08-17T14:32:33.0266263Z AssertionError: Dynamo attempts to add additional input during export: value=0.832506537437439, source=RandomValueSource(random_call_index=0)
2023-08-17T14:32:33.0266578Z 
2023-08-17T14:32:33.0266670Z from user code:
2023-08-17T14:32:33.0267184Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 930, in forward
2023-08-17T14:32:33.0267592Z     outputs = self.model.decoder(
2023-08-17T14:32:33.0268453Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:32:33.0268942Z     return forward_call(*args, **kwargs)
2023-08-17T14:32:33.0269507Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 673, in forward
2023-08-17T14:32:33.0269943Z     dropout_probability = random.uniform(0, 1)
2023-08-17T14:32:33.0270178Z 
2023-08-17T14:32:33.0270472Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-08-17T14:32:33.0270813Z 
2023-08-17T14:32:33.0271002Z TorchDynamo optimized model failed to run because of following error
2023-08-17T14:32:33.0271323Z fail_to_run
2023-08-17T14:32:37.0303307Z 
2023-08-17T14:32:39.8868429Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:32:39.8868773Z loading model: 0it [00:02, ?it/s]
2023-08-17T14:32:39.8906213Z cuda eval  PLBartForCausalLM                  
2023-08-17T14:32:41.6769967Z ERROR:common:Dynamo attempts to add additional input during export: value=0.6562776565551758, source=RandomValueSource(random_call_index=0)
2023-08-17T14:32:41.6770519Z 
2023-08-17T14:32:41.6770659Z from user code:
2023-08-17T14:32:41.6772048Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1692, in forward
2023-08-17T14:32:41.6772653Z     outputs = self.model.decoder(
2023-08-17T14:32:41.6773295Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:32:41.6773691Z     return forward_call(*args, **kwargs)
2023-08-17T14:32:41.6774272Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1050, in forward
2023-08-17T14:32:41.6774708Z     dropout_probability = random.uniform(0, 1)
2023-08-17T14:32:41.6775274Z 
2023-08-17T14:32:41.6775464Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-08-17T14:32:41.6775790Z Traceback (most recent call last):
2023-08-17T14:32:41.6776169Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2125, in check_accuracy
2023-08-17T14:32:41.6776622Z     new_result = optimized_model_iter_fn(model_copy, example_inputs)
2023-08-17T14:32:41.6777221Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1186, in opt_aot_inductor
2023-08-17T14:32:41.6777851Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-08-17T14:32:41.6778301Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1131, in load
2023-08-17T14:32:41.6778678Z     so_path, exported = torch._export.aot_compile(
2023-08-17T14:32:41.6779271Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 460, in aot_compile
2023-08-17T14:32:41.6779662Z     ep = export(f, args, kwargs, constraints)
2023-08-17T14:32:41.6780204Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 222, in export
2023-08-17T14:32:41.6780598Z     gm_torch_level, _ = torch._dynamo.export(
2023-08-17T14:32:41.6781134Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1201, in inner
2023-08-17T14:32:41.6781516Z     result_traced = opt_f(*args, **kwargs)
2023-08-17T14:32:41.6782065Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T14:32:41.6782472Z     return self._call_impl(*args, **kwargs)
2023-08-17T14:32:41.6783006Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:32:41.6783449Z     return forward_call(*args, **kwargs)
2023-08-17T14:32:41.6783956Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 333, in _fn
2023-08-17T14:32:41.6784331Z     return fn(*args, **kwargs)
2023-08-17T14:32:41.6784877Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T14:32:41.6785283Z     return self._call_impl(*args, **kwargs)
2023-08-17T14:32:41.6785817Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:32:41.6786190Z     return forward_call(*args, **kwargs)
2023-08-17T14:32:41.6786736Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 493, in catch_errors
2023-08-17T14:32:41.6787160Z     return callback(frame, cache_size, hooks, frame_state)
2023-08-17T14:32:41.6787708Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 132, in _fn
2023-08-17T14:32:41.6788055Z     return fn(*args, **kwargs)
2023-08-17T14:32:41.6788603Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 370, in _convert_frame_assert
2023-08-17T14:32:41.6788998Z     return _compile(
2023-08-17T14:32:41.6789509Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 554, in _compile
2023-08-17T14:32:41.6789955Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-08-17T14:32:41.6790511Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 180, in time_wrapper
2023-08-17T14:32:41.6790876Z     r = func(*args, **kwargs)
2023-08-17T14:32:41.6791411Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 465, in compile_inner
2023-08-17T14:32:41.6791834Z     out_code = transform_code_object(code, transform)
2023-08-17T14:32:41.6792438Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-08-17T14:32:41.6792896Z     transformations(instructions, code_options)
2023-08-17T14:32:41.6793655Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 432, in transform
2023-08-17T14:32:41.6794015Z     tracer.run()
2023-08-17T14:32:41.6794524Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2071, in run
2023-08-17T14:32:41.6794865Z     super().run()
2023-08-17T14:32:41.6795370Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:32:41.6795849Z     and self.step()
2023-08-17T14:32:41.6796366Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:32:41.6796738Z     getattr(self, inst.opname)(inst)
2023-08-17T14:32:41.6797282Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:32:41.6797680Z     return inner_fn(self, inst)
2023-08-17T14:32:41.6798247Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1167, in CALL_FUNCTION_KW
2023-08-17T14:32:41.6798640Z     self.call_function(fn, args, kwargs)
2023-08-17T14:32:41.6799198Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:32:41.6799618Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:32:41.6800203Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 331, in call_function
2023-08-17T14:32:41.6800616Z     return tx.inline_user_function_return(
2023-08-17T14:32:41.6801195Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T14:32:41.6801712Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T14:32:41.6802354Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T14:32:41.6802990Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T14:32:41.6803610Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T14:32:41.6803965Z     tracer.run()
2023-08-17T14:32:41.6804470Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:32:41.6804831Z     and self.step()
2023-08-17T14:32:41.6805340Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:32:41.6805715Z     getattr(self, inst.opname)(inst)
2023-08-17T14:32:41.6806264Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:32:41.6806647Z     return inner_fn(self, inst)
2023-08-17T14:32:41.6807201Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1155, in CALL_FUNCTION_EX
2023-08-17T14:32:41.6807647Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-08-17T14:32:41.6808248Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:32:41.6808672Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:32:41.6809258Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-08-17T14:32:41.6809678Z     return super().call_function(tx, args, kwargs)
2023-08-17T14:32:41.6810239Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-08-17T14:32:41.6810656Z     return super().call_function(tx, args, kwargs)
2023-08-17T14:32:41.6811229Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-08-17T14:32:41.6811997Z     return tx.inline_user_function_return(
2023-08-17T14:32:41.6812596Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T14:32:41.6813114Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T14:32:41.6813808Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T14:32:41.6814373Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T14:32:41.6814963Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T14:32:41.6815316Z     tracer.run()
2023-08-17T14:32:41.6815829Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:32:41.6816190Z     and self.step()
2023-08-17T14:32:41.6816710Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:32:41.6817099Z     getattr(self, inst.opname)(inst)
2023-08-17T14:32:41.6817633Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:32:41.6818011Z     return inner_fn(self, inst)
2023-08-17T14:32:41.6818598Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1115, in CALL_FUNCTION
2023-08-17T14:32:41.6819003Z     self.call_function(fn, args, {})
2023-08-17T14:32:41.6819562Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:32:41.6819989Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:32:41.6820583Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 328, in call_function
2023-08-17T14:32:41.6821069Z     return VariableBuilder(tx, source).wrap_unspecialized_primitive(
2023-08-17T14:32:41.6821726Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1119, in wrap_unspecialized_primitive
2023-08-17T14:32:41.6822121Z     raise AssertionError(
2023-08-17T14:32:41.6822564Z AssertionError: Dynamo attempts to add additional input during export: value=0.6562776565551758, source=RandomValueSource(random_call_index=0)
2023-08-17T14:32:41.6822876Z 
2023-08-17T14:32:41.6822968Z from user code:
2023-08-17T14:32:41.6823556Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1692, in forward
2023-08-17T14:32:41.6823969Z     outputs = self.model.decoder(
2023-08-17T14:32:41.6824488Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:32:41.6824880Z     return forward_call(*args, **kwargs)
2023-08-17T14:32:41.6825454Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1050, in forward
2023-08-17T14:32:41.6825897Z     dropout_probability = random.uniform(0, 1)
2023-08-17T14:32:41.6826089Z 
2023-08-17T14:32:41.6826258Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-08-17T14:32:41.6826482Z 
2023-08-17T14:32:41.6826670Z TorchDynamo optimized model failed to run because of following error
2023-08-17T14:32:41.6826979Z fail_to_run
2023-08-17T14:32:45.6809095Z 
2023-08-17T14:32:50.4336289Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:32:50.4337110Z loading model: 0it [00:04, ?it/s]
2023-08-17T14:32:50.4403776Z cuda eval  PLBartForConditionalGeneration     
2023-08-17T14:32:52.4253541Z ERROR:common:Dynamo attempts to add additional input during export: value=0.832506537437439, source=RandomValueSource(random_call_index=0)
2023-08-17T14:32:52.4254108Z 
2023-08-17T14:32:52.4254204Z from user code:
2023-08-17T14:32:52.4256675Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1323, in forward
2023-08-17T14:32:52.4257479Z     outputs = self.model(
2023-08-17T14:32:52.4259684Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:32:52.4260399Z     return forward_call(*args, **kwargs)
2023-08-17T14:32:52.4261366Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward
2023-08-17T14:32:52.4262433Z     encoder_outputs = self.encoder(
2023-08-17T14:32:52.4263432Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:32:52.4263819Z     return forward_call(*args, **kwargs)
2023-08-17T14:32:52.4264401Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 799, in forward
2023-08-17T14:32:52.4264834Z     dropout_probability = random.uniform(0, 1)
2023-08-17T14:32:52.4265040Z 
2023-08-17T14:32:52.4265228Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-08-17T14:32:52.4265552Z Traceback (most recent call last):
2023-08-17T14:32:52.4265927Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2125, in check_accuracy
2023-08-17T14:32:52.4266344Z     new_result = optimized_model_iter_fn(model_copy, example_inputs)
2023-08-17T14:32:52.4266773Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1186, in opt_aot_inductor
2023-08-17T14:32:52.4267229Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-08-17T14:32:52.4267655Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1131, in load
2023-08-17T14:32:52.4268032Z     so_path, exported = torch._export.aot_compile(
2023-08-17T14:32:52.4268591Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 460, in aot_compile
2023-08-17T14:32:52.4268994Z     ep = export(f, args, kwargs, constraints)
2023-08-17T14:32:52.4269510Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 222, in export
2023-08-17T14:32:52.4269975Z     gm_torch_level, _ = torch._dynamo.export(
2023-08-17T14:32:52.4270519Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1201, in inner
2023-08-17T14:32:52.4270903Z     result_traced = opt_f(*args, **kwargs)
2023-08-17T14:32:52.4271468Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T14:32:52.4271860Z     return self._call_impl(*args, **kwargs)
2023-08-17T14:32:52.4272489Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:32:52.4272863Z     return forward_call(*args, **kwargs)
2023-08-17T14:32:52.4273381Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 333, in _fn
2023-08-17T14:32:52.4273747Z     return fn(*args, **kwargs)
2023-08-17T14:32:52.4274279Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T14:32:52.4274678Z     return self._call_impl(*args, **kwargs)
2023-08-17T14:32:52.4275194Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:32:52.4275576Z     return forward_call(*args, **kwargs)
2023-08-17T14:32:52.4276122Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 493, in catch_errors
2023-08-17T14:32:52.4276543Z     return callback(frame, cache_size, hooks, frame_state)
2023-08-17T14:32:52.4277079Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 132, in _fn
2023-08-17T14:32:52.4277443Z     return fn(*args, **kwargs)
2023-08-17T14:32:52.4277991Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 370, in _convert_frame_assert
2023-08-17T14:32:52.4278576Z     return _compile(
2023-08-17T14:32:52.4279076Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 554, in _compile
2023-08-17T14:32:52.4279514Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-08-17T14:32:52.4280124Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 180, in time_wrapper
2023-08-17T14:32:52.4280598Z     r = func(*args, **kwargs)
2023-08-17T14:32:52.4281140Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 465, in compile_inner
2023-08-17T14:32:52.4281549Z     out_code = transform_code_object(code, transform)
2023-08-17T14:32:52.4282169Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-08-17T14:32:52.4282842Z     transformations(instructions, code_options)
2023-08-17T14:32:52.4283418Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 432, in transform
2023-08-17T14:32:52.4283760Z     tracer.run()
2023-08-17T14:32:52.4284267Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2071, in run
2023-08-17T14:32:52.4284624Z     super().run()
2023-08-17T14:32:52.4285131Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:32:52.4285488Z     and self.step()
2023-08-17T14:32:52.4285980Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:32:52.4286365Z     getattr(self, inst.opname)(inst)
2023-08-17T14:32:52.4286907Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:32:52.4287294Z     return inner_fn(self, inst)
2023-08-17T14:32:52.4287838Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1167, in CALL_FUNCTION_KW
2023-08-17T14:32:52.4288251Z     self.call_function(fn, args, kwargs)
2023-08-17T14:32:52.4288807Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:32:52.4289226Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:32:52.4289850Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 331, in call_function
2023-08-17T14:32:52.4290243Z     return tx.inline_user_function_return(
2023-08-17T14:32:52.4290836Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T14:32:52.4291354Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T14:32:52.4291998Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T14:32:52.4292406Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T14:32:52.4292985Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T14:32:52.4293350Z     tracer.run()
2023-08-17T14:32:52.4293855Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:32:52.4294218Z     and self.step()
2023-08-17T14:32:52.4294710Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:32:52.4295096Z     getattr(self, inst.opname)(inst)
2023-08-17T14:32:52.4295631Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:32:52.4296010Z     return inner_fn(self, inst)
2023-08-17T14:32:52.4296712Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1155, in CALL_FUNCTION_EX
2023-08-17T14:32:52.4297170Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-08-17T14:32:52.4297770Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:32:52.4298188Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:32:52.4298913Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-08-17T14:32:52.4299324Z     return super().call_function(tx, args, kwargs)
2023-08-17T14:32:52.4299944Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-08-17T14:32:52.4300364Z     return super().call_function(tx, args, kwargs)
2023-08-17T14:32:52.4300940Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-08-17T14:32:52.4301346Z     return tx.inline_user_function_return(
2023-08-17T14:32:52.4301944Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T14:32:52.4302458Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T14:32:52.4303102Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T14:32:52.4303528Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T14:32:52.4304085Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T14:32:52.4304450Z     tracer.run()
2023-08-17T14:32:52.4304953Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:32:52.4305319Z     and self.step()
2023-08-17T14:32:52.4305809Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:32:52.4306195Z     getattr(self, inst.opname)(inst)
2023-08-17T14:32:52.4306738Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:32:52.4307118Z     return inner_fn(self, inst)
2023-08-17T14:32:52.4307676Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1167, in CALL_FUNCTION_KW
2023-08-17T14:32:52.4308196Z     self.call_function(fn, args, kwargs)
2023-08-17T14:32:52.4308759Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:32:52.4309182Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:32:52.4309763Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 331, in call_function
2023-08-17T14:32:52.4310207Z     return tx.inline_user_function_return(
2023-08-17T14:32:52.4310802Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T14:32:52.4311321Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T14:32:52.4311959Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T14:32:52.4312389Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T14:32:52.4312951Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T14:32:52.4313319Z     tracer.run()
2023-08-17T14:32:52.4313820Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:32:52.4314175Z     and self.step()
2023-08-17T14:32:52.4314791Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:32:52.4315180Z     getattr(self, inst.opname)(inst)
2023-08-17T14:32:52.4315718Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:32:52.4316099Z     return inner_fn(self, inst)
2023-08-17T14:32:52.4316652Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1155, in CALL_FUNCTION_EX
2023-08-17T14:32:52.4317187Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-08-17T14:32:52.4317809Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:32:52.4318232Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:32:52.4318822Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-08-17T14:32:52.4319234Z     return super().call_function(tx, args, kwargs)
2023-08-17T14:32:52.4319845Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-08-17T14:32:52.4320268Z     return super().call_function(tx, args, kwargs)
2023-08-17T14:32:52.4320841Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-08-17T14:32:52.4321254Z     return tx.inline_user_function_return(
2023-08-17T14:32:52.4321840Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T14:32:52.4322352Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T14:32:52.4323139Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T14:32:52.4323562Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T14:32:52.4324143Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T14:32:52.4324493Z     tracer.run()
2023-08-17T14:32:52.4324996Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:32:52.4325354Z     and self.step()
2023-08-17T14:32:52.4325865Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:32:52.4326239Z     getattr(self, inst.opname)(inst)
2023-08-17T14:32:52.4326781Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:32:52.4327156Z     return inner_fn(self, inst)
2023-08-17T14:32:52.4327699Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1115, in CALL_FUNCTION
2023-08-17T14:32:52.4328085Z     self.call_function(fn, args, {})
2023-08-17T14:32:52.4328631Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:32:52.4329058Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:32:52.4329641Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 328, in call_function
2023-08-17T14:32:52.4330178Z     return VariableBuilder(tx, source).wrap_unspecialized_primitive(
2023-08-17T14:32:52.4330829Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1119, in wrap_unspecialized_primitive
2023-08-17T14:32:52.4331241Z     raise AssertionError(
2023-08-17T14:32:52.4331677Z AssertionError: Dynamo attempts to add additional input during export: value=0.832506537437439, source=RandomValueSource(random_call_index=0)
2023-08-17T14:32:52.4331994Z 
2023-08-17T14:32:52.4332085Z from user code:
2023-08-17T14:32:52.4332775Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1323, in forward
2023-08-17T14:32:52.4333155Z     outputs = self.model(
2023-08-17T14:32:52.4333678Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:32:52.4334063Z     return forward_call(*args, **kwargs)
2023-08-17T14:32:52.4334746Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward
2023-08-17T14:32:52.4335145Z     encoder_outputs = self.encoder(
2023-08-17T14:32:52.4335679Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:32:52.4336064Z     return forward_call(*args, **kwargs)
2023-08-17T14:32:52.4336634Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 799, in forward
2023-08-17T14:32:52.4337072Z     dropout_probability = random.uniform(0, 1)
2023-08-17T14:32:52.4337253Z 
2023-08-17T14:32:52.4337435Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-08-17T14:32:52.4337657Z 
2023-08-17T14:32:52.4337848Z TorchDynamo optimized model failed to run because of following error
2023-08-17T14:32:52.4338166Z fail_to_run
2023-08-17T14:32:56.4030916Z 
2023-08-17T14:33:04.3268509Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:33:04.3269051Z loading model: 0it [00:07, ?it/s]
2023-08-17T14:33:04.3352123Z cuda eval  PegasusForCausalLM                 
2023-08-17T14:33:06.2846468Z ERROR:common:Dynamo attempts to add additional input during export: value=0.832506537437439, source=RandomValueSource(random_call_index=0)
2023-08-17T14:33:06.2847005Z 
2023-08-17T14:33:06.2847147Z from user code:
2023-08-17T14:33:06.2848143Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1673, in forward
2023-08-17T14:33:06.2848930Z     outputs = self.model.decoder(
2023-08-17T14:33:06.2849794Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:33:06.2851834Z     return forward_call(*args, **kwargs)
2023-08-17T14:33:06.2853142Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1072, in forward
2023-08-17T14:33:06.2853933Z     dropout_probability = random.uniform(0, 1)
2023-08-17T14:33:06.2854303Z 
2023-08-17T14:33:06.2854597Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-08-17T14:33:06.2855224Z Traceback (most recent call last):
2023-08-17T14:33:06.2855829Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2125, in check_accuracy
2023-08-17T14:33:06.2856534Z     new_result = optimized_model_iter_fn(model_copy, example_inputs)
2023-08-17T14:33:06.2857247Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1186, in opt_aot_inductor
2023-08-17T14:33:06.2858058Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-08-17T14:33:06.2859148Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1131, in load
2023-08-17T14:33:06.2859819Z     so_path, exported = torch._export.aot_compile(
2023-08-17T14:33:06.2860779Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 460, in aot_compile
2023-08-17T14:33:06.2861500Z     ep = export(f, args, kwargs, constraints)
2023-08-17T14:33:06.2862433Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 222, in export
2023-08-17T14:33:06.2863135Z     gm_torch_level, _ = torch._dynamo.export(
2023-08-17T14:33:06.2864073Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1201, in inner
2023-08-17T14:33:06.2864684Z     result_traced = opt_f(*args, **kwargs)
2023-08-17T14:33:06.2865647Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T14:33:06.2866877Z     return self._call_impl(*args, **kwargs)
2023-08-17T14:33:06.2867811Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:33:06.2868506Z     return forward_call(*args, **kwargs)
2023-08-17T14:33:06.2869435Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 333, in _fn
2023-08-17T14:33:06.2870249Z     return fn(*args, **kwargs)
2023-08-17T14:33:06.2871224Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T14:33:06.2871891Z     return self._call_impl(*args, **kwargs)
2023-08-17T14:33:06.2872821Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:33:06.2873526Z     return forward_call(*args, **kwargs)
2023-08-17T14:33:06.2874462Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 493, in catch_errors
2023-08-17T14:33:06.2875185Z     return callback(frame, cache_size, hooks, frame_state)
2023-08-17T14:33:06.2876153Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 132, in _fn
2023-08-17T14:33:06.2876771Z     return fn(*args, **kwargs)
2023-08-17T14:33:06.2877720Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 370, in _convert_frame_assert
2023-08-17T14:33:06.2878423Z     return _compile(
2023-08-17T14:33:06.2879303Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 554, in _compile
2023-08-17T14:33:06.2880049Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-08-17T14:33:06.2881056Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 180, in time_wrapper
2023-08-17T14:33:06.2881625Z     r = func(*args, **kwargs)
2023-08-17T14:33:06.2882538Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 465, in compile_inner
2023-08-17T14:33:06.2883546Z     out_code = transform_code_object(code, transform)
2023-08-17T14:33:06.2884665Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-08-17T14:33:06.2885421Z     transformations(instructions, code_options)
2023-08-17T14:33:06.2886431Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 432, in transform
2023-08-17T14:33:06.2886983Z     tracer.run()
2023-08-17T14:33:06.2887812Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2071, in run
2023-08-17T14:33:06.2888222Z     super().run()
2023-08-17T14:33:06.2888805Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:33:06.2889469Z     and self.step()
2023-08-17T14:33:06.2889995Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:33:06.2890381Z     getattr(self, inst.opname)(inst)
2023-08-17T14:33:06.2890912Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:33:06.2891431Z     return inner_fn(self, inst)
2023-08-17T14:33:06.2892004Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1167, in CALL_FUNCTION_KW
2023-08-17T14:33:06.2892412Z     self.call_function(fn, args, kwargs)
2023-08-17T14:33:06.2892997Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:33:06.2893422Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:33:06.2894002Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 331, in call_function
2023-08-17T14:33:06.2894647Z     return tx.inline_user_function_return(
2023-08-17T14:33:06.2895422Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T14:33:06.2895929Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T14:33:06.2896732Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T14:33:06.2897172Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T14:33:06.2897761Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T14:33:06.2898133Z     tracer.run()
2023-08-17T14:33:06.2898625Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:33:06.2898991Z     and self.step()
2023-08-17T14:33:06.2899499Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:33:06.2899886Z     getattr(self, inst.opname)(inst)
2023-08-17T14:33:06.2900427Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:33:06.2900809Z     return inner_fn(self, inst)
2023-08-17T14:33:06.2901368Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1155, in CALL_FUNCTION_EX
2023-08-17T14:33:06.2901829Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-08-17T14:33:06.2902588Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:33:06.2903048Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:33:06.2903640Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-08-17T14:33:06.2904072Z     return super().call_function(tx, args, kwargs)
2023-08-17T14:33:06.2904661Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-08-17T14:33:06.2905071Z     return super().call_function(tx, args, kwargs)
2023-08-17T14:33:06.2905640Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-08-17T14:33:06.2906058Z     return tx.inline_user_function_return(
2023-08-17T14:33:06.2906650Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T14:33:06.2907168Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T14:33:06.2907792Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T14:33:06.2908343Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T14:33:06.2908930Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T14:33:06.2909298Z     tracer.run()
2023-08-17T14:33:06.2909785Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:33:06.2910146Z     and self.step()
2023-08-17T14:33:06.2910659Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:33:06.2911043Z     getattr(self, inst.opname)(inst)
2023-08-17T14:33:06.2911581Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:33:06.2911946Z     return inner_fn(self, inst)
2023-08-17T14:33:06.2912494Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1115, in CALL_FUNCTION
2023-08-17T14:33:06.2913152Z     self.call_function(fn, args, {})
2023-08-17T14:33:06.2913711Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:33:06.2914123Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:33:06.2914718Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 328, in call_function
2023-08-17T14:33:06.2915303Z     return VariableBuilder(tx, source).wrap_unspecialized_primitive(
2023-08-17T14:33:06.2916158Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1119, in wrap_unspecialized_primitive
2023-08-17T14:33:06.2916584Z     raise AssertionError(
2023-08-17T14:33:06.2917065Z AssertionError: Dynamo attempts to add additional input during export: value=0.832506537437439, source=RandomValueSource(random_call_index=0)
2023-08-17T14:33:06.2917522Z 
2023-08-17T14:33:06.2917652Z from user code:
2023-08-17T14:33:06.2918250Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1673, in forward
2023-08-17T14:33:06.2918760Z     outputs = self.model.decoder(
2023-08-17T14:33:06.2919637Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:33:06.2920038Z     return forward_call(*args, **kwargs)
2023-08-17T14:33:06.2920652Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1072, in forward
2023-08-17T14:33:06.2921093Z     dropout_probability = random.uniform(0, 1)
2023-08-17T14:33:06.2921288Z 
2023-08-17T14:33:06.2921472Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-08-17T14:33:06.2921698Z 
2023-08-17T14:33:06.2921876Z TorchDynamo optimized model failed to run because of following error
2023-08-17T14:33:06.2922203Z fail_to_run
2023-08-17T14:33:10.2901305Z 
2023-08-17T14:33:23.1046315Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:33:23.1046822Z loading model: 0it [00:12, ?it/s]
2023-08-17T14:33:23.1177632Z cuda eval  PegasusForConditionalGeneration    
2023-08-17T14:33:25.4467154Z ERROR:common:Dynamo attempts to add additional input during export: value=0.9165307879447937, source=RandomValueSource(random_call_index=0)
2023-08-17T14:33:25.4467830Z 
2023-08-17T14:33:25.4468040Z from user code:
2023-08-17T14:33:25.4470801Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1411, in forward
2023-08-17T14:33:25.4471213Z     outputs = self.model(
2023-08-17T14:33:25.4471748Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:33:25.4472126Z     return forward_call(*args, **kwargs)
2023-08-17T14:33:25.4472978Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1248, in forward
2023-08-17T14:33:25.4473807Z     encoder_outputs = self.encoder(
2023-08-17T14:33:25.4474669Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:33:25.4475226Z     return forward_call(*args, **kwargs)
2023-08-17T14:33:25.4476316Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 794, in forward
2023-08-17T14:33:25.4477731Z     dropout_probability = random.uniform(0, 1)
2023-08-17T14:33:25.4478024Z 
2023-08-17T14:33:25.4478215Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-08-17T14:33:25.4478559Z Traceback (most recent call last):
2023-08-17T14:33:25.4478928Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2125, in check_accuracy
2023-08-17T14:33:25.4479349Z     new_result = optimized_model_iter_fn(model_copy, example_inputs)
2023-08-17T14:33:25.4479776Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1186, in opt_aot_inductor
2023-08-17T14:33:25.4480623Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-08-17T14:33:25.4481055Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1131, in load
2023-08-17T14:33:25.4481425Z     so_path, exported = torch._export.aot_compile(
2023-08-17T14:33:25.4482239Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 460, in aot_compile
2023-08-17T14:33:25.4483030Z     ep = export(f, args, kwargs, constraints)
2023-08-17T14:33:25.4483590Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 222, in export
2023-08-17T14:33:25.4483985Z     gm_torch_level, _ = torch._dynamo.export(
2023-08-17T14:33:25.4484514Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1201, in inner
2023-08-17T14:33:25.4484880Z     result_traced = opt_f(*args, **kwargs)
2023-08-17T14:33:25.4485448Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T14:33:25.4485854Z     return self._call_impl(*args, **kwargs)
2023-08-17T14:33:25.4486391Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:33:25.4486773Z     return forward_call(*args, **kwargs)
2023-08-17T14:33:25.4487280Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 333, in _fn
2023-08-17T14:33:25.4487641Z     return fn(*args, **kwargs)
2023-08-17T14:33:25.4488237Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T14:33:25.4488642Z     return self._call_impl(*args, **kwargs)
2023-08-17T14:33:25.4489160Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:33:25.4489549Z     return forward_call(*args, **kwargs)
2023-08-17T14:33:25.4490080Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 493, in catch_errors
2023-08-17T14:33:25.4490501Z     return callback(frame, cache_size, hooks, frame_state)
2023-08-17T14:33:25.4491052Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 132, in _fn
2023-08-17T14:33:25.4491403Z     return fn(*args, **kwargs)
2023-08-17T14:33:25.4491957Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 370, in _convert_frame_assert
2023-08-17T14:33:25.4492335Z     return _compile(
2023-08-17T14:33:25.4492841Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 554, in _compile
2023-08-17T14:33:25.4493274Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-08-17T14:33:25.4493838Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 180, in time_wrapper
2023-08-17T14:33:25.4494200Z     r = func(*args, **kwargs)
2023-08-17T14:33:25.4494726Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 465, in compile_inner
2023-08-17T14:33:25.4495132Z     out_code = transform_code_object(code, transform)
2023-08-17T14:33:25.4495746Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-08-17T14:33:25.4496203Z     transformations(instructions, code_options)
2023-08-17T14:33:25.4496762Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 432, in transform
2023-08-17T14:33:25.4497119Z     tracer.run()
2023-08-17T14:33:25.4497605Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2071, in run
2023-08-17T14:33:25.4497988Z     super().run()
2023-08-17T14:33:25.4498487Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:33:25.4498992Z     and self.step()
2023-08-17T14:33:25.4499489Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:33:25.4499876Z     getattr(self, inst.opname)(inst)
2023-08-17T14:33:25.4500424Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:33:25.4500802Z     return inner_fn(self, inst)
2023-08-17T14:33:25.4501450Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1167, in CALL_FUNCTION_KW
2023-08-17T14:33:25.4501842Z     self.call_function(fn, args, kwargs)
2023-08-17T14:33:25.4502399Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:33:25.4502821Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:33:25.4503401Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 331, in call_function
2023-08-17T14:33:25.4503793Z     return tx.inline_user_function_return(
2023-08-17T14:33:25.4504385Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T14:33:25.4504901Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T14:33:25.4505545Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T14:33:25.4505966Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T14:33:25.4506524Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T14:33:25.4506889Z     tracer.run()
2023-08-17T14:33:25.4507391Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:33:25.4507757Z     and self.step()
2023-08-17T14:33:25.4508418Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:33:25.4508803Z     getattr(self, inst.opname)(inst)
2023-08-17T14:33:25.4509344Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:33:25.4509720Z     return inner_fn(self, inst)
2023-08-17T14:33:25.4510278Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1155, in CALL_FUNCTION_EX
2023-08-17T14:33:25.4510719Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-08-17T14:33:25.4511310Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:33:25.4511728Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:33:25.4512311Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-08-17T14:33:25.4512714Z     return super().call_function(tx, args, kwargs)
2023-08-17T14:33:25.4513283Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-08-17T14:33:25.4513704Z     return super().call_function(tx, args, kwargs)
2023-08-17T14:33:25.4514284Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-08-17T14:33:25.4514689Z     return tx.inline_user_function_return(
2023-08-17T14:33:25.4515264Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T14:33:25.4515777Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T14:33:25.4516412Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T14:33:25.4516970Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T14:33:25.4517546Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T14:33:25.4517933Z     tracer.run()
2023-08-17T14:33:25.4518437Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:33:25.4518796Z     and self.step()
2023-08-17T14:33:25.4519406Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:33:25.4519785Z     getattr(self, inst.opname)(inst)
2023-08-17T14:33:25.4520329Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:33:25.4520706Z     return inner_fn(self, inst)
2023-08-17T14:33:25.4521253Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1167, in CALL_FUNCTION_KW
2023-08-17T14:33:25.4521650Z     self.call_function(fn, args, kwargs)
2023-08-17T14:33:25.4522202Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:33:25.4522778Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:33:25.4523366Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 331, in call_function
2023-08-17T14:33:25.4523770Z     return tx.inline_user_function_return(
2023-08-17T14:33:25.4524347Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T14:33:25.4524858Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T14:33:25.4525493Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T14:33:25.4525917Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T14:33:25.4526488Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T14:33:25.4526844Z     tracer.run()
2023-08-17T14:33:25.4527343Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:33:25.4527705Z     and self.step()
2023-08-17T14:33:25.4528249Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:33:25.4528618Z     getattr(self, inst.opname)(inst)
2023-08-17T14:33:25.4529158Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:33:25.4529532Z     return inner_fn(self, inst)
2023-08-17T14:33:25.4530083Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1155, in CALL_FUNCTION_EX
2023-08-17T14:33:25.4530531Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-08-17T14:33:25.4531123Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:33:25.4531543Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:33:25.4532120Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-08-17T14:33:25.4532539Z     return super().call_function(tx, args, kwargs)
2023-08-17T14:33:25.4533098Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-08-17T14:33:25.4533515Z     return super().call_function(tx, args, kwargs)
2023-08-17T14:33:25.4534084Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-08-17T14:33:25.4534632Z     return tx.inline_user_function_return(
2023-08-17T14:33:25.4535229Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T14:33:25.4535725Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T14:33:25.4536364Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T14:33:25.4536916Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T14:33:25.4537497Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T14:33:25.4537890Z     tracer.run()
2023-08-17T14:33:25.4538388Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:33:25.4538746Z     and self.step()
2023-08-17T14:33:25.4539250Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:33:25.4539638Z     getattr(self, inst.opname)(inst)
2023-08-17T14:33:25.4540166Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:33:25.4540548Z     return inner_fn(self, inst)
2023-08-17T14:33:25.4541098Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1115, in CALL_FUNCTION
2023-08-17T14:33:25.4541490Z     self.call_function(fn, args, {})
2023-08-17T14:33:25.4542020Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:33:25.4542444Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:33:25.4543023Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 328, in call_function
2023-08-17T14:33:25.4543493Z     return VariableBuilder(tx, source).wrap_unspecialized_primitive(
2023-08-17T14:33:25.4544151Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1119, in wrap_unspecialized_primitive
2023-08-17T14:33:25.4544548Z     raise AssertionError(
2023-08-17T14:33:25.4544989Z AssertionError: Dynamo attempts to add additional input during export: value=0.9165307879447937, source=RandomValueSource(random_call_index=0)
2023-08-17T14:33:25.4545301Z 
2023-08-17T14:33:25.4545393Z from user code:
2023-08-17T14:33:25.4545947Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1411, in forward
2023-08-17T14:33:25.4546326Z     outputs = self.model(
2023-08-17T14:33:25.4546840Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:33:25.4547225Z     return forward_call(*args, **kwargs)
2023-08-17T14:33:25.4547801Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1248, in forward
2023-08-17T14:33:25.4548261Z     encoder_outputs = self.encoder(
2023-08-17T14:33:25.4548779Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:33:25.4549160Z     return forward_call(*args, **kwargs)
2023-08-17T14:33:25.4549738Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 794, in forward
2023-08-17T14:33:25.4550178Z     dropout_probability = random.uniform(0, 1)
2023-08-17T14:33:25.4550368Z 
2023-08-17T14:33:25.4550537Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-08-17T14:33:25.4550756Z 
2023-08-17T14:33:25.4550941Z TorchDynamo optimized model failed to run because of following error
2023-08-17T14:33:25.4596341Z fail_to_run
2023-08-17T14:33:29.4039205Z 
2023-08-17T14:33:29.4084514Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`
2023-08-17T14:33:32.4702177Z 
2023-08-17T14:33:32.4702854Z loading model: 0it [00:03, ?it/s]
2023-08-17T14:33:32.4758572Z cuda eval  RobertaForCausalLM                 
2023-08-17T14:34:13.3297044Z pass
2023-08-17T14:34:17.6367732Z 
2023-08-17T14:34:20.2975360Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:34:20.2975693Z loading model: 0it [00:02, ?it/s]
2023-08-17T14:34:20.3028893Z cuda eval  RobertaForQuestionAnswering        
2023-08-17T14:35:00.9756159Z pass
2023-08-17T14:35:05.3256401Z 
2023-08-17T14:35:06.4741196Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:35:06.4741719Z loading model: 0it [00:01, ?it/s]
2023-08-17T14:35:06.4783430Z cuda eval  Speech2Text2ForCausalLM            
2023-08-17T14:35:08.3614680Z ERROR:common:Dynamo attempts to add additional input during export: value=0.6562776565551758, source=RandomValueSource(random_call_index=0)
2023-08-17T14:35:08.3615627Z 
2023-08-17T14:35:08.3615893Z from user code:
2023-08-17T14:35:08.3618615Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/speech_to_text_2/modeling_speech_to_text_2.py", line 918, in forward
2023-08-17T14:35:08.3619082Z     outputs = self.model.decoder(
2023-08-17T14:35:08.3619973Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:35:08.3620615Z     return forward_call(*args, **kwargs)
2023-08-17T14:35:08.3621690Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/speech_to_text_2/modeling_speech_to_text_2.py", line 660, in forward
2023-08-17T14:35:08.3622135Z     dropout_probability = random.uniform(0, 1)
2023-08-17T14:35:08.3622328Z 
2023-08-17T14:35:08.3624190Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-08-17T14:35:08.3624700Z Traceback (most recent call last):
2023-08-17T14:35:08.3625097Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2125, in check_accuracy
2023-08-17T14:35:08.3625565Z     new_result = optimized_model_iter_fn(model_copy, example_inputs)
2023-08-17T14:35:08.3625993Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1186, in opt_aot_inductor
2023-08-17T14:35:08.3626446Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-08-17T14:35:08.3626897Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1131, in load
2023-08-17T14:35:08.3627285Z     so_path, exported = torch._export.aot_compile(
2023-08-17T14:35:08.3628320Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 460, in aot_compile
2023-08-17T14:35:08.3628990Z     ep = export(f, args, kwargs, constraints)
2023-08-17T14:35:08.3630101Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 222, in export
2023-08-17T14:35:08.3630740Z     gm_torch_level, _ = torch._dynamo.export(
2023-08-17T14:35:08.3631473Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1201, in inner
2023-08-17T14:35:08.3631876Z     result_traced = opt_f(*args, **kwargs)
2023-08-17T14:35:08.3632432Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T14:35:08.3632824Z     return self._call_impl(*args, **kwargs)
2023-08-17T14:35:08.3633358Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:35:08.3633755Z     return forward_call(*args, **kwargs)
2023-08-17T14:35:08.3634275Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 333, in _fn
2023-08-17T14:35:08.3634633Z     return fn(*args, **kwargs)
2023-08-17T14:35:08.3635153Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T14:35:08.3635553Z     return self._call_impl(*args, **kwargs)
2023-08-17T14:35:08.3636460Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:35:08.3636844Z     return forward_call(*args, **kwargs)
2023-08-17T14:35:08.3637369Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 493, in catch_errors
2023-08-17T14:35:08.3637792Z     return callback(frame, cache_size, hooks, frame_state)
2023-08-17T14:35:08.3638339Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 132, in _fn
2023-08-17T14:35:08.3638854Z     return fn(*args, **kwargs)
2023-08-17T14:35:08.3639420Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 370, in _convert_frame_assert
2023-08-17T14:35:08.3639850Z     return _compile(
2023-08-17T14:35:08.3640366Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 554, in _compile
2023-08-17T14:35:08.3640798Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-08-17T14:35:08.3641374Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 180, in time_wrapper
2023-08-17T14:35:08.3641712Z     r = func(*args, **kwargs)
2023-08-17T14:35:08.3642242Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 465, in compile_inner
2023-08-17T14:35:08.3642854Z     out_code = transform_code_object(code, transform)
2023-08-17T14:35:08.3643493Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-08-17T14:35:08.3643949Z     transformations(instructions, code_options)
2023-08-17T14:35:08.3644491Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 432, in transform
2023-08-17T14:35:08.3644850Z     tracer.run()
2023-08-17T14:35:08.3645347Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2071, in run
2023-08-17T14:35:08.3645703Z     super().run()
2023-08-17T14:35:08.3646188Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:35:08.3646539Z     and self.step()
2023-08-17T14:35:08.3647044Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:35:08.3647429Z     getattr(self, inst.opname)(inst)
2023-08-17T14:35:08.3647958Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:35:08.3648334Z     return inner_fn(self, inst)
2023-08-17T14:35:08.3648883Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1167, in CALL_FUNCTION_KW
2023-08-17T14:35:08.3649284Z     self.call_function(fn, args, kwargs)
2023-08-17T14:35:08.3649874Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:35:08.3650291Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:35:08.3650865Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 331, in call_function
2023-08-17T14:35:08.3651275Z     return tx.inline_user_function_return(
2023-08-17T14:35:08.3651863Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T14:35:08.3652384Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T14:35:08.3653006Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T14:35:08.3653425Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T14:35:08.3654038Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T14:35:08.3654554Z     tracer.run()
2023-08-17T14:35:08.3655050Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:35:08.3655434Z     and self.step()
2023-08-17T14:35:08.3656206Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:35:08.3656601Z     getattr(self, inst.opname)(inst)
2023-08-17T14:35:08.3657313Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:35:08.3657700Z     return inner_fn(self, inst)
2023-08-17T14:35:08.3658255Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1155, in CALL_FUNCTION_EX
2023-08-17T14:35:08.3658708Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-08-17T14:35:08.3659303Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:35:08.3659772Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:35:08.3660356Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-08-17T14:35:08.3660771Z     return super().call_function(tx, args, kwargs)
2023-08-17T14:35:08.3661340Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-08-17T14:35:08.3661769Z     return super().call_function(tx, args, kwargs)
2023-08-17T14:35:08.3662327Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-08-17T14:35:08.3662736Z     return tx.inline_user_function_return(
2023-08-17T14:35:08.3663322Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T14:35:08.3663840Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T14:35:08.3664466Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T14:35:08.3664889Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T14:35:08.3665459Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T14:35:08.3665822Z     tracer.run()
2023-08-17T14:35:08.3666327Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:35:08.3666671Z     and self.step()
2023-08-17T14:35:08.3667206Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:35:08.3667590Z     getattr(self, inst.opname)(inst)
2023-08-17T14:35:08.3668173Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:35:08.3668557Z     return inner_fn(self, inst)
2023-08-17T14:35:08.3669088Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1115, in CALL_FUNCTION
2023-08-17T14:35:08.3669488Z     self.call_function(fn, args, {})
2023-08-17T14:35:08.3670072Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:35:08.3670493Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:35:08.3671068Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 328, in call_function
2023-08-17T14:35:08.3671536Z     return VariableBuilder(tx, source).wrap_unspecialized_primitive(
2023-08-17T14:35:08.3672195Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1119, in wrap_unspecialized_primitive
2023-08-17T14:35:08.3672603Z     raise AssertionError(
2023-08-17T14:35:08.3673169Z AssertionError: Dynamo attempts to add additional input during export: value=0.6562776565551758, source=RandomValueSource(random_call_index=0)
2023-08-17T14:35:08.3673462Z 
2023-08-17T14:35:08.3673552Z from user code:
2023-08-17T14:35:08.3674139Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/speech_to_text_2/modeling_speech_to_text_2.py", line 918, in forward
2023-08-17T14:35:08.3674571Z     outputs = self.model.decoder(
2023-08-17T14:35:08.3675218Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:35:08.3675595Z     return forward_call(*args, **kwargs)
2023-08-17T14:35:08.3676219Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/speech_to_text_2/modeling_speech_to_text_2.py", line 660, in forward
2023-08-17T14:35:08.3676668Z     dropout_probability = random.uniform(0, 1)
2023-08-17T14:35:08.3676857Z 
2023-08-17T14:35:08.3677040Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-08-17T14:35:08.3677297Z 
2023-08-17T14:35:08.3677480Z TorchDynamo optimized model failed to run because of following error
2023-08-17T14:35:08.3677773Z fail_to_run
2023-08-17T14:35:12.2841192Z 
2023-08-17T14:35:14.3838471Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:35:14.3838799Z loading model: 0it [00:02, ?it/s]
2023-08-17T14:35:14.3880181Z cuda eval  T5ForConditionalGeneration         
2023-08-17T14:36:00.6663703Z fail_accuracy
2023-08-17T14:36:05.1902782Z 
2023-08-17T14:36:07.3019750Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:36:07.3020302Z loading model: 0it [00:02, ?it/s]
2023-08-17T14:36:07.3060396Z cuda eval  T5Small                            
2023-08-17T14:36:31.7796032Z fail_accuracy
2023-08-17T14:36:36.0569125Z 
2023-08-17T14:36:41.7068074Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:36:41.7069130Z loading model: 0it [00:05, ?it/s]
2023-08-17T14:36:41.7150809Z cuda eval  TrOCRForCausalLM                   
2023-08-17T14:36:43.6735249Z ERROR:common:Dynamo attempts to add additional input during export: value=0.832506537437439, source=RandomValueSource(random_call_index=0)
2023-08-17T14:36:43.6735907Z 
2023-08-17T14:36:43.6736077Z from user code:
2023-08-17T14:36:43.6739103Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 958, in forward
2023-08-17T14:36:43.6739887Z     outputs = self.model.decoder(
2023-08-17T14:36:43.6740952Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:36:43.6741659Z     return forward_call(*args, **kwargs)
2023-08-17T14:36:43.6776562Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 692, in forward
2023-08-17T14:36:43.6777448Z     dropout_probability = random.uniform(0, 1)
2023-08-17T14:36:43.6777748Z 
2023-08-17T14:36:43.6778037Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-08-17T14:36:43.6778575Z Traceback (most recent call last):
2023-08-17T14:36:43.6779119Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2125, in check_accuracy
2023-08-17T14:36:43.6779757Z     new_result = optimized_model_iter_fn(model_copy, example_inputs)
2023-08-17T14:36:43.6780391Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1186, in opt_aot_inductor
2023-08-17T14:36:43.6781061Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-08-17T14:36:43.6781724Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1131, in load
2023-08-17T14:36:43.6782287Z     so_path, exported = torch._export.aot_compile(
2023-08-17T14:36:43.6783380Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 460, in aot_compile
2023-08-17T14:36:43.6783973Z     ep = export(f, args, kwargs, constraints)
2023-08-17T14:36:43.6784784Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 222, in export
2023-08-17T14:36:43.6785861Z     gm_torch_level, _ = torch._dynamo.export(
2023-08-17T14:36:43.6786793Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1201, in inner
2023-08-17T14:36:43.6787367Z     result_traced = opt_f(*args, **kwargs)
2023-08-17T14:36:43.6788219Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T14:36:43.6788824Z     return self._call_impl(*args, **kwargs)
2023-08-17T14:36:43.6789948Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:36:43.6790544Z     return forward_call(*args, **kwargs)
2023-08-17T14:36:43.6791365Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 333, in _fn
2023-08-17T14:36:43.6791911Z     return fn(*args, **kwargs)
2023-08-17T14:36:43.6792736Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T14:36:43.6793426Z     return self._call_impl(*args, **kwargs)
2023-08-17T14:36:43.6794262Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:36:43.6794833Z     return forward_call(*args, **kwargs)
2023-08-17T14:36:43.6795672Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 493, in catch_errors
2023-08-17T14:36:43.6796302Z     return callback(frame, cache_size, hooks, frame_state)
2023-08-17T14:36:43.6797154Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 132, in _fn
2023-08-17T14:36:43.6797694Z     return fn(*args, **kwargs)
2023-08-17T14:36:43.6798539Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 370, in _convert_frame_assert
2023-08-17T14:36:43.6799100Z     return _compile(
2023-08-17T14:36:43.6799871Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 554, in _compile
2023-08-17T14:36:43.6800519Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-08-17T14:36:43.6801378Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 180, in time_wrapper
2023-08-17T14:36:43.6801908Z     r = func(*args, **kwargs)
2023-08-17T14:36:43.6802964Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 465, in compile_inner
2023-08-17T14:36:43.6803612Z     out_code = transform_code_object(code, transform)
2023-08-17T14:36:43.6804561Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-08-17T14:36:43.6805231Z     transformations(instructions, code_options)
2023-08-17T14:36:43.6806064Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 432, in transform
2023-08-17T14:36:43.6806584Z     tracer.run()
2023-08-17T14:36:43.6807353Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2071, in run
2023-08-17T14:36:43.6807885Z     super().run()
2023-08-17T14:36:43.6808642Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:36:43.6809152Z     and self.step()
2023-08-17T14:36:43.6809920Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:36:43.6810519Z     getattr(self, inst.opname)(inst)
2023-08-17T14:36:43.6811321Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:36:43.6811876Z     return inner_fn(self, inst)
2023-08-17T14:36:43.6812733Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1167, in CALL_FUNCTION_KW
2023-08-17T14:36:43.6813789Z     self.call_function(fn, args, kwargs)
2023-08-17T14:36:43.6814718Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:36:43.6815360Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:36:43.6816267Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 331, in call_function
2023-08-17T14:36:43.6816895Z     return tx.inline_user_function_return(
2023-08-17T14:36:43.6818085Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T14:36:43.6818883Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T14:36:43.6819884Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T14:36:43.6820550Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T14:36:43.6821512Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T14:36:43.6822073Z     tracer.run()
2023-08-17T14:36:43.6822919Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:36:43.6823482Z     and self.step()
2023-08-17T14:36:43.6824304Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:36:43.6824907Z     getattr(self, inst.opname)(inst)
2023-08-17T14:36:43.6825726Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:36:43.6826272Z     return inner_fn(self, inst)
2023-08-17T14:36:43.6827117Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1155, in CALL_FUNCTION_EX
2023-08-17T14:36:43.6827808Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-08-17T14:36:43.6828716Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:36:43.6829337Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:36:43.6830201Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-08-17T14:36:43.6830839Z     return super().call_function(tx, args, kwargs)
2023-08-17T14:36:43.6831779Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-08-17T14:36:43.6832407Z     return super().call_function(tx, args, kwargs)
2023-08-17T14:36:43.6833402Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-08-17T14:36:43.6834001Z     return tx.inline_user_function_return(
2023-08-17T14:36:43.6834935Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T14:36:43.6835733Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T14:36:43.6836719Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T14:36:43.6837341Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T14:36:43.6838237Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T14:36:43.6838783Z     tracer.run()
2023-08-17T14:36:43.6839554Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:36:43.6840089Z     and self.step()
2023-08-17T14:36:43.6840840Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:36:43.6841694Z     getattr(self, inst.opname)(inst)
2023-08-17T14:36:43.6842535Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:36:43.6843374Z     return inner_fn(self, inst)
2023-08-17T14:36:43.6844224Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1115, in CALL_FUNCTION
2023-08-17T14:36:43.6844808Z     self.call_function(fn, args, {})
2023-08-17T14:36:43.6845901Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:36:43.6846573Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:36:43.6847517Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 328, in call_function
2023-08-17T14:36:43.6848220Z     return VariableBuilder(tx, source).wrap_unspecialized_primitive(
2023-08-17T14:36:43.6849270Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1119, in wrap_unspecialized_primitive
2023-08-17T14:36:43.6849895Z     raise AssertionError(
2023-08-17T14:36:43.6850529Z AssertionError: Dynamo attempts to add additional input during export: value=0.832506537437439, source=RandomValueSource(random_call_index=0)
2023-08-17T14:36:43.6850991Z 
2023-08-17T14:36:43.6851104Z from user code:
2023-08-17T14:36:43.6851960Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 958, in forward
2023-08-17T14:36:43.6852569Z     outputs = self.model.decoder(
2023-08-17T14:36:43.6853471Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:36:43.6854057Z     return forward_call(*args, **kwargs)
2023-08-17T14:36:43.6854959Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 692, in forward
2023-08-17T14:36:43.6855692Z     dropout_probability = random.uniform(0, 1)
2023-08-17T14:36:43.6855970Z 
2023-08-17T14:36:43.6856256Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-08-17T14:36:43.6856621Z 
2023-08-17T14:36:43.6856917Z TorchDynamo optimized model failed to run because of following error
2023-08-17T14:36:43.6857440Z fail_to_run
2023-08-17T14:36:47.6372194Z 
2023-08-17T14:37:01.1806490Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:37:01.1806973Z loading model: 0it [00:13, ?it/s]
2023-08-17T14:37:01.1905686Z cuda eval  XGLMForCausalLM                    
2023-08-17T14:37:03.4224352Z ERROR:common:Dynamo attempts to add additional input during export: value=0.9165307879447937, source=RandomValueSource(random_call_index=0)
2023-08-17T14:37:03.4224870Z 
2023-08-17T14:37:03.4225006Z from user code:
2023-08-17T14:37:03.4227741Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 889, in forward
2023-08-17T14:37:03.4228432Z     outputs = self.model(
2023-08-17T14:37:03.4229352Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:37:03.4230051Z     return forward_call(*args, **kwargs)
2023-08-17T14:37:03.4231090Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 743, in forward
2023-08-17T14:37:03.4231879Z     dropout_probability = random.uniform(0, 1)
2023-08-17T14:37:03.4232146Z 
2023-08-17T14:37:03.4232512Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-08-17T14:37:03.4233130Z Traceback (most recent call last):
2023-08-17T14:37:03.4233772Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2125, in check_accuracy
2023-08-17T14:37:03.4234521Z     new_result = optimized_model_iter_fn(model_copy, example_inputs)
2023-08-17T14:37:03.4235231Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1186, in opt_aot_inductor
2023-08-17T14:37:03.4236527Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-08-17T14:37:03.4237312Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1131, in load
2023-08-17T14:37:03.4237974Z     so_path, exported = torch._export.aot_compile(
2023-08-17T14:37:03.4238973Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 460, in aot_compile
2023-08-17T14:37:03.4239623Z     ep = export(f, args, kwargs, constraints)
2023-08-17T14:37:03.4240812Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 222, in export
2023-08-17T14:37:03.4241485Z     gm_torch_level, _ = torch._dynamo.export(
2023-08-17T14:37:03.4242418Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1201, in inner
2023-08-17T14:37:03.4243410Z     result_traced = opt_f(*args, **kwargs)
2023-08-17T14:37:03.4244372Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T14:37:03.4245118Z     return self._call_impl(*args, **kwargs)
2023-08-17T14:37:03.4246034Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:37:03.4246659Z     return forward_call(*args, **kwargs)
2023-08-17T14:37:03.4247558Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 333, in _fn
2023-08-17T14:37:03.4248163Z     return fn(*args, **kwargs)
2023-08-17T14:37:03.4249092Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T14:37:03.4249745Z     return self._call_impl(*args, **kwargs)
2023-08-17T14:37:03.4250681Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:37:03.4251390Z     return forward_call(*args, **kwargs)
2023-08-17T14:37:03.4252319Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 493, in catch_errors
2023-08-17T14:37:03.4253068Z     return callback(frame, cache_size, hooks, frame_state)
2023-08-17T14:37:03.4254019Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 132, in _fn
2023-08-17T14:37:03.4254607Z     return fn(*args, **kwargs)
2023-08-17T14:37:03.4255562Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 370, in _convert_frame_assert
2023-08-17T14:37:03.4256177Z     return _compile(
2023-08-17T14:37:03.4257032Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 554, in _compile
2023-08-17T14:37:03.4257775Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-08-17T14:37:03.4258809Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 180, in time_wrapper
2023-08-17T14:37:03.4259459Z     r = func(*args, **kwargs)
2023-08-17T14:37:03.4260360Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 465, in compile_inner
2023-08-17T14:37:03.4261130Z     out_code = transform_code_object(code, transform)
2023-08-17T14:37:03.4262159Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-08-17T14:37:03.4263020Z     transformations(instructions, code_options)
2023-08-17T14:37:03.4263991Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 432, in transform
2023-08-17T14:37:03.4264626Z     tracer.run()
2023-08-17T14:37:03.4266767Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2071, in run
2023-08-17T14:37:03.4267350Z     super().run()
2023-08-17T14:37:03.4268052Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:37:03.4268924Z     and self.step()
2023-08-17T14:37:03.4269612Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:37:03.4270004Z     getattr(self, inst.opname)(inst)
2023-08-17T14:37:03.4270552Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:37:03.4271033Z     return inner_fn(self, inst)
2023-08-17T14:37:03.4271774Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1167, in CALL_FUNCTION_KW
2023-08-17T14:37:03.4272189Z     self.call_function(fn, args, kwargs)
2023-08-17T14:37:03.4272756Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:37:03.4273268Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:37:03.4273838Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 331, in call_function
2023-08-17T14:37:03.4274257Z     return tx.inline_user_function_return(
2023-08-17T14:37:03.4274987Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T14:37:03.4275529Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T14:37:03.4276168Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T14:37:03.4276604Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T14:37:03.4277185Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T14:37:03.4277548Z     tracer.run()
2023-08-17T14:37:03.4278049Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:37:03.4278400Z     and self.step()
2023-08-17T14:37:03.4278923Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:37:03.4279312Z     getattr(self, inst.opname)(inst)
2023-08-17T14:37:03.4279863Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:37:03.4280229Z     return inner_fn(self, inst)
2023-08-17T14:37:03.4280789Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1155, in CALL_FUNCTION_EX
2023-08-17T14:37:03.4281247Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-08-17T14:37:03.4281860Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:37:03.4282412Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:37:03.4283455Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-08-17T14:37:03.4283893Z     return super().call_function(tx, args, kwargs)
2023-08-17T14:37:03.4284480Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-08-17T14:37:03.4284901Z     return super().call_function(tx, args, kwargs)
2023-08-17T14:37:03.4285466Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-08-17T14:37:03.4285883Z     return tx.inline_user_function_return(
2023-08-17T14:37:03.4286470Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T14:37:03.4286982Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T14:37:03.4287615Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T14:37:03.4288237Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T14:37:03.4288821Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T14:37:03.4289191Z     tracer.run()
2023-08-17T14:37:03.4289702Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T14:37:03.4290062Z     and self.step()
2023-08-17T14:37:03.4290678Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T14:37:03.4291069Z     getattr(self, inst.opname)(inst)
2023-08-17T14:37:03.4291617Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T14:37:03.4291999Z     return inner_fn(self, inst)
2023-08-17T14:37:03.4292531Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1115, in CALL_FUNCTION
2023-08-17T14:37:03.4292938Z     self.call_function(fn, args, {})
2023-08-17T14:37:03.4293564Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T14:37:03.4293992Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T14:37:03.4294566Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 328, in call_function
2023-08-17T14:37:03.4295048Z     return VariableBuilder(tx, source).wrap_unspecialized_primitive(
2023-08-17T14:37:03.4295903Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1119, in wrap_unspecialized_primitive
2023-08-17T14:37:03.4296323Z     raise AssertionError(
2023-08-17T14:37:03.4296770Z AssertionError: Dynamo attempts to add additional input during export: value=0.9165307879447937, source=RandomValueSource(random_call_index=0)
2023-08-17T14:37:03.4297091Z 
2023-08-17T14:37:03.4297167Z from user code:
2023-08-17T14:37:03.4297714Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 889, in forward
2023-08-17T14:37:03.4298096Z     outputs = self.model(
2023-08-17T14:37:03.4298615Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T14:37:03.4299000Z     return forward_call(*args, **kwargs)
2023-08-17T14:37:03.4299551Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 743, in forward
2023-08-17T14:37:03.4299975Z     dropout_probability = random.uniform(0, 1)
2023-08-17T14:37:03.4300162Z 
2023-08-17T14:37:03.4300346Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-08-17T14:37:03.4300568Z 
2023-08-17T14:37:03.4300751Z TorchDynamo optimized model failed to run because of following error
2023-08-17T14:37:03.4330964Z fail_to_run
2023-08-17T14:37:07.4404777Z 
2023-08-17T14:37:13.7752270Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:37:13.7752630Z loading model: 0it [00:06, ?it/s]
2023-08-17T14:37:13.7857663Z cuda eval  XLNetLMHeadModel                   
2023-08-17T14:37:42.9455123Z [2023-08-17 14:37:42,943] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-08-17T14:38:02.0536356Z [2023-08-17 14:38:02,051] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-08-17T14:38:54.5907776Z pass
2023-08-17T14:38:59.3119565Z 
2023-08-17T14:39:02.3211380Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:39:02.3212188Z loading model: 0it [00:03, ?it/s]
2023-08-17T14:39:02.3292173Z cuda eval  YituTechConvBert                   
2023-08-17T14:40:01.7940219Z pass
2023-08-17T14:40:03.0845968Z accuracy            pass_rate=41.18%
2023-08-17T14:40:03.0846539Z calls_captured      gmean=0.00x mean=441.000x
2023-08-17T14:40:03.0847648Z unique_graphs       gmean=0.00x mean=0.529x
2023-08-17T14:40:03.0854936Z graph_breaks        gmean=0.00x mean=0.000x
2023-08-17T14:40:03.0856817Z unique_graph_breaks gmean=0.00x mean=0.000x
2023-08-17T14:40:03.5962505Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *maxautotune-true* ]]
2023-08-17T14:40:03.5963346Z + for target in "${targets[@]}"
2023-08-17T14:40:03.5963663Z + target_flag=("--${target}")
2023-08-17T14:40:03.5966411Z + local target_flag
2023-08-17T14:40:03.5966901Z + [[ performance == \p\e\r\f\o\r\m\a\n\c\e ]]
2023-08-17T14:40:03.5967639Z + target_flag+=(--cold-start-latency)
2023-08-17T14:40:03.5969547Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *default-true* ]]
2023-08-17T14:40:03.5971199Z + python benchmarks/dynamo/huggingface.py --performance --cold-start-latency --inference --bfloat16 --backend inductor --disable-cudagraphs --device cuda --total-partitions 3 --partition-id 2 --output /var/lib/jenkins/workspace/test/test-reports/inductor_no_cudagraphs_huggingface_bfloat16_inference_cuda_performance.csv
2023-08-17T14:40:10.2210141Z 
2023-08-17T14:40:16.8091478Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:40:16.8091992Z loading model: 0it [00:06, ?it/s]
2023-08-17T14:40:16.8176681Z cuda eval  MegatronBertForQuestionAnswering   
2023-08-17T14:40:54.1365594Z 
2023-08-17T14:40:54.2565625Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T14:40:54.3724381Z running benchmark:   3% 1/30 [00:00<00:03,  8.30it/s]
2023-08-17T14:40:54.4882914Z running benchmark:  10% 3/30 [00:00<00:02, 13.48it/s]
2023-08-17T14:40:54.6046073Z running benchmark:  17% 5/30 [00:00<00:01, 15.18it/s]
2023-08-17T14:40:54.7206093Z running benchmark:  23% 7/30 [00:00<00:01, 15.98it/s]
2023-08-17T14:40:54.8364793Z running benchmark:  30% 9/30 [00:00<00:01, 16.43it/s]
2023-08-17T14:40:54.9522290Z running benchmark:  37% 11/30 [00:00<00:01, 16.71it/s]
2023-08-17T14:40:55.0679822Z running benchmark:  43% 13/30 [00:00<00:01, 16.89it/s]
2023-08-17T14:40:55.1892376Z running benchmark:  50% 15/30 [00:00<00:00, 17.01it/s]
2023-08-17T14:40:55.3048976Z running benchmark:  57% 17/30 [00:01<00:00, 16.86it/s]
2023-08-17T14:40:55.4206939Z running benchmark:  63% 19/30 [00:01<00:00, 16.98it/s]
2023-08-17T14:40:55.5365166Z running benchmark:  70% 21/30 [00:01<00:00, 17.07it/s]
2023-08-17T14:40:55.6523121Z running benchmark:  77% 23/30 [00:01<00:00, 17.13it/s]
2023-08-17T14:40:55.7684065Z running benchmark:  83% 25/30 [00:01<00:00, 17.17it/s]
2023-08-17T14:40:55.8842750Z running benchmark:  90% 27/30 [00:01<00:00, 17.19it/s]
2023-08-17T14:40:55.9423140Z running benchmark:  97% 29/30 [00:01<00:00, 17.21it/s]
2023-08-17T14:40:55.9423617Z running benchmark: 100% 30/30 [00:01<00:00, 16.61it/s]
2023-08-17T14:40:55.9445717Z 1.398x
2023-08-17T14:41:00.5296810Z 
2023-08-17T14:41:02.4025503Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:41:02.4025804Z loading model: 0it [00:01, ?it/s]
2023-08-17T14:41:02.4277122Z cuda eval  MobileBertForMaskedLM              
2023-08-17T14:42:29.3963016Z 
2023-08-17T14:42:29.5433378Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T14:42:29.6752091Z running benchmark:   7% 2/30 [00:00<00:02, 13.55it/s]
2023-08-17T14:42:29.8050450Z running benchmark:  13% 4/30 [00:00<00:01, 14.43it/s]
2023-08-17T14:42:29.9330844Z running benchmark:  20% 6/30 [00:00<00:01, 14.85it/s]
2023-08-17T14:42:30.0653089Z running benchmark:  27% 8/30 [00:00<00:01, 15.15it/s]
2023-08-17T14:42:30.1937646Z running benchmark:  33% 10/30 [00:00<00:01, 15.14it/s]
2023-08-17T14:42:30.3324140Z running benchmark:  40% 12/30 [00:00<00:01, 15.28it/s]
2023-08-17T14:42:30.4601851Z running benchmark:  47% 14/30 [00:00<00:01, 15.00it/s]
2023-08-17T14:42:30.5886161Z running benchmark:  53% 16/30 [00:01<00:00, 15.19it/s]
2023-08-17T14:42:30.7177113Z running benchmark:  60% 18/30 [00:01<00:00, 15.31it/s]
2023-08-17T14:42:30.8472305Z running benchmark:  67% 20/30 [00:01<00:00, 15.37it/s]
2023-08-17T14:42:30.9752017Z running benchmark:  73% 22/30 [00:01<00:00, 15.39it/s]
2023-08-17T14:42:31.1031062Z running benchmark:  80% 24/30 [00:01<00:00, 15.46it/s]
2023-08-17T14:42:31.2312592Z running benchmark:  87% 26/30 [00:01<00:00, 15.51it/s]
2023-08-17T14:42:31.3726977Z running benchmark:  93% 28/30 [00:01<00:00, 15.54it/s]
2023-08-17T14:42:31.3727583Z running benchmark: 100% 30/30 [00:01<00:00, 15.10it/s]
2023-08-17T14:42:31.3727881Z running benchmark: 100% 30/30 [00:01<00:00, 15.17it/s]
2023-08-17T14:42:31.3762322Z 2.215x
2023-08-17T14:42:36.4110635Z 
2023-08-17T14:42:37.9972523Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:42:37.9973035Z loading model: 0it [00:01, ?it/s]
2023-08-17T14:42:38.0215691Z cuda eval  MobileBertForQuestionAnswering     
2023-08-17T14:44:01.8900525Z 
2023-08-17T14:44:01.9939937Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T14:44:02.1422803Z running benchmark:   7% 2/30 [00:00<00:01, 19.15it/s]
2023-08-17T14:44:02.2448894Z running benchmark:  17% 5/30 [00:00<00:01, 19.87it/s]
2023-08-17T14:44:02.3934688Z running benchmark:  23% 7/30 [00:00<00:01, 19.71it/s]
2023-08-17T14:44:02.5073983Z running benchmark:  33% 10/30 [00:00<00:01, 19.93it/s]
2023-08-17T14:44:02.6535683Z running benchmark:  40% 12/30 [00:00<00:00, 19.15it/s]
2023-08-17T14:44:02.7996208Z running benchmark:  50% 15/30 [00:00<00:00, 19.66it/s]
2023-08-17T14:44:02.9464941Z running benchmark:  60% 18/30 [00:00<00:00, 19.97it/s]
2023-08-17T14:44:03.0935213Z running benchmark:  70% 21/30 [00:01<00:00, 20.12it/s]
2023-08-17T14:44:03.2449347Z running benchmark:  80% 24/30 [00:01<00:00, 20.22it/s]
2023-08-17T14:44:03.3972743Z running benchmark:  90% 27/30 [00:01<00:00, 20.09it/s]
2023-08-17T14:44:03.3973367Z running benchmark: 100% 30/30 [00:01<00:00, 19.96it/s]
2023-08-17T14:44:03.3973807Z running benchmark: 100% 30/30 [00:01<00:00, 19.89it/s]
2023-08-17T14:44:03.4012387Z 2.175x
2023-08-17T14:44:08.4749213Z 
2023-08-17T14:44:12.1536382Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:44:12.1536897Z loading model: 0it [00:03, ?it/s]
2023-08-17T14:44:12.1574426Z cuda eval  OPTForCausalLM                     
2023-08-17T14:44:34.2778672Z 
2023-08-17T14:44:34.3848010Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T14:44:34.4892099Z running benchmark:   3% 1/30 [00:00<00:03,  9.32it/s]
2023-08-17T14:44:34.5993740Z running benchmark:  10% 3/30 [00:00<00:01, 15.04it/s]
2023-08-17T14:44:34.7066715Z running benchmark:  17% 5/30 [00:00<00:01, 16.49it/s]
2023-08-17T14:44:34.8101228Z running benchmark:  23% 7/30 [00:00<00:01, 17.34it/s]
2023-08-17T14:44:34.9152933Z running benchmark:  30% 9/30 [00:00<00:01, 18.04it/s]
2023-08-17T14:44:35.0219706Z running benchmark:  37% 11/30 [00:00<00:01, 18.37it/s]
2023-08-17T14:44:35.1261554Z running benchmark:  43% 13/30 [00:00<00:00, 18.50it/s]
2023-08-17T14:44:35.2296219Z running benchmark:  50% 15/30 [00:00<00:00, 18.71it/s]
2023-08-17T14:44:35.3503867Z running benchmark:  57% 17/30 [00:00<00:00, 18.91it/s]
2023-08-17T14:44:35.4528014Z running benchmark:  63% 19/30 [00:01<00:00, 18.11it/s]
2023-08-17T14:44:35.5612729Z running benchmark:  70% 21/30 [00:01<00:00, 18.52it/s]
2023-08-17T14:44:35.6673569Z running benchmark:  77% 23/30 [00:01<00:00, 18.49it/s]
2023-08-17T14:44:35.7770966Z running benchmark:  83% 25/30 [00:01<00:00, 18.60it/s]
2023-08-17T14:44:35.8814110Z running benchmark:  90% 27/30 [00:01<00:00, 18.49it/s]
2023-08-17T14:44:35.9351054Z running benchmark:  97% 29/30 [00:01<00:00, 18.68it/s]
2023-08-17T14:44:35.9351398Z running benchmark: 100% 30/30 [00:01<00:00, 18.10it/s]
2023-08-17T14:44:35.9373122Z 2.399x
2023-08-17T14:44:40.2330147Z 
2023-08-17T14:44:43.3555051Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:44:43.3555420Z loading model: 0it [00:03, ?it/s]
2023-08-17T14:44:43.3586741Z cuda eval  PLBartForCausalLM                  
2023-08-17T14:45:00.1105899Z 
2023-08-17T14:45:00.2445175Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T14:45:00.3767644Z running benchmark:   7% 2/30 [00:00<00:01, 14.90it/s]
2023-08-17T14:45:00.5108915Z running benchmark:  17% 5/30 [00:00<00:01, 19.41it/s]
2023-08-17T14:45:00.6469779Z running benchmark:  27% 8/30 [00:00<00:01, 20.77it/s]
2023-08-17T14:45:00.7787617Z running benchmark:  37% 11/30 [00:00<00:00, 21.28it/s]
2023-08-17T14:45:00.9146549Z running benchmark:  47% 14/30 [00:00<00:00, 21.81it/s]
2023-08-17T14:45:01.0488747Z running benchmark:  57% 17/30 [00:00<00:00, 21.90it/s]
2023-08-17T14:45:01.1831785Z running benchmark:  67% 20/30 [00:00<00:00, 22.05it/s]
2023-08-17T14:45:01.3157039Z running benchmark:  77% 23/30 [00:01<00:00, 22.14it/s]
2023-08-17T14:45:01.4490817Z running benchmark:  87% 26/30 [00:01<00:00, 22.30it/s]
2023-08-17T14:45:01.4936000Z running benchmark:  97% 29/30 [00:01<00:00, 22.36it/s]
2023-08-17T14:45:01.4936365Z running benchmark: 100% 30/30 [00:01<00:00, 21.68it/s]
2023-08-17T14:45:01.4957871Z 2.312x
2023-08-17T14:45:05.6724484Z 
2023-08-17T14:45:10.9173840Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:45:10.9174625Z loading model: 0it [00:05, ?it/s]
2023-08-17T14:45:10.9230241Z cuda eval  PLBartForConditionalGeneration     
2023-08-17T14:45:37.4529086Z 
2023-08-17T14:45:37.5784749Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T14:45:37.7024279Z running benchmark:   7% 2/30 [00:00<00:01, 15.90it/s]
2023-08-17T14:45:37.8421898Z running benchmark:  17% 5/30 [00:00<00:01, 20.73it/s]
2023-08-17T14:45:37.9696067Z running benchmark:  27% 8/30 [00:00<00:01, 21.09it/s]
2023-08-17T14:45:38.0983297Z running benchmark:  37% 11/30 [00:00<00:00, 22.05it/s]
2023-08-17T14:45:38.2222423Z running benchmark:  47% 14/30 [00:00<00:00, 22.49it/s]
2023-08-17T14:45:38.3472745Z running benchmark:  57% 17/30 [00:00<00:00, 23.06it/s]
2023-08-17T14:45:38.4725419Z running benchmark:  67% 20/30 [00:00<00:00, 23.36it/s]
2023-08-17T14:45:38.5959387Z running benchmark:  77% 23/30 [00:01<00:00, 23.54it/s]
2023-08-17T14:45:38.7188882Z running benchmark:  87% 26/30 [00:01<00:00, 23.78it/s]
2023-08-17T14:45:38.7673671Z running benchmark:  97% 29/30 [00:01<00:00, 23.97it/s]
2023-08-17T14:45:38.7674014Z running benchmark: 100% 30/30 [00:01<00:00, 22.82it/s]
2023-08-17T14:45:38.7696444Z 1.572x
2023-08-17T14:45:43.2047549Z 
2023-08-17T14:45:51.5129021Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:45:51.5129496Z loading model: 0it [00:08, ?it/s]
2023-08-17T14:45:51.5192045Z cuda eval  PegasusForCausalLM                 
2023-08-17T14:46:14.1683376Z 
2023-08-17T14:46:14.3004499Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T14:46:14.4331700Z running benchmark:  10% 3/30 [00:00<00:01, 22.64it/s]
2023-08-17T14:46:14.5660565Z running benchmark:  23% 7/30 [00:00<00:00, 27.04it/s]
2023-08-17T14:46:14.6752751Z running benchmark:  37% 11/30 [00:00<00:00, 28.44it/s]
2023-08-17T14:46:14.8084517Z running benchmark:  47% 14/30 [00:00<00:00, 28.13it/s]
2023-08-17T14:46:14.9083174Z running benchmark:  60% 18/30 [00:00<00:00, 28.85it/s]
2023-08-17T14:46:15.0410881Z running benchmark:  70% 21/30 [00:00<00:00, 29.17it/s]
2023-08-17T14:46:15.1739196Z running benchmark:  83% 25/30 [00:00<00:00, 29.51it/s]
2023-08-17T14:46:15.2072431Z running benchmark:  97% 29/30 [00:01<00:00, 29.71it/s]
2023-08-17T14:46:15.2072771Z running benchmark: 100% 30/30 [00:01<00:00, 28.86it/s]
2023-08-17T14:46:15.2100332Z 1.568x
2023-08-17T14:46:19.5450707Z 
2023-08-17T14:46:32.9507671Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:46:32.9508298Z loading model: 0it [00:13, ?it/s]
2023-08-17T14:46:32.9613234Z cuda eval  PegasusForConditionalGeneration    
2023-08-17T14:47:16.1518078Z 
2023-08-17T14:47:16.2716027Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T14:47:16.4044939Z running benchmark:   3% 1/30 [00:00<00:03,  8.29it/s]
2023-08-17T14:47:16.5458922Z running benchmark:  10% 3/30 [00:00<00:02, 12.41it/s]
2023-08-17T14:47:16.6786206Z running benchmark:  17% 5/30 [00:00<00:01, 13.24it/s]
2023-08-17T14:47:16.8112909Z running benchmark:  23% 7/30 [00:00<00:01, 13.96it/s]
2023-08-17T14:47:16.9439599Z running benchmark:  30% 9/30 [00:00<00:01, 14.36it/s]
2023-08-17T14:47:17.0862132Z running benchmark:  37% 11/30 [00:00<00:01, 14.60it/s]
2023-08-17T14:47:17.2191505Z running benchmark:  43% 13/30 [00:00<00:01, 14.42it/s]
2023-08-17T14:47:17.3531326Z running benchmark:  50% 15/30 [00:01<00:01, 14.61it/s]
2023-08-17T14:47:17.4863864Z running benchmark:  57% 17/30 [00:01<00:00, 14.72it/s]
2023-08-17T14:47:17.6195575Z running benchmark:  63% 19/30 [00:01<00:00, 14.81it/s]
2023-08-17T14:47:17.7523340Z running benchmark:  70% 21/30 [00:01<00:00, 14.87it/s]
2023-08-17T14:47:17.8850376Z running benchmark:  77% 23/30 [00:01<00:00, 14.92it/s]
2023-08-17T14:47:18.0182360Z running benchmark:  83% 25/30 [00:01<00:00, 14.97it/s]
2023-08-17T14:47:18.1511209Z running benchmark:  90% 27/30 [00:01<00:00, 14.98it/s]
2023-08-17T14:47:18.2183312Z running benchmark:  97% 29/30 [00:02<00:00, 15.00it/s]
2023-08-17T14:47:18.2183652Z running benchmark: 100% 30/30 [00:02<00:00, 14.51it/s]
2023-08-17T14:47:18.2207669Z 1.331x
2023-08-17T14:47:22.9575003Z 
2023-08-17T14:47:22.9606803Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`
2023-08-17T14:47:26.0878440Z 
2023-08-17T14:47:26.0879169Z loading model: 0it [00:03, ?it/s]
2023-08-17T14:47:26.0927553Z cuda eval  RobertaForCausalLM                 
2023-08-17T14:47:49.4443061Z 
2023-08-17T14:47:49.5809862Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T14:47:49.7112152Z running benchmark:   7% 2/30 [00:00<00:01, 14.58it/s]
2023-08-17T14:47:49.8527646Z running benchmark:  17% 5/30 [00:00<00:01, 19.42it/s]
2023-08-17T14:47:49.9830467Z running benchmark:  27% 8/30 [00:00<00:01, 20.26it/s]
2023-08-17T14:47:50.1131433Z running benchmark:  37% 11/30 [00:00<00:00, 21.32it/s]
2023-08-17T14:47:50.2435578Z running benchmark:  47% 14/30 [00:00<00:00, 21.93it/s]
2023-08-17T14:47:50.3737532Z running benchmark:  57% 17/30 [00:00<00:00, 22.29it/s]
2023-08-17T14:47:50.5045181Z running benchmark:  67% 20/30 [00:00<00:00, 22.53it/s]
2023-08-17T14:47:50.6348584Z running benchmark:  77% 23/30 [00:01<00:00, 22.67it/s]
2023-08-17T14:47:50.7651200Z running benchmark:  87% 26/30 [00:01<00:00, 22.77it/s]
2023-08-17T14:47:50.8084925Z running benchmark:  97% 29/30 [00:01<00:00, 22.86it/s]
2023-08-17T14:47:50.8091124Z running benchmark: 100% 30/30 [00:01<00:00, 21.98it/s]
2023-08-17T14:47:50.8110380Z 1.536x
2023-08-17T14:47:55.5463032Z 
2023-08-17T14:47:58.2903289Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:47:58.2904281Z loading model: 0it [00:02, ?it/s]
2023-08-17T14:47:58.2946011Z cuda eval  RobertaForQuestionAnswering        
2023-08-17T14:48:20.8749041Z 
2023-08-17T14:48:20.9880379Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T14:48:21.0916777Z running benchmark:   7% 2/30 [00:00<00:01, 17.61it/s]
2023-08-17T14:48:21.1953847Z running benchmark:  17% 5/30 [00:00<00:01, 23.99it/s]
2023-08-17T14:48:21.2994422Z running benchmark:  27% 8/30 [00:00<00:00, 26.19it/s]
2023-08-17T14:48:21.4033782Z running benchmark:  37% 11/30 [00:00<00:00, 27.22it/s]
2023-08-17T14:48:21.5077916Z running benchmark:  47% 14/30 [00:00<00:00, 27.81it/s]
2023-08-17T14:48:21.6213276Z running benchmark:  57% 17/30 [00:00<00:00, 28.13it/s]
2023-08-17T14:48:21.7252806Z running benchmark:  67% 20/30 [00:00<00:00, 27.55it/s]
2023-08-17T14:48:21.8291858Z running benchmark:  77% 23/30 [00:00<00:00, 27.94it/s]
2023-08-17T14:48:21.9330737Z running benchmark:  87% 26/30 [00:00<00:00, 28.23it/s]
2023-08-17T14:48:21.9677250Z running benchmark:  97% 29/30 [00:01<00:00, 28.43it/s]
2023-08-17T14:48:21.9677583Z running benchmark: 100% 30/30 [00:01<00:00, 27.43it/s]
2023-08-17T14:48:21.9702761Z 1.536x
2023-08-17T14:48:26.3041391Z 
2023-08-17T14:48:27.4240713Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:48:27.4241050Z loading model: 0it [00:01, ?it/s]
2023-08-17T14:48:27.4275306Z cuda eval  Speech2Text2ForCausalLM            
2023-08-17T14:48:41.1665591Z 
2023-08-17T14:48:41.2896308Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T14:48:41.4065343Z running benchmark:  13% 4/30 [00:00<00:00, 32.46it/s]
2023-08-17T14:48:41.5221967Z running benchmark:  30% 9/30 [00:00<00:00, 38.34it/s]
2023-08-17T14:48:41.6374400Z running benchmark:  47% 14/30 [00:00<00:00, 40.52it/s]
2023-08-17T14:48:41.7531046Z running benchmark:  63% 19/30 [00:00<00:00, 41.64it/s]
2023-08-17T14:48:41.8687177Z running benchmark:  80% 24/30 [00:00<00:00, 42.21it/s]
2023-08-17T14:48:41.8918615Z running benchmark:  97% 29/30 [00:00<00:00, 42.56it/s]
2023-08-17T14:48:41.8919016Z running benchmark: 100% 30/30 [00:00<00:00, 41.33it/s]
2023-08-17T14:48:41.8940263Z 1.819x
2023-08-17T14:48:46.0830155Z 
2023-08-17T14:48:48.3608345Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:48:48.3608837Z loading model: 0it [00:02, ?it/s]
2023-08-17T14:48:48.3639442Z cuda eval  T5ForConditionalGeneration         
2023-08-17T14:49:16.4122868Z 
2023-08-17T14:49:16.5140794Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T14:49:16.6318628Z running benchmark:   7% 2/30 [00:00<00:01, 19.52it/s]
2023-08-17T14:49:16.7495321Z running benchmark:  17% 5/30 [00:00<00:01, 23.20it/s]
2023-08-17T14:49:16.8754985Z running benchmark:  27% 8/30 [00:00<00:00, 24.27it/s]
2023-08-17T14:49:16.9932298Z running benchmark:  37% 11/30 [00:00<00:00, 24.09it/s]
2023-08-17T14:49:17.1107448Z running benchmark:  47% 14/30 [00:00<00:00, 24.58it/s]
2023-08-17T14:49:17.2287462Z running benchmark:  57% 17/30 [00:00<00:00, 24.90it/s]
2023-08-17T14:49:17.3469388Z running benchmark:  67% 20/30 [00:00<00:00, 25.08it/s]
2023-08-17T14:49:17.4646476Z running benchmark:  77% 23/30 [00:00<00:00, 25.17it/s]
2023-08-17T14:49:17.5824602Z running benchmark:  87% 26/30 [00:01<00:00, 25.27it/s]
2023-08-17T14:49:17.6217315Z running benchmark:  97% 29/30 [00:01<00:00, 25.33it/s]
2023-08-17T14:49:17.6221118Z running benchmark: 100% 30/30 [00:01<00:00, 24.78it/s]
2023-08-17T14:49:17.6242002Z 2.163x
2023-08-17T14:49:21.9972732Z 
2023-08-17T14:49:24.1851555Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:49:24.1851890Z loading model: 0it [00:02, ?it/s]
2023-08-17T14:49:24.1884990Z cuda eval  T5Small                            
2023-08-17T14:49:52.1829775Z 
2023-08-17T14:49:52.2834869Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T14:49:52.4009477Z running benchmark:   7% 2/30 [00:00<00:01, 19.46it/s]
2023-08-17T14:49:52.5186940Z running benchmark:  17% 5/30 [00:00<00:01, 23.18it/s]
2023-08-17T14:49:52.6363289Z running benchmark:  27% 8/30 [00:00<00:00, 24.26it/s]
2023-08-17T14:49:52.7636770Z running benchmark:  37% 11/30 [00:00<00:00, 24.76it/s]
2023-08-17T14:49:52.8813631Z running benchmark:  47% 14/30 [00:00<00:00, 24.30it/s]
2023-08-17T14:49:52.9990164Z running benchmark:  57% 17/30 [00:00<00:00, 24.70it/s]
2023-08-17T14:49:53.1167323Z running benchmark:  67% 20/30 [00:00<00:00, 24.96it/s]
2023-08-17T14:49:53.2346160Z running benchmark:  77% 23/30 [00:00<00:00, 25.13it/s]
2023-08-17T14:49:53.3522297Z running benchmark:  87% 26/30 [00:01<00:00, 25.23it/s]
2023-08-17T14:49:53.3915086Z running benchmark:  97% 29/30 [00:01<00:00, 25.31it/s]
2023-08-17T14:49:53.3915431Z running benchmark: 100% 30/30 [00:01<00:00, 24.77it/s]
2023-08-17T14:49:53.3940574Z 2.163x
2023-08-17T14:49:57.7761329Z 
2023-08-17T14:50:03.9946926Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:50:03.9947804Z loading model: 0it [00:06, ?it/s]
2023-08-17T14:50:04.0012660Z cuda eval  TrOCRForCausalLM                   
2023-08-17T14:50:26.8209593Z 
2023-08-17T14:50:26.9557164Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T14:50:27.0795623Z running benchmark:   3% 1/30 [00:00<00:03,  7.39it/s]
2023-08-17T14:50:27.2044250Z running benchmark:  10% 3/30 [00:00<00:02, 12.35it/s]
2023-08-17T14:50:27.3280015Z running benchmark:  17% 5/30 [00:00<00:01, 13.99it/s]
2023-08-17T14:50:27.4513082Z running benchmark:  23% 7/30 [00:00<00:01, 14.84it/s]
2023-08-17T14:50:27.5787286Z running benchmark:  30% 9/30 [00:00<00:01, 15.33it/s]
2023-08-17T14:50:27.7023469Z running benchmark:  37% 11/30 [00:00<00:01, 15.46it/s]
2023-08-17T14:50:27.8256148Z running benchmark:  43% 13/30 [00:00<00:01, 15.69it/s]
2023-08-17T14:50:27.9487786Z running benchmark:  50% 15/30 [00:01<00:00, 15.86it/s]
2023-08-17T14:50:28.0719403Z running benchmark:  57% 17/30 [00:01<00:00, 15.98it/s]
2023-08-17T14:50:28.1953932Z running benchmark:  63% 19/30 [00:01<00:00, 16.06it/s]
2023-08-17T14:50:28.3195766Z running benchmark:  70% 21/30 [00:01<00:00, 16.11it/s]
2023-08-17T14:50:28.4429008Z running benchmark:  77% 23/30 [00:01<00:00, 16.10it/s]
2023-08-17T14:50:28.5664582Z running benchmark:  83% 25/30 [00:01<00:00, 16.13it/s]
2023-08-17T14:50:28.6900803Z running benchmark:  90% 27/30 [00:01<00:00, 16.16it/s]
2023-08-17T14:50:28.7518818Z running benchmark:  97% 29/30 [00:01<00:00, 16.16it/s]
2023-08-17T14:50:28.7524275Z running benchmark: 100% 30/30 [00:01<00:00, 15.53it/s]
2023-08-17T14:50:28.7549335Z 1.673x
2023-08-17T14:50:33.0388087Z 
2023-08-17T14:50:47.7197613Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:50:47.7198150Z loading model: 0it [00:14, ?it/s]
2023-08-17T14:50:47.7272575Z cuda eval  XGLMForCausalLM                    
2023-08-17T14:51:20.8134862Z 
2023-08-17T14:51:20.9212440Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T14:51:21.0226621Z running benchmark:  10% 3/30 [00:00<00:00, 27.73it/s]
2023-08-17T14:51:21.1235817Z running benchmark:  20% 6/30 [00:00<00:00, 28.76it/s]
2023-08-17T14:51:21.2258003Z running benchmark:  30% 9/30 [00:00<00:00, 29.20it/s]
2023-08-17T14:51:21.3265503Z running benchmark:  40% 12/30 [00:00<00:00, 29.26it/s]
2023-08-17T14:51:21.4270474Z running benchmark:  50% 15/30 [00:00<00:00, 29.44it/s]
2023-08-17T14:51:21.5276996Z running benchmark:  60% 18/30 [00:00<00:00, 29.57it/s]
2023-08-17T14:51:21.6292249Z running benchmark:  70% 21/30 [00:00<00:00, 29.65it/s]
2023-08-17T14:51:21.7323308Z running benchmark:  80% 24/30 [00:00<00:00, 29.63it/s]
2023-08-17T14:51:21.8452514Z running benchmark:  90% 27/30 [00:00<00:00, 29.47it/s]
2023-08-17T14:51:21.8452877Z running benchmark: 100% 30/30 [00:01<00:00, 28.51it/s]
2023-08-17T14:51:21.8453180Z running benchmark: 100% 30/30 [00:01<00:00, 29.06it/s]
2023-08-17T14:51:21.8482039Z 1.709x
2023-08-17T14:51:26.4236523Z 
2023-08-17T14:51:33.2418525Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:51:33.2419423Z loading model: 0it [00:06, ?it/s]
2023-08-17T14:51:33.2503490Z cuda eval  XLNetLMHeadModel                   
2023-08-17T14:52:03.4944140Z [2023-08-17 14:52:03,492] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-08-17T14:52:21.9036897Z 
2023-08-17T14:52:22.0909207Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T14:52:22.2655647Z running benchmark:   3% 1/30 [00:00<00:05,  5.34it/s]
2023-08-17T14:52:22.4291093Z running benchmark:   7% 2/30 [00:00<00:05,  5.56it/s]
2023-08-17T14:52:22.5929461Z running benchmark:  10% 3/30 [00:00<00:04,  5.80it/s]
2023-08-17T14:52:22.7534630Z running benchmark:  13% 4/30 [00:00<00:04,  5.91it/s]
2023-08-17T14:52:22.9120573Z running benchmark:  17% 5/30 [00:00<00:04,  6.02it/s]
2023-08-17T14:52:23.0698569Z running benchmark:  20% 6/30 [00:01<00:03,  6.12it/s]
2023-08-17T14:52:23.2366746Z running benchmark:  23% 7/30 [00:01<00:03,  6.19it/s]
2023-08-17T14:52:23.3945811Z running benchmark:  27% 8/30 [00:01<00:03,  6.13it/s]
2023-08-17T14:52:23.5603314Z running benchmark:  30% 9/30 [00:01<00:03,  6.19it/s]
2023-08-17T14:52:23.7244777Z running benchmark:  33% 10/30 [00:01<00:03,  6.14it/s]
2023-08-17T14:52:23.8866084Z running benchmark:  37% 11/30 [00:01<00:03,  6.12it/s]
2023-08-17T14:52:24.0476544Z running benchmark:  40% 12/30 [00:01<00:02,  6.14it/s]
2023-08-17T14:52:24.2059597Z running benchmark:  43% 13/30 [00:02<00:02,  6.16it/s]
2023-08-17T14:52:24.3638751Z running benchmark:  47% 14/30 [00:02<00:02,  6.21it/s]
2023-08-17T14:52:24.5232691Z running benchmark:  50% 15/30 [00:02<00:02,  6.24it/s]
2023-08-17T14:52:24.6819960Z running benchmark:  53% 16/30 [00:02<00:02,  6.25it/s]
2023-08-17T14:52:24.8428650Z running benchmark:  57% 17/30 [00:02<00:02,  6.27it/s]
2023-08-17T14:52:25.0009368Z running benchmark:  60% 18/30 [00:02<00:01,  6.25it/s]
2023-08-17T14:52:25.1619375Z running benchmark:  63% 19/30 [00:03<00:01,  6.27it/s]
2023-08-17T14:52:25.3319211Z running benchmark:  67% 20/30 [00:03<00:01,  6.25it/s]
2023-08-17T14:52:25.4942462Z running benchmark:  70% 21/30 [00:03<00:01,  6.14it/s]
2023-08-17T14:52:25.6564170Z running benchmark:  73% 22/30 [00:03<00:01,  6.14it/s]
2023-08-17T14:52:25.8165332Z running benchmark:  77% 23/30 [00:03<00:01,  6.15it/s]
2023-08-17T14:52:25.9795793Z running benchmark:  80% 24/30 [00:03<00:00,  6.18it/s]
2023-08-17T14:52:26.1421424Z running benchmark:  83% 25/30 [00:04<00:00,  6.16it/s]
2023-08-17T14:52:26.3042468Z running benchmark:  87% 26/30 [00:04<00:00,  6.16it/s]
2023-08-17T14:52:26.4661879Z running benchmark:  90% 27/30 [00:04<00:00,  6.16it/s]
2023-08-17T14:52:26.6270389Z running benchmark:  93% 28/30 [00:04<00:00,  6.17it/s]
2023-08-17T14:52:26.7849867Z running benchmark:  97% 29/30 [00:04<00:00,  6.18it/s]
2023-08-17T14:52:26.7853756Z running benchmark: 100% 30/30 [00:04<00:00,  6.23it/s]
2023-08-17T14:52:26.7854562Z running benchmark: 100% 30/30 [00:04<00:00,  6.15it/s]
2023-08-17T14:52:26.7885138Z 2.078x
2023-08-17T14:52:31.5110800Z 
2023-08-17T14:52:34.7416402Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:52:34.7417179Z loading model: 0it [00:03, ?it/s]
2023-08-17T14:52:34.7480950Z cuda eval  YituTechConvBert                   
2023-08-17T14:53:06.3426815Z 
2023-08-17T14:53:06.4495546Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T14:53:06.5544250Z running benchmark:   3% 1/30 [00:00<00:03,  9.30it/s]
2023-08-17T14:53:06.6591936Z running benchmark:  10% 3/30 [00:00<00:01, 14.97it/s]
2023-08-17T14:53:06.7644943Z running benchmark:  17% 5/30 [00:00<00:01, 16.84it/s]
2023-08-17T14:53:06.8692388Z running benchmark:  23% 7/30 [00:00<00:01, 17.68it/s]
2023-08-17T14:53:06.9740480Z running benchmark:  30% 9/30 [00:00<00:01, 18.19it/s]
2023-08-17T14:53:07.0789920Z running benchmark:  37% 11/30 [00:00<00:01, 18.50it/s]
2023-08-17T14:53:07.1842135Z running benchmark:  43% 13/30 [00:00<00:00, 18.69it/s]
2023-08-17T14:53:07.2979675Z running benchmark:  50% 15/30 [00:00<00:00, 18.79it/s]
2023-08-17T14:53:07.4027475Z running benchmark:  57% 17/30 [00:00<00:00, 18.39it/s]
2023-08-17T14:53:07.5075187Z running benchmark:  63% 19/30 [00:01<00:00, 18.59it/s]
2023-08-17T14:53:07.6123282Z running benchmark:  70% 21/30 [00:01<00:00, 18.74it/s]
2023-08-17T14:53:07.7171318Z running benchmark:  77% 23/30 [00:01<00:00, 18.85it/s]
2023-08-17T14:53:07.8218088Z running benchmark:  83% 25/30 [00:01<00:00, 18.92it/s]
2023-08-17T14:53:07.9265500Z running benchmark:  90% 27/30 [00:01<00:00, 18.97it/s]
2023-08-17T14:53:07.9790162Z running benchmark:  97% 29/30 [00:01<00:00, 19.01it/s]
2023-08-17T14:53:07.9796171Z running benchmark: 100% 30/30 [00:01<00:00, 18.32it/s]
2023-08-17T14:53:07.9818490Z 1.651x
2023-08-17T14:53:09.2285786Z speedup             gmean=1.81x mean=1.841x
2023-08-17T14:53:09.2286378Z abs_latency         gmean=16.83x mean=18.537x
2023-08-17T14:53:09.2286850Z compilation_latency mean=34.364 seconds
2023-08-17T14:53:09.2288119Z compression_ratio   mean=1.156x
2023-08-17T14:53:09.2290556Z eager_peak_mem      gmean=1.58x mean=1.798x
2023-08-17T14:53:09.2292780Z dynamo_peak_mem     gmean=1.41x mean=1.597x
2023-08-17T14:53:09.2296376Z calls_captured      gmean=642.21x mean=730.412x
2023-08-17T14:53:09.2297560Z unique_graphs       gmean=1.00x mean=1.000x
2023-08-17T14:53:09.2299849Z graph_breaks        gmean=0.00x mean=0.000x
2023-08-17T14:53:09.2301808Z unique_graph_breaks gmean=0.00x mean=0.000x
2023-08-17T14:53:09.7360220Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *cudagraphs-true* ]]
2023-08-17T14:53:09.7362124Z + python benchmarks/dynamo/huggingface.py --performance --cold-start-latency --inference --bfloat16 --backend inductor --device cuda --total-partitions 3 --partition-id 2 --output /var/lib/jenkins/workspace/test/test-reports/inductor_with_cudagraphs_huggingface_bfloat16_inference_cuda_performance.csv
2023-08-17T14:53:16.6236807Z 
2023-08-17T14:53:23.2503318Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:53:23.2503654Z loading model: 0it [00:06, ?it/s]
2023-08-17T14:53:23.2592029Z cuda eval  MegatronBertForQuestionAnswering   
2023-08-17T14:54:01.0849090Z 
2023-08-17T14:54:01.2110503Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T14:54:01.3262657Z running benchmark:   3% 1/30 [00:00<00:03,  7.90it/s]
2023-08-17T14:54:01.4416863Z running benchmark:  10% 3/30 [00:00<00:02, 13.23it/s]
2023-08-17T14:54:01.5610361Z running benchmark:  17% 5/30 [00:00<00:01, 15.06it/s]
2023-08-17T14:54:01.6766572Z running benchmark:  23% 7/30 [00:00<00:01, 15.75it/s]
2023-08-17T14:54:01.7925150Z running benchmark:  30% 9/30 [00:00<00:01, 16.29it/s]
2023-08-17T14:54:01.9079650Z running benchmark:  37% 11/30 [00:00<00:01, 16.61it/s]
2023-08-17T14:54:02.0232036Z running benchmark:  43% 13/30 [00:00<00:01, 16.85it/s]
2023-08-17T14:54:02.1383819Z running benchmark:  50% 15/30 [00:00<00:00, 17.00it/s]
2023-08-17T14:54:02.2539220Z running benchmark:  57% 17/30 [00:01<00:00, 17.12it/s]
2023-08-17T14:54:02.3697900Z running benchmark:  63% 19/30 [00:01<00:00, 17.18it/s]
2023-08-17T14:54:02.4852197Z running benchmark:  70% 21/30 [00:01<00:00, 17.20it/s]
2023-08-17T14:54:02.6208780Z running benchmark:  77% 23/30 [00:01<00:00, 17.24it/s]
2023-08-17T14:54:02.7358781Z running benchmark:  83% 25/30 [00:01<00:00, 16.41it/s]
2023-08-17T14:54:02.8514336Z running benchmark:  90% 27/30 [00:01<00:00, 16.68it/s]
2023-08-17T14:54:02.9095374Z running benchmark:  97% 29/30 [00:01<00:00, 16.87it/s]
2023-08-17T14:54:02.9095694Z running benchmark: 100% 30/30 [00:01<00:00, 16.43it/s]
2023-08-17T14:54:02.9119973Z 1.421x
2023-08-17T14:54:07.4329530Z 
2023-08-17T14:54:09.2946129Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:54:09.2946904Z loading model: 0it [00:01, ?it/s]
2023-08-17T14:54:09.3190458Z cuda eval  MobileBertForMaskedLM              
2023-08-17T14:55:37.2543712Z 
2023-08-17T14:55:37.3716042Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T14:55:37.4942781Z running benchmark:   3% 1/30 [00:00<00:03,  8.49it/s]
2023-08-17T14:55:37.6164156Z running benchmark:  10% 3/30 [00:00<00:02, 13.14it/s]
2023-08-17T14:55:37.7384865Z running benchmark:  17% 5/30 [00:00<00:01, 14.63it/s]
2023-08-17T14:55:37.8776633Z running benchmark:  23% 7/30 [00:00<00:01, 15.32it/s]
2023-08-17T14:55:38.0006494Z running benchmark:  30% 9/30 [00:00<00:01, 14.96it/s]
2023-08-17T14:55:38.1221215Z running benchmark:  37% 11/30 [00:00<00:01, 15.38it/s]
2023-08-17T14:55:38.2435903Z running benchmark:  43% 13/30 [00:00<00:01, 15.72it/s]
2023-08-17T14:55:38.3646583Z running benchmark:  50% 15/30 [00:00<00:00, 15.96it/s]
2023-08-17T14:55:38.4859541Z running benchmark:  57% 17/30 [00:01<00:00, 16.13it/s]
2023-08-17T14:55:38.6070337Z running benchmark:  63% 19/30 [00:01<00:00, 16.24it/s]
2023-08-17T14:55:38.7279749Z running benchmark:  70% 21/30 [00:01<00:00, 16.32it/s]
2023-08-17T14:55:38.8570054Z running benchmark:  77% 23/30 [00:01<00:00, 16.39it/s]
2023-08-17T14:55:38.9805735Z running benchmark:  83% 25/30 [00:01<00:00, 16.11it/s]
2023-08-17T14:55:39.1052332Z running benchmark:  90% 27/30 [00:01<00:00, 16.14it/s]
2023-08-17T14:55:39.1668831Z running benchmark:  97% 29/30 [00:01<00:00, 16.11it/s]
2023-08-17T14:55:39.1669170Z running benchmark: 100% 30/30 [00:01<00:00, 15.68it/s]
2023-08-17T14:55:39.1704668Z 2.351x
2023-08-17T14:55:44.2946998Z 
2023-08-17T14:55:45.8710017Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:55:45.8710518Z loading model: 0it [00:01, ?it/s]
2023-08-17T14:55:45.8949455Z cuda eval  MobileBertForQuestionAnswering     
2023-08-17T14:57:11.1198129Z 
2023-08-17T14:57:11.2543562Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T14:57:11.3927911Z running benchmark:   7% 2/30 [00:00<00:01, 14.79it/s]
2023-08-17T14:57:11.5306798Z running benchmark:  17% 5/30 [00:00<00:01, 18.86it/s]
2023-08-17T14:57:11.6702662Z running benchmark:  27% 8/30 [00:00<00:01, 20.18it/s]
2023-08-17T14:57:11.8131194Z running benchmark:  37% 11/30 [00:00<00:00, 20.71it/s]
2023-08-17T14:57:11.9530851Z running benchmark:  47% 14/30 [00:00<00:00, 20.82it/s]
2023-08-17T14:57:12.1095527Z running benchmark:  57% 17/30 [00:00<00:00, 21.03it/s]
2023-08-17T14:57:12.2478078Z running benchmark:  67% 20/30 [00:00<00:00, 20.37it/s]
2023-08-17T14:57:12.3846733Z running benchmark:  77% 23/30 [00:01<00:00, 20.78it/s]
2023-08-17T14:57:12.5215204Z running benchmark:  87% 26/30 [00:01<00:00, 21.12it/s]
2023-08-17T14:57:12.5673636Z running benchmark:  97% 29/30 [00:01<00:00, 21.36it/s]
2023-08-17T14:57:12.5674404Z running benchmark: 100% 30/30 [00:01<00:00, 20.71it/s]
2023-08-17T14:57:12.5708584Z 2.415x
2023-08-17T14:57:17.6529119Z 
2023-08-17T14:57:21.2450662Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:57:21.2451406Z loading model: 0it [00:03, ?it/s]
2023-08-17T14:57:21.2487171Z cuda eval  OPTForCausalLM                     
2023-08-17T14:57:43.8235714Z 
2023-08-17T14:57:43.9243930Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T14:57:44.0298094Z running benchmark:   3% 1/30 [00:00<00:02,  9.90it/s]
2023-08-17T14:57:44.1328422Z running benchmark:  10% 3/30 [00:00<00:01, 15.32it/s]
2023-08-17T14:57:44.2367395Z running benchmark:  17% 5/30 [00:00<00:01, 17.18it/s]
2023-08-17T14:57:44.3398833Z running benchmark:  23% 7/30 [00:00<00:01, 18.00it/s]
2023-08-17T14:57:44.4426021Z running benchmark:  30% 9/30 [00:00<00:01, 18.49it/s]
2023-08-17T14:57:44.5469860Z running benchmark:  37% 11/30 [00:00<00:01, 18.82it/s]
2023-08-17T14:57:44.6498379Z running benchmark:  43% 13/30 [00:00<00:00, 18.94it/s]
2023-08-17T14:57:44.7687271Z running benchmark:  50% 15/30 [00:00<00:00, 19.10it/s]
2023-08-17T14:57:44.8731432Z running benchmark:  57% 17/30 [00:00<00:00, 19.16it/s]
2023-08-17T14:57:44.9805754Z running benchmark:  63% 19/30 [00:01<00:00, 18.33it/s]
2023-08-17T14:57:45.0859236Z running benchmark:  70% 21/30 [00:01<00:00, 18.42it/s]
2023-08-17T14:57:45.1895372Z running benchmark:  77% 23/30 [00:01<00:00, 18.59it/s]
2023-08-17T14:57:45.2917185Z running benchmark:  83% 25/30 [00:01<00:00, 18.80it/s]
2023-08-17T14:57:45.3953018Z running benchmark:  90% 27/30 [00:01<00:00, 19.03it/s]
2023-08-17T14:57:45.4462142Z running benchmark:  97% 29/30 [00:01<00:00, 19.11it/s]
2023-08-17T14:57:45.4490161Z running benchmark: 100% 30/30 [00:01<00:00, 18.48it/s]
2023-08-17T14:57:45.4490466Z 2.383x
2023-08-17T14:57:49.8279365Z 
2023-08-17T14:57:52.8609803Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:57:52.8610141Z loading model: 0it [00:03, ?it/s]
2023-08-17T14:57:52.8659930Z cuda eval  PLBartForCausalLM                  
2023-08-17T14:58:09.9802816Z 
2023-08-17T14:58:10.1088930Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T14:58:10.2406854Z running benchmark:   7% 2/30 [00:00<00:01, 15.52it/s]
2023-08-17T14:58:10.3860677Z running benchmark:  17% 5/30 [00:00<00:01, 19.81it/s]
2023-08-17T14:58:10.5198275Z running benchmark:  27% 8/30 [00:00<00:01, 20.22it/s]
2023-08-17T14:58:10.6528078Z running benchmark:  37% 11/30 [00:00<00:00, 21.07it/s]
2023-08-17T14:58:10.7854749Z running benchmark:  47% 14/30 [00:00<00:00, 21.60it/s]
2023-08-17T14:58:10.9232995Z running benchmark:  57% 17/30 [00:00<00:00, 21.94it/s]
2023-08-17T14:58:11.0568005Z running benchmark:  67% 20/30 [00:00<00:00, 21.88it/s]
2023-08-17T14:58:11.1889671Z running benchmark:  77% 23/30 [00:01<00:00, 22.07it/s]
2023-08-17T14:58:11.3254741Z running benchmark:  87% 26/30 [00:01<00:00, 22.26it/s]
2023-08-17T14:58:11.3696231Z running benchmark:  97% 29/30 [00:01<00:00, 22.17it/s]
2023-08-17T14:58:11.3701610Z running benchmark: 100% 30/30 [00:01<00:00, 21.58it/s]
2023-08-17T14:58:11.3721610Z 2.336x
2023-08-17T14:58:15.5898627Z 
2023-08-17T14:58:20.8838774Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:58:20.8839261Z loading model: 0it [00:05, ?it/s]
2023-08-17T14:58:20.8891646Z cuda eval  PLBartForConditionalGeneration     
2023-08-17T14:58:47.9862974Z 
2023-08-17T14:58:48.1164229Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T14:58:48.2568704Z running benchmark:   7% 2/30 [00:00<00:01, 15.33it/s]
2023-08-17T14:58:48.3834861Z running benchmark:  17% 5/30 [00:00<00:01, 19.66it/s]
2023-08-17T14:58:48.5070944Z running benchmark:  27% 8/30 [00:00<00:01, 20.87it/s]
2023-08-17T14:58:48.6310308Z running benchmark:  37% 11/30 [00:00<00:00, 22.15it/s]
2023-08-17T14:58:48.7570384Z running benchmark:  47% 14/30 [00:00<00:00, 22.87it/s]
2023-08-17T14:58:48.8795144Z running benchmark:  57% 17/30 [00:00<00:00, 23.18it/s]
2023-08-17T14:58:49.0019990Z running benchmark:  67% 20/30 [00:00<00:00, 23.60it/s]
2023-08-17T14:58:49.1243940Z running benchmark:  77% 23/30 [00:01<00:00, 23.88it/s]
2023-08-17T14:58:49.2488560Z running benchmark:  87% 26/30 [00:01<00:00, 24.08it/s]
2023-08-17T14:58:49.3004448Z running benchmark:  97% 29/30 [00:01<00:00, 24.09it/s]
2023-08-17T14:58:49.3004794Z running benchmark: 100% 30/30 [00:01<00:00, 22.82it/s]
2023-08-17T14:58:49.3047445Z 1.609x
2023-08-17T14:58:53.6670505Z 
2023-08-17T14:59:02.0136132Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:59:02.0137327Z loading model: 0it [00:08, ?it/s]
2023-08-17T14:59:02.0198691Z cuda eval  PegasusForCausalLM                 
2023-08-17T14:59:25.1930120Z 
2023-08-17T14:59:25.2974656Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T14:59:25.4016461Z running benchmark:   7% 2/30 [00:00<00:01, 19.04it/s]
2023-08-17T14:59:25.5335002Z running benchmark:  17% 5/30 [00:00<00:01, 24.73it/s]
2023-08-17T14:59:25.6649703Z running benchmark:  30% 9/30 [00:00<00:00, 27.61it/s]
2023-08-17T14:59:25.7964328Z running benchmark:  43% 13/30 [00:00<00:00, 28.80it/s]
2023-08-17T14:59:25.9278868Z running benchmark:  57% 17/30 [00:00<00:00, 29.41it/s]
2023-08-17T14:59:26.0593664Z running benchmark:  70% 21/30 [00:00<00:00, 29.77it/s]
2023-08-17T14:59:26.1727480Z running benchmark:  83% 25/30 [00:00<00:00, 29.99it/s]
2023-08-17T14:59:26.2389840Z running benchmark:  93% 28/30 [00:00<00:00, 28.97it/s]
2023-08-17T14:59:26.2393988Z running benchmark: 100% 30/30 [00:01<00:00, 28.66it/s]
2023-08-17T14:59:26.2417434Z 1.600x
2023-08-17T14:59:30.5596227Z 
2023-08-17T14:59:44.0028680Z loading model: 0it [00:00, ?it/s]
2023-08-17T14:59:44.0029199Z loading model: 0it [00:13, ?it/s]
2023-08-17T14:59:44.0133611Z cuda eval  PegasusForConditionalGeneration    
2023-08-17T15:00:27.8308691Z 
2023-08-17T15:00:27.9771619Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:00:28.1084144Z running benchmark:   3% 1/30 [00:00<00:04,  6.81it/s]
2023-08-17T15:00:28.2398140Z running benchmark:  10% 3/30 [00:00<00:02, 11.52it/s]
2023-08-17T15:00:28.3712748Z running benchmark:  17% 5/30 [00:00<00:01, 13.17it/s]
2023-08-17T15:00:28.5026938Z running benchmark:  23% 7/30 [00:00<00:01, 13.96it/s]
2023-08-17T15:00:28.6462287Z running benchmark:  30% 9/30 [00:00<00:01, 14.41it/s]
2023-08-17T15:00:28.7781392Z running benchmark:  37% 11/30 [00:00<00:01, 14.24it/s]
2023-08-17T15:00:28.9107466Z running benchmark:  43% 13/30 [00:00<00:01, 14.54it/s]
2023-08-17T15:00:29.0425595Z running benchmark:  50% 15/30 [00:01<00:01, 14.71it/s]
2023-08-17T15:00:29.1738993Z running benchmark:  57% 17/30 [00:01<00:00, 14.85it/s]
2023-08-17T15:00:29.3052700Z running benchmark:  63% 19/30 [00:01<00:00, 14.96it/s]
2023-08-17T15:00:29.4365960Z running benchmark:  70% 21/30 [00:01<00:00, 15.04it/s]
2023-08-17T15:00:29.5680584Z running benchmark:  77% 23/30 [00:01<00:00, 15.10it/s]
2023-08-17T15:00:29.7002485Z running benchmark:  83% 25/30 [00:01<00:00, 15.13it/s]
2023-08-17T15:00:29.8320759Z running benchmark:  90% 27/30 [00:01<00:00, 15.14it/s]
2023-08-17T15:00:29.8979730Z running benchmark:  97% 29/30 [00:02<00:00, 15.14it/s]
2023-08-17T15:00:29.8980060Z running benchmark: 100% 30/30 [00:02<00:00, 14.51it/s]
2023-08-17T15:00:29.9004266Z 1.355x
2023-08-17T15:00:34.5929951Z 
2023-08-17T15:00:34.5961142Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`
2023-08-17T15:00:37.7015762Z 
2023-08-17T15:00:37.7016673Z loading model: 0it [00:03, ?it/s]
2023-08-17T15:00:37.7061353Z cuda eval  RobertaForCausalLM                 
2023-08-17T15:01:01.5631672Z 
2023-08-17T15:01:01.6984937Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:01:01.8284183Z running benchmark:   7% 2/30 [00:00<00:01, 14.72it/s]
2023-08-17T15:01:01.9582805Z running benchmark:  17% 5/30 [00:00<00:01, 19.54it/s]
2023-08-17T15:01:02.0881816Z running benchmark:  27% 8/30 [00:00<00:01, 21.15it/s]
2023-08-17T15:01:02.2184791Z running benchmark:  37% 11/30 [00:00<00:00, 21.91it/s]
2023-08-17T15:01:02.3482792Z running benchmark:  47% 14/30 [00:00<00:00, 22.32it/s]
2023-08-17T15:01:02.4891676Z running benchmark:  57% 17/30 [00:00<00:00, 22.59it/s]
2023-08-17T15:01:02.6192943Z running benchmark:  67% 20/30 [00:00<00:00, 22.14it/s]
2023-08-17T15:01:02.7494158Z running benchmark:  77% 23/30 [00:01<00:00, 22.43it/s]
2023-08-17T15:01:02.8791115Z running benchmark:  87% 26/30 [00:01<00:00, 22.62it/s]
2023-08-17T15:01:02.9223270Z running benchmark:  97% 29/30 [00:01<00:00, 22.77it/s]
2023-08-17T15:01:02.9226217Z running benchmark: 100% 30/30 [00:01<00:00, 22.06it/s]
2023-08-17T15:01:02.9268728Z 1.563x
2023-08-17T15:01:07.3047562Z 
2023-08-17T15:01:10.0404657Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:01:10.0404978Z loading model: 0it [00:02, ?it/s]
2023-08-17T15:01:10.0447134Z cuda eval  RobertaForQuestionAnswering        
2023-08-17T15:01:33.2010964Z 
2023-08-17T15:01:33.3088375Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:01:33.4116926Z running benchmark:   7% 2/30 [00:00<00:01, 18.50it/s]
2023-08-17T15:01:33.5146591Z running benchmark:  17% 5/30 [00:00<00:01, 24.61it/s]
2023-08-17T15:01:33.6177767Z running benchmark:  27% 8/30 [00:00<00:00, 26.64it/s]
2023-08-17T15:01:33.7207665Z running benchmark:  37% 11/30 [00:00<00:00, 27.60it/s]
2023-08-17T15:01:33.8237156Z running benchmark:  47% 14/30 [00:00<00:00, 28.15it/s]
2023-08-17T15:01:33.9269310Z running benchmark:  57% 17/30 [00:00<00:00, 28.48it/s]
2023-08-17T15:01:34.0299594Z running benchmark:  67% 20/30 [00:00<00:00, 28.69it/s]
2023-08-17T15:01:34.1572024Z running benchmark:  77% 23/30 [00:00<00:00, 28.82it/s]
2023-08-17T15:01:34.2607515Z running benchmark:  87% 26/30 [00:00<00:00, 26.96it/s]
2023-08-17T15:01:34.2952204Z running benchmark:  97% 29/30 [00:01<00:00, 27.53it/s]
2023-08-17T15:01:34.2952527Z running benchmark: 100% 30/30 [00:01<00:00, 27.40it/s]
2023-08-17T15:01:34.3000846Z 1.569x
2023-08-17T15:01:38.5743409Z 
2023-08-17T15:01:39.6111302Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:01:39.6111794Z loading model: 0it [00:01, ?it/s]
2023-08-17T15:01:39.6142414Z cuda eval  Speech2Text2ForCausalLM            
2023-08-17T15:01:53.7254065Z 
2023-08-17T15:01:53.8439999Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:01:53.9590624Z running benchmark:  13% 4/30 [00:00<00:00, 33.68it/s]
2023-08-17T15:01:54.0734292Z running benchmark:  30% 9/30 [00:00<00:00, 39.34it/s]
2023-08-17T15:01:54.1893960Z running benchmark:  47% 14/30 [00:00<00:00, 41.30it/s]
2023-08-17T15:01:54.3089111Z running benchmark:  63% 19/30 [00:00<00:00, 42.03it/s]
2023-08-17T15:01:54.4262188Z running benchmark:  80% 24/30 [00:00<00:00, 41.95it/s]
2023-08-17T15:01:54.4491461Z running benchmark:  97% 29/30 [00:00<00:00, 42.18it/s]
2023-08-17T15:01:54.4491790Z running benchmark: 100% 30/30 [00:00<00:00, 41.42it/s]
2023-08-17T15:01:54.4512489Z 1.850x
2023-08-17T15:01:58.6132372Z 
2023-08-17T15:02:00.8258721Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:02:00.8259288Z loading model: 0it [00:02, ?it/s]
2023-08-17T15:02:00.8288590Z cuda eval  T5ForConditionalGeneration         
2023-08-17T15:02:29.3393540Z 
2023-08-17T15:02:29.4608159Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:02:29.5769537Z running benchmark:   7% 2/30 [00:00<00:01, 16.30it/s]
2023-08-17T15:02:29.6933594Z running benchmark:  17% 5/30 [00:00<00:01, 21.74it/s]
2023-08-17T15:02:29.8097906Z running benchmark:  27% 8/30 [00:00<00:00, 23.55it/s]
2023-08-17T15:02:29.9260402Z running benchmark:  37% 11/30 [00:00<00:00, 24.42it/s]
2023-08-17T15:02:30.0424120Z running benchmark:  47% 14/30 [00:00<00:00, 24.92it/s]
2023-08-17T15:02:30.1588175Z running benchmark:  57% 17/30 [00:00<00:00, 25.21it/s]
2023-08-17T15:02:30.2757665Z running benchmark:  67% 20/30 [00:00<00:00, 25.40it/s]
2023-08-17T15:02:30.3925159Z running benchmark:  77% 23/30 [00:00<00:00, 25.49it/s]
2023-08-17T15:02:30.5144015Z running benchmark:  87% 26/30 [00:01<00:00, 25.55it/s]
2023-08-17T15:02:30.5531820Z running benchmark:  97% 29/30 [00:01<00:00, 25.26it/s]
2023-08-17T15:02:30.5532140Z running benchmark: 100% 30/30 [00:01<00:00, 24.68it/s]
2023-08-17T15:02:30.5558041Z 2.229x
2023-08-17T15:02:34.9026590Z 
2023-08-17T15:02:37.0627787Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:02:37.0628271Z loading model: 0it [00:02, ?it/s]
2023-08-17T15:02:37.0659562Z cuda eval  T5Small                            
2023-08-17T15:03:05.6027575Z 
2023-08-17T15:03:05.7147786Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:03:05.8310150Z running benchmark:   7% 2/30 [00:00<00:01, 17.75it/s]
2023-08-17T15:03:05.9474040Z running benchmark:  17% 5/30 [00:00<00:01, 22.51it/s]
2023-08-17T15:03:06.0638743Z running benchmark:  27% 8/30 [00:00<00:00, 24.01it/s]
2023-08-17T15:03:06.1803325Z running benchmark:  37% 11/30 [00:00<00:00, 24.70it/s]
2023-08-17T15:03:06.3070202Z running benchmark:  47% 14/30 [00:00<00:00, 25.09it/s]
2023-08-17T15:03:06.4236880Z running benchmark:  57% 17/30 [00:00<00:00, 24.61it/s]
2023-08-17T15:03:06.5400627Z running benchmark:  67% 20/30 [00:00<00:00, 24.93it/s]
2023-08-17T15:03:06.6563864Z running benchmark:  77% 23/30 [00:00<00:00, 25.20it/s]
2023-08-17T15:03:06.7727274Z running benchmark:  87% 26/30 [00:01<00:00, 25.38it/s]
2023-08-17T15:03:06.8116059Z running benchmark:  97% 29/30 [00:01<00:00, 25.50it/s]
2023-08-17T15:03:06.8116646Z running benchmark: 100% 30/30 [00:01<00:00, 24.79it/s]
2023-08-17T15:03:06.8142646Z 2.227x
2023-08-17T15:03:11.1851537Z 
2023-08-17T15:03:17.3030176Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:03:17.3030486Z loading model: 0it [00:06, ?it/s]
2023-08-17T15:03:17.3092006Z cuda eval  TrOCRForCausalLM                   
2023-08-17T15:03:40.6970168Z 
2023-08-17T15:03:40.8252561Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:03:40.9486724Z running benchmark:   3% 1/30 [00:00<00:03,  7.77it/s]
2023-08-17T15:03:41.0716637Z running benchmark:  10% 3/30 [00:00<00:02, 12.64it/s]
2023-08-17T15:03:41.1998858Z running benchmark:  17% 5/30 [00:00<00:01, 14.27it/s]
2023-08-17T15:03:41.3225531Z running benchmark:  23% 7/30 [00:00<00:01, 14.81it/s]
2023-08-17T15:03:41.4453260Z running benchmark:  30% 9/30 [00:00<00:01, 15.33it/s]
2023-08-17T15:03:41.5684379Z running benchmark:  37% 11/30 [00:00<00:01, 15.66it/s]
2023-08-17T15:03:41.6915408Z running benchmark:  43% 13/30 [00:00<00:01, 15.85it/s]
2023-08-17T15:03:41.8143928Z running benchmark:  50% 15/30 [00:00<00:00, 15.97it/s]
2023-08-17T15:03:41.9372040Z running benchmark:  57% 17/30 [00:01<00:00, 16.07it/s]
2023-08-17T15:03:42.0597821Z running benchmark:  63% 19/30 [00:01<00:00, 16.14it/s]
2023-08-17T15:03:42.1825564Z running benchmark:  70% 21/30 [00:01<00:00, 16.19it/s]
2023-08-17T15:03:42.3053546Z running benchmark:  77% 23/30 [00:01<00:00, 16.22it/s]
2023-08-17T15:03:42.4280190Z running benchmark:  83% 25/30 [00:01<00:00, 16.24it/s]
2023-08-17T15:03:42.5505417Z running benchmark:  90% 27/30 [00:01<00:00, 16.26it/s]
2023-08-17T15:03:42.6121834Z running benchmark:  97% 29/30 [00:01<00:00, 16.28it/s]
2023-08-17T15:03:42.6122182Z running benchmark: 100% 30/30 [00:01<00:00, 15.66it/s]
2023-08-17T15:03:42.6171808Z 1.686x
2023-08-17T15:03:47.0205944Z 
2023-08-17T15:04:01.7850801Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:04:01.7851294Z loading model: 0it [00:14, ?it/s]
2023-08-17T15:04:01.7927421Z cuda eval  XGLMForCausalLM                    
2023-08-17T15:04:35.5227063Z 
2023-08-17T15:04:35.6476586Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:04:35.7725448Z running benchmark:  10% 3/30 [00:00<00:01, 23.91it/s]
2023-08-17T15:04:35.8962983Z running benchmark:  23% 7/30 [00:00<00:00, 28.66it/s]
2023-08-17T15:04:36.0208465Z running benchmark:  37% 11/30 [00:00<00:00, 30.31it/s]
2023-08-17T15:04:36.1454364Z running benchmark:  50% 15/30 [00:00<00:00, 31.04it/s]
2023-08-17T15:04:36.2700720Z running benchmark:  63% 19/30 [00:00<00:00, 31.42it/s]
2023-08-17T15:04:36.3932340Z running benchmark:  77% 23/30 [00:00<00:00, 31.66it/s]
2023-08-17T15:04:36.5065641Z running benchmark:  90% 27/30 [00:00<00:00, 31.91it/s]
2023-08-17T15:04:36.5065982Z running benchmark: 100% 30/30 [00:00<00:00, 30.48it/s]
2023-08-17T15:04:36.5088861Z 2.133x
2023-08-17T15:04:41.0218480Z 
2023-08-17T15:04:47.8217818Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:04:47.8218632Z loading model: 0it [00:06, ?it/s]
2023-08-17T15:04:47.8297246Z cuda eval  XLNetLMHeadModel                   
2023-08-17T15:05:18.1474883Z [2023-08-17 15:05:18,145] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-08-17T15:05:35.3564692Z skipping cudagraphs due to multiple devices
2023-08-17T15:05:36.4353224Z 
2023-08-17T15:05:36.6178255Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:05:36.7807033Z running benchmark:   3% 1/30 [00:00<00:05,  5.49it/s]
2023-08-17T15:05:36.9449132Z running benchmark:   7% 2/30 [00:00<00:04,  5.85it/s]
2023-08-17T15:05:37.1074361Z running benchmark:  10% 3/30 [00:00<00:04,  5.96it/s]
2023-08-17T15:05:37.2682330Z running benchmark:  13% 4/30 [00:00<00:04,  6.03it/s]
2023-08-17T15:05:37.4265566Z running benchmark:  17% 5/30 [00:00<00:04,  6.10it/s]
2023-08-17T15:05:37.5855095Z running benchmark:  20% 6/30 [00:00<00:03,  6.17it/s]
2023-08-17T15:05:37.7436965Z running benchmark:  23% 7/30 [00:01<00:03,  6.21it/s]
2023-08-17T15:05:37.9090393Z running benchmark:  27% 8/30 [00:01<00:03,  6.24it/s]
2023-08-17T15:05:38.0728808Z running benchmark:  30% 9/30 [00:01<00:03,  6.18it/s]
2023-08-17T15:05:38.2356776Z running benchmark:  33% 10/30 [00:01<00:03,  6.16it/s]
2023-08-17T15:05:38.3982473Z running benchmark:  37% 11/30 [00:01<00:03,  6.15it/s]
2023-08-17T15:05:38.5577104Z running benchmark:  40% 12/30 [00:01<00:02,  6.15it/s]
2023-08-17T15:05:38.7154130Z running benchmark:  43% 13/30 [00:02<00:02,  6.19it/s]
2023-08-17T15:05:38.8844343Z running benchmark:  47% 14/30 [00:02<00:02,  6.23it/s]
2023-08-17T15:05:39.0450290Z running benchmark:  50% 15/30 [00:02<00:02,  6.14it/s]
2023-08-17T15:05:39.2053770Z running benchmark:  53% 16/30 [00:02<00:02,  6.16it/s]
2023-08-17T15:05:39.3640302Z running benchmark:  57% 17/30 [00:02<00:02,  6.18it/s]
2023-08-17T15:05:39.5223471Z running benchmark:  60% 18/30 [00:02<00:01,  6.22it/s]
2023-08-17T15:05:39.6799821Z running benchmark:  63% 19/30 [00:03<00:01,  6.25it/s]
2023-08-17T15:05:39.8389073Z running benchmark:  67% 20/30 [00:03<00:01,  6.28it/s]
2023-08-17T15:05:39.9976211Z running benchmark:  70% 21/30 [00:03<00:01,  6.28it/s]
2023-08-17T15:05:40.1560022Z running benchmark:  73% 22/30 [00:03<00:01,  6.29it/s]
2023-08-17T15:05:40.3159095Z running benchmark:  77% 23/30 [00:03<00:01,  6.29it/s]
2023-08-17T15:05:40.4737892Z running benchmark:  80% 24/30 [00:03<00:00,  6.28it/s]
2023-08-17T15:05:40.6317866Z running benchmark:  83% 25/30 [00:04<00:00,  6.30it/s]
2023-08-17T15:05:40.7895642Z running benchmark:  87% 26/30 [00:04<00:00,  6.31it/s]
2023-08-17T15:05:40.9589629Z running benchmark:  90% 27/30 [00:04<00:00,  6.32it/s]
2023-08-17T15:05:41.1168856Z running benchmark:  93% 28/30 [00:04<00:00,  6.19it/s]
2023-08-17T15:05:41.2741273Z running benchmark:  97% 29/30 [00:04<00:00,  6.23it/s]
2023-08-17T15:05:41.2745242Z running benchmark: 100% 30/30 [00:04<00:00,  6.27it/s]
2023-08-17T15:05:41.2745571Z running benchmark: 100% 30/30 [00:04<00:00,  6.20it/s]
2023-08-17T15:05:41.2774200Z 2.147x
2023-08-17T15:05:46.1142833Z 
2023-08-17T15:05:49.3677699Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:05:49.3678181Z loading model: 0it [00:03, ?it/s]
2023-08-17T15:05:49.3743461Z cuda eval  YituTechConvBert                   
2023-08-17T15:06:21.3990601Z 
2023-08-17T15:06:21.5072540Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:06:21.6111050Z running benchmark:   3% 1/30 [00:00<00:03,  9.18it/s]
2023-08-17T15:06:21.7152005Z running benchmark:  10% 3/30 [00:00<00:01, 14.96it/s]
2023-08-17T15:06:21.8193135Z running benchmark:  17% 5/30 [00:00<00:01, 16.88it/s]
2023-08-17T15:06:21.9366863Z running benchmark:  23% 7/30 [00:00<00:01, 17.80it/s]
2023-08-17T15:06:22.0405322Z running benchmark:  30% 9/30 [00:00<00:01, 17.51it/s]
2023-08-17T15:06:22.1444460Z running benchmark:  37% 11/30 [00:00<00:01, 18.08it/s]
2023-08-17T15:06:22.2484229Z running benchmark:  43% 13/30 [00:00<00:00, 18.45it/s]
2023-08-17T15:06:22.3522675Z running benchmark:  50% 15/30 [00:00<00:00, 18.70it/s]
2023-08-17T15:06:22.4560722Z running benchmark:  57% 17/30 [00:00<00:00, 18.87it/s]
2023-08-17T15:06:22.5602196Z running benchmark:  63% 19/30 [00:01<00:00, 18.99it/s]
2023-08-17T15:06:22.6646884Z running benchmark:  70% 21/30 [00:01<00:00, 19.06it/s]
2023-08-17T15:06:22.7686891Z running benchmark:  77% 23/30 [00:01<00:00, 19.09it/s]
2023-08-17T15:06:22.8724933Z running benchmark:  83% 25/30 [00:01<00:00, 19.13it/s]
2023-08-17T15:06:22.9851922Z running benchmark:  90% 27/30 [00:01<00:00, 19.17it/s]
2023-08-17T15:06:23.0370848Z running benchmark:  97% 29/30 [00:01<00:00, 18.72it/s]
2023-08-17T15:06:23.0371204Z running benchmark: 100% 30/30 [00:01<00:00, 18.30it/s]
2023-08-17T15:06:23.0396480Z 1.681x
2023-08-17T15:06:24.2413408Z speedup             gmean=1.88x mean=1.915x
2023-08-17T15:06:24.2413774Z abs_latency         gmean=16.11x mean=17.823x
2023-08-17T15:06:24.2414654Z compilation_latency mean=34.919 seconds
2023-08-17T15:06:24.2415416Z compression_ratio   mean=1.156x
2023-08-17T15:06:24.2420700Z eager_peak_mem      gmean=1.58x mean=1.798x
2023-08-17T15:06:24.2424145Z dynamo_peak_mem     gmean=1.41x mean=1.597x
2023-08-17T15:06:24.2426857Z calls_captured      gmean=642.21x mean=730.412x
2023-08-17T15:06:24.2429459Z unique_graphs       gmean=1.00x mean=1.000x
2023-08-17T15:06:24.2431722Z graph_breaks        gmean=0.00x mean=0.000x
2023-08-17T15:06:24.2433945Z unique_graph_breaks gmean=0.00x mean=0.000x
2023-08-17T15:06:24.7377566Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *dynamic-true* ]]
2023-08-17T15:06:24.7379370Z + python benchmarks/dynamo/huggingface.py --performance --cold-start-latency --inference --bfloat16 --backend inductor --dynamic-shapes --dynamic-batch-only --device cuda --total-partitions 3 --partition-id 2 --output /var/lib/jenkins/workspace/test/test-reports/inductor_dynamic_huggingface_bfloat16_inference_cuda_performance.csv
2023-08-17T15:06:31.3002819Z 
2023-08-17T15:06:37.9008974Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:06:37.9009520Z loading model: 0it [00:06, ?it/s]
2023-08-17T15:06:37.9095856Z cuda eval  MegatronBertForQuestionAnswering   
2023-08-17T15:07:36.8197060Z 
2023-08-17T15:07:36.9471151Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:07:37.0622454Z running benchmark:   3% 1/30 [00:00<00:03,  7.82it/s]
2023-08-17T15:07:37.1773902Z running benchmark:  10% 3/30 [00:00<00:02, 13.17it/s]
2023-08-17T15:07:37.2929673Z running benchmark:  17% 5/30 [00:00<00:01, 15.04it/s]
2023-08-17T15:07:37.4087823Z running benchmark:  23% 7/30 [00:00<00:01, 15.93it/s]
2023-08-17T15:07:37.5244798Z running benchmark:  30% 9/30 [00:00<00:01, 16.41it/s]
2023-08-17T15:07:37.6397069Z running benchmark:  37% 11/30 [00:00<00:01, 16.70it/s]
2023-08-17T15:07:37.7703289Z running benchmark:  43% 13/30 [00:00<00:01, 16.91it/s]
2023-08-17T15:07:37.8853478Z running benchmark:  50% 15/30 [00:00<00:00, 16.37it/s]
2023-08-17T15:07:38.0007283Z running benchmark:  57% 17/30 [00:01<00:00, 16.67it/s]
2023-08-17T15:07:38.1160693Z running benchmark:  63% 19/30 [00:01<00:00, 16.87it/s]
2023-08-17T15:07:38.2313764Z running benchmark:  70% 21/30 [00:01<00:00, 17.01it/s]
2023-08-17T15:07:38.3466896Z running benchmark:  77% 23/30 [00:01<00:00, 17.11it/s]
2023-08-17T15:07:38.4624798Z running benchmark:  83% 25/30 [00:01<00:00, 17.18it/s]
2023-08-17T15:07:38.5787815Z running benchmark:  90% 27/30 [00:01<00:00, 17.22it/s]
2023-08-17T15:07:38.6367197Z running benchmark:  97% 29/30 [00:01<00:00, 17.21it/s]
2023-08-17T15:07:38.6367680Z running benchmark: 100% 30/30 [00:01<00:00, 16.50it/s]
2023-08-17T15:07:38.6390663Z 1.412x
2023-08-17T15:07:43.7690844Z 
2023-08-17T15:07:45.5987742Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:07:45.5988108Z loading model: 0it [00:01, ?it/s]
2023-08-17T15:07:45.6238896Z cuda eval  MobileBertForMaskedLM              
2023-08-17T15:09:42.6529364Z 
2023-08-17T15:09:42.7811413Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:09:42.9104580Z running benchmark:   3% 1/30 [00:00<00:03,  7.77it/s]
2023-08-17T15:09:43.0373707Z running benchmark:  10% 3/30 [00:00<00:02, 12.29it/s]
2023-08-17T15:09:43.1634703Z running benchmark:  17% 5/30 [00:00<00:01, 13.86it/s]
2023-08-17T15:09:43.2894699Z running benchmark:  23% 7/30 [00:00<00:01, 14.64it/s]
2023-08-17T15:09:43.4155999Z running benchmark:  30% 9/30 [00:00<00:01, 15.08it/s]
2023-08-17T15:09:43.5406251Z running benchmark:  37% 11/30 [00:00<00:01, 15.34it/s]
2023-08-17T15:09:43.6817759Z running benchmark:  43% 13/30 [00:00<00:01, 15.55it/s]
2023-08-17T15:09:43.8061644Z running benchmark:  50% 15/30 [00:01<00:00, 15.08it/s]
2023-08-17T15:09:43.9319775Z running benchmark:  57% 17/30 [00:01<00:00, 15.38it/s]
2023-08-17T15:09:44.0605845Z running benchmark:  63% 19/30 [00:01<00:00, 15.54it/s]
2023-08-17T15:09:44.1875041Z running benchmark:  70% 21/30 [00:01<00:00, 15.54it/s]
2023-08-17T15:09:44.3143558Z running benchmark:  77% 23/30 [00:01<00:00, 15.61it/s]
2023-08-17T15:09:44.4403315Z running benchmark:  83% 25/30 [00:01<00:00, 15.65it/s]
2023-08-17T15:09:44.5659676Z running benchmark:  90% 27/30 [00:01<00:00, 15.72it/s]
2023-08-17T15:09:44.6325389Z running benchmark:  97% 29/30 [00:01<00:00, 15.78it/s]
2023-08-17T15:09:44.6325708Z running benchmark: 100% 30/30 [00:01<00:00, 15.15it/s]
2023-08-17T15:09:44.6362667Z 2.066x
2023-08-17T15:09:49.8212950Z 
2023-08-17T15:09:51.4003482Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:09:51.4247515Z loading model: 0it [00:01, ?it/s]
2023-08-17T15:09:51.4248096Z cuda eval  MobileBertForQuestionAnswering     
2023-08-17T15:11:46.4187611Z 
2023-08-17T15:11:46.5592215Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:11:46.6976387Z running benchmark:   7% 2/30 [00:00<00:01, 14.17it/s]
2023-08-17T15:11:46.8352117Z running benchmark:  17% 5/30 [00:00<00:01, 18.53it/s]
2023-08-17T15:11:46.9784012Z running benchmark:  27% 8/30 [00:00<00:01, 20.01it/s]
2023-08-17T15:11:47.1207358Z running benchmark:  37% 11/30 [00:00<00:00, 20.40it/s]
2023-08-17T15:11:47.2594226Z running benchmark:  47% 14/30 [00:00<00:00, 20.64it/s]
2023-08-17T15:11:47.3965880Z running benchmark:  57% 17/30 [00:00<00:00, 20.97it/s]
2023-08-17T15:11:47.5353238Z running benchmark:  67% 20/30 [00:00<00:00, 21.26it/s]
2023-08-17T15:11:47.6723147Z running benchmark:  77% 23/30 [00:01<00:00, 21.38it/s]
2023-08-17T15:11:47.8097004Z running benchmark:  87% 26/30 [00:01<00:00, 21.54it/s]
2023-08-17T15:11:47.8553233Z running benchmark:  97% 29/30 [00:01<00:00, 21.63it/s]
2023-08-17T15:11:47.8553632Z running benchmark: 100% 30/30 [00:01<00:00, 20.87it/s]
2023-08-17T15:11:47.8589143Z 2.370x
2023-08-17T15:11:53.0370390Z 
2023-08-17T15:11:56.7847774Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:11:56.7848108Z loading model: 0it [00:03, ?it/s]
2023-08-17T15:11:56.7888579Z cuda eval  OPTForCausalLM                     
2023-08-17T15:12:30.4443429Z 
2023-08-17T15:12:30.5503430Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:12:30.6559865Z running benchmark:   3% 1/30 [00:00<00:03,  9.41it/s]
2023-08-17T15:12:30.7612920Z running benchmark:  10% 3/30 [00:00<00:01, 14.97it/s]
2023-08-17T15:12:30.8643381Z running benchmark:  17% 5/30 [00:00<00:01, 16.79it/s]
2023-08-17T15:12:30.9679547Z running benchmark:  23% 7/30 [00:00<00:01, 17.81it/s]
2023-08-17T15:12:31.0731232Z running benchmark:  30% 9/30 [00:00<00:01, 18.35it/s]
2023-08-17T15:12:31.1765477Z running benchmark:  37% 11/30 [00:00<00:01, 18.58it/s]
2023-08-17T15:12:31.2825202Z running benchmark:  43% 13/30 [00:00<00:00, 18.82it/s]
2023-08-17T15:12:31.3880448Z running benchmark:  50% 15/30 [00:00<00:00, 18.84it/s]
2023-08-17T15:12:31.4912427Z running benchmark:  57% 17/30 [00:00<00:00, 18.87it/s]
2023-08-17T15:12:31.5990238Z running benchmark:  63% 19/30 [00:01<00:00, 19.03it/s]
2023-08-17T15:12:31.7072179Z running benchmark:  70% 21/30 [00:01<00:00, 18.89it/s]
2023-08-17T15:12:31.8137810Z running benchmark:  77% 23/30 [00:01<00:00, 18.75it/s]
2023-08-17T15:12:31.9197626Z running benchmark:  83% 25/30 [00:01<00:00, 18.76it/s]
2023-08-17T15:12:32.0251100Z running benchmark:  90% 27/30 [00:01<00:00, 18.79it/s]
2023-08-17T15:12:32.0800758Z running benchmark:  97% 29/30 [00:01<00:00, 18.85it/s]
2023-08-17T15:12:32.0801170Z running benchmark: 100% 30/30 [00:01<00:00, 18.33it/s]
2023-08-17T15:12:32.0823496Z 2.451x
2023-08-17T15:12:36.4953156Z 
2023-08-17T15:12:39.5685432Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:12:39.5685773Z loading model: 0it [00:03, ?it/s]
2023-08-17T15:12:39.5716328Z cuda eval  PLBartForCausalLM                  
2023-08-17T15:13:01.3554026Z 
2023-08-17T15:13:01.4677848Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:13:01.5712292Z running benchmark:   3% 1/30 [00:00<00:03,  8.86it/s]
2023-08-17T15:13:01.6738729Z running benchmark:  10% 3/30 [00:00<00:01, 14.79it/s]
2023-08-17T15:13:01.7746063Z running benchmark:  17% 5/30 [00:00<00:01, 16.88it/s]
2023-08-17T15:13:01.8753466Z running benchmark:  23% 7/30 [00:00<00:01, 18.02it/s]
2023-08-17T15:13:01.9764804Z running benchmark:  30% 9/30 [00:00<00:01, 18.68it/s]
2023-08-17T15:13:02.0773028Z running benchmark:  37% 11/30 [00:00<00:00, 19.05it/s]
2023-08-17T15:13:02.1774460Z running benchmark:  43% 13/30 [00:00<00:00, 19.30it/s]
2023-08-17T15:13:02.2959874Z running benchmark:  50% 15/30 [00:00<00:00, 19.51it/s]
2023-08-17T15:13:02.3978164Z running benchmark:  57% 17/30 [00:00<00:00, 18.59it/s]
2023-08-17T15:13:02.5474042Z running benchmark:  63% 19/30 [00:01<00:00, 18.91it/s]
2023-08-17T15:13:02.6490219Z running benchmark:  73% 22/30 [00:01<00:00, 19.35it/s]
2023-08-17T15:13:02.7550021Z running benchmark:  80% 24/30 [00:01<00:00, 19.44it/s]
2023-08-17T15:13:02.8558470Z running benchmark:  87% 26/30 [00:01<00:00, 19.28it/s]
2023-08-17T15:13:02.9628138Z running benchmark:  93% 28/30 [00:01<00:00, 19.43it/s]
2023-08-17T15:13:02.9628816Z running benchmark: 100% 30/30 [00:01<00:00, 19.21it/s]
2023-08-17T15:13:02.9629099Z running benchmark: 100% 30/30 [00:01<00:00, 18.65it/s]
2023-08-17T15:13:02.9654503Z 1.580x
2023-08-17T15:13:07.2953129Z 
2023-08-17T15:13:12.4370153Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:13:12.4370889Z loading model: 0it [00:05, ?it/s]
2023-08-17T15:13:12.4439365Z cuda eval  PLBartForConditionalGeneration     
2023-08-17T15:13:53.2227209Z 
2023-08-17T15:13:53.3233955Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:13:53.4690831Z running benchmark:   3% 1/30 [00:00<00:02,  9.87it/s]
2023-08-17T15:13:53.5587209Z running benchmark:  13% 4/30 [00:00<00:01, 18.29it/s]
2023-08-17T15:13:53.6895206Z running benchmark:  20% 6/30 [00:00<00:01, 18.68it/s]
2023-08-17T15:13:53.8204114Z running benchmark:  30% 9/30 [00:00<00:01, 20.52it/s]
2023-08-17T15:13:53.9518773Z running benchmark:  40% 12/30 [00:00<00:00, 21.43it/s]
2023-08-17T15:13:54.0845629Z running benchmark:  50% 15/30 [00:00<00:00, 21.93it/s]
2023-08-17T15:13:54.2172244Z running benchmark:  60% 18/30 [00:00<00:00, 22.16it/s]
2023-08-17T15:13:54.3507595Z running benchmark:  70% 21/30 [00:00<00:00, 22.31it/s]
2023-08-17T15:13:54.4995773Z running benchmark:  80% 24/30 [00:01<00:00, 22.36it/s]
2023-08-17T15:13:54.6330311Z running benchmark:  90% 27/30 [00:01<00:00, 21.63it/s]
2023-08-17T15:13:54.6330726Z running benchmark: 100% 30/30 [00:01<00:00, 21.87it/s]
2023-08-17T15:13:54.6331038Z running benchmark: 100% 30/30 [00:01<00:00, 21.26it/s]
2023-08-17T15:13:54.6355363Z 1.328x
2023-08-17T15:13:59.0892827Z 
2023-08-17T15:14:07.5304999Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:14:07.5305675Z loading model: 0it [00:08, ?it/s]
2023-08-17T15:14:07.5371086Z cuda eval  PegasusForCausalLM                 
2023-08-17T15:14:40.7454644Z 
2023-08-17T15:14:40.8713861Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:14:40.9866772Z running benchmark:   7% 2/30 [00:00<00:01, 15.83it/s]
2023-08-17T15:14:41.0976585Z running benchmark:  17% 5/30 [00:00<00:01, 21.58it/s]
2023-08-17T15:14:41.2093884Z running benchmark:  27% 8/30 [00:00<00:00, 23.96it/s]
2023-08-17T15:14:41.3213942Z running benchmark:  37% 11/30 [00:00<00:00, 25.08it/s]
2023-08-17T15:14:41.4329397Z running benchmark:  47% 14/30 [00:00<00:00, 25.69it/s]
2023-08-17T15:14:41.5443938Z running benchmark:  57% 17/30 [00:00<00:00, 26.09it/s]
2023-08-17T15:14:41.6723372Z running benchmark:  67% 20/30 [00:00<00:00, 26.36it/s]
2023-08-17T15:14:41.7836305Z running benchmark:  77% 23/30 [00:00<00:00, 25.35it/s]
2023-08-17T15:14:41.8949028Z running benchmark:  87% 26/30 [00:01<00:00, 25.84it/s]
2023-08-17T15:14:41.9320537Z running benchmark:  97% 29/30 [00:01<00:00, 26.17it/s]
2023-08-17T15:14:41.9321109Z running benchmark: 100% 30/30 [00:01<00:00, 25.27it/s]
2023-08-17T15:14:41.9349800Z 1.212x
2023-08-17T15:14:46.4107766Z 
2023-08-17T15:14:59.8929453Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:14:59.8929900Z loading model: 0it [00:13, ?it/s]
2023-08-17T15:14:59.9034707Z cuda eval  PegasusForConditionalGeneration    
2023-08-17T15:16:10.6077871Z 
2023-08-17T15:16:10.7730338Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:16:10.9133051Z running benchmark:   3% 1/30 [00:00<00:04,  6.03it/s]
2023-08-17T15:16:11.0537263Z running benchmark:  10% 3/30 [00:00<00:02, 10.51it/s]
2023-08-17T15:16:11.1940257Z running benchmark:  17% 5/30 [00:00<00:02, 12.15it/s]
2023-08-17T15:16:11.3344378Z running benchmark:  23% 7/30 [00:00<00:01, 12.96it/s]
2023-08-17T15:16:11.4747980Z running benchmark:  30% 9/30 [00:00<00:01, 13.41it/s]
2023-08-17T15:16:11.6150211Z running benchmark:  37% 11/30 [00:00<00:01, 13.70it/s]
2023-08-17T15:16:11.7734696Z running benchmark:  43% 13/30 [00:01<00:01, 13.88it/s]
2023-08-17T15:16:11.9150699Z running benchmark:  50% 15/30 [00:01<00:01, 13.46it/s]
2023-08-17T15:16:12.0562797Z running benchmark:  57% 17/30 [00:01<00:00, 13.66it/s]
2023-08-17T15:16:12.1970445Z running benchmark:  63% 19/30 [00:01<00:00, 13.81it/s]
2023-08-17T15:16:12.3379339Z running benchmark:  70% 21/30 [00:01<00:00, 13.93it/s]
2023-08-17T15:16:12.4784603Z running benchmark:  77% 23/30 [00:01<00:00, 14.01it/s]
2023-08-17T15:16:12.6189343Z running benchmark:  83% 25/30 [00:01<00:00, 14.07it/s]
2023-08-17T15:16:12.7634266Z running benchmark:  90% 27/30 [00:02<00:00, 14.12it/s]
2023-08-17T15:16:12.8339150Z running benchmark:  97% 29/30 [00:02<00:00, 14.04it/s]
2023-08-17T15:16:12.8339707Z running benchmark: 100% 30/30 [00:02<00:00, 13.47it/s]
2023-08-17T15:16:12.8365489Z 1.170x
2023-08-17T15:16:17.7505357Z 
2023-08-17T15:16:17.7536031Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`
2023-08-17T15:16:20.8467810Z 
2023-08-17T15:16:20.8468394Z loading model: 0it [00:03, ?it/s]
2023-08-17T15:16:20.8515615Z cuda eval  RobertaForCausalLM                 
2023-08-17T15:16:55.6872499Z 
2023-08-17T15:16:55.7886199Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:16:55.9179805Z running benchmark:   3% 1/30 [00:00<00:02,  9.82it/s]
2023-08-17T15:16:56.0474370Z running benchmark:  13% 4/30 [00:00<00:01, 18.41it/s]
2023-08-17T15:16:56.1769360Z running benchmark:  23% 7/30 [00:00<00:01, 20.69it/s]
2023-08-17T15:16:56.3068789Z running benchmark:  33% 10/30 [00:00<00:00, 21.70it/s]
2023-08-17T15:16:56.4367874Z running benchmark:  43% 13/30 [00:00<00:00, 22.22it/s]
2023-08-17T15:16:56.5661632Z running benchmark:  53% 16/30 [00:00<00:00, 22.52it/s]
2023-08-17T15:16:56.6957839Z running benchmark:  63% 19/30 [00:00<00:00, 22.73it/s]
2023-08-17T15:16:56.8256524Z running benchmark:  73% 22/30 [00:01<00:00, 22.86it/s]
2023-08-17T15:16:56.9549087Z running benchmark:  83% 25/30 [00:01<00:00, 22.95it/s]
2023-08-17T15:16:57.0413242Z running benchmark:  93% 28/30 [00:01<00:00, 23.02it/s]
2023-08-17T15:16:57.0413580Z running benchmark: 100% 30/30 [00:01<00:00, 22.14it/s]
2023-08-17T15:16:57.0438616Z 1.561x
2023-08-17T15:17:01.4670370Z 
2023-08-17T15:17:04.1972379Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:17:04.1972715Z loading model: 0it [00:02, ?it/s]
2023-08-17T15:17:04.2018060Z cuda eval  RobertaForQuestionAnswering        
2023-08-17T15:17:37.9898491Z 
2023-08-17T15:17:38.0971221Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:17:38.2000089Z running benchmark:   7% 2/30 [00:00<00:01, 18.55it/s]
2023-08-17T15:17:38.3032318Z running benchmark:  17% 5/30 [00:00<00:01, 24.64it/s]
2023-08-17T15:17:38.4063414Z running benchmark:  27% 8/30 [00:00<00:00, 26.63it/s]
2023-08-17T15:17:38.5096024Z running benchmark:  37% 11/30 [00:00<00:00, 27.60it/s]
2023-08-17T15:17:38.6129506Z running benchmark:  47% 14/30 [00:00<00:00, 28.13it/s]
2023-08-17T15:17:38.7158935Z running benchmark:  57% 17/30 [00:00<00:00, 28.43it/s]
2023-08-17T15:17:38.8396236Z running benchmark:  67% 20/30 [00:00<00:00, 28.65it/s]
2023-08-17T15:17:38.9426700Z running benchmark:  77% 23/30 [00:00<00:00, 27.08it/s]
2023-08-17T15:17:39.0457428Z running benchmark:  87% 26/30 [00:00<00:00, 27.69it/s]
2023-08-17T15:17:39.0801579Z running benchmark:  97% 29/30 [00:01<00:00, 28.11it/s]
2023-08-17T15:17:39.0801895Z running benchmark: 100% 30/30 [00:01<00:00, 27.49it/s]
2023-08-17T15:17:39.0825820Z 1.570x
2023-08-17T15:17:43.4132676Z 
2023-08-17T15:17:44.4723606Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:17:44.4724482Z loading model: 0it [00:01, ?it/s]
2023-08-17T15:17:44.4759126Z cuda eval  Speech2Text2ForCausalLM            
2023-08-17T15:18:04.3617521Z 
2023-08-17T15:18:04.4809255Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:18:04.5954163Z running benchmark:  13% 4/30 [00:00<00:00, 33.52it/s]
2023-08-17T15:18:04.7097602Z running benchmark:  30% 9/30 [00:00<00:00, 39.35it/s]
2023-08-17T15:18:04.8242925Z running benchmark:  47% 14/30 [00:00<00:00, 41.32it/s]
2023-08-17T15:18:04.9385625Z running benchmark:  63% 19/30 [00:00<00:00, 42.25it/s]
2023-08-17T15:18:05.0530048Z running benchmark:  80% 24/30 [00:00<00:00, 42.79it/s]
2023-08-17T15:18:05.0761919Z running benchmark:  97% 29/30 [00:00<00:00, 43.10it/s]
2023-08-17T15:18:05.0762252Z running benchmark: 100% 30/30 [00:00<00:00, 41.97it/s]
2023-08-17T15:18:05.0804542Z 1.832x
2023-08-17T15:18:09.3033194Z 
2023-08-17T15:18:11.4826531Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:18:11.4826846Z loading model: 0it [00:02, ?it/s]
2023-08-17T15:18:11.4860653Z cuda eval  T5ForConditionalGeneration         
2023-08-17T15:18:51.7375650Z 
2023-08-17T15:18:51.8470236Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:18:51.9616297Z running benchmark:   7% 2/30 [00:00<00:01, 18.10it/s]
2023-08-17T15:18:52.0765422Z running benchmark:  17% 5/30 [00:00<00:01, 22.88it/s]
2023-08-17T15:18:52.1914513Z running benchmark:  27% 8/30 [00:00<00:00, 24.37it/s]
2023-08-17T15:18:52.3062790Z running benchmark:  37% 11/30 [00:00<00:00, 25.06it/s]
2023-08-17T15:18:52.4211942Z running benchmark:  47% 14/30 [00:00<00:00, 25.44it/s]
2023-08-17T15:18:52.5559872Z running benchmark:  57% 17/30 [00:00<00:00, 25.67it/s]
2023-08-17T15:18:52.6708043Z running benchmark:  67% 20/30 [00:00<00:00, 24.45it/s]
2023-08-17T15:18:52.7856622Z running benchmark:  77% 23/30 [00:00<00:00, 24.95it/s]
2023-08-17T15:18:52.9004329Z running benchmark:  87% 26/30 [00:01<00:00, 25.30it/s]
2023-08-17T15:18:52.9388119Z running benchmark:  97% 29/30 [00:01<00:00, 25.55it/s]
2023-08-17T15:18:52.9388684Z running benchmark: 100% 30/30 [00:01<00:00, 24.94it/s]
2023-08-17T15:18:52.9414477Z 2.312x
2023-08-17T15:18:57.3860982Z 
2023-08-17T15:18:59.6735094Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:18:59.6735476Z loading model: 0it [00:02, ?it/s]
2023-08-17T15:18:59.6786256Z cuda eval  T5Small                            
2023-08-17T15:19:40.0680128Z 
2023-08-17T15:19:40.1777476Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:19:40.2926107Z running benchmark:   7% 2/30 [00:00<00:01, 18.13it/s]
2023-08-17T15:19:40.4075370Z running benchmark:  17% 5/30 [00:00<00:01, 22.87it/s]
2023-08-17T15:19:40.5224357Z running benchmark:  27% 8/30 [00:00<00:00, 24.36it/s]
2023-08-17T15:19:40.6398009Z running benchmark:  37% 11/30 [00:00<00:00, 25.05it/s]
2023-08-17T15:19:40.7547949Z running benchmark:  47% 14/30 [00:00<00:00, 25.25it/s]
2023-08-17T15:19:40.8699668Z running benchmark:  57% 17/30 [00:00<00:00, 25.53it/s]
2023-08-17T15:19:40.9853076Z running benchmark:  67% 20/30 [00:00<00:00, 25.70it/s]
2023-08-17T15:19:41.1001768Z running benchmark:  77% 23/30 [00:00<00:00, 25.80it/s]
2023-08-17T15:19:41.2150607Z running benchmark:  87% 26/30 [00:01<00:00, 25.90it/s]
2023-08-17T15:19:41.2534472Z running benchmark:  97% 29/30 [00:01<00:00, 25.96it/s]
2023-08-17T15:19:41.2534807Z running benchmark: 100% 30/30 [00:01<00:00, 25.29it/s]
2023-08-17T15:19:41.2561093Z 2.313x
2023-08-17T15:19:45.7407786Z 
2023-08-17T15:19:51.7545283Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:19:51.7545962Z loading model: 0it [00:06, ?it/s]
2023-08-17T15:19:51.7610296Z cuda eval  TrOCRForCausalLM                   
2023-08-17T15:20:24.6439937Z 
2023-08-17T15:20:24.8128966Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:20:24.9535509Z running benchmark:   3% 1/30 [00:00<00:04,  5.90it/s]
2023-08-17T15:20:25.0948977Z running benchmark:  10% 3/30 [00:00<00:02, 10.40it/s]
2023-08-17T15:20:25.2350124Z running benchmark:  17% 5/30 [00:00<00:02, 12.04it/s]
2023-08-17T15:20:25.3749714Z running benchmark:  23% 7/30 [00:00<00:01, 12.89it/s]
2023-08-17T15:20:25.5147435Z running benchmark:  30% 9/30 [00:00<00:01, 13.39it/s]
2023-08-17T15:20:25.6566397Z running benchmark:  37% 11/30 [00:00<00:01, 13.69it/s]
2023-08-17T15:20:25.7968814Z running benchmark:  43% 13/30 [00:01<00:01, 13.83it/s]
2023-08-17T15:20:25.9370523Z running benchmark:  50% 15/30 [00:01<00:01, 13.96it/s]
2023-08-17T15:20:26.0778489Z running benchmark:  57% 17/30 [00:01<00:00, 14.06it/s]
2023-08-17T15:20:26.2179890Z running benchmark:  63% 19/30 [00:01<00:00, 14.11it/s]
2023-08-17T15:20:26.3580788Z running benchmark:  70% 21/30 [00:01<00:00, 14.15it/s]
2023-08-17T15:20:26.4980007Z running benchmark:  77% 23/30 [00:01<00:00, 14.19it/s]
2023-08-17T15:20:26.6378952Z running benchmark:  83% 25/30 [00:01<00:00, 14.22it/s]
2023-08-17T15:20:26.7796176Z running benchmark:  90% 27/30 [00:01<00:00, 14.24it/s]
2023-08-17T15:20:26.8496395Z running benchmark:  97% 29/30 [00:02<00:00, 14.20it/s]
2023-08-17T15:20:26.8496911Z running benchmark: 100% 30/30 [00:02<00:00, 13.60it/s]
2023-08-17T15:20:26.8526827Z 1.225x
2023-08-17T15:20:31.2055315Z 
2023-08-17T15:20:45.9717908Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:20:45.9718296Z loading model: 0it [00:14, ?it/s]
2023-08-17T15:20:45.9798403Z cuda eval  XGLMForCausalLM                    
2023-08-17T15:21:41.6004917Z 
2023-08-17T15:21:41.7227661Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:21:41.8453395Z running benchmark:  10% 3/30 [00:00<00:01, 24.38it/s]
2023-08-17T15:21:41.9682235Z running benchmark:  23% 7/30 [00:00<00:00, 29.20it/s]
2023-08-17T15:21:42.0938187Z running benchmark:  37% 11/30 [00:00<00:00, 30.73it/s]
2023-08-17T15:21:42.2194847Z running benchmark:  50% 15/30 [00:00<00:00, 31.19it/s]
2023-08-17T15:21:42.3430792Z running benchmark:  63% 19/30 [00:00<00:00, 31.41it/s]
2023-08-17T15:21:42.4866652Z running benchmark:  77% 23/30 [00:00<00:00, 31.73it/s]
2023-08-17T15:21:42.5779857Z running benchmark:  90% 27/30 [00:00<00:00, 30.34it/s]
2023-08-17T15:21:42.5780216Z running benchmark: 100% 30/30 [00:00<00:00, 30.66it/s]
2023-08-17T15:21:42.5804228Z 2.190x
2023-08-17T15:21:47.1989635Z 
2023-08-17T15:21:54.0160542Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:21:54.0161227Z loading model: 0it [00:06, ?it/s]
2023-08-17T15:21:54.0246560Z cuda eval  XLNetLMHeadModel                   
2023-08-17T15:23:08.8287272Z [2023-08-17 15:23:08,826] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-08-17T15:23:25.7694288Z skipping cudagraphs due to multiple devices
2023-08-17T15:23:27.0207745Z 
2023-08-17T15:23:27.2033226Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:23:27.3619562Z running benchmark:   3% 1/30 [00:00<00:05,  5.48it/s]
2023-08-17T15:23:27.5271558Z running benchmark:   7% 2/30 [00:00<00:04,  5.93it/s]
2023-08-17T15:23:27.6890508Z running benchmark:  10% 3/30 [00:00<00:04,  5.99it/s]
2023-08-17T15:23:27.8491079Z running benchmark:  13% 4/30 [00:00<00:04,  6.06it/s]
2023-08-17T15:23:28.0079473Z running benchmark:  17% 5/30 [00:00<00:04,  6.13it/s]
2023-08-17T15:23:28.1654422Z running benchmark:  20% 6/30 [00:00<00:03,  6.19it/s]
2023-08-17T15:23:28.3243528Z running benchmark:  23% 7/30 [00:01<00:03,  6.24it/s]
2023-08-17T15:23:28.4815630Z running benchmark:  27% 8/30 [00:01<00:03,  6.25it/s]
2023-08-17T15:23:28.6426558Z running benchmark:  30% 9/30 [00:01<00:03,  6.29it/s]
2023-08-17T15:23:28.8005251Z running benchmark:  33% 10/30 [00:01<00:03,  6.26it/s]
2023-08-17T15:23:28.9588651Z running benchmark:  37% 11/30 [00:01<00:03,  6.28it/s]
2023-08-17T15:23:29.1168156Z running benchmark:  40% 12/30 [00:01<00:02,  6.29it/s]
2023-08-17T15:23:29.2741208Z running benchmark:  43% 13/30 [00:02<00:02,  6.30it/s]
2023-08-17T15:23:29.4313893Z running benchmark:  47% 14/30 [00:02<00:02,  6.32it/s]
2023-08-17T15:23:29.5884157Z running benchmark:  50% 15/30 [00:02<00:02,  6.33it/s]
2023-08-17T15:23:29.7490314Z running benchmark:  53% 16/30 [00:02<00:02,  6.34it/s]
2023-08-17T15:23:29.9080570Z running benchmark:  57% 17/30 [00:02<00:02,  6.31it/s]
2023-08-17T15:23:30.0655824Z running benchmark:  60% 18/30 [00:02<00:01,  6.30it/s]
2023-08-17T15:23:30.2229507Z running benchmark:  63% 19/30 [00:03<00:01,  6.32it/s]
2023-08-17T15:23:30.3808796Z running benchmark:  67% 20/30 [00:03<00:01,  6.33it/s]
2023-08-17T15:23:30.5395716Z running benchmark:  70% 21/30 [00:03<00:01,  6.33it/s]
2023-08-17T15:23:30.6973480Z running benchmark:  73% 22/30 [00:03<00:01,  6.32it/s]
2023-08-17T15:23:30.8612640Z running benchmark:  77% 23/30 [00:03<00:01,  6.33it/s]
2023-08-17T15:23:31.0204551Z running benchmark:  80% 24/30 [00:03<00:00,  6.26it/s]
2023-08-17T15:23:31.1770837Z running benchmark:  83% 25/30 [00:03<00:00,  6.27it/s]
2023-08-17T15:23:31.3344544Z running benchmark:  87% 26/30 [00:04<00:00,  6.30it/s]
2023-08-17T15:23:31.4922172Z running benchmark:  90% 27/30 [00:04<00:00,  6.32it/s]
2023-08-17T15:23:31.6499542Z running benchmark:  93% 28/30 [00:04<00:00,  6.32it/s]
2023-08-17T15:23:31.8077331Z running benchmark:  97% 29/30 [00:04<00:00,  6.33it/s]
2023-08-17T15:23:31.8077662Z running benchmark: 100% 30/30 [00:04<00:00,  6.33it/s]
2023-08-17T15:23:31.8081867Z running benchmark: 100% 30/30 [00:04<00:00,  6.27it/s]
2023-08-17T15:23:31.8111552Z 2.141x
2023-08-17T15:23:36.7879636Z 
2023-08-17T15:23:39.9670722Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:23:39.9671217Z loading model: 0it [00:03, ?it/s]
2023-08-17T15:23:39.9735026Z cuda eval  YituTechConvBert                   
2023-08-17T15:24:28.5658127Z 
2023-08-17T15:24:28.6740095Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:24:28.7780834Z running benchmark:   3% 1/30 [00:00<00:03,  9.18it/s]
2023-08-17T15:24:28.8818802Z running benchmark:  10% 3/30 [00:00<00:01, 14.96it/s]
2023-08-17T15:24:28.9858126Z running benchmark:  17% 5/30 [00:00<00:01, 16.90it/s]
2023-08-17T15:24:29.0897061Z running benchmark:  23% 7/30 [00:00<00:01, 17.82it/s]
2023-08-17T15:24:29.1938830Z running benchmark:  30% 9/30 [00:00<00:01, 18.34it/s]
2023-08-17T15:24:29.2982925Z running benchmark:  37% 11/30 [00:00<00:01, 18.63it/s]
2023-08-17T15:24:29.4062894Z running benchmark:  43% 13/30 [00:00<00:00, 18.80it/s]
2023-08-17T15:24:29.5100587Z running benchmark:  50% 15/30 [00:00<00:00, 18.71it/s]
2023-08-17T15:24:29.6139372Z running benchmark:  57% 17/30 [00:00<00:00, 18.88it/s]
2023-08-17T15:24:29.7177781Z running benchmark:  63% 19/30 [00:01<00:00, 18.99it/s]
2023-08-17T15:24:29.8216240Z running benchmark:  70% 21/30 [00:01<00:00, 19.07it/s]
2023-08-17T15:24:29.9256585Z running benchmark:  77% 23/30 [00:01<00:00, 19.13it/s]
2023-08-17T15:24:30.0295968Z running benchmark:  83% 25/30 [00:01<00:00, 19.16it/s]
2023-08-17T15:24:30.1334123Z running benchmark:  90% 27/30 [00:01<00:00, 19.19it/s]
2023-08-17T15:24:30.1854094Z running benchmark:  97% 29/30 [00:01<00:00, 19.21it/s]
2023-08-17T15:24:30.1854420Z running benchmark: 100% 30/30 [00:01<00:00, 18.51it/s]
2023-08-17T15:24:30.1879366Z 1.676x
2023-08-17T15:24:31.5098901Z speedup             gmean=1.73x mean=1.789x
2023-08-17T15:24:31.5099639Z abs_latency         gmean=17.45x mean=19.461x
2023-08-17T15:24:31.5100465Z compilation_latency mean=51.947 seconds
2023-08-17T15:24:31.5102964Z compression_ratio   mean=1.232x
2023-08-17T15:24:31.5106306Z eager_peak_mem      gmean=1.58x mean=1.798x
2023-08-17T15:24:31.5108463Z dynamo_peak_mem     gmean=1.33x mean=1.484x
2023-08-17T15:24:31.5111221Z calls_captured      gmean=865.89x mean=965.471x
2023-08-17T15:24:31.5114696Z unique_graphs       gmean=1.00x mean=1.000x
2023-08-17T15:24:31.5117142Z graph_breaks        gmean=0.00x mean=0.000x
2023-08-17T15:24:31.5118560Z unique_graph_breaks gmean=0.00x mean=0.000x
2023-08-17T15:24:32.0051495Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *cppwrapper-true* ]]
2023-08-17T15:24:32.0053748Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *freezing_cudagraphs-true* ]]
2023-08-17T15:24:32.0055125Z + [[ inference == \i\n\f\e\r\e\n\c\e ]]
2023-08-17T15:24:32.0057531Z + python benchmarks/dynamo/huggingface.py --performance --cold-start-latency --inference --bfloat16 --backend inductor --device cuda --total-partitions 3 --partition-id 2 --freezing --output /var/lib/jenkins/workspace/test/test-reports/inductor_with_cudagraphs_freezing_huggingface_bfloat16_inference_cuda_performance.csv
2023-08-17T15:24:38.6624984Z 
2023-08-17T15:24:45.2951648Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:24:45.2952131Z loading model: 0it [00:06, ?it/s]
2023-08-17T15:24:45.3037709Z cuda eval  MegatronBertForQuestionAnswering   
2023-08-17T15:25:38.7195535Z 
2023-08-17T15:25:38.8418766Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:25:38.9542773Z running benchmark:   3% 1/30 [00:00<00:03,  8.14it/s]
2023-08-17T15:25:39.0665219Z running benchmark:  10% 3/30 [00:00<00:01, 13.59it/s]
2023-08-17T15:25:39.1873229Z running benchmark:  17% 5/30 [00:00<00:01, 15.48it/s]
2023-08-17T15:25:39.2995781Z running benchmark:  23% 7/30 [00:00<00:01, 15.92it/s]
2023-08-17T15:25:39.4117535Z running benchmark:  30% 9/30 [00:00<00:01, 16.58it/s]
2023-08-17T15:25:39.5239625Z running benchmark:  37% 11/30 [00:00<00:01, 17.00it/s]
2023-08-17T15:25:39.6364571Z running benchmark:  43% 13/30 [00:00<00:00, 17.26it/s]
2023-08-17T15:25:39.7488998Z running benchmark:  50% 15/30 [00:00<00:00, 17.43it/s]
2023-08-17T15:25:39.8611637Z running benchmark:  57% 17/30 [00:01<00:00, 17.54it/s]
2023-08-17T15:25:39.9736374Z running benchmark:  63% 19/30 [00:01<00:00, 17.62it/s]
2023-08-17T15:25:40.0861111Z running benchmark:  70% 21/30 [00:01<00:00, 17.68it/s]
2023-08-17T15:25:40.2187637Z running benchmark:  77% 23/30 [00:01<00:00, 17.71it/s]
2023-08-17T15:25:40.3310589Z running benchmark:  83% 25/30 [00:01<00:00, 16.82it/s]
2023-08-17T15:25:40.4433465Z running benchmark:  90% 27/30 [00:01<00:00, 17.10it/s]
2023-08-17T15:25:40.4995761Z running benchmark:  97% 29/30 [00:01<00:00, 17.31it/s]
2023-08-17T15:25:40.4996288Z running benchmark: 100% 30/30 [00:01<00:00, 16.85it/s]
2023-08-17T15:25:40.5019873Z 1.513x
2023-08-17T15:25:45.0491516Z 
2023-08-17T15:25:46.9888399Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:25:46.9888723Z loading model: 0it [00:01, ?it/s]
2023-08-17T15:25:47.0140897Z cuda eval  MobileBertForMaskedLM              
2023-08-17T15:27:38.1856527Z 
2023-08-17T15:27:38.2983268Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:27:38.4187098Z running benchmark:   3% 1/30 [00:00<00:03,  8.72it/s]
2023-08-17T15:27:38.5389169Z running benchmark:  10% 3/30 [00:00<00:02, 13.44it/s]
2023-08-17T15:27:38.6597646Z running benchmark:  17% 5/30 [00:00<00:01, 14.91it/s]
2023-08-17T15:27:38.7810834Z running benchmark:  23% 7/30 [00:00<00:01, 15.57it/s]
2023-08-17T15:27:38.9003675Z running benchmark:  30% 9/30 [00:00<00:01, 15.90it/s]
2023-08-17T15:27:39.0293913Z running benchmark:  37% 11/30 [00:00<00:01, 16.19it/s]
2023-08-17T15:27:39.1474166Z running benchmark:  43% 13/30 [00:00<00:01, 15.95it/s]
2023-08-17T15:27:39.2659051Z running benchmark:  50% 15/30 [00:00<00:00, 16.26it/s]
2023-08-17T15:27:39.3849791Z running benchmark:  57% 17/30 [00:01<00:00, 16.45it/s]
2023-08-17T15:27:39.5044776Z running benchmark:  63% 19/30 [00:01<00:00, 16.56it/s]
2023-08-17T15:27:39.6233668Z running benchmark:  70% 21/30 [00:01<00:00, 16.61it/s]
2023-08-17T15:27:39.7422735Z running benchmark:  77% 23/30 [00:01<00:00, 16.67it/s]
2023-08-17T15:27:39.8615577Z running benchmark:  83% 25/30 [00:01<00:00, 16.72it/s]
2023-08-17T15:27:40.0004409Z running benchmark:  90% 27/30 [00:01<00:00, 16.73it/s]
2023-08-17T15:27:40.0602323Z running benchmark:  97% 29/30 [00:01<00:00, 15.96it/s]
2023-08-17T15:27:40.0602787Z running benchmark: 100% 30/30 [00:01<00:00, 15.98it/s]
2023-08-17T15:27:40.0637343Z 2.550x
2023-08-17T15:27:45.1247117Z 
2023-08-17T15:27:46.7199579Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:27:46.7199886Z loading model: 0it [00:01, ?it/s]
2023-08-17T15:27:46.7438480Z cuda eval  MobileBertForQuestionAnswering     
2023-08-17T15:29:37.4106511Z 
2023-08-17T15:29:37.5418396Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:29:37.6807228Z running benchmark:   7% 2/30 [00:00<00:01, 15.16it/s]
2023-08-17T15:29:37.7893963Z running benchmark:  17% 5/30 [00:00<00:01, 19.03it/s]
2023-08-17T15:29:37.9264382Z running benchmark:  23% 7/30 [00:00<00:01, 18.78it/s]
2023-08-17T15:29:38.0623257Z running benchmark:  33% 10/30 [00:00<00:00, 20.07it/s]
2023-08-17T15:29:38.1981573Z running benchmark:  43% 13/30 [00:00<00:00, 20.82it/s]
2023-08-17T15:29:38.3338620Z running benchmark:  53% 16/30 [00:00<00:00, 21.26it/s]
2023-08-17T15:29:38.4685125Z running benchmark:  63% 19/30 [00:00<00:00, 21.54it/s]
2023-08-17T15:29:38.6033447Z running benchmark:  73% 22/30 [00:01<00:00, 21.77it/s]
2023-08-17T15:29:38.7488999Z running benchmark:  83% 25/30 [00:01<00:00, 21.92it/s]
2023-08-17T15:29:38.8412685Z running benchmark:  93% 28/30 [00:01<00:00, 21.52it/s]
2023-08-17T15:29:38.8413316Z running benchmark: 100% 30/30 [00:01<00:00, 20.96it/s]
2023-08-17T15:29:38.8448400Z 2.698x
2023-08-17T15:29:43.7646958Z 
2023-08-17T15:29:47.4622923Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:29:47.4623412Z loading model: 0it [00:03, ?it/s]
2023-08-17T15:29:47.4665327Z cuda eval  OPTForCausalLM                     
2023-08-17T15:30:17.2791593Z 
2023-08-17T15:30:17.4319110Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:30:17.5336019Z running benchmark:   7% 2/30 [00:00<00:02, 13.06it/s]
2023-08-17T15:30:17.6366940Z running benchmark:  13% 4/30 [00:00<00:01, 16.25it/s]
2023-08-17T15:30:17.7653771Z running benchmark:  20% 6/30 [00:00<00:01, 17.55it/s]
2023-08-17T15:30:17.8699590Z running benchmark:  27% 8/30 [00:00<00:01, 16.72it/s]
2023-08-17T15:30:17.9721160Z running benchmark:  33% 10/30 [00:00<00:01, 17.50it/s]
2023-08-17T15:30:18.0770092Z running benchmark:  40% 12/30 [00:00<00:00, 18.15it/s]
2023-08-17T15:30:18.1828351Z running benchmark:  47% 14/30 [00:00<00:00, 18.44it/s]
2023-08-17T15:30:18.2867988Z running benchmark:  53% 16/30 [00:00<00:00, 18.58it/s]
2023-08-17T15:30:18.3908264Z running benchmark:  60% 18/30 [00:01<00:00, 18.78it/s]
2023-08-17T15:30:18.4942275Z running benchmark:  67% 20/30 [00:01<00:00, 18.92it/s]
2023-08-17T15:30:18.5982312Z running benchmark:  73% 22/30 [00:01<00:00, 19.04it/s]
2023-08-17T15:30:18.7012031Z running benchmark:  80% 24/30 [00:01<00:00, 19.10it/s]
2023-08-17T15:30:18.8032989Z running benchmark:  87% 26/30 [00:01<00:00, 19.21it/s]
2023-08-17T15:30:18.9061435Z running benchmark:  93% 28/30 [00:01<00:00, 19.31it/s]
2023-08-17T15:30:18.9061789Z running benchmark: 100% 30/30 [00:01<00:00, 19.35it/s]
2023-08-17T15:30:18.9067152Z running benchmark: 100% 30/30 [00:01<00:00, 18.43it/s]
2023-08-17T15:30:18.9086748Z 2.429x
2023-08-17T15:30:23.2153205Z 
2023-08-17T15:30:26.3317728Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:30:26.3318231Z loading model: 0it [00:03, ?it/s]
2023-08-17T15:30:26.3349902Z cuda eval  PLBartForCausalLM                  
2023-08-17T15:30:47.2680973Z 
2023-08-17T15:30:47.3726197Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:30:47.5053572Z running benchmark:   3% 1/30 [00:00<00:03,  9.56it/s]
2023-08-17T15:30:47.6402192Z running benchmark:  13% 4/30 [00:00<00:01, 17.91it/s]
2023-08-17T15:30:47.7720429Z running benchmark:  23% 7/30 [00:00<00:01, 20.01it/s]
2023-08-17T15:30:47.9036763Z running benchmark:  33% 10/30 [00:00<00:00, 21.11it/s]
2023-08-17T15:30:48.0346399Z running benchmark:  43% 13/30 [00:00<00:00, 21.73it/s]
2023-08-17T15:30:48.1677421Z running benchmark:  53% 16/30 [00:00<00:00, 22.13it/s]
2023-08-17T15:30:48.3003677Z running benchmark:  63% 19/30 [00:00<00:00, 22.26it/s]
2023-08-17T15:30:48.4449978Z running benchmark:  73% 22/30 [00:01<00:00, 22.38it/s]
2023-08-17T15:30:48.5807327Z running benchmark:  83% 25/30 [00:01<00:00, 21.84it/s]
2023-08-17T15:30:48.6706868Z running benchmark:  93% 28/30 [00:01<00:00, 21.91it/s]
2023-08-17T15:30:48.6707241Z running benchmark: 100% 30/30 [00:01<00:00, 21.38it/s]
2023-08-17T15:30:48.6730498Z 2.347x
2023-08-17T15:30:52.9235785Z 
2023-08-17T15:30:58.0238202Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:30:58.0238752Z loading model: 0it [00:05, ?it/s]
2023-08-17T15:30:58.0289335Z cuda eval  PLBartForConditionalGeneration     
2023-08-17T15:31:35.2780907Z 
2023-08-17T15:31:35.4082253Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:31:35.5414364Z running benchmark:   7% 2/30 [00:00<00:01, 15.29it/s]
2023-08-17T15:31:35.6641211Z running benchmark:  17% 5/30 [00:00<00:01, 19.56it/s]
2023-08-17T15:31:35.7860099Z running benchmark:  27% 8/30 [00:00<00:01, 21.69it/s]
2023-08-17T15:31:35.9083963Z running benchmark:  37% 11/30 [00:00<00:00, 22.81it/s]
2023-08-17T15:31:36.0331095Z running benchmark:  47% 14/30 [00:00<00:00, 23.42it/s]
2023-08-17T15:31:36.1570973Z running benchmark:  57% 17/30 [00:00<00:00, 23.63it/s]
2023-08-17T15:31:36.2810056Z running benchmark:  67% 20/30 [00:00<00:00, 23.82it/s]
2023-08-17T15:31:36.4036761Z running benchmark:  77% 23/30 [00:01<00:00, 23.94it/s]
2023-08-17T15:31:36.5407009Z running benchmark:  87% 26/30 [00:01<00:00, 24.10it/s]
2023-08-17T15:31:36.5815677Z running benchmark:  97% 29/30 [00:01<00:00, 23.38it/s]
2023-08-17T15:31:36.5816262Z running benchmark: 100% 30/30 [00:01<00:00, 23.00it/s]
2023-08-17T15:31:36.5838676Z 1.612x
2023-08-17T15:31:41.0665482Z 
2023-08-17T15:31:49.3454867Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:31:49.3455208Z loading model: 0it [00:08, ?it/s]
2023-08-17T15:31:49.3528960Z cuda eval  PegasusForCausalLM                 
2023-08-17T15:32:19.4988672Z 
2023-08-17T15:32:19.6049479Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:32:19.7058627Z running benchmark:   7% 2/30 [00:00<00:01, 18.82it/s]
2023-08-17T15:32:19.8076118Z running benchmark:  17% 5/30 [00:00<00:00, 25.01it/s]
2023-08-17T15:32:19.9368867Z running benchmark:  27% 8/30 [00:00<00:00, 27.03it/s]
2023-08-17T15:32:20.0662626Z running benchmark:  40% 12/30 [00:00<00:00, 28.80it/s]
2023-08-17T15:32:20.1958345Z running benchmark:  53% 16/30 [00:00<00:00, 29.64it/s]
2023-08-17T15:32:20.3251914Z running benchmark:  67% 20/30 [00:00<00:00, 30.08it/s]
2023-08-17T15:32:20.4543530Z running benchmark:  80% 24/30 [00:00<00:00, 30.36it/s]
2023-08-17T15:32:20.5189592Z running benchmark:  93% 28/30 [00:00<00:00, 30.56it/s]
2023-08-17T15:32:20.5190159Z running benchmark: 100% 30/30 [00:01<00:00, 29.38it/s]
2023-08-17T15:32:20.5217574Z 1.686x
2023-08-17T15:32:24.8102028Z 
2023-08-17T15:32:38.1692753Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:32:38.1693452Z loading model: 0it [00:13, ?it/s]
2023-08-17T15:32:38.1795280Z cuda eval  PegasusForConditionalGeneration    
2023-08-17T15:33:42.3736447Z 
2023-08-17T15:33:42.5293468Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:33:42.6576356Z running benchmark:   3% 1/30 [00:00<00:04,  6.39it/s]
2023-08-17T15:33:42.7864606Z running benchmark:  10% 3/30 [00:00<00:02, 11.34it/s]
2023-08-17T15:33:42.9151261Z running benchmark:  17% 5/30 [00:00<00:01, 13.16it/s]
2023-08-17T15:33:43.0432574Z running benchmark:  23% 7/30 [00:00<00:01, 14.09it/s]
2023-08-17T15:33:43.1718382Z running benchmark:  30% 9/30 [00:00<00:01, 14.61it/s]
2023-08-17T15:33:43.3014303Z running benchmark:  37% 11/30 [00:00<00:01, 14.93it/s]
2023-08-17T15:33:43.4299447Z running benchmark:  43% 13/30 [00:00<00:01, 15.09it/s]
2023-08-17T15:33:43.5589462Z running benchmark:  50% 15/30 [00:01<00:00, 15.24it/s]
2023-08-17T15:33:43.6876885Z running benchmark:  57% 17/30 [00:01<00:00, 15.33it/s]
2023-08-17T15:33:43.8162219Z running benchmark:  63% 19/30 [00:01<00:00, 15.38it/s]
2023-08-17T15:33:43.9452542Z running benchmark:  70% 21/30 [00:01<00:00, 15.44it/s]
2023-08-17T15:33:44.0735536Z running benchmark:  77% 23/30 [00:01<00:00, 15.46it/s]
2023-08-17T15:33:44.2019449Z running benchmark:  83% 25/30 [00:01<00:00, 15.50it/s]
2023-08-17T15:33:44.3306337Z running benchmark:  90% 27/30 [00:01<00:00, 15.52it/s]
2023-08-17T15:33:44.3953637Z running benchmark:  97% 29/30 [00:01<00:00, 15.53it/s]
2023-08-17T15:33:44.3953977Z running benchmark: 100% 30/30 [00:02<00:00, 14.83it/s]
2023-08-17T15:33:44.3979912Z 1.457x
2023-08-17T15:33:49.1896992Z 
2023-08-17T15:33:49.1930330Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`
2023-08-17T15:33:52.4648124Z 
2023-08-17T15:33:52.4648910Z loading model: 0it [00:03, ?it/s]
2023-08-17T15:33:52.4693659Z cuda eval  RobertaForCausalLM                 
2023-08-17T15:34:23.5012165Z 
2023-08-17T15:34:23.6378580Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:34:23.7667624Z running benchmark:   7% 2/30 [00:00<00:01, 14.60it/s]
2023-08-17T15:34:23.8954781Z running benchmark:  17% 5/30 [00:00<00:01, 19.55it/s]
2023-08-17T15:34:24.0241622Z running benchmark:  27% 8/30 [00:00<00:01, 21.24it/s]
2023-08-17T15:34:24.1528300Z running benchmark:  37% 11/30 [00:00<00:00, 22.05it/s]
2023-08-17T15:34:24.2816849Z running benchmark:  47% 14/30 [00:00<00:00, 22.50it/s]
2023-08-17T15:34:24.4105089Z running benchmark:  57% 17/30 [00:00<00:00, 22.77it/s]
2023-08-17T15:34:24.5498338Z running benchmark:  67% 20/30 [00:00<00:00, 22.94it/s]
2023-08-17T15:34:24.6784534Z running benchmark:  77% 23/30 [00:01<00:00, 22.48it/s]
2023-08-17T15:34:24.8076796Z running benchmark:  87% 26/30 [00:01<00:00, 22.73it/s]
2023-08-17T15:34:24.8507887Z running benchmark:  97% 29/30 [00:01<00:00, 22.88it/s]
2023-08-17T15:34:24.8508659Z running benchmark: 100% 30/30 [00:01<00:00, 22.22it/s]
2023-08-17T15:34:24.8532790Z 1.589x
2023-08-17T15:34:29.1977161Z 
2023-08-17T15:34:32.0106230Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:34:32.0106740Z loading model: 0it [00:02, ?it/s]
2023-08-17T15:34:32.0151098Z cuda eval  RobertaForQuestionAnswering        
2023-08-17T15:35:02.4370942Z 
2023-08-17T15:35:02.5437575Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:35:02.6460614Z running benchmark:   7% 2/30 [00:00<00:01, 18.63it/s]
2023-08-17T15:35:02.7483585Z running benchmark:  17% 5/30 [00:00<00:01, 24.77it/s]
2023-08-17T15:35:02.8512126Z running benchmark:  27% 8/30 [00:00<00:00, 26.82it/s]
2023-08-17T15:35:02.9543182Z running benchmark:  37% 11/30 [00:00<00:00, 27.75it/s]
2023-08-17T15:35:03.0568639Z running benchmark:  47% 14/30 [00:00<00:00, 28.24it/s]
2023-08-17T15:35:03.1595497Z running benchmark:  57% 17/30 [00:00<00:00, 28.58it/s]
2023-08-17T15:35:03.2627992Z running benchmark:  67% 20/30 [00:00<00:00, 28.79it/s]
2023-08-17T15:35:03.3658240Z running benchmark:  77% 23/30 [00:00<00:00, 28.88it/s]
2023-08-17T15:35:03.4685458Z running benchmark:  87% 26/30 [00:00<00:00, 28.95it/s]
2023-08-17T15:35:03.5026727Z running benchmark:  97% 29/30 [00:01<00:00, 29.03it/s]
2023-08-17T15:35:03.5032155Z running benchmark: 100% 30/30 [00:01<00:00, 28.13it/s]
2023-08-17T15:35:03.5054571Z 1.600x
2023-08-17T15:35:07.9172904Z 
2023-08-17T15:35:09.0293534Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:35:09.0293898Z loading model: 0it [00:01, ?it/s]
2023-08-17T15:35:09.0347690Z cuda eval  Speech2Text2ForCausalLM            
2023-08-17T15:35:26.7269760Z 
2023-08-17T15:35:26.8433977Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:35:26.9586048Z running benchmark:  13% 4/30 [00:00<00:00, 34.00it/s]
2023-08-17T15:35:27.0717710Z running benchmark:  27% 8/30 [00:00<00:00, 34.44it/s]
2023-08-17T15:35:27.1846216Z running benchmark:  43% 13/30 [00:00<00:00, 38.78it/s]
2023-08-17T15:35:27.2993326Z running benchmark:  60% 18/30 [00:00<00:00, 40.93it/s]
2023-08-17T15:35:27.4162002Z running benchmark:  77% 23/30 [00:00<00:00, 41.91it/s]
2023-08-17T15:35:27.4624636Z running benchmark:  93% 28/30 [00:00<00:00, 42.20it/s]
2023-08-17T15:35:27.4625011Z running benchmark: 100% 30/30 [00:00<00:00, 40.71it/s]
2023-08-17T15:35:27.4646624Z 1.873x
2023-08-17T15:35:31.7053996Z 
2023-08-17T15:35:33.9003109Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:35:33.9003634Z loading model: 0it [00:02, ?it/s]
2023-08-17T15:35:33.9036753Z cuda eval  T5ForConditionalGeneration         
2023-08-17T15:36:09.8457362Z 
2023-08-17T15:36:09.9605099Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:36:10.0758662Z running benchmark:   7% 2/30 [00:00<00:01, 17.37it/s]
2023-08-17T15:36:10.1913991Z running benchmark:  17% 5/30 [00:00<00:01, 22.42it/s]
2023-08-17T15:36:10.3075870Z running benchmark:  27% 8/30 [00:00<00:00, 24.03it/s]
2023-08-17T15:36:10.4231264Z running benchmark:  37% 11/30 [00:00<00:00, 24.75it/s]
2023-08-17T15:36:10.5386873Z running benchmark:  47% 14/30 [00:00<00:00, 25.18it/s]
2023-08-17T15:36:10.6543018Z running benchmark:  57% 17/30 [00:00<00:00, 25.44it/s]
2023-08-17T15:36:10.7699430Z running benchmark:  67% 20/30 [00:00<00:00, 25.61it/s]
2023-08-17T15:36:10.8857216Z running benchmark:  77% 23/30 [00:00<00:00, 25.72it/s]
2023-08-17T15:36:11.0150888Z running benchmark:  87% 26/30 [00:01<00:00, 25.78it/s]
2023-08-17T15:36:11.0535035Z running benchmark:  97% 29/30 [00:01<00:00, 24.93it/s]
2023-08-17T15:36:11.0535393Z running benchmark: 100% 30/30 [00:01<00:00, 24.82it/s]
2023-08-17T15:36:11.0561533Z 2.279x
2023-08-17T15:36:15.4923355Z 
2023-08-17T15:36:17.6624268Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:36:17.6624774Z loading model: 0it [00:02, ?it/s]
2023-08-17T15:36:17.6658908Z cuda eval  T5Small                            
2023-08-17T15:36:53.8581121Z 
2023-08-17T15:36:53.9872669Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:36:54.1026285Z running benchmark:   7% 2/30 [00:00<00:01, 15.45it/s]
2023-08-17T15:36:54.2183490Z running benchmark:  17% 5/30 [00:00<00:01, 21.32it/s]
2023-08-17T15:36:54.3345906Z running benchmark:  27% 8/30 [00:00<00:00, 23.37it/s]
2023-08-17T15:36:54.4530247Z running benchmark:  37% 11/30 [00:00<00:00, 24.32it/s]
2023-08-17T15:36:54.5660895Z running benchmark:  47% 14/30 [00:00<00:00, 24.88it/s]
2023-08-17T15:36:54.6816476Z running benchmark:  57% 17/30 [00:00<00:00, 25.24it/s]
2023-08-17T15:36:54.7972427Z running benchmark:  67% 20/30 [00:00<00:00, 25.47it/s]
2023-08-17T15:36:54.9127629Z running benchmark:  77% 23/30 [00:00<00:00, 25.62it/s]
2023-08-17T15:36:55.0430332Z running benchmark:  87% 26/30 [00:01<00:00, 25.73it/s]
2023-08-17T15:36:55.0814175Z running benchmark:  97% 29/30 [00:01<00:00, 24.84it/s]
2023-08-17T15:36:55.0814515Z running benchmark: 100% 30/30 [00:01<00:00, 24.51it/s]
2023-08-17T15:36:55.0843471Z 2.280x
2023-08-17T15:36:59.4613521Z 
2023-08-17T15:37:05.6796077Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:37:05.6796579Z loading model: 0it [00:06, ?it/s]
2023-08-17T15:37:05.6858379Z cuda eval  TrOCRForCausalLM                   
2023-08-17T15:37:36.0746631Z 
2023-08-17T15:37:36.2019001Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:37:36.3246393Z running benchmark:   3% 1/30 [00:00<00:03,  7.82it/s]
2023-08-17T15:37:36.4466450Z running benchmark:  10% 3/30 [00:00<00:02, 12.72it/s]
2023-08-17T15:37:36.5681354Z running benchmark:  17% 5/30 [00:00<00:01, 14.37it/s]
2023-08-17T15:37:36.6897487Z running benchmark:  23% 7/30 [00:00<00:01, 15.18it/s]
2023-08-17T15:37:36.8113999Z running benchmark:  30% 9/30 [00:00<00:01, 15.64it/s]
2023-08-17T15:37:36.9328610Z running benchmark:  37% 11/30 [00:00<00:01, 15.91it/s]
2023-08-17T15:37:37.0619046Z running benchmark:  43% 13/30 [00:00<00:01, 16.09it/s]
2023-08-17T15:37:37.1840260Z running benchmark:  50% 15/30 [00:00<00:00, 15.90it/s]
2023-08-17T15:37:37.3054924Z running benchmark:  57% 17/30 [00:01<00:00, 16.04it/s]
2023-08-17T15:37:37.4270274Z running benchmark:  63% 19/30 [00:01<00:00, 16.17it/s]
2023-08-17T15:37:37.5489186Z running benchmark:  70% 21/30 [00:01<00:00, 16.26it/s]
2023-08-17T15:37:37.6705616Z running benchmark:  77% 23/30 [00:01<00:00, 16.30it/s]
2023-08-17T15:37:37.7920523Z running benchmark:  83% 25/30 [00:01<00:00, 16.34it/s]
2023-08-17T15:37:37.9135403Z running benchmark:  90% 27/30 [00:01<00:00, 16.38it/s]
2023-08-17T15:37:37.9818930Z running benchmark:  97% 29/30 [00:01<00:00, 16.40it/s]
2023-08-17T15:37:37.9819287Z running benchmark: 100% 30/30 [00:01<00:00, 15.72it/s]
2023-08-17T15:37:37.9847320Z 1.736x
2023-08-17T15:37:42.2610842Z 
2023-08-17T15:37:57.1135362Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:37:57.1136374Z loading model: 0it [00:14, ?it/s]
2023-08-17T15:37:57.1210738Z cuda eval  XGLMForCausalLM                    
2023-08-17T15:38:44.9091819Z 
2023-08-17T15:38:45.0294135Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:38:45.1500344Z running benchmark:  10% 3/30 [00:00<00:01, 24.82it/s]
2023-08-17T15:38:45.2741256Z running benchmark:  23% 7/30 [00:00<00:00, 29.69it/s]
2023-08-17T15:38:45.3959497Z running benchmark:  37% 11/30 [00:00<00:00, 30.86it/s]
2023-08-17T15:38:45.5161504Z running benchmark:  50% 15/30 [00:00<00:00, 31.64it/s]
2023-08-17T15:38:45.6480585Z running benchmark:  63% 19/30 [00:00<00:00, 32.22it/s]
2023-08-17T15:38:45.7691104Z running benchmark:  77% 23/30 [00:00<00:00, 31.56it/s]
2023-08-17T15:38:45.8598752Z running benchmark:  90% 27/30 [00:00<00:00, 32.03it/s]
2023-08-17T15:38:45.8599233Z running benchmark: 100% 30/30 [00:00<00:00, 31.53it/s]
2023-08-17T15:38:45.8624303Z 2.318x
2023-08-17T15:38:50.3845672Z 
2023-08-17T15:38:57.1980005Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:38:57.1980352Z loading model: 0it [00:06, ?it/s]
2023-08-17T15:38:57.2065494Z cuda eval  XLNetLMHeadModel                   
2023-08-17T15:39:43.1616696Z [2023-08-17 15:39:43,159] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-08-17T15:40:01.3130886Z skipping cudagraphs due to multiple devices
2023-08-17T15:40:02.4148041Z 
2023-08-17T15:40:02.6036170Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:40:02.7670207Z running benchmark:   3% 1/30 [00:00<00:05,  5.30it/s]
2023-08-17T15:40:02.9292426Z running benchmark:   7% 2/30 [00:00<00:04,  5.75it/s]
2023-08-17T15:40:03.0904956Z running benchmark:  10% 3/30 [00:00<00:04,  5.93it/s]
2023-08-17T15:40:03.2534692Z running benchmark:  13% 4/30 [00:00<00:04,  6.04it/s]
2023-08-17T15:40:03.4339038Z running benchmark:  17% 5/30 [00:00<00:04,  6.07it/s]
2023-08-17T15:40:03.6022299Z running benchmark:  20% 6/30 [00:01<00:04,  5.88it/s]
2023-08-17T15:40:03.7645997Z running benchmark:  23% 7/30 [00:01<00:03,  5.90it/s]
2023-08-17T15:40:03.9254358Z running benchmark:  27% 8/30 [00:01<00:03,  5.98it/s]
2023-08-17T15:40:04.0863111Z running benchmark:  30% 9/30 [00:01<00:03,  6.05it/s]
2023-08-17T15:40:04.2481572Z running benchmark:  33% 10/30 [00:01<00:03,  6.10it/s]
2023-08-17T15:40:04.4143505Z running benchmark:  37% 11/30 [00:01<00:03,  6.12it/s]
2023-08-17T15:40:04.5718259Z running benchmark:  40% 12/30 [00:01<00:02,  6.09it/s]
2023-08-17T15:40:04.7291038Z running benchmark:  43% 13/30 [00:02<00:02,  6.17it/s]
2023-08-17T15:40:04.8879089Z running benchmark:  47% 14/30 [00:02<00:02,  6.22it/s]
2023-08-17T15:40:05.0456593Z running benchmark:  50% 15/30 [00:02<00:02,  6.25it/s]
2023-08-17T15:40:05.2035157Z running benchmark:  53% 16/30 [00:02<00:02,  6.27it/s]
2023-08-17T15:40:05.3605889Z running benchmark:  57% 17/30 [00:02<00:02,  6.29it/s]
2023-08-17T15:40:05.5175515Z running benchmark:  60% 18/30 [00:02<00:01,  6.31it/s]
2023-08-17T15:40:05.6744773Z running benchmark:  63% 19/30 [00:03<00:01,  6.33it/s]
2023-08-17T15:40:05.8314462Z running benchmark:  67% 20/30 [00:03<00:01,  6.34it/s]
2023-08-17T15:40:05.9914504Z running benchmark:  70% 21/30 [00:03<00:01,  6.35it/s]
2023-08-17T15:40:06.1490017Z running benchmark:  73% 22/30 [00:03<00:01,  6.32it/s]
2023-08-17T15:40:06.3058005Z running benchmark:  77% 23/30 [00:03<00:01,  6.33it/s]
2023-08-17T15:40:06.4628475Z running benchmark:  80% 24/30 [00:03<00:00,  6.34it/s]
2023-08-17T15:40:06.6198933Z running benchmark:  83% 25/30 [00:04<00:00,  6.35it/s]
2023-08-17T15:40:06.7771907Z running benchmark:  87% 26/30 [00:04<00:00,  6.35it/s]
2023-08-17T15:40:06.9343893Z running benchmark:  90% 27/30 [00:04<00:00,  6.36it/s]
2023-08-17T15:40:07.0939711Z running benchmark:  93% 28/30 [00:04<00:00,  6.36it/s]
2023-08-17T15:40:07.2584507Z running benchmark:  97% 29/30 [00:04<00:00,  6.33it/s]
2023-08-17T15:40:07.2584861Z running benchmark: 100% 30/30 [00:04<00:00,  6.25it/s]
2023-08-17T15:40:07.2590630Z running benchmark: 100% 30/30 [00:04<00:00,  6.19it/s]
2023-08-17T15:40:07.2618704Z 2.134x
2023-08-17T15:40:11.9912970Z 
2023-08-17T15:40:15.2647996Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:40:15.2648482Z loading model: 0it [00:03, ?it/s]
2023-08-17T15:40:15.2715520Z cuda eval  YituTechConvBert                   
2023-08-17T15:41:04.7992698Z 
2023-08-17T15:41:04.9077284Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:41:05.0115354Z running benchmark:   3% 1/30 [00:00<00:03,  9.18it/s]
2023-08-17T15:41:05.1158587Z running benchmark:  10% 3/30 [00:00<00:01, 14.97it/s]
2023-08-17T15:41:05.2197444Z running benchmark:  17% 5/30 [00:00<00:01, 16.87it/s]
2023-08-17T15:41:05.3238537Z running benchmark:  23% 7/30 [00:00<00:01, 17.80it/s]
2023-08-17T15:41:05.4275511Z running benchmark:  30% 9/30 [00:00<00:01, 18.31it/s]
2023-08-17T15:41:05.5311963Z running benchmark:  37% 11/30 [00:00<00:01, 18.64it/s]
2023-08-17T15:41:05.6350941Z running benchmark:  43% 13/30 [00:00<00:00, 18.86it/s]
2023-08-17T15:41:05.7391123Z running benchmark:  50% 15/30 [00:00<00:00, 18.99it/s]
2023-08-17T15:41:05.8530992Z running benchmark:  57% 17/30 [00:00<00:00, 19.06it/s]
2023-08-17T15:41:05.9571778Z running benchmark:  63% 19/30 [00:01<00:00, 18.57it/s]
2023-08-17T15:41:06.0608202Z running benchmark:  70% 21/30 [00:01<00:00, 18.75it/s]
2023-08-17T15:41:06.1644627Z running benchmark:  77% 23/30 [00:01<00:00, 18.91it/s]
2023-08-17T15:41:06.2682027Z running benchmark:  83% 25/30 [00:01<00:00, 19.03it/s]
2023-08-17T15:41:06.3718667Z running benchmark:  90% 27/30 [00:01<00:00, 19.10it/s]
2023-08-17T15:41:06.4238114Z running benchmark:  97% 29/30 [00:01<00:00, 19.16it/s]
2023-08-17T15:41:06.4238443Z running benchmark: 100% 30/30 [00:01<00:00, 18.46it/s]
2023-08-17T15:41:06.4281442Z 1.699x
2023-08-17T15:41:07.6687779Z speedup             gmean=1.95x mean=1.988x
2023-08-17T15:41:07.6688139Z abs_latency         gmean=15.57x mean=17.295x
2023-08-17T15:41:07.6688722Z compilation_latency mean=46.751 seconds
2023-08-17T15:41:07.6690933Z compression_ratio   mean=1.113x
2023-08-17T15:41:07.6693684Z eager_peak_mem      gmean=1.58x mean=1.798x
2023-08-17T15:41:07.6696391Z dynamo_peak_mem     gmean=1.47x mean=1.659x
2023-08-17T15:41:07.6699210Z calls_captured      gmean=642.21x mean=730.412x
2023-08-17T15:41:07.6701753Z unique_graphs       gmean=1.00x mean=1.000x
2023-08-17T15:41:07.6704635Z graph_breaks        gmean=0.00x mean=0.000x
2023-08-17T15:41:07.6707317Z unique_graph_breaks gmean=0.00x mean=0.000x
2023-08-17T15:41:08.1826737Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *aotinductor-true* ]]
2023-08-17T15:41:08.1827507Z + [[ inference == \i\n\f\e\r\e\n\c\e ]]
2023-08-17T15:41:08.1828661Z + python benchmarks/dynamo/huggingface.py --performance --cold-start-latency --inference --bfloat16 --export-aot-inductor --disable-cudagraphs --device cuda --total-partitions 3 --partition-id 2 --output /var/lib/jenkins/workspace/test/test-reports/inductor_aot_inductor_huggingface_bfloat16_inference_cuda_performance.csv
2023-08-17T15:41:14.7481032Z 
2023-08-17T15:41:21.4509979Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:41:21.4510869Z loading model: 0it [00:06, ?it/s]
2023-08-17T15:41:21.4613257Z cuda eval  MegatronBertForQuestionAnswering   
2023-08-17T15:42:47.6467077Z 
2023-08-17T15:42:47.7693907Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:42:47.8851815Z running benchmark:   3% 1/30 [00:00<00:03,  8.11it/s]
2023-08-17T15:42:48.0013931Z running benchmark:  10% 3/30 [00:00<00:02, 13.35it/s]
2023-08-17T15:42:48.1175817Z running benchmark:  17% 5/30 [00:00<00:01, 15.10it/s]
2023-08-17T15:42:48.2336188Z running benchmark:  23% 7/30 [00:00<00:01, 15.92it/s]
2023-08-17T15:42:48.3554514Z running benchmark:  30% 9/30 [00:00<00:01, 16.39it/s]
2023-08-17T15:42:48.4712742Z running benchmark:  37% 11/30 [00:00<00:01, 16.40it/s]
2023-08-17T15:42:48.5870698Z running benchmark:  43% 13/30 [00:00<00:01, 16.68it/s]
2023-08-17T15:42:48.7031015Z running benchmark:  50% 15/30 [00:00<00:00, 16.86it/s]
2023-08-17T15:42:48.8189101Z running benchmark:  57% 17/30 [00:01<00:00, 16.98it/s]
2023-08-17T15:42:48.9347552Z running benchmark:  63% 19/30 [00:01<00:00, 17.07it/s]
2023-08-17T15:42:49.0507101Z running benchmark:  70% 21/30 [00:01<00:00, 17.13it/s]
2023-08-17T15:42:49.1668963Z running benchmark:  77% 23/30 [00:01<00:00, 17.17it/s]
2023-08-17T15:42:49.2831803Z running benchmark:  83% 25/30 [00:01<00:00, 17.18it/s]
2023-08-17T15:42:49.3992306Z running benchmark:  90% 27/30 [00:01<00:00, 17.19it/s]
2023-08-17T15:42:49.4573271Z running benchmark:  97% 29/30 [00:01<00:00, 17.20it/s]
2023-08-17T15:42:49.4573832Z running benchmark: 100% 30/30 [00:01<00:00, 16.56it/s]
2023-08-17T15:42:49.4597421Z 1.387x
2023-08-17T15:42:53.9569483Z 
2023-08-17T15:42:55.7811511Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:42:55.7812225Z loading model: 0it [00:01, ?it/s]
2023-08-17T15:42:55.8061887Z cuda eval  MobileBertForMaskedLM              
2023-08-17T15:46:35.7649034Z 
2023-08-17T15:46:35.8648063Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:46:35.9957214Z running benchmark:   3% 1/30 [00:00<00:02,  9.94it/s]
2023-08-17T15:46:36.1273612Z running benchmark:  10% 3/30 [00:00<00:02, 13.40it/s]
2023-08-17T15:46:36.2563072Z running benchmark:  17% 5/30 [00:00<00:01, 14.26it/s]
2023-08-17T15:46:36.3861206Z running benchmark:  23% 7/30 [00:00<00:01, 14.77it/s]
2023-08-17T15:46:36.5134389Z running benchmark:  30% 9/30 [00:00<00:01, 15.00it/s]
2023-08-17T15:46:36.6447725Z running benchmark:  37% 11/30 [00:00<00:01, 15.24it/s]
2023-08-17T15:46:36.7721971Z running benchmark:  43% 13/30 [00:00<00:01, 15.24it/s]
2023-08-17T15:46:36.9000982Z running benchmark:  50% 15/30 [00:01<00:00, 15.38it/s]
2023-08-17T15:46:37.0291353Z running benchmark:  57% 17/30 [00:01<00:00, 15.47it/s]
2023-08-17T15:46:37.1570499Z running benchmark:  63% 19/30 [00:01<00:00, 15.48it/s]
2023-08-17T15:46:37.2848264Z running benchmark:  70% 21/30 [00:01<00:00, 15.52it/s]
2023-08-17T15:46:37.4144344Z running benchmark:  77% 23/30 [00:01<00:00, 15.56it/s]
2023-08-17T15:46:37.5414848Z running benchmark:  83% 25/30 [00:01<00:00, 15.52it/s]
2023-08-17T15:46:37.6776751Z running benchmark:  90% 27/30 [00:01<00:00, 15.59it/s]
2023-08-17T15:46:37.7420684Z running benchmark:  97% 29/30 [00:01<00:00, 15.31it/s]
2023-08-17T15:46:37.7421148Z running benchmark: 100% 30/30 [00:01<00:00, 15.17it/s]
2023-08-17T15:46:37.7472277Z 2.072x
2023-08-17T15:46:42.5477107Z 
2023-08-17T15:46:44.1195373Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:46:44.1195697Z loading model: 0it [00:01, ?it/s]
2023-08-17T15:46:44.1438270Z cuda eval  MobileBertForQuestionAnswering     
2023-08-17T15:50:19.4415008Z 
2023-08-17T15:50:19.5589195Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:50:19.7002095Z running benchmark:   7% 2/30 [00:00<00:01, 16.92it/s]
2023-08-17T15:50:19.8487636Z running benchmark:  17% 5/30 [00:00<00:01, 19.63it/s]
2023-08-17T15:50:19.9906350Z running benchmark:  27% 8/30 [00:00<00:01, 19.91it/s]
2023-08-17T15:50:20.1320259Z running benchmark:  37% 11/30 [00:00<00:00, 20.40it/s]
2023-08-17T15:50:20.2780836Z running benchmark:  47% 14/30 [00:00<00:00, 20.69it/s]
2023-08-17T15:50:20.4225713Z running benchmark:  57% 17/30 [00:00<00:00, 20.64it/s]
2023-08-17T15:50:20.5657723Z running benchmark:  67% 20/30 [00:00<00:00, 20.69it/s]
2023-08-17T15:50:20.7069981Z running benchmark:  77% 23/30 [00:01<00:00, 20.77it/s]
2023-08-17T15:50:20.8537335Z running benchmark:  87% 26/30 [00:01<00:00, 20.91it/s]
2023-08-17T15:50:20.9011376Z running benchmark:  97% 29/30 [00:01<00:00, 21.02it/s]
2023-08-17T15:50:20.9011683Z running benchmark: 100% 30/30 [00:01<00:00, 20.54it/s]
2023-08-17T15:50:20.9049920Z 2.325x
2023-08-17T15:50:25.8449714Z 
2023-08-17T15:50:29.4076637Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:50:29.4077135Z loading model: 0it [00:03, ?it/s]
2023-08-17T15:50:29.4117460Z cuda eval  OPTForCausalLM                     
2023-08-17T15:50:31.3176416Z ERROR:common:Backend dynamo failed in warmup()
2023-08-17T15:50:31.3177762Z Traceback (most recent call last):
2023-08-17T15:50:31.3178159Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2278, in warmup
2023-08-17T15:50:31.3178587Z     fn(model, example_inputs)
2023-08-17T15:50:31.3180631Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1186, in opt_aot_inductor
2023-08-17T15:50:31.3181348Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-08-17T15:50:31.3182102Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1131, in load
2023-08-17T15:50:31.3182583Z     so_path, exported = torch._export.aot_compile(
2023-08-17T15:50:31.3184388Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 460, in aot_compile
2023-08-17T15:50:31.3184784Z     ep = export(f, args, kwargs, constraints)
2023-08-17T15:50:31.3185307Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 222, in export
2023-08-17T15:50:31.3185707Z     gm_torch_level, _ = torch._dynamo.export(
2023-08-17T15:50:31.3186254Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1201, in inner
2023-08-17T15:50:31.3186636Z     result_traced = opt_f(*args, **kwargs)
2023-08-17T15:50:31.3187181Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T15:50:31.3187584Z     return self._call_impl(*args, **kwargs)
2023-08-17T15:50:31.3188127Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:50:31.3188507Z     return forward_call(*args, **kwargs)
2023-08-17T15:50:31.3189024Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 333, in _fn
2023-08-17T15:50:31.3189368Z     return fn(*args, **kwargs)
2023-08-17T15:50:31.3194909Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T15:50:31.3195689Z     return self._call_impl(*args, **kwargs)
2023-08-17T15:50:31.3196710Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:50:31.3197427Z     return forward_call(*args, **kwargs)
2023-08-17T15:50:31.3198444Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 493, in catch_errors
2023-08-17T15:50:31.3199234Z     return callback(frame, cache_size, hooks, frame_state)
2023-08-17T15:50:31.3199796Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 132, in _fn
2023-08-17T15:50:31.3200160Z     return fn(*args, **kwargs)
2023-08-17T15:50:31.3200714Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 370, in _convert_frame_assert
2023-08-17T15:50:31.3201093Z     return _compile(
2023-08-17T15:50:31.3201592Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 554, in _compile
2023-08-17T15:50:31.3202031Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-08-17T15:50:31.3202747Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 180, in time_wrapper
2023-08-17T15:50:31.3203106Z     r = func(*args, **kwargs)
2023-08-17T15:50:31.3203729Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 465, in compile_inner
2023-08-17T15:50:31.3204818Z     out_code = transform_code_object(code, transform)
2023-08-17T15:50:31.3205951Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-08-17T15:50:31.3206651Z     transformations(instructions, code_options)
2023-08-17T15:50:31.3207512Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 432, in transform
2023-08-17T15:50:31.3208035Z     tracer.run()
2023-08-17T15:50:31.3209125Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2071, in run
2023-08-17T15:50:31.3209685Z     super().run()
2023-08-17T15:50:31.3210466Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:50:31.3210990Z     and self.step()
2023-08-17T15:50:31.3211757Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:50:31.3212372Z     getattr(self, inst.opname)(inst)
2023-08-17T15:50:31.3213191Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:50:31.3213763Z     return inner_fn(self, inst)
2023-08-17T15:50:31.3214700Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1167, in CALL_FUNCTION_KW
2023-08-17T15:50:31.3215377Z     self.call_function(fn, args, kwargs)
2023-08-17T15:50:31.3215979Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:50:31.3216399Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:50:31.3216973Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 331, in call_function
2023-08-17T15:50:31.3217364Z     return tx.inline_user_function_return(
2023-08-17T15:50:31.3217968Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T15:50:31.3218482Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T15:50:31.3219555Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T15:50:31.3220261Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T15:50:31.3221170Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T15:50:31.3221692Z     tracer.run()
2023-08-17T15:50:31.3222401Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:50:31.3222900Z     and self.step()
2023-08-17T15:50:31.3223616Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:50:31.3224176Z     getattr(self, inst.opname)(inst)
2023-08-17T15:50:31.3225028Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:50:31.3225564Z     return inner_fn(self, inst)
2023-08-17T15:50:31.3226341Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1155, in CALL_FUNCTION_EX
2023-08-17T15:50:31.3226978Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-08-17T15:50:31.3266883Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:50:31.3267609Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:50:31.3268531Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-08-17T15:50:31.3269102Z     return super().call_function(tx, args, kwargs)
2023-08-17T15:50:31.3270282Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-08-17T15:50:31.3270840Z     return super().call_function(tx, args, kwargs)
2023-08-17T15:50:31.3271624Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-08-17T15:50:31.3272181Z     return tx.inline_user_function_return(
2023-08-17T15:50:31.3273204Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T15:50:31.3273982Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T15:50:31.3275167Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T15:50:31.3275779Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T15:50:31.3276629Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T15:50:31.3277164Z     tracer.run()
2023-08-17T15:50:31.3277907Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:50:31.3278426Z     and self.step()
2023-08-17T15:50:31.3279171Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:50:31.3279778Z     getattr(self, inst.opname)(inst)
2023-08-17T15:50:31.3280585Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:50:31.3281137Z     return inner_fn(self, inst)
2023-08-17T15:50:31.3281974Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1115, in CALL_FUNCTION
2023-08-17T15:50:31.3282542Z     self.call_function(fn, args, {})
2023-08-17T15:50:31.3283441Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:50:31.3284028Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:50:31.3284827Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 328, in call_function
2023-08-17T15:50:31.3285504Z     return VariableBuilder(tx, source).wrap_unspecialized_primitive(
2023-08-17T15:50:31.3286411Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1119, in wrap_unspecialized_primitive
2023-08-17T15:50:31.3286940Z     raise AssertionError(
2023-08-17T15:50:31.3287548Z AssertionError: Dynamo attempts to add additional input during export: value=0.9526665806770325, source=RandomValueSource(random_call_index=0)
2023-08-17T15:50:31.3288004Z 
2023-08-17T15:50:31.3288134Z from user code:
2023-08-17T15:50:31.3288923Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 930, in forward
2023-08-17T15:50:31.3289517Z     outputs = self.model.decoder(
2023-08-17T15:50:31.3290292Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:50:31.3290858Z     return forward_call(*args, **kwargs)
2023-08-17T15:50:31.3291684Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/opt/modeling_opt.py", line 673, in forward
2023-08-17T15:50:31.3292321Z     dropout_probability = random.uniform(0, 1)
2023-08-17T15:50:31.3292619Z 
2023-08-17T15:50:31.3292879Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-08-17T15:50:31.3293212Z 
2023-08-17T15:50:32.0762952Z ERROR
2023-08-17T15:50:35.2764143Z 
2023-08-17T15:50:38.3519755Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:50:38.3520480Z loading model: 0it [00:03, ?it/s]
2023-08-17T15:50:38.3551734Z cuda eval  PLBartForCausalLM                  
2023-08-17T15:50:39.9950017Z ERROR:common:Backend dynamo failed in warmup()
2023-08-17T15:50:39.9951316Z Traceback (most recent call last):
2023-08-17T15:50:39.9952011Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2278, in warmup
2023-08-17T15:50:39.9952718Z     fn(model, example_inputs)
2023-08-17T15:50:39.9953391Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1186, in opt_aot_inductor
2023-08-17T15:50:39.9954121Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-08-17T15:50:39.9955301Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1131, in load
2023-08-17T15:50:39.9955706Z     so_path, exported = torch._export.aot_compile(
2023-08-17T15:50:39.9956518Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 460, in aot_compile
2023-08-17T15:50:39.9956931Z     ep = export(f, args, kwargs, constraints)
2023-08-17T15:50:39.9957463Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 222, in export
2023-08-17T15:50:39.9957877Z     gm_torch_level, _ = torch._dynamo.export(
2023-08-17T15:50:39.9958426Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1201, in inner
2023-08-17T15:50:39.9958819Z     result_traced = opt_f(*args, **kwargs)
2023-08-17T15:50:39.9959382Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T15:50:39.9959784Z     return self._call_impl(*args, **kwargs)
2023-08-17T15:50:39.9960340Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:50:39.9960740Z     return forward_call(*args, **kwargs)
2023-08-17T15:50:39.9961259Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 333, in _fn
2023-08-17T15:50:39.9961619Z     return fn(*args, **kwargs)
2023-08-17T15:50:39.9962160Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T15:50:39.9962858Z     return self._call_impl(*args, **kwargs)
2023-08-17T15:50:39.9963417Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:50:39.9963812Z     return forward_call(*args, **kwargs)
2023-08-17T15:50:39.9964355Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 493, in catch_errors
2023-08-17T15:50:39.9964785Z     return callback(frame, cache_size, hooks, frame_state)
2023-08-17T15:50:39.9965400Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 132, in _fn
2023-08-17T15:50:39.9965777Z     return fn(*args, **kwargs)
2023-08-17T15:50:39.9966336Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 370, in _convert_frame_assert
2023-08-17T15:50:39.9966708Z     return _compile(
2023-08-17T15:50:39.9967242Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 554, in _compile
2023-08-17T15:50:39.9967692Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-08-17T15:50:39.9968276Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 180, in time_wrapper
2023-08-17T15:50:39.9968645Z     r = func(*args, **kwargs)
2023-08-17T15:50:39.9969177Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 465, in compile_inner
2023-08-17T15:50:39.9969614Z     out_code = transform_code_object(code, transform)
2023-08-17T15:50:39.9970249Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-08-17T15:50:39.9970714Z     transformations(instructions, code_options)
2023-08-17T15:50:39.9971269Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 432, in transform
2023-08-17T15:50:39.9971826Z     tracer.run()
2023-08-17T15:50:39.9972354Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2071, in run
2023-08-17T15:50:39.9972719Z     super().run()
2023-08-17T15:50:39.9973227Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:50:39.9973583Z     and self.step()
2023-08-17T15:50:39.9974211Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:50:39.9974613Z     getattr(self, inst.opname)(inst)
2023-08-17T15:50:39.9975207Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:50:39.9975580Z     return inner_fn(self, inst)
2023-08-17T15:50:39.9976149Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1167, in CALL_FUNCTION_KW
2023-08-17T15:50:39.9976578Z     self.call_function(fn, args, kwargs)
2023-08-17T15:50:39.9977141Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:50:39.9977571Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:50:39.9978142Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 331, in call_function
2023-08-17T15:50:39.9978559Z     return tx.inline_user_function_return(
2023-08-17T15:50:39.9979163Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T15:50:39.9979694Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T15:50:39.9980325Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T15:50:39.9980762Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T15:50:39.9981355Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T15:50:39.9981737Z     tracer.run()
2023-08-17T15:50:39.9982249Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:50:39.9982595Z     and self.step()
2023-08-17T15:50:39.9983109Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:50:39.9983505Z     getattr(self, inst.opname)(inst)
2023-08-17T15:50:39.9984057Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:50:39.9984426Z     return inner_fn(self, inst)
2023-08-17T15:50:39.9984987Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1155, in CALL_FUNCTION_EX
2023-08-17T15:50:39.9985499Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-08-17T15:50:39.9986110Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:50:39.9986540Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:50:39.9987109Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-08-17T15:50:39.9987541Z     return super().call_function(tx, args, kwargs)
2023-08-17T15:50:39.9988130Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-08-17T15:50:39.9988560Z     return super().call_function(tx, args, kwargs)
2023-08-17T15:50:39.9989132Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-08-17T15:50:39.9989553Z     return tx.inline_user_function_return(
2023-08-17T15:50:39.9990149Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T15:50:39.9990782Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T15:50:39.9991430Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T15:50:39.9991849Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T15:50:39.9992528Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T15:50:39.9992905Z     tracer.run()
2023-08-17T15:50:39.9993420Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:50:39.9993822Z     and self.step()
2023-08-17T15:50:39.9994701Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:50:39.9995482Z     getattr(self, inst.opname)(inst)
2023-08-17T15:50:39.9996340Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:50:39.9996883Z     return inner_fn(self, inst)
2023-08-17T15:50:39.9997654Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1115, in CALL_FUNCTION
2023-08-17T15:50:39.9998219Z     self.call_function(fn, args, {})
2023-08-17T15:50:39.9999042Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:50:39.9999633Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:50:40.0000476Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 328, in call_function
2023-08-17T15:50:40.0001148Z     return VariableBuilder(tx, source).wrap_unspecialized_primitive(
2023-08-17T15:50:40.0002100Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1119, in wrap_unspecialized_primitive
2023-08-17T15:50:40.0002884Z     raise AssertionError(
2023-08-17T15:50:40.0003516Z AssertionError: Dynamo attempts to add additional input during export: value=0.6197455525398254, source=RandomValueSource(random_call_index=0)
2023-08-17T15:50:40.0003965Z 
2023-08-17T15:50:40.0004081Z from user code:
2023-08-17T15:50:40.0004912Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1692, in forward
2023-08-17T15:50:40.0005548Z     outputs = self.model.decoder(
2023-08-17T15:50:40.0006339Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:50:40.0006879Z     return forward_call(*args, **kwargs)
2023-08-17T15:50:40.0007727Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1050, in forward
2023-08-17T15:50:40.0008347Z     dropout_probability = random.uniform(0, 1)
2023-08-17T15:50:40.0008612Z 
2023-08-17T15:50:40.0008863Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-08-17T15:50:40.0009165Z 
2023-08-17T15:50:40.7743580Z ERROR
2023-08-17T15:50:43.9528438Z 
2023-08-17T15:50:49.1128755Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:50:49.1129098Z loading model: 0it [00:05, ?it/s]
2023-08-17T15:50:49.1181628Z cuda eval  PLBartForConditionalGeneration     
2023-08-17T15:50:50.9521208Z ERROR:common:Backend dynamo failed in warmup()
2023-08-17T15:50:50.9522179Z Traceback (most recent call last):
2023-08-17T15:50:50.9522891Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2278, in warmup
2023-08-17T15:50:50.9523341Z     fn(model, example_inputs)
2023-08-17T15:50:50.9523993Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1186, in opt_aot_inductor
2023-08-17T15:50:50.9524891Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-08-17T15:50:50.9526414Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1131, in load
2023-08-17T15:50:50.9527023Z     so_path, exported = torch._export.aot_compile(
2023-08-17T15:50:50.9528710Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 460, in aot_compile
2023-08-17T15:50:50.9529123Z     ep = export(f, args, kwargs, constraints)
2023-08-17T15:50:50.9529893Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 222, in export
2023-08-17T15:50:50.9530302Z     gm_torch_level, _ = torch._dynamo.export(
2023-08-17T15:50:50.9530856Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1201, in inner
2023-08-17T15:50:50.9531250Z     result_traced = opt_f(*args, **kwargs)
2023-08-17T15:50:50.9531808Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T15:50:50.9532236Z     return self._call_impl(*args, **kwargs)
2023-08-17T15:50:50.9532780Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:50:50.9533174Z     return forward_call(*args, **kwargs)
2023-08-17T15:50:50.9533699Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 333, in _fn
2023-08-17T15:50:50.9534056Z     return fn(*args, **kwargs)
2023-08-17T15:50:50.9534602Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T15:50:50.9535013Z     return self._call_impl(*args, **kwargs)
2023-08-17T15:50:50.9535550Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:50:50.9535924Z     return forward_call(*args, **kwargs)
2023-08-17T15:50:50.9536467Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 493, in catch_errors
2023-08-17T15:50:50.9536904Z     return callback(frame, cache_size, hooks, frame_state)
2023-08-17T15:50:50.9537460Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 132, in _fn
2023-08-17T15:50:50.9537826Z     return fn(*args, **kwargs)
2023-08-17T15:50:50.9538421Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 370, in _convert_frame_assert
2023-08-17T15:50:50.9538810Z     return _compile(
2023-08-17T15:50:50.9539339Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 554, in _compile
2023-08-17T15:50:50.9539789Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-08-17T15:50:50.9540353Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 180, in time_wrapper
2023-08-17T15:50:50.9540718Z     r = func(*args, **kwargs)
2023-08-17T15:50:50.9541259Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 465, in compile_inner
2023-08-17T15:50:50.9541694Z     out_code = transform_code_object(code, transform)
2023-08-17T15:50:50.9542317Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-08-17T15:50:50.9542761Z     transformations(instructions, code_options)
2023-08-17T15:50:50.9543331Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 432, in transform
2023-08-17T15:50:50.9543692Z     tracer.run()
2023-08-17T15:50:50.9544237Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2071, in run
2023-08-17T15:50:50.9544731Z     super().run()
2023-08-17T15:50:50.9545511Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:50:50.9546056Z     and self.step()
2023-08-17T15:50:50.9547230Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:50:50.9547909Z     getattr(self, inst.opname)(inst)
2023-08-17T15:50:50.9548878Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:50:50.9549434Z     return inner_fn(self, inst)
2023-08-17T15:50:50.9550205Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1167, in CALL_FUNCTION_KW
2023-08-17T15:50:50.9551052Z     self.call_function(fn, args, kwargs)
2023-08-17T15:50:50.9551911Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:50:50.9552357Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:50:50.9553175Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 331, in call_function
2023-08-17T15:50:50.9553889Z     return tx.inline_user_function_return(
2023-08-17T15:50:50.9554807Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T15:50:50.9555581Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T15:50:50.9556566Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T15:50:50.9557233Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T15:50:50.9558103Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T15:50:50.9558719Z     tracer.run()
2023-08-17T15:50:50.9559523Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:50:50.9560064Z     and self.step()
2023-08-17T15:50:50.9560834Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:50:50.9561400Z     getattr(self, inst.opname)(inst)
2023-08-17T15:50:50.9562345Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:50:50.9563106Z     return inner_fn(self, inst)
2023-08-17T15:50:50.9563979Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1155, in CALL_FUNCTION_EX
2023-08-17T15:50:50.9564660Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-08-17T15:50:50.9565699Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:50:50.9566124Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:50:50.9566722Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-08-17T15:50:50.9567169Z     return super().call_function(tx, args, kwargs)
2023-08-17T15:50:50.9567915Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-08-17T15:50:50.9568637Z     return super().call_function(tx, args, kwargs)
2023-08-17T15:50:50.9569474Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-08-17T15:50:50.9570099Z     return tx.inline_user_function_return(
2023-08-17T15:50:50.9571008Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T15:50:50.9571782Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T15:50:50.9572754Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T15:50:50.9573411Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T15:50:50.9574653Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T15:50:50.9575200Z     tracer.run()
2023-08-17T15:50:50.9575976Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:50:50.9576501Z     and self.step()
2023-08-17T15:50:50.9577284Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:50:50.9578081Z     getattr(self, inst.opname)(inst)
2023-08-17T15:50:50.9578748Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:50:50.9579192Z     return inner_fn(self, inst)
2023-08-17T15:50:50.9580174Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1167, in CALL_FUNCTION_KW
2023-08-17T15:50:50.9580862Z     self.call_function(fn, args, kwargs)
2023-08-17T15:50:50.9581720Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:50:50.9582312Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:50:50.9583139Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 331, in call_function
2023-08-17T15:50:50.9583734Z     return tx.inline_user_function_return(
2023-08-17T15:50:50.9584618Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T15:50:50.9585349Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T15:50:50.9586252Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T15:50:50.9586856Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T15:50:50.9587688Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T15:50:50.9588223Z     tracer.run()
2023-08-17T15:50:50.9589029Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:50:50.9589527Z     and self.step()
2023-08-17T15:50:50.9590264Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:50:50.9590811Z     getattr(self, inst.opname)(inst)
2023-08-17T15:50:50.9591607Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:50:50.9592131Z     return inner_fn(self, inst)
2023-08-17T15:50:50.9592906Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1155, in CALL_FUNCTION_EX
2023-08-17T15:50:50.9593545Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-08-17T15:50:50.9594368Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:50:50.9594930Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:50:50.9595696Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-08-17T15:50:50.9596259Z     return super().call_function(tx, args, kwargs)
2023-08-17T15:50:50.9597040Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-08-17T15:50:50.9597629Z     return super().call_function(tx, args, kwargs)
2023-08-17T15:50:50.9598512Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-08-17T15:50:50.9599106Z     return tx.inline_user_function_return(
2023-08-17T15:50:50.9599978Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T15:50:50.9600945Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T15:50:50.9601875Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T15:50:50.9602462Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T15:50:50.9603695Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T15:50:50.9604230Z     tracer.run()
2023-08-17T15:50:50.9604963Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:50:50.9605461Z     and self.step()
2023-08-17T15:50:50.9606192Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:50:50.9606744Z     getattr(self, inst.opname)(inst)
2023-08-17T15:50:50.9607527Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:50:50.9608065Z     return inner_fn(self, inst)
2023-08-17T15:50:50.9608892Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1115, in CALL_FUNCTION
2023-08-17T15:50:50.9609416Z     self.call_function(fn, args, {})
2023-08-17T15:50:50.9610176Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:50:50.9610738Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:50:50.9611521Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 328, in call_function
2023-08-17T15:50:50.9612171Z     return VariableBuilder(tx, source).wrap_unspecialized_primitive(
2023-08-17T15:50:50.9613049Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1119, in wrap_unspecialized_primitive
2023-08-17T15:50:50.9613624Z     raise AssertionError(
2023-08-17T15:50:50.9614229Z AssertionError: Dynamo attempts to add additional input during export: value=0.9526665806770325, source=RandomValueSource(random_call_index=0)
2023-08-17T15:50:50.9614670Z 
2023-08-17T15:50:50.9614787Z from user code:
2023-08-17T15:50:50.9615578Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1323, in forward
2023-08-17T15:50:50.9616138Z     outputs = self.model(
2023-08-17T15:50:50.9616876Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:50:50.9617427Z     return forward_call(*args, **kwargs)
2023-08-17T15:50:50.9618236Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward
2023-08-17T15:50:50.9618872Z     encoder_outputs = self.encoder(
2023-08-17T15:50:50.9619653Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:50:50.9620205Z     return forward_call(*args, **kwargs)
2023-08-17T15:50:50.9621006Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/plbart/modeling_plbart.py", line 799, in forward
2023-08-17T15:50:50.9621615Z     dropout_probability = random.uniform(0, 1)
2023-08-17T15:50:50.9621884Z 
2023-08-17T15:50:50.9622149Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-08-17T15:50:50.9622465Z 
2023-08-17T15:50:51.7165601Z ERROR
2023-08-17T15:50:54.8868274Z 
2023-08-17T15:51:03.1421126Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:51:03.1421664Z loading model: 0it [00:08, ?it/s]
2023-08-17T15:51:03.1487732Z cuda eval  PegasusForCausalLM                 
2023-08-17T15:51:04.9539279Z ERROR:common:Backend dynamo failed in warmup()
2023-08-17T15:51:04.9539942Z Traceback (most recent call last):
2023-08-17T15:51:04.9541272Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2278, in warmup
2023-08-17T15:51:04.9541886Z     fn(model, example_inputs)
2023-08-17T15:51:04.9542899Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1186, in opt_aot_inductor
2023-08-17T15:51:04.9543768Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-08-17T15:51:04.9544606Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1131, in load
2023-08-17T15:51:04.9546129Z     so_path, exported = torch._export.aot_compile(
2023-08-17T15:51:04.9548345Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 460, in aot_compile
2023-08-17T15:51:04.9548781Z     ep = export(f, args, kwargs, constraints)
2023-08-17T15:51:04.9549337Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 222, in export
2023-08-17T15:51:04.9549743Z     gm_torch_level, _ = torch._dynamo.export(
2023-08-17T15:51:04.9550299Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1201, in inner
2023-08-17T15:51:04.9550696Z     result_traced = opt_f(*args, **kwargs)
2023-08-17T15:51:04.9551265Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T15:51:04.9551680Z     return self._call_impl(*args, **kwargs)
2023-08-17T15:51:04.9552224Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:51:04.9552622Z     return forward_call(*args, **kwargs)
2023-08-17T15:51:04.9553154Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 333, in _fn
2023-08-17T15:51:04.9553526Z     return fn(*args, **kwargs)
2023-08-17T15:51:04.9554070Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T15:51:04.9554479Z     return self._call_impl(*args, **kwargs)
2023-08-17T15:51:04.9555024Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:51:04.9555416Z     return forward_call(*args, **kwargs)
2023-08-17T15:51:04.9555967Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 493, in catch_errors
2023-08-17T15:51:04.9556389Z     return callback(frame, cache_size, hooks, frame_state)
2023-08-17T15:51:04.9556966Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 132, in _fn
2023-08-17T15:51:04.9557340Z     return fn(*args, **kwargs)
2023-08-17T15:51:04.9557905Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 370, in _convert_frame_assert
2023-08-17T15:51:04.9558447Z     return _compile(
2023-08-17T15:51:04.9559176Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 554, in _compile
2023-08-17T15:51:04.9559849Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-08-17T15:51:04.9560763Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 180, in time_wrapper
2023-08-17T15:51:04.9561358Z     r = func(*args, **kwargs)
2023-08-17T15:51:04.9562299Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 465, in compile_inner
2023-08-17T15:51:04.9563358Z     out_code = transform_code_object(code, transform)
2023-08-17T15:51:04.9564372Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-08-17T15:51:04.9564963Z     transformations(instructions, code_options)
2023-08-17T15:51:04.9565954Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 432, in transform
2023-08-17T15:51:04.9566324Z     tracer.run()
2023-08-17T15:51:04.9566938Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2071, in run
2023-08-17T15:51:04.9567994Z     super().run()
2023-08-17T15:51:04.9568998Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:51:04.9569609Z     and self.step()
2023-08-17T15:51:04.9570491Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:51:04.9570944Z     getattr(self, inst.opname)(inst)
2023-08-17T15:51:04.9571714Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:51:04.9572100Z     return inner_fn(self, inst)
2023-08-17T15:51:04.9572686Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1167, in CALL_FUNCTION_KW
2023-08-17T15:51:04.9573107Z     self.call_function(fn, args, kwargs)
2023-08-17T15:51:04.9573681Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:51:04.9574128Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:51:04.9574757Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 331, in call_function
2023-08-17T15:51:04.9575189Z     return tx.inline_user_function_return(
2023-08-17T15:51:04.9575800Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T15:51:04.9576334Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T15:51:04.9577028Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T15:51:04.9577453Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T15:51:04.9578048Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T15:51:04.9578479Z     tracer.run()
2023-08-17T15:51:04.9578995Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:51:04.9579351Z     and self.step()
2023-08-17T15:51:04.9579872Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:51:04.9580274Z     getattr(self, inst.opname)(inst)
2023-08-17T15:51:04.9580864Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:51:04.9581401Z     return inner_fn(self, inst)
2023-08-17T15:51:04.9582263Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1155, in CALL_FUNCTION_EX
2023-08-17T15:51:04.9582944Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-08-17T15:51:04.9583898Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:51:04.9584548Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:51:04.9585424Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-08-17T15:51:04.9586073Z     return super().call_function(tx, args, kwargs)
2023-08-17T15:51:04.9586985Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-08-17T15:51:04.9587621Z     return super().call_function(tx, args, kwargs)
2023-08-17T15:51:04.9588584Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-08-17T15:51:04.9589187Z     return tx.inline_user_function_return(
2023-08-17T15:51:04.9590122Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T15:51:04.9591280Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T15:51:04.9592042Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T15:51:04.9592675Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T15:51:04.9593743Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T15:51:04.9594563Z     tracer.run()
2023-08-17T15:51:04.9595318Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:51:04.9595827Z     and self.step()
2023-08-17T15:51:04.9596550Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:51:04.9597108Z     getattr(self, inst.opname)(inst)
2023-08-17T15:51:04.9597895Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:51:04.9598531Z     return inner_fn(self, inst)
2023-08-17T15:51:04.9599317Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1115, in CALL_FUNCTION
2023-08-17T15:51:04.9599890Z     self.call_function(fn, args, {})
2023-08-17T15:51:04.9600686Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:51:04.9601298Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:51:04.9602163Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 328, in call_function
2023-08-17T15:51:04.9603004Z     return VariableBuilder(tx, source).wrap_unspecialized_primitive(
2023-08-17T15:51:04.9603976Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1119, in wrap_unspecialized_primitive
2023-08-17T15:51:04.9604578Z     raise AssertionError(
2023-08-17T15:51:04.9605201Z AssertionError: Dynamo attempts to add additional input during export: value=0.9526665806770325, source=RandomValueSource(random_call_index=0)
2023-08-17T15:51:04.9605661Z 
2023-08-17T15:51:04.9605777Z from user code:
2023-08-17T15:51:04.9606581Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1673, in forward
2023-08-17T15:51:04.9607170Z     outputs = self.model.decoder(
2023-08-17T15:51:04.9607948Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:51:04.9608534Z     return forward_call(*args, **kwargs)
2023-08-17T15:51:04.9609307Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1072, in forward
2023-08-17T15:51:04.9609901Z     dropout_probability = random.uniform(0, 1)
2023-08-17T15:51:04.9610161Z 
2023-08-17T15:51:04.9610424Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-08-17T15:51:04.9610723Z 
2023-08-17T15:51:05.7117073Z ERROR
2023-08-17T15:51:08.9292539Z 
2023-08-17T15:51:22.4092478Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:51:22.4092797Z loading model: 0it [00:13, ?it/s]
2023-08-17T15:51:22.4194901Z cuda eval  PegasusForConditionalGeneration    
2023-08-17T15:51:24.3833026Z ERROR:common:Backend dynamo failed in warmup()
2023-08-17T15:51:24.3833538Z Traceback (most recent call last):
2023-08-17T15:51:24.3834562Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2278, in warmup
2023-08-17T15:51:24.3835132Z     fn(model, example_inputs)
2023-08-17T15:51:24.3835680Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1186, in opt_aot_inductor
2023-08-17T15:51:24.3836317Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-08-17T15:51:24.3838245Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1131, in load
2023-08-17T15:51:24.3839681Z     so_path, exported = torch._export.aot_compile(
2023-08-17T15:51:24.3840893Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 460, in aot_compile
2023-08-17T15:51:24.3841572Z     ep = export(f, args, kwargs, constraints)
2023-08-17T15:51:24.3842503Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 222, in export
2023-08-17T15:51:24.3843373Z     gm_torch_level, _ = torch._dynamo.export(
2023-08-17T15:51:24.3844818Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1201, in inner
2023-08-17T15:51:24.3845230Z     result_traced = opt_f(*args, **kwargs)
2023-08-17T15:51:24.3845792Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T15:51:24.3846217Z     return self._call_impl(*args, **kwargs)
2023-08-17T15:51:24.3846757Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:51:24.3847168Z     return forward_call(*args, **kwargs)
2023-08-17T15:51:24.3847691Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 333, in _fn
2023-08-17T15:51:24.3848041Z     return fn(*args, **kwargs)
2023-08-17T15:51:24.3848665Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T15:51:24.3849077Z     return self._call_impl(*args, **kwargs)
2023-08-17T15:51:24.3849618Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:51:24.3849993Z     return forward_call(*args, **kwargs)
2023-08-17T15:51:24.3850539Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 493, in catch_errors
2023-08-17T15:51:24.3850968Z     return callback(frame, cache_size, hooks, frame_state)
2023-08-17T15:51:24.3851531Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 132, in _fn
2023-08-17T15:51:24.3851897Z     return fn(*args, **kwargs)
2023-08-17T15:51:24.3852442Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 370, in _convert_frame_assert
2023-08-17T15:51:24.3852830Z     return _compile(
2023-08-17T15:51:24.3853351Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 554, in _compile
2023-08-17T15:51:24.3853800Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-08-17T15:51:24.3854359Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 180, in time_wrapper
2023-08-17T15:51:24.3854726Z     r = func(*args, **kwargs)
2023-08-17T15:51:24.3855274Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 465, in compile_inner
2023-08-17T15:51:24.3855711Z     out_code = transform_code_object(code, transform)
2023-08-17T15:51:24.3856351Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-08-17T15:51:24.3856798Z     transformations(instructions, code_options)
2023-08-17T15:51:24.3857358Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 432, in transform
2023-08-17T15:51:24.3857723Z     tracer.run()
2023-08-17T15:51:24.3858294Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2071, in run
2023-08-17T15:51:24.3858642Z     super().run()
2023-08-17T15:51:24.3859150Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:51:24.3859509Z     and self.step()
2023-08-17T15:51:24.3860023Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:51:24.3860563Z     getattr(self, inst.opname)(inst)
2023-08-17T15:51:24.3861113Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:51:24.3861497Z     return inner_fn(self, inst)
2023-08-17T15:51:24.3862062Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1167, in CALL_FUNCTION_KW
2023-08-17T15:51:24.3862478Z     self.call_function(fn, args, kwargs)
2023-08-17T15:51:24.3863125Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:51:24.3863564Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:51:24.3864151Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 331, in call_function
2023-08-17T15:51:24.3864568Z     return tx.inline_user_function_return(
2023-08-17T15:51:24.3865154Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T15:51:24.3865788Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T15:51:24.3866425Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T15:51:24.3866861Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T15:51:24.3867456Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T15:51:24.3867839Z     tracer.run()
2023-08-17T15:51:24.3868388Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:51:24.3868757Z     and self.step()
2023-08-17T15:51:24.3869277Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:51:24.3869685Z     getattr(self, inst.opname)(inst)
2023-08-17T15:51:24.3870239Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:51:24.3870614Z     return inner_fn(self, inst)
2023-08-17T15:51:24.3871180Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1155, in CALL_FUNCTION_EX
2023-08-17T15:51:24.3871655Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-08-17T15:51:24.3872271Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:51:24.3872687Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:51:24.3873277Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-08-17T15:51:24.3873756Z     return super().call_function(tx, args, kwargs)
2023-08-17T15:51:24.3874640Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-08-17T15:51:24.3875286Z     return super().call_function(tx, args, kwargs)
2023-08-17T15:51:24.3876185Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-08-17T15:51:24.3876814Z     return tx.inline_user_function_return(
2023-08-17T15:51:24.3877748Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T15:51:24.3878592Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T15:51:24.3879604Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T15:51:24.3880238Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T15:51:24.3881123Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T15:51:24.3882013Z     tracer.run()
2023-08-17T15:51:24.3882984Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:51:24.3883535Z     and self.step()
2023-08-17T15:51:24.3884347Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:51:24.3885628Z     getattr(self, inst.opname)(inst)
2023-08-17T15:51:24.3886953Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:51:24.3887490Z     return inner_fn(self, inst)
2023-08-17T15:51:24.3888324Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1167, in CALL_FUNCTION_KW
2023-08-17T15:51:24.3888920Z     self.call_function(fn, args, kwargs)
2023-08-17T15:51:24.3889740Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:51:24.3890373Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:51:24.3891206Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 331, in call_function
2023-08-17T15:51:24.3891797Z     return tx.inline_user_function_return(
2023-08-17T15:51:24.3892680Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T15:51:24.3893503Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T15:51:24.3894429Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T15:51:24.3895020Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T15:51:24.3895853Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T15:51:24.3896390Z     tracer.run()
2023-08-17T15:51:24.3897133Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:51:24.3897629Z     and self.step()
2023-08-17T15:51:24.3898361Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:51:24.3898908Z     getattr(self, inst.opname)(inst)
2023-08-17T15:51:24.3899643Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:51:24.3900130Z     return inner_fn(self, inst)
2023-08-17T15:51:24.3900876Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1155, in CALL_FUNCTION_EX
2023-08-17T15:51:24.3901472Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-08-17T15:51:24.3902276Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:51:24.3902854Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:51:24.3903756Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-08-17T15:51:24.3904365Z     return super().call_function(tx, args, kwargs)
2023-08-17T15:51:24.3905195Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-08-17T15:51:24.3905808Z     return super().call_function(tx, args, kwargs)
2023-08-17T15:51:24.3906652Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-08-17T15:51:24.3907223Z     return tx.inline_user_function_return(
2023-08-17T15:51:24.3908221Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T15:51:24.3908968Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T15:51:24.3910264Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T15:51:24.3910850Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T15:51:24.3911684Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T15:51:24.3912202Z     tracer.run()
2023-08-17T15:51:24.3913126Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:51:24.3913641Z     and self.step()
2023-08-17T15:51:24.3914365Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:51:24.3914911Z     getattr(self, inst.opname)(inst)
2023-08-17T15:51:24.3915642Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:51:24.3916163Z     return inner_fn(self, inst)
2023-08-17T15:51:24.3916879Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1115, in CALL_FUNCTION
2023-08-17T15:51:24.3917413Z     self.call_function(fn, args, {})
2023-08-17T15:51:24.3918179Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:51:24.3918755Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:51:24.3919544Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 328, in call_function
2023-08-17T15:51:24.3920207Z     return VariableBuilder(tx, source).wrap_unspecialized_primitive(
2023-08-17T15:51:24.3921153Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1119, in wrap_unspecialized_primitive
2023-08-17T15:51:24.3921794Z     raise AssertionError(
2023-08-17T15:51:24.3922441Z AssertionError: Dynamo attempts to add additional input during export: value=0.9152050018310547, source=RandomValueSource(random_call_index=0)
2023-08-17T15:51:24.3923151Z 
2023-08-17T15:51:24.3923274Z from user code:
2023-08-17T15:51:24.3924123Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1411, in forward
2023-08-17T15:51:24.3924719Z     outputs = self.model(
2023-08-17T15:51:24.3925525Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:51:24.3926094Z     return forward_call(*args, **kwargs)
2023-08-17T15:51:24.3926954Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1248, in forward
2023-08-17T15:51:24.3927566Z     encoder_outputs = self.encoder(
2023-08-17T15:51:24.3928351Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:51:24.3928898Z     return forward_call(*args, **kwargs)
2023-08-17T15:51:24.3929698Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 794, in forward
2023-08-17T15:51:24.3930323Z     dropout_probability = random.uniform(0, 1)
2023-08-17T15:51:24.3930593Z 
2023-08-17T15:51:24.3930849Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-08-17T15:51:24.3931146Z 
2023-08-17T15:51:25.1565509Z ERROR
2023-08-17T15:51:28.3807862Z 
2023-08-17T15:51:28.3856026Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`
2023-08-17T15:51:31.4479244Z 
2023-08-17T15:51:31.4479847Z loading model: 0it [00:03, ?it/s]
2023-08-17T15:51:31.4546337Z cuda eval  RobertaForCausalLM                 
2023-08-17T15:52:18.2043073Z 
2023-08-17T15:52:18.3081370Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:52:18.4386038Z running benchmark:   3% 1/30 [00:00<00:03,  9.53it/s]
2023-08-17T15:52:18.5689761Z running benchmark:  13% 4/30 [00:00<00:01, 18.12it/s]
2023-08-17T15:52:18.7015160Z running benchmark:  23% 7/30 [00:00<00:01, 20.45it/s]
2023-08-17T15:52:18.8316651Z running benchmark:  33% 10/30 [00:00<00:00, 21.35it/s]
2023-08-17T15:52:18.9621773Z running benchmark:  43% 13/30 [00:00<00:00, 21.97it/s]
2023-08-17T15:52:19.0928017Z running benchmark:  53% 16/30 [00:00<00:00, 22.32it/s]
2023-08-17T15:52:19.2230167Z running benchmark:  63% 19/30 [00:00<00:00, 22.53it/s]
2023-08-17T15:52:19.3532375Z running benchmark:  73% 22/30 [00:01<00:00, 22.69it/s]
2023-08-17T15:52:19.4834395Z running benchmark:  83% 25/30 [00:01<00:00, 22.80it/s]
2023-08-17T15:52:19.5702717Z running benchmark:  93% 28/30 [00:01<00:00, 22.87it/s]
2023-08-17T15:52:19.5703045Z running benchmark: 100% 30/30 [00:01<00:00, 21.94it/s]
2023-08-17T15:52:19.5727164Z 1.536x
2023-08-17T15:52:23.8924584Z 
2023-08-17T15:52:26.6519420Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:52:26.6519786Z loading model: 0it [00:02, ?it/s]
2023-08-17T15:52:26.6562369Z cuda eval  RobertaForQuestionAnswering        
2023-08-17T15:53:12.9253413Z 
2023-08-17T15:53:13.0417498Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:53:13.1456587Z running benchmark:   7% 2/30 [00:00<00:01, 17.09it/s]
2023-08-17T15:53:13.2495537Z running benchmark:  17% 5/30 [00:00<00:01, 23.67it/s]
2023-08-17T15:53:13.3534688Z running benchmark:  27% 8/30 [00:00<00:00, 25.97it/s]
2023-08-17T15:53:13.4573576Z running benchmark:  37% 11/30 [00:00<00:00, 27.09it/s]
2023-08-17T15:53:13.5611567Z running benchmark:  47% 14/30 [00:00<00:00, 27.73it/s]
2023-08-17T15:53:13.6653396Z running benchmark:  57% 17/30 [00:00<00:00, 28.13it/s]
2023-08-17T15:53:13.7692098Z running benchmark:  67% 20/30 [00:00<00:00, 28.35it/s]
2023-08-17T15:53:13.8729409Z running benchmark:  77% 23/30 [00:00<00:00, 28.51it/s]
2023-08-17T15:53:13.9858677Z running benchmark:  87% 26/30 [00:00<00:00, 28.64it/s]
2023-08-17T15:53:14.0204358Z running benchmark:  97% 29/30 [00:01<00:00, 27.98it/s]
2023-08-17T15:53:14.0204726Z running benchmark: 100% 30/30 [00:01<00:00, 27.37it/s]
2023-08-17T15:53:14.0228652Z 1.539x
2023-08-17T15:53:18.3347424Z 
2023-08-17T15:53:19.3859457Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:53:19.3859990Z loading model: 0it [00:01, ?it/s]
2023-08-17T15:53:19.3892857Z cuda eval  Speech2Text2ForCausalLM            
2023-08-17T15:53:20.9714695Z ERROR:common:Backend dynamo failed in warmup()
2023-08-17T15:53:20.9715234Z Traceback (most recent call last):
2023-08-17T15:53:20.9715631Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2278, in warmup
2023-08-17T15:53:20.9719117Z     fn(model, example_inputs)
2023-08-17T15:53:20.9719803Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1186, in opt_aot_inductor
2023-08-17T15:53:20.9720507Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-08-17T15:53:20.9720967Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1131, in load
2023-08-17T15:53:20.9721369Z     so_path, exported = torch._export.aot_compile(
2023-08-17T15:53:20.9722800Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 460, in aot_compile
2023-08-17T15:53:20.9723502Z     ep = export(f, args, kwargs, constraints)
2023-08-17T15:53:20.9724448Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 222, in export
2023-08-17T15:53:20.9724847Z     gm_torch_level, _ = torch._dynamo.export(
2023-08-17T15:53:20.9725409Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1201, in inner
2023-08-17T15:53:20.9725794Z     result_traced = opt_f(*args, **kwargs)
2023-08-17T15:53:20.9726342Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T15:53:20.9726769Z     return self._call_impl(*args, **kwargs)
2023-08-17T15:53:20.9727308Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:53:20.9728274Z     return forward_call(*args, **kwargs)
2023-08-17T15:53:20.9729168Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 333, in _fn
2023-08-17T15:53:20.9730192Z     return fn(*args, **kwargs)
2023-08-17T15:53:20.9731644Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T15:53:20.9733153Z     return self._call_impl(*args, **kwargs)
2023-08-17T15:53:20.9734525Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:53:20.9735259Z     return forward_call(*args, **kwargs)
2023-08-17T15:53:20.9735996Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 493, in catch_errors
2023-08-17T15:53:20.9736420Z     return callback(frame, cache_size, hooks, frame_state)
2023-08-17T15:53:20.9736988Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 132, in _fn
2023-08-17T15:53:20.9737354Z     return fn(*args, **kwargs)
2023-08-17T15:53:20.9737907Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 370, in _convert_frame_assert
2023-08-17T15:53:20.9738272Z     return _compile(
2023-08-17T15:53:20.9738786Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 554, in _compile
2023-08-17T15:53:20.9739229Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-08-17T15:53:20.9739801Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 180, in time_wrapper
2023-08-17T15:53:20.9740200Z     r = func(*args, **kwargs)
2023-08-17T15:53:20.9740732Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 465, in compile_inner
2023-08-17T15:53:20.9741158Z     out_code = transform_code_object(code, transform)
2023-08-17T15:53:20.9741783Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-08-17T15:53:20.9742241Z     transformations(instructions, code_options)
2023-08-17T15:53:20.9742784Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 432, in transform
2023-08-17T15:53:20.9743136Z     tracer.run()
2023-08-17T15:53:20.9743641Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2071, in run
2023-08-17T15:53:20.9743997Z     super().run()
2023-08-17T15:53:20.9744488Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:53:20.9744842Z     and self.step()
2023-08-17T15:53:20.9745346Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:53:20.9745737Z     getattr(self, inst.opname)(inst)
2023-08-17T15:53:20.9746262Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:53:20.9746637Z     return inner_fn(self, inst)
2023-08-17T15:53:20.9747185Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1167, in CALL_FUNCTION_KW
2023-08-17T15:53:20.9747589Z     self.call_function(fn, args, kwargs)
2023-08-17T15:53:20.9748150Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:53:20.9748557Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:53:20.9749136Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 331, in call_function
2023-08-17T15:53:20.9749537Z     return tx.inline_user_function_return(
2023-08-17T15:53:20.9750172Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T15:53:20.9750881Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T15:53:20.9751513Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T15:53:20.9751938Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T15:53:20.9752622Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T15:53:20.9752990Z     tracer.run()
2023-08-17T15:53:20.9753484Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:53:20.9753840Z     and self.step()
2023-08-17T15:53:20.9754348Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:53:20.9754745Z     getattr(self, inst.opname)(inst)
2023-08-17T15:53:20.9755273Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:53:20.9755659Z     return inner_fn(self, inst)
2023-08-17T15:53:20.9756213Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1155, in CALL_FUNCTION_EX
2023-08-17T15:53:20.9756666Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-08-17T15:53:20.9757269Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:53:20.9757678Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:53:20.9758261Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-08-17T15:53:20.9758681Z     return super().call_function(tx, args, kwargs)
2023-08-17T15:53:20.9759254Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-08-17T15:53:20.9759676Z     return super().call_function(tx, args, kwargs)
2023-08-17T15:53:20.9760286Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-08-17T15:53:20.9760695Z     return tx.inline_user_function_return(
2023-08-17T15:53:20.9761294Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T15:53:20.9761897Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T15:53:20.9762519Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T15:53:20.9763355Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T15:53:20.9763957Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T15:53:20.9764329Z     tracer.run()
2023-08-17T15:53:20.9764836Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:53:20.9765183Z     and self.step()
2023-08-17T15:53:20.9765694Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:53:20.9766079Z     getattr(self, inst.opname)(inst)
2023-08-17T15:53:20.9766627Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:53:20.9766989Z     return inner_fn(self, inst)
2023-08-17T15:53:20.9767534Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1115, in CALL_FUNCTION
2023-08-17T15:53:20.9767925Z     self.call_function(fn, args, {})
2023-08-17T15:53:20.9768469Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:53:20.9769079Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:53:20.9769656Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 328, in call_function
2023-08-17T15:53:20.9770178Z     return VariableBuilder(tx, source).wrap_unspecialized_primitive(
2023-08-17T15:53:20.9770841Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1119, in wrap_unspecialized_primitive
2023-08-17T15:53:20.9771377Z     raise AssertionError(
2023-08-17T15:53:20.9771802Z AssertionError: Dynamo attempts to add additional input during export: value=0.6197455525398254, source=RandomValueSource(random_call_index=0)
2023-08-17T15:53:20.9772109Z 
2023-08-17T15:53:20.9772201Z from user code:
2023-08-17T15:53:20.9772800Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/speech_to_text_2/modeling_speech_to_text_2.py", line 918, in forward
2023-08-17T15:53:20.9773242Z     outputs = self.model.decoder(
2023-08-17T15:53:20.9773776Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:53:20.9774146Z     return forward_call(*args, **kwargs)
2023-08-17T15:53:20.9774762Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/speech_to_text_2/modeling_speech_to_text_2.py", line 660, in forward
2023-08-17T15:53:20.9775214Z     dropout_probability = random.uniform(0, 1)
2023-08-17T15:53:20.9775409Z 
2023-08-17T15:53:20.9775593Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-08-17T15:53:20.9775813Z 
2023-08-17T15:53:21.7254513Z ERROR
2023-08-17T15:53:25.1188684Z 
2023-08-17T15:53:27.3042756Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:53:27.3043090Z loading model: 0it [00:02, ?it/s]
2023-08-17T15:53:27.3073891Z cuda eval  T5ForConditionalGeneration         
2023-08-17T15:54:19.6756347Z 
2023-08-17T15:54:19.7915598Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:54:19.9071410Z running benchmark:   7% 2/30 [00:00<00:01, 17.15it/s]
2023-08-17T15:54:20.0227811Z running benchmark:  17% 5/30 [00:00<00:01, 22.29it/s]
2023-08-17T15:54:20.1383991Z running benchmark:  27% 8/30 [00:00<00:00, 23.95it/s]
2023-08-17T15:54:20.2541836Z running benchmark:  37% 11/30 [00:00<00:00, 24.74it/s]
2023-08-17T15:54:20.3699737Z running benchmark:  47% 14/30 [00:00<00:00, 25.16it/s]
2023-08-17T15:54:20.4918613Z running benchmark:  57% 17/30 [00:00<00:00, 25.42it/s]
2023-08-17T15:54:20.6075507Z running benchmark:  67% 20/30 [00:00<00:00, 25.15it/s]
2023-08-17T15:54:20.7234325Z running benchmark:  77% 23/30 [00:00<00:00, 25.39it/s]
2023-08-17T15:54:20.8393726Z running benchmark:  87% 26/30 [00:01<00:00, 25.55it/s]
2023-08-17T15:54:20.8781317Z running benchmark:  97% 29/30 [00:01<00:00, 25.65it/s]
2023-08-17T15:54:20.8781742Z running benchmark: 100% 30/30 [00:01<00:00, 24.93it/s]
2023-08-17T15:54:20.8807655Z 2.274x
2023-08-17T15:54:25.7082364Z 
2023-08-17T15:54:27.8989105Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:54:27.8989595Z loading model: 0it [00:02, ?it/s]
2023-08-17T15:54:27.9022355Z cuda eval  T5Small                            
2023-08-17T15:55:20.1879634Z 
2023-08-17T15:55:20.3041460Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:55:20.4195414Z running benchmark:   7% 2/30 [00:00<00:01, 17.14it/s]
2023-08-17T15:55:20.5354516Z running benchmark:  17% 5/30 [00:00<00:01, 22.30it/s]
2023-08-17T15:55:20.6516060Z running benchmark:  27% 8/30 [00:00<00:00, 23.94it/s]
2023-08-17T15:55:20.7675215Z running benchmark:  37% 11/30 [00:00<00:00, 24.68it/s]
2023-08-17T15:55:20.8930653Z running benchmark:  47% 14/30 [00:00<00:00, 25.11it/s]
2023-08-17T15:55:21.0092488Z running benchmark:  57% 17/30 [00:00<00:00, 24.69it/s]
2023-08-17T15:55:21.1248651Z running benchmark:  67% 20/30 [00:00<00:00, 25.04it/s]
2023-08-17T15:55:21.2406919Z running benchmark:  77% 23/30 [00:00<00:00, 25.32it/s]
2023-08-17T15:55:21.3562361Z running benchmark:  87% 26/30 [00:01<00:00, 25.50it/s]
2023-08-17T15:55:21.3949286Z running benchmark:  97% 29/30 [00:01<00:00, 25.64it/s]
2023-08-17T15:55:21.3950404Z running benchmark: 100% 30/30 [00:01<00:00, 24.84it/s]
2023-08-17T15:55:21.3977602Z 2.275x
2023-08-17T15:55:25.7768463Z 
2023-08-17T15:55:31.7536333Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:55:31.7536840Z loading model: 0it [00:05, ?it/s]
2023-08-17T15:55:31.7597632Z cuda eval  TrOCRForCausalLM                   
2023-08-17T15:55:33.6454977Z ERROR:common:Backend dynamo failed in warmup()
2023-08-17T15:55:33.6455573Z Traceback (most recent call last):
2023-08-17T15:55:33.6455955Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2278, in warmup
2023-08-17T15:55:33.6456385Z     fn(model, example_inputs)
2023-08-17T15:55:33.6457745Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1186, in opt_aot_inductor
2023-08-17T15:55:33.6458666Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-08-17T15:55:33.6459620Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1131, in load
2023-08-17T15:55:33.6460451Z     so_path, exported = torch._export.aot_compile(
2023-08-17T15:55:33.6461618Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 460, in aot_compile
2023-08-17T15:55:33.6462178Z     ep = export(f, args, kwargs, constraints)
2023-08-17T15:55:33.6463230Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 222, in export
2023-08-17T15:55:33.6463813Z     gm_torch_level, _ = torch._dynamo.export(
2023-08-17T15:55:33.6464813Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1201, in inner
2023-08-17T15:55:33.6465428Z     result_traced = opt_f(*args, **kwargs)
2023-08-17T15:55:33.6465999Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T15:55:33.6466421Z     return self._call_impl(*args, **kwargs)
2023-08-17T15:55:33.6466956Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:55:33.6467323Z     return forward_call(*args, **kwargs)
2023-08-17T15:55:33.6467843Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 333, in _fn
2023-08-17T15:55:33.6468202Z     return fn(*args, **kwargs)
2023-08-17T15:55:33.6468742Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T15:55:33.6469149Z     return self._call_impl(*args, **kwargs)
2023-08-17T15:55:33.6469665Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:55:33.6470101Z     return forward_call(*args, **kwargs)
2023-08-17T15:55:33.6470638Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 493, in catch_errors
2023-08-17T15:55:33.6471067Z     return callback(frame, cache_size, hooks, frame_state)
2023-08-17T15:55:33.6471601Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 132, in _fn
2023-08-17T15:55:33.6471961Z     return fn(*args, **kwargs)
2023-08-17T15:55:33.6472556Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 370, in _convert_frame_assert
2023-08-17T15:55:33.6472933Z     return _compile(
2023-08-17T15:55:33.6473448Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 554, in _compile
2023-08-17T15:55:33.6473890Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-08-17T15:55:33.6474448Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 180, in time_wrapper
2023-08-17T15:55:33.6474807Z     r = func(*args, **kwargs)
2023-08-17T15:55:33.6475333Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 465, in compile_inner
2023-08-17T15:55:33.6476080Z     out_code = transform_code_object(code, transform)
2023-08-17T15:55:33.6476697Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-08-17T15:55:33.6477152Z     transformations(instructions, code_options)
2023-08-17T15:55:33.6477714Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 432, in transform
2023-08-17T15:55:33.6478190Z     tracer.run()
2023-08-17T15:55:33.6478701Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2071, in run
2023-08-17T15:55:33.6479038Z     super().run()
2023-08-17T15:55:33.6479534Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:55:33.6479886Z     and self.step()
2023-08-17T15:55:33.6480455Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:55:33.6480823Z     getattr(self, inst.opname)(inst)
2023-08-17T15:55:33.6481364Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:55:33.6481741Z     return inner_fn(self, inst)
2023-08-17T15:55:33.6482296Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1167, in CALL_FUNCTION_KW
2023-08-17T15:55:33.6483033Z     self.call_function(fn, args, kwargs)
2023-08-17T15:55:33.6483585Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:55:33.6484007Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:55:33.6484583Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 331, in call_function
2023-08-17T15:55:33.6484996Z     return tx.inline_user_function_return(
2023-08-17T15:55:33.6485577Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T15:55:33.6486095Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T15:55:33.6486731Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T15:55:33.6487151Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T15:55:33.6487732Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T15:55:33.6488078Z     tracer.run()
2023-08-17T15:55:33.6488585Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:55:33.6488940Z     and self.step()
2023-08-17T15:55:33.6489446Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:55:33.6489823Z     getattr(self, inst.opname)(inst)
2023-08-17T15:55:33.6490403Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:55:33.6490781Z     return inner_fn(self, inst)
2023-08-17T15:55:33.6491336Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1155, in CALL_FUNCTION_EX
2023-08-17T15:55:33.6491802Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-08-17T15:55:33.6492380Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:55:33.6492795Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:55:33.6493579Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-08-17T15:55:33.6494198Z     return super().call_function(tx, args, kwargs)
2023-08-17T15:55:33.6495423Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-08-17T15:55:33.6496058Z     return super().call_function(tx, args, kwargs)
2023-08-17T15:55:33.6496942Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-08-17T15:55:33.6497561Z     return tx.inline_user_function_return(
2023-08-17T15:55:33.6498731Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T15:55:33.6499517Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T15:55:33.6500584Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T15:55:33.6501224Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T15:55:33.6502120Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T15:55:33.6502658Z     tracer.run()
2023-08-17T15:55:33.6503497Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:55:33.6504026Z     and self.step()
2023-08-17T15:55:33.6504812Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:55:33.6505392Z     getattr(self, inst.opname)(inst)
2023-08-17T15:55:33.6506218Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:55:33.6506775Z     return inner_fn(self, inst)
2023-08-17T15:55:33.6507619Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1115, in CALL_FUNCTION
2023-08-17T15:55:33.6508376Z     self.call_function(fn, args, {})
2023-08-17T15:55:33.6509252Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:55:33.6509886Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:55:33.6510844Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 328, in call_function
2023-08-17T15:55:33.6511552Z     return VariableBuilder(tx, source).wrap_unspecialized_primitive(
2023-08-17T15:55:33.6512565Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1119, in wrap_unspecialized_primitive
2023-08-17T15:55:33.6513207Z     raise AssertionError(
2023-08-17T15:55:33.6513979Z AssertionError: Dynamo attempts to add additional input during export: value=0.9526665806770325, source=RandomValueSource(random_call_index=0)
2023-08-17T15:55:33.6514530Z 
2023-08-17T15:55:33.6514668Z from user code:
2023-08-17T15:55:33.6515554Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 958, in forward
2023-08-17T15:55:33.6516118Z     outputs = self.model.decoder(
2023-08-17T15:55:33.6516896Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:55:33.6517461Z     return forward_call(*args, **kwargs)
2023-08-17T15:55:33.6518289Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/trocr/modeling_trocr.py", line 692, in forward
2023-08-17T15:55:33.6518909Z     dropout_probability = random.uniform(0, 1)
2023-08-17T15:55:33.6519179Z 
2023-08-17T15:55:33.6519424Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-08-17T15:55:33.6519732Z 
2023-08-17T15:55:34.4180781Z ERROR
2023-08-17T15:55:37.6090799Z 
2023-08-17T15:55:52.3257781Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:55:52.3258455Z loading model: 0it [00:14, ?it/s]
2023-08-17T15:55:52.3333620Z cuda eval  XGLMForCausalLM                    
2023-08-17T15:55:54.1680457Z ERROR:common:Backend dynamo failed in warmup()
2023-08-17T15:55:54.1680816Z Traceback (most recent call last):
2023-08-17T15:55:54.1681200Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2278, in warmup
2023-08-17T15:55:54.1683111Z     fn(model, example_inputs)
2023-08-17T15:55:54.1683495Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1186, in opt_aot_inductor
2023-08-17T15:55:54.1684099Z     module, exported, output_tensors, output_spec = AOTInductorModelCache.load(
2023-08-17T15:55:54.1685073Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1131, in load
2023-08-17T15:55:54.1685881Z     so_path, exported = torch._export.aot_compile(
2023-08-17T15:55:54.1688229Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 460, in aot_compile
2023-08-17T15:55:54.1689001Z     ep = export(f, args, kwargs, constraints)
2023-08-17T15:55:54.1689953Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_export/__init__.py", line 222, in export
2023-08-17T15:55:54.1691557Z     gm_torch_level, _ = torch._dynamo.export(
2023-08-17T15:55:54.1692503Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1201, in inner
2023-08-17T15:55:54.1692892Z     result_traced = opt_f(*args, **kwargs)
2023-08-17T15:55:54.1693469Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T15:55:54.1693895Z     return self._call_impl(*args, **kwargs)
2023-08-17T15:55:54.1694443Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:55:54.1694834Z     return forward_call(*args, **kwargs)
2023-08-17T15:55:54.1695343Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 333, in _fn
2023-08-17T15:55:54.1695705Z     return fn(*args, **kwargs)
2023-08-17T15:55:54.1696258Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
2023-08-17T15:55:54.1696666Z     return self._call_impl(*args, **kwargs)
2023-08-17T15:55:54.1697186Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:55:54.1697571Z     return forward_call(*args, **kwargs)
2023-08-17T15:55:54.1698115Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 493, in catch_errors
2023-08-17T15:55:54.1698647Z     return callback(frame, cache_size, hooks, frame_state)
2023-08-17T15:55:54.1699204Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 132, in _fn
2023-08-17T15:55:54.1699551Z     return fn(*args, **kwargs)
2023-08-17T15:55:54.1700097Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 370, in _convert_frame_assert
2023-08-17T15:55:54.1700490Z     return _compile(
2023-08-17T15:55:54.1701001Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 554, in _compile
2023-08-17T15:55:54.1701618Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2023-08-17T15:55:54.1702471Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 180, in time_wrapper
2023-08-17T15:55:54.1703014Z     r = func(*args, **kwargs)
2023-08-17T15:55:54.1703868Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 465, in compile_inner
2023-08-17T15:55:54.1704669Z     out_code = transform_code_object(code, transform)
2023-08-17T15:55:54.1705631Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
2023-08-17T15:55:54.1706301Z     transformations(instructions, code_options)
2023-08-17T15:55:54.1706942Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 432, in transform
2023-08-17T15:55:54.1707624Z     tracer.run()
2023-08-17T15:55:54.1708269Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2071, in run
2023-08-17T15:55:54.1708674Z     super().run()
2023-08-17T15:55:54.1709184Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:55:54.1709539Z     and self.step()
2023-08-17T15:55:54.1710171Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:55:54.1710572Z     getattr(self, inst.opname)(inst)
2023-08-17T15:55:54.1711117Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:55:54.1711499Z     return inner_fn(self, inst)
2023-08-17T15:55:54.1712057Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1167, in CALL_FUNCTION_KW
2023-08-17T15:55:54.1712459Z     self.call_function(fn, args, kwargs)
2023-08-17T15:55:54.1713012Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:55:54.1713440Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:55:54.1714016Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py", line 331, in call_function
2023-08-17T15:55:54.1714412Z     return tx.inline_user_function_return(
2023-08-17T15:55:54.1715009Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T15:55:54.1715529Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T15:55:54.1716167Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T15:55:54.1716597Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T15:55:54.1717156Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T15:55:54.1717520Z     tracer.run()
2023-08-17T15:55:54.1718024Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:55:54.1718387Z     and self.step()
2023-08-17T15:55:54.1718937Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:55:54.1719323Z     getattr(self, inst.opname)(inst)
2023-08-17T15:55:54.1719871Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:55:54.1720251Z     return inner_fn(self, inst)
2023-08-17T15:55:54.1720818Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1155, in CALL_FUNCTION_EX
2023-08-17T15:55:54.1721266Z     self.call_function(fn, argsvars.items, kwargsvars.items)
2023-08-17T15:55:54.1721867Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:55:54.1722375Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:55:54.1723327Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 307, in call_function
2023-08-17T15:55:54.1723740Z     return super().call_function(tx, args, kwargs)
2023-08-17T15:55:54.1724322Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 261, in call_function
2023-08-17T15:55:54.1724749Z     return super().call_function(tx, args, kwargs)
2023-08-17T15:55:54.1725321Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 90, in call_function
2023-08-17T15:55:54.1725743Z     return tx.inline_user_function_return(
2023-08-17T15:55:54.1726510Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 598, in inline_user_function_return
2023-08-17T15:55:54.1727028Z     result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
2023-08-17T15:55:54.1727669Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2176, in inline_call
2023-08-17T15:55:54.1728096Z     return cls.inline_call_(parent, func, args, kwargs)
2023-08-17T15:55:54.1728844Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2283, in inline_call_
2023-08-17T15:55:54.1729203Z     tracer.run()
2023-08-17T15:55:54.1729714Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 724, in run
2023-08-17T15:55:54.1730071Z     and self.step()
2023-08-17T15:55:54.1730587Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 688, in step
2023-08-17T15:55:54.1730968Z     getattr(self, inst.opname)(inst)
2023-08-17T15:55:54.1731511Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 392, in wrapper
2023-08-17T15:55:54.1731894Z     return inner_fn(self, inst)
2023-08-17T15:55:54.1732447Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1115, in CALL_FUNCTION
2023-08-17T15:55:54.1732831Z     self.call_function(fn, args, {})
2023-08-17T15:55:54.1733390Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 562, in call_function
2023-08-17T15:55:54.1733816Z     self.push(fn.call_function(self, args, kwargs))
2023-08-17T15:55:54.1734398Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/user_defined.py", line 328, in call_function
2023-08-17T15:55:54.1734876Z     return VariableBuilder(tx, source).wrap_unspecialized_primitive(
2023-08-17T15:55:54.1735528Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py", line 1119, in wrap_unspecialized_primitive
2023-08-17T15:55:54.1735937Z     raise AssertionError(
2023-08-17T15:55:54.1736373Z AssertionError: Dynamo attempts to add additional input during export: value=0.9152050018310547, source=RandomValueSource(random_call_index=0)
2023-08-17T15:55:54.1736687Z 
2023-08-17T15:55:54.1736778Z from user code:
2023-08-17T15:55:54.1737315Z    File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 889, in forward
2023-08-17T15:55:54.1737688Z     outputs = self.model(
2023-08-17T15:55:54.1738209Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
2023-08-17T15:55:54.1738636Z     return forward_call(*args, **kwargs)
2023-08-17T15:55:54.1739199Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xglm/modeling_xglm.py", line 743, in forward
2023-08-17T15:55:54.1739618Z     dropout_probability = random.uniform(0, 1)
2023-08-17T15:55:54.1739809Z 
2023-08-17T15:55:54.1739996Z Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
2023-08-17T15:55:54.1740218Z 
2023-08-17T15:55:54.9410989Z ERROR
2023-08-17T15:55:58.1614221Z 
2023-08-17T15:56:04.8593696Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:56:04.8594015Z loading model: 0it [00:06, ?it/s]
2023-08-17T15:56:04.8677306Z cuda eval  XLNetLMHeadModel                   
2023-08-17T15:56:34.1707197Z [2023-08-17 15:56:34,168] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-08-17T15:56:57.0708059Z [2023-08-17 15:56:57,069] torch._inductor.utils: [WARNING] DeviceCopy in input program
2023-08-17T15:57:50.3569312Z 
2023-08-17T15:57:50.5518820Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:57:50.7096106Z running benchmark:   3% 1/30 [00:00<00:05,  5.12it/s]
2023-08-17T15:57:50.8684922Z running benchmark:   7% 2/30 [00:00<00:04,  5.77it/s]
2023-08-17T15:57:51.0278922Z running benchmark:  10% 3/30 [00:00<00:04,  6.00it/s]
2023-08-17T15:57:51.1859682Z running benchmark:  13% 4/30 [00:00<00:04,  6.10it/s]
2023-08-17T15:57:51.3438228Z running benchmark:  17% 5/30 [00:00<00:04,  6.18it/s]
2023-08-17T15:57:51.5019487Z running benchmark:  20% 6/30 [00:00<00:03,  6.23it/s]
2023-08-17T15:57:51.6597335Z running benchmark:  23% 7/30 [00:01<00:03,  6.27it/s]
2023-08-17T15:57:51.8186119Z running benchmark:  27% 8/30 [00:01<00:03,  6.29it/s]
2023-08-17T15:57:51.9799402Z running benchmark:  30% 9/30 [00:01<00:03,  6.29it/s]
2023-08-17T15:57:52.1392999Z running benchmark:  33% 10/30 [00:01<00:03,  6.26it/s]
2023-08-17T15:57:52.2969908Z running benchmark:  37% 11/30 [00:01<00:03,  6.26it/s]
2023-08-17T15:57:52.4544294Z running benchmark:  40% 12/30 [00:01<00:02,  6.29it/s]
2023-08-17T15:57:52.6128266Z running benchmark:  43% 13/30 [00:02<00:02,  6.31it/s]
2023-08-17T15:57:52.7704045Z running benchmark:  47% 14/30 [00:02<00:02,  6.31it/s]
2023-08-17T15:57:52.9285580Z running benchmark:  50% 15/30 [00:02<00:02,  6.32it/s]
2023-08-17T15:57:53.0860476Z running benchmark:  53% 16/30 [00:02<00:02,  6.32it/s]
2023-08-17T15:57:53.2473788Z running benchmark:  57% 17/30 [00:02<00:02,  6.33it/s]
2023-08-17T15:57:53.4055631Z running benchmark:  60% 18/30 [00:02<00:01,  6.29it/s]
2023-08-17T15:57:53.5641415Z running benchmark:  63% 19/30 [00:03<00:01,  6.30it/s]
2023-08-17T15:57:53.7214117Z running benchmark:  67% 20/30 [00:03<00:01,  6.30it/s]
2023-08-17T15:57:53.8834802Z running benchmark:  70% 21/30 [00:03<00:01,  6.32it/s]
2023-08-17T15:57:54.0448342Z running benchmark:  73% 22/30 [00:03<00:01,  6.27it/s]
2023-08-17T15:57:54.2035060Z running benchmark:  77% 23/30 [00:03<00:01,  6.25it/s]
2023-08-17T15:57:54.3641215Z running benchmark:  80% 24/30 [00:03<00:00,  6.27it/s]
2023-08-17T15:57:54.5272578Z running benchmark:  83% 25/30 [00:04<00:00,  6.25it/s]
2023-08-17T15:57:54.6852474Z running benchmark:  87% 26/30 [00:04<00:00,  6.22it/s]
2023-08-17T15:57:54.8435477Z running benchmark:  90% 27/30 [00:04<00:00,  6.25it/s]
2023-08-17T15:57:55.0007098Z running benchmark:  93% 28/30 [00:04<00:00,  6.27it/s]
2023-08-17T15:57:55.1586861Z running benchmark:  97% 29/30 [00:04<00:00,  6.30it/s]
2023-08-17T15:57:55.1587369Z running benchmark: 100% 30/30 [00:04<00:00,  6.31it/s]
2023-08-17T15:57:55.1591213Z running benchmark: 100% 30/30 [00:04<00:00,  6.25it/s]
2023-08-17T15:57:55.1619551Z 2.162x
2023-08-17T15:57:59.9237629Z 
2023-08-17T15:58:03.0714658Z loading model: 0it [00:00, ?it/s]
2023-08-17T15:58:03.0715021Z loading model: 0it [00:03, ?it/s]
2023-08-17T15:58:03.0780672Z cuda eval  YituTechConvBert                   
2023-08-17T15:59:09.3771844Z 
2023-08-17T15:59:09.4947890Z running benchmark:   0% 0/30 [00:00<?, ?it/s]
2023-08-17T15:59:09.5992161Z running benchmark:   3% 1/30 [00:00<00:03,  8.46it/s]
2023-08-17T15:59:09.7039730Z running benchmark:  10% 3/30 [00:00<00:01, 14.41it/s]
2023-08-17T15:59:09.8084921Z running benchmark:  17% 5/30 [00:00<00:01, 16.49it/s]
2023-08-17T15:59:09.9131634Z running benchmark:  23% 7/30 [00:00<00:01, 17.51it/s]
2023-08-17T15:59:10.0184373Z running benchmark:  30% 9/30 [00:00<00:01, 18.08it/s]
2023-08-17T15:59:10.1231856Z running benchmark:  37% 11/30 [00:00<00:01, 18.41it/s]
2023-08-17T15:59:10.2280116Z running benchmark:  43% 13/30 [00:00<00:00, 18.62it/s]
2023-08-17T15:59:10.3420202Z running benchmark:  50% 15/30 [00:00<00:00, 18.77it/s]
2023-08-17T15:59:10.4465472Z running benchmark:  57% 17/30 [00:00<00:00, 18.36it/s]
2023-08-17T15:59:10.5510946Z running benchmark:  63% 19/30 [00:01<00:00, 18.59it/s]
2023-08-17T15:59:10.6556466Z running benchmark:  70% 21/30 [00:01<00:00, 18.75it/s]
2023-08-17T15:59:10.7604453Z running benchmark:  77% 23/30 [00:01<00:00, 18.87it/s]
2023-08-17T15:59:10.8650361Z running benchmark:  83% 25/30 [00:01<00:00, 18.93it/s]
2023-08-17T15:59:10.9696303Z running benchmark:  90% 27/30 [00:01<00:00, 18.99it/s]
2023-08-17T15:59:11.0220170Z running benchmark:  97% 29/30 [00:01<00:00, 19.03it/s]
2023-08-17T15:59:11.0221220Z running benchmark: 100% 30/30 [00:01<00:00, 18.23it/s]
2023-08-17T15:59:11.0246372Z 1.653x
2023-08-17T15:59:12.2489629Z speedup             gmean=0.00x mean=1.013x
2023-08-17T15:59:12.2490007Z abs_latency         gmean=0.00x mean=10.726x
2023-08-17T15:59:12.2493929Z compilation_latency mean=52.156 seconds
2023-08-17T15:59:12.2494770Z compression_ratio   mean=0.371x
2023-08-17T15:59:12.2497264Z eager_peak_mem      gmean=0.00x mean=0.718x
2023-08-17T15:59:12.2501497Z dynamo_peak_mem     gmean=0.00x mean=1.041x
2023-08-17T15:59:12.2503060Z calls_captured      gmean=0.00x mean=441.000x
2023-08-17T15:59:12.2505088Z unique_graphs       gmean=0.00x mean=0.529x
2023-08-17T15:59:12.2507645Z graph_breaks        gmean=0.00x mean=0.000x
2023-08-17T15:59:12.2510467Z unique_graph_breaks gmean=0.00x mean=0.000x
2023-08-17T15:59:12.7435026Z + [[ training-true-inference-true-default-true-dynamic-true-cudagraphs-true-aotinductor-true-freezing_cudagraphs-true == *maxautotune-true* ]]
2023-08-17T15:59:12.7609505Z ##[group]Run cat test/**/*.log || true
2023-08-17T15:59:12.7609839Z cat test/**/*.log || true
2023-08-17T15:59:12.7629039Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-08-17T15:59:12.7629335Z env:
2023-08-17T15:59:12.7629546Z   GIT_DEFAULT_BRANCH: main
2023-08-17T15:59:12.7629866Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T15:59:12.7630282Z   DOCKER_CONTAINER_ID: 8fe9204520fb1e14a5fbb249a5da40cb451cc64386cf92c59420d7165072e296
2023-08-17T15:59:12.7630624Z ##[endgroup]
2023-08-17T15:59:12.7691003Z cat: 'test/**/*.log': No such file or directory
2023-08-17T15:59:12.7720277Z Prepare all required actions
2023-08-17T15:59:12.7744557Z ##[group]Run ./.github/actions/get-workflow-job-id
2023-08-17T15:59:12.7744838Z with:
2023-08-17T15:59:12.7745764Z   github-token: ***
2023-08-17T15:59:12.7745987Z env:
2023-08-17T15:59:12.7746194Z   GIT_DEFAULT_BRANCH: main
2023-08-17T15:59:12.7746515Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T15:59:12.7746950Z   DOCKER_CONTAINER_ID: 8fe9204520fb1e14a5fbb249a5da40cb451cc64386cf92c59420d7165072e296
2023-08-17T15:59:12.7747292Z ##[endgroup]
2023-08-17T15:59:12.7763266Z ##[group]Run set -eux
2023-08-17T15:59:12.7763525Z set -eux
2023-08-17T15:59:12.7763898Z GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}")
2023-08-17T15:59:12.7764334Z echo "job-id=${GHA_WORKFLOW_JOB_ID}" >> "${GITHUB_OUTPUT}"
2023-08-17T15:59:12.7781497Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-08-17T15:59:12.7781789Z env:
2023-08-17T15:59:12.7782080Z   GIT_DEFAULT_BRANCH: main
2023-08-17T15:59:12.7782384Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T15:59:12.7782800Z   DOCKER_CONTAINER_ID: 8fe9204520fb1e14a5fbb249a5da40cb451cc64386cf92c59420d7165072e296
2023-08-17T15:59:12.7783356Z   GITHUB_TOKEN: ***
2023-08-17T15:59:12.7783589Z ##[endgroup]
2023-08-17T15:59:12.7824166Z ++ python3 .github/scripts/get_workflow_job_id.py 5887783871 gh-ci-gcp-a100-12
2023-08-17T15:59:13.4344325Z + GHA_WORKFLOW_JOB_ID=15968424899
2023-08-17T15:59:13.4344878Z + echo job-id=15968424899
2023-08-17T15:59:13.4378022Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2023-08-17T15:59:13.4378429Z kill "$MONITOR_SCRIPT_PID"
2023-08-17T15:59:13.4397036Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-08-17T15:59:13.4397332Z env:
2023-08-17T15:59:13.4397546Z   GIT_DEFAULT_BRANCH: main
2023-08-17T15:59:13.4397859Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T15:59:13.4398330Z   DOCKER_CONTAINER_ID: 8fe9204520fb1e14a5fbb249a5da40cb451cc64386cf92c59420d7165072e296
2023-08-17T15:59:13.4398695Z   MONITOR_SCRIPT_PID: 3517818
2023-08-17T15:59:13.4398924Z ##[endgroup]
2023-08-17T15:59:13.4509635Z Prepare all required actions
2023-08-17T15:59:13.4509987Z Getting action download info
2023-08-17T15:59:13.6449694Z Download action repository 'actions/upload-artifact@v3' (SHA:0b7f8abb1508181956e8e162db84b466c27e18ce)
2023-08-17T15:59:14.0404161Z ##[group]Run ./.github/actions/upload-test-artifacts
2023-08-17T15:59:14.0404461Z with:
2023-08-17T15:59:14.0404799Z   file-suffix: test-inductor_huggingface_perf-3-3-linux.gcp.a100.large_15968424899
2023-08-17T15:59:14.0405206Z   use-gha: anything-non-empty-to-use-gha
2023-08-17T15:59:14.0405461Z env:
2023-08-17T15:59:14.0405683Z   GIT_DEFAULT_BRANCH: main
2023-08-17T15:59:14.0405997Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T15:59:14.0406409Z   DOCKER_CONTAINER_ID: 8fe9204520fb1e14a5fbb249a5da40cb451cc64386cf92c59420d7165072e296
2023-08-17T15:59:14.0406732Z ##[endgroup]
2023-08-17T15:59:14.0502343Z ##[group]Run actions/upload-artifact@v3
2023-08-17T15:59:14.0502616Z with:
2023-08-17T15:59:14.0503048Z   name: test-jsons-runattempt1-test-inductor_huggingface_perf-3-3-linux.gcp.a100.large_15968424899.zip
2023-08-17T15:59:14.0503458Z   retention-days: 14
2023-08-17T15:59:14.0503716Z   if-no-files-found: warn
2023-08-17T15:59:14.0503963Z   path: test/**/*.json
2023-08-17T15:59:14.0504184Z env:
2023-08-17T15:59:14.0504402Z   GIT_DEFAULT_BRANCH: main
2023-08-17T15:59:14.0504709Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T15:59:14.0505116Z   DOCKER_CONTAINER_ID: 8fe9204520fb1e14a5fbb249a5da40cb451cc64386cf92c59420d7165072e296
2023-08-17T15:59:14.0505441Z ##[endgroup]
2023-08-17T15:59:14.2978312Z With the provided path, there will be 3 files uploaded
2023-08-17T15:59:14.2981166Z Starting artifact upload
2023-08-17T15:59:14.2982743Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging
2023-08-17T15:59:14.2983358Z Artifact name is valid!
2023-08-17T15:59:14.4308942Z Container for artifact "test-jsons-runattempt1-test-inductor_huggingface_perf-3-3-linux.gcp.a100.large_15968424899.zip" successfully created. Starting upload of file(s)
2023-08-17T15:59:14.7286244Z Total size of all the files uploaded is 29080 bytes
2023-08-17T15:59:14.7287036Z File upload process has finished. Finalizing the artifact upload
2023-08-17T15:59:14.8102615Z Artifact has been finalized. All files have been successfully uploaded!
2023-08-17T15:59:14.8103028Z 
2023-08-17T15:59:14.8103221Z The raw size of all the files that were specified for upload is 300200 bytes
2023-08-17T15:59:14.8103747Z The size of all the files that were uploaded is 29080 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage
2023-08-17T15:59:14.8104073Z 
2023-08-17T15:59:14.8104673Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 
2023-08-17T15:59:14.8105100Z 
2023-08-17T15:59:14.8105589Z Artifact test-jsons-runattempt1-test-inductor_huggingface_perf-3-3-linux.gcp.a100.large_15968424899.zip has been successfully uploaded!
2023-08-17T15:59:14.8199076Z ##[group]Run actions/upload-artifact@v3
2023-08-17T15:59:14.8199367Z with:
2023-08-17T15:59:14.8199771Z   name: test-reports-runattempt1-test-inductor_huggingface_perf-3-3-linux.gcp.a100.large_15968424899.zip
2023-08-17T15:59:14.8200170Z   retention-days: 14
2023-08-17T15:59:14.8200426Z   if-no-files-found: ignore
2023-08-17T15:59:14.8200695Z   path: test/**/*.xml
test/**/*.csv

2023-08-17T15:59:14.8200928Z env:
2023-08-17T15:59:14.8201132Z   GIT_DEFAULT_BRANCH: main
2023-08-17T15:59:14.8201443Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T15:59:14.8201853Z   DOCKER_CONTAINER_ID: 8fe9204520fb1e14a5fbb249a5da40cb451cc64386cf92c59420d7165072e296
2023-08-17T15:59:14.8202187Z ##[endgroup]
2023-08-17T15:59:15.0533148Z With the provided path, there will be 24 files uploaded
2023-08-17T15:59:15.0536230Z Starting artifact upload
2023-08-17T15:59:15.0537187Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging
2023-08-17T15:59:15.0537999Z Artifact name is valid!
2023-08-17T15:59:15.1734521Z Container for artifact "test-reports-runattempt1-test-inductor_huggingface_perf-3-3-linux.gcp.a100.large_15968424899.zip" successfully created. Starting upload of file(s)
2023-08-17T15:59:17.3421993Z Total size of all the files uploaded is 20080 bytes
2023-08-17T15:59:17.3422442Z File upload process has finished. Finalizing the artifact upload
2023-08-17T15:59:17.4102961Z Artifact has been finalized. All files have been successfully uploaded!
2023-08-17T15:59:17.4103216Z 
2023-08-17T15:59:17.4103407Z The raw size of all the files that were specified for upload is 40653 bytes
2023-08-17T15:59:17.4103947Z The size of all the files that were uploaded is 20080 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage
2023-08-17T15:59:17.4104277Z 
2023-08-17T15:59:17.4104900Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 
2023-08-17T15:59:17.4105375Z 
2023-08-17T15:59:17.4105878Z Artifact test-reports-runattempt1-test-inductor_huggingface_perf-3-3-linux.gcp.a100.large_15968424899.zip has been successfully uploaded!
2023-08-17T15:59:17.4199078Z ##[group]Run actions/upload-artifact@v3
2023-08-17T15:59:17.4199342Z with:
2023-08-17T15:59:17.4199719Z   name: usage-log-runattempt1-test-inductor_huggingface_perf-3-3-linux.gcp.a100.large_15968424899.zip
2023-08-17T15:59:17.4200334Z   retention-days: 14
2023-08-17T15:59:17.4200592Z   if-no-files-found: ignore
2023-08-17T15:59:17.4200867Z   path: usage_log.txt
test/**/*.log

2023-08-17T15:59:17.4201095Z env:
2023-08-17T15:59:17.4201328Z   GIT_DEFAULT_BRANCH: main
2023-08-17T15:59:17.4201635Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T15:59:17.4202045Z   DOCKER_CONTAINER_ID: 8fe9204520fb1e14a5fbb249a5da40cb451cc64386cf92c59420d7165072e296
2023-08-17T15:59:17.4202395Z ##[endgroup]
2023-08-17T15:59:17.6733327Z Multiple search paths detected. Calculating the least common ancestor of all paths
2023-08-17T15:59:17.6736297Z The least common ancestor is /home/weiwangmeta/actions-runner/_work/pytorch/pytorch. This will be the root directory of the artifact
2023-08-17T15:59:17.6737328Z With the provided path, there will be 1 file uploaded
2023-08-17T15:59:17.6740600Z Starting artifact upload
2023-08-17T15:59:17.6742103Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging
2023-08-17T15:59:17.6742701Z Artifact name is valid!
2023-08-17T15:59:17.7981011Z Container for artifact "usage-log-runattempt1-test-inductor_huggingface_perf-3-3-linux.gcp.a100.large_15968424899.zip" successfully created. Starting upload of file(s)
2023-08-17T15:59:19.1743783Z Total size of all the files uploaded is 1434581 bytes
2023-08-17T15:59:19.1744216Z File upload process has finished. Finalizing the artifact upload
2023-08-17T15:59:19.3468122Z Artifact has been finalized. All files have been successfully uploaded!
2023-08-17T15:59:19.3468412Z 
2023-08-17T15:59:19.3468609Z The raw size of all the files that were specified for upload is 82896315 bytes
2023-08-17T15:59:19.3469138Z The size of all the files that were uploaded is 1434581 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage
2023-08-17T15:59:19.3469471Z 
2023-08-17T15:59:19.3470205Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 
2023-08-17T15:59:19.3470613Z 
2023-08-17T15:59:19.3471491Z Artifact usage-log-runattempt1-test-inductor_huggingface_perf-3-3-linux.gcp.a100.large_15968424899.zip has been successfully uploaded!
2023-08-17T15:59:19.3590583Z ##[group]Run # shellcheck disable=SC2156
2023-08-17T15:59:19.3590917Z # shellcheck disable=SC2156
2023-08-17T15:59:19.3591529Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2023-08-17T15:59:19.3610090Z shell: /usr/bin/bash -e {0}
2023-08-17T15:59:19.3610333Z env:
2023-08-17T15:59:19.3610555Z   GIT_DEFAULT_BRANCH: main
2023-08-17T15:59:19.3610874Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T15:59:19.3611278Z   DOCKER_CONTAINER_ID: 8fe9204520fb1e14a5fbb249a5da40cb451cc64386cf92c59420d7165072e296
2023-08-17T15:59:19.3611620Z ##[endgroup]
2023-08-17T15:59:19.6844946Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main
2023-08-17T15:59:19.6845291Z with:
2023-08-17T15:59:19.6845479Z env:
2023-08-17T15:59:19.6845706Z   GIT_DEFAULT_BRANCH: main
2023-08-17T15:59:19.6846071Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T15:59:19.6846520Z   DOCKER_CONTAINER_ID: 8fe9204520fb1e14a5fbb249a5da40cb451cc64386cf92c59420d7165072e296
2023-08-17T15:59:19.6846886Z ##[endgroup]
2023-08-17T15:59:19.6863737Z ##[group]Run set -eou pipefail
2023-08-17T15:59:19.6864030Z set -eou pipefail
2023-08-17T15:59:19.6864265Z 
2023-08-17T15:59:19.6864571Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2023-08-17T15:59:19.6864885Z for _ in $(seq 1440); do
2023-08-17T15:59:19.6865172Z     # Break if no ssh session exists anymore
2023-08-17T15:59:19.6865457Z     if [ "$(who)" = "" ]; then
2023-08-17T15:59:19.6865690Z       break
2023-08-17T15:59:19.6865949Z     fi
2023-08-17T15:59:19.6866167Z     echo "."
2023-08-17T15:59:19.6866432Z     sleep 5
2023-08-17T15:59:19.6866654Z done
2023-08-17T15:59:19.6885402Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-08-17T15:59:19.6885690Z env:
2023-08-17T15:59:19.6885900Z   GIT_DEFAULT_BRANCH: main
2023-08-17T15:59:19.6886215Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T15:59:19.6886651Z   DOCKER_CONTAINER_ID: 8fe9204520fb1e14a5fbb249a5da40cb451cc64386cf92c59420d7165072e296
2023-08-17T15:59:19.6886992Z ##[endgroup]
2023-08-17T15:59:19.6924649Z Holding runner for 2 hours until all ssh sessions have logged out
2023-08-17T15:59:19.6973835Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2023-08-17T15:59:19.6974252Z # ignore expansion of "docker ps -q" since it could be empty
2023-08-17T15:59:19.6974582Z # shellcheck disable=SC2046
2023-08-17T15:59:19.6974888Z docker stop $(docker ps -q) || true
2023-08-17T15:59:19.6975187Z # Prune all of the docker images
2023-08-17T15:59:19.6975456Z docker system prune -af
2023-08-17T15:59:19.6991921Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-08-17T15:59:19.6992209Z env:
2023-08-17T15:59:19.6992436Z   GIT_DEFAULT_BRANCH: main
2023-08-17T15:59:19.6992741Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2023-08-17T15:59:19.6993167Z   DOCKER_CONTAINER_ID: 8fe9204520fb1e14a5fbb249a5da40cb451cc64386cf92c59420d7165072e296
2023-08-17T15:59:19.6993562Z ##[endgroup]
2023-08-17T15:59:20.0867557Z 8fe9204520fb
2023-08-17T15:59:20.7297277Z Deleted Containers:
2023-08-17T15:59:20.7297675Z 8fe9204520fb1e14a5fbb249a5da40cb451cc64386cf92c59420d7165072e296
2023-08-17T15:59:20.7297917Z 
2023-08-17T15:59:27.1118462Z Deleted Images:
2023-08-17T15:59:27.1120572Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks:f89d62c8626a24aae4b1a64d42a3caf9cd95bddd
2023-08-17T15:59:27.1122233Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn8-py3-gcc9-inductor-benchmarks@sha256:2fb6aa86ece66a8b83e06455dc98242ec47d8bc0bdb0edad8232d40ed23c8576
2023-08-17T15:59:27.1123440Z deleted: sha256:808d86a8c9535f1195b348920a0142956c41b85dbcfdf4cc80d453060e812b6f
2023-08-17T15:59:27.1123891Z deleted: sha256:bb2ca1d8e1197076cf17bc4273393149fa0e600e4638d4fa1df78feff1001046
2023-08-17T15:59:27.1125007Z deleted: sha256:d0720383fdd7af7973595fc7e2aec34500f4d504ef15b72f1aed14b13e893442
2023-08-17T15:59:27.1125489Z deleted: sha256:64b90fa6abe5473b972b6d22e3f601f02d4fd7472062cf9bfa2e0334182c0e0b
2023-08-17T15:59:27.1125925Z deleted: sha256:964d74abf9898b8903094ca5e310216a2cc81b71d29abd76ee9e483e407b9442
2023-08-17T15:59:27.1126349Z deleted: sha256:220d0c5d750f08150e2f377b7590418ab737af5950a9bbe07ae23ecbccc31d4f
2023-08-17T15:59:27.1126772Z deleted: sha256:b34d8663db07ae972a220b394056924ab3444a08b3bd1e938cfd192c8dfa3f95
2023-08-17T15:59:27.1127364Z deleted: sha256:198645ab0164d822432631285ba524b17217472c48fa0dbfb74207e26c43d3b4
2023-08-17T15:59:27.1127786Z deleted: sha256:73845daa53b4403cfca106ed48db4eb834e6afb382d5cf00645383cd108bdcfd
2023-08-17T15:59:27.1128243Z deleted: sha256:ac4ed563908bdd71576cfb6c199fa203f8fee9b7c24130d8a62fb0bc52fd2978
2023-08-17T15:59:27.1128664Z deleted: sha256:191fa6a59b1477e1983642761327d1575e57278a872848b8d266674a2d51a011
2023-08-17T15:59:27.1129062Z deleted: sha256:c851275b98ac5f9742faea0c8e2705dc788359be90ac6386c11b105e85f757d4
2023-08-17T15:59:27.1129502Z deleted: sha256:e9d043ad90d332db62f45cfd7f7ac8ad718647af8515f8df3e97d24806d06c71
2023-08-17T15:59:27.1129943Z deleted: sha256:7e624b50d6e8975715a4fffa577d6a9900313735d8db2cddcbe3a140c3170c15
2023-08-17T15:59:27.1130465Z deleted: sha256:31fad6f7bbad4ad38771a67b16e56b1d080d77b8dfa18f2dd03e710c34436388
2023-08-17T15:59:27.1130926Z deleted: sha256:5ac9b2da640ec6aeb3fbd873135b61429cf93b4a70ac940927f87f55a48436ed
2023-08-17T15:59:27.1131356Z deleted: sha256:bf4d3e70c2d0eb12960953c5647c186796411eed2998c372b825bc43c1ba6fa5
2023-08-17T15:59:27.1131779Z deleted: sha256:b0334a6042345da3b7d6fcb82e5665fe3cff8287ddf0e8626306b8d665ef1b0d
2023-08-17T15:59:27.1132227Z deleted: sha256:36f89cf698d40ae2d92decdd0482e0c27aa1db2d107d4ad567fd819f4b2d18fc
2023-08-17T15:59:27.1132678Z deleted: sha256:dff44c314f8fd03333872fec97ba89ac593f00703d6879f4e323bd9f998d6902
2023-08-17T15:59:27.1133091Z deleted: sha256:d08d2fd2815bea5988021562e1d9d4107f21646a3a8345eaa44b0b2b63e8dbde
2023-08-17T15:59:27.1133530Z deleted: sha256:5eefb3f1c4cac8443701291cee93ad85786af175d36ec7bfe2d0793035b97150
2023-08-17T15:59:27.1133960Z deleted: sha256:6dd68da849412e80f6f809907836617524ccd62acada6f1826199a9b75b72b37
2023-08-17T15:59:27.1134381Z deleted: sha256:353f7e662d9effd1bf2d39835c5b6dc7921f69e09c71c646b603796f2c51d7ae
2023-08-17T15:59:27.1134781Z deleted: sha256:d99f859c8842828f928231c30688c0faa1880e38d41586e263b2c34b7ae49399
2023-08-17T15:59:27.1135184Z deleted: sha256:0b02ea2969082d7ac0a51691805b4d3f823af2b885818992b9b8bfdcca51fd46
2023-08-17T15:59:27.1135601Z deleted: sha256:74614f674388db4a529ac54a2cada0cc6310258a1c551a1f46d8d95a55544b75
2023-08-17T15:59:27.1136213Z deleted: sha256:cf0a7ede6c0f49e81bd1bb834afcec73abc33a5b7d619fabb169bd149d556034
2023-08-17T15:59:27.1136679Z deleted: sha256:6567fe09e4e658b0c2f7656d69af56297f912904c63dd5dc10495ec456bdf562
2023-08-17T15:59:27.1137285Z deleted: sha256:688b88e5fb61756bc6ef38e73606a39b3d638fbbab5e43f5746a00a736c38ab9
2023-08-17T15:59:27.1137858Z deleted: sha256:7c3598609067a237f01491980b0b442208c2a759867e59963f257a29197e252e
2023-08-17T15:59:27.1138466Z deleted: sha256:82559b282c96acf0942461f185a974a8bb70f3ab8ec371dcb766ffc46b782b78
2023-08-17T15:59:27.1138890Z deleted: sha256:3048794780bd6acebfe36a057755c62346f4ca52fb429563d750d94a806495bd
2023-08-17T15:59:27.1139301Z deleted: sha256:c812ff25661196a1aa6e5c1163d49cb3c32c0b67cdc7d2d9bf0c62ec240f3756
2023-08-17T15:59:27.1139744Z deleted: sha256:097b4f9cf943613cf840461998a36190b9a2c3fd8e2dd68afd5a7525959e5264
2023-08-17T15:59:27.1140221Z deleted: sha256:2b0f080135613f90e286f5c4316ded888706048b1da6c67fff4428bfa3977690
2023-08-17T15:59:27.1140654Z deleted: sha256:aa893b66af8e71a4d97ce488c924885a8cd0cf9c42f26cde5a2b4f348232d058
2023-08-17T15:59:27.1141088Z deleted: sha256:2e273910885e7275d924cef8856e117c52a36ccc611ad0b939d8ed02179d5ba3
2023-08-17T15:59:27.1141491Z deleted: sha256:f4db39de569468fe4b0be51740820b6110ec67fa269901142d78ab51d2ae1ebc
2023-08-17T15:59:27.1142133Z deleted: sha256:d8c5c4ed5bf58dac0c9f4844e220c171fe7f6f637e3ba007864390cfed285778
2023-08-17T15:59:27.1142576Z deleted: sha256:97458aa46141b894b796d3a3d0f7dd7148a8b25416bc7bd26eb29ccabde5d519
2023-08-17T15:59:27.1143001Z deleted: sha256:034293e4fddc57d51db119f616e99085f0d745705e6ab585e2abd854b384e8e4
2023-08-17T15:59:27.1143392Z deleted: sha256:9c912744f04342b7b0a1965b323353553dd19a44b79d5d29446fbc980c644494
2023-08-17T15:59:27.1143804Z deleted: sha256:90fb4c6f199e1b8e28ffd437b3664614d21d6930a8af59b91b3fa8e00e480a24
2023-08-17T15:59:27.1144326Z deleted: sha256:478439e1091d3378b8efeee1f214a7b8581d79c3f53bf3366c47855e7c5bae6a
2023-08-17T15:59:27.1144761Z deleted: sha256:741c481d3a17d3eb83137be0fbb4f9733c9e1236d5657c371bb1e8ddffd6ad35
2023-08-17T15:59:27.1145204Z deleted: sha256:c88c9dc3806d63617e02cc7f8000f2a3ba192dc6ea425a53a3ff8aecc427d5c6
2023-08-17T15:59:27.1145628Z deleted: sha256:44e244abe9d84724036b10e2e4525e0dd5c4c38aaf604b089f52377993af8a71
2023-08-17T15:59:27.1146046Z deleted: sha256:5904e4cb4678684a8185072fcd5cfed783a4c069ab6562382a691d9e69c6768e
2023-08-17T15:59:27.1146453Z deleted: sha256:471d49704052d70843a29043e843dcf2ff2bb45f20a5f743320f6b8930ab83e0
2023-08-17T15:59:27.1146872Z deleted: sha256:e32f7778fad2a4d38c6dc8f947c819b372a32b2f61220be3bcd367059d71ef5b
2023-08-17T15:59:27.1147285Z deleted: sha256:8faf2202b72e741b4876ff44a351c4b62e3537ca7435b3da26019df67fa29a0c
2023-08-17T15:59:27.1147711Z deleted: sha256:afdbc94880b21011b8644d87f58a7f2921bd8699349e1cd1b7d7f091de6258ef
2023-08-17T15:59:27.1148143Z deleted: sha256:57a007b555987ae7cbc3d25b94a3f621b75a7bac5e6e1f18632e11229bcf0d0f
2023-08-17T15:59:27.1148610Z deleted: sha256:629e4c8b1530f13738faa0d73346540ca532493fc0a65a02410b2c5865cd668b
2023-08-17T15:59:27.1149105Z deleted: sha256:fedad22659b7c4e6845dddd4241cdc1bf1acb5e89c96946a4faf2e17c9355aa7
2023-08-17T15:59:27.1149553Z deleted: sha256:556cace179eb61750f06d283426e49b6ecc83a8d00f523a15a1a46ce1c1e403d
2023-08-17T15:59:27.1149989Z deleted: sha256:a512dcb57f2d5b50af1f70ec0c10c5003f3469b86287344ff84d5fdf49bdb212
2023-08-17T15:59:27.1150498Z deleted: sha256:f31b46dab152ddba4794c72c740de2be4951eb49fbf3d25af059d9e6cdd530b8
2023-08-17T15:59:27.1150957Z deleted: sha256:eb468c10edc9a6686f2fc02febe7d2656f190202ccd625c47f94282650cb3c1d
2023-08-17T15:59:27.1151406Z deleted: sha256:b4f9d4cfbc08ee015dc40390a10da9da7bdbf8c55d583df9643163b0bc5f8aab
2023-08-17T15:59:27.1151880Z deleted: sha256:bccf4a5c6cc0860a9cae88e67817b6c0c1e6f2cd9c18cbfbd597e21f5626e7b8
2023-08-17T15:59:27.1152338Z deleted: sha256:b5d0238a705556dc64f513b87d24fca4a9519f004a4eedbe6b2ef474360ba240
2023-08-17T15:59:27.1152770Z deleted: sha256:e47a56d0da4b3f5c81108e574ca111a50177281ea1114b1a5b6a2ffc1e08a4f6
2023-08-17T15:59:27.1153182Z deleted: sha256:64a3f6af42576a985454f12b64530985d762810d5540aa8324aa804a4c3c4fd8
2023-08-17T15:59:27.1153584Z deleted: sha256:74fcea7ec29d6298921b1d0b8302aaba2b036ec65bcb4300214c06e680b4708a
2023-08-17T15:59:27.1154036Z deleted: sha256:ec66d8cea54a2f4dfbbd8342ce082503bf8541e996a800c0d724b8dd2fea7f6a
2023-08-17T15:59:27.1154301Z 
2023-08-17T15:59:27.1154442Z Total reclaimed space: 24.58GB
2023-08-17T15:59:27.1242196Z Post job cleanup.
2023-08-17T15:59:27.1295195Z Post job cleanup.
2023-08-17T15:59:27.2415552Z Unexpected error attempting to determine if executable file exists '/home/weiwangmeta/.local/bin/git': Error: EACCES: permission denied, stat '/home/weiwangmeta/.local/bin/git'
2023-08-17T15:59:27.2450680Z [command]/usr/bin/git version
2023-08-17T15:59:27.2496577Z git version 2.25.1
2023-08-17T15:59:27.2549303Z Temporarily overriding HOME='/home/weiwangmeta/actions-runner/_work/_temp/8ac24340-afd1-41de-80c6-4a6dffe5a341' before making global git config changes
2023-08-17T15:59:27.2550352Z Adding repository directory to the temporary git global config as a safe directory
2023-08-17T15:59:27.2556472Z [command]/usr/bin/git config --global --add safe.directory /home/weiwangmeta/actions-runner/_work/pytorch/pytorch
2023-08-17T15:59:27.2602942Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2023-08-17T15:59:27.2640649Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2023-08-17T15:59:27.2913596Z Entering 'android/libs/fbjni'
2023-08-17T15:59:27.2958997Z Entering 'third_party/FP16'
2023-08-17T15:59:27.3004717Z Entering 'third_party/FXdiv'
2023-08-17T15:59:27.3048458Z Entering 'third_party/NNPACK'
2023-08-17T15:59:27.3093885Z Entering 'third_party/QNNPACK'
2023-08-17T15:59:27.3139967Z Entering 'third_party/VulkanMemoryAllocator'
2023-08-17T15:59:27.3184184Z Entering 'third_party/XNNPACK'
2023-08-17T15:59:27.3245290Z Entering 'third_party/benchmark'
2023-08-17T15:59:27.3289195Z Entering 'third_party/cpuinfo'
2023-08-17T15:59:27.3335480Z Entering 'third_party/cub'
2023-08-17T15:59:27.3380762Z Entering 'third_party/cudnn_frontend'
2023-08-17T15:59:27.3434838Z Entering 'third_party/cutlass'
2023-08-17T15:59:27.3488861Z Entering 'third_party/eigen'
2023-08-17T15:59:27.3536401Z Entering 'third_party/fbgemm'
2023-08-17T15:59:27.3579959Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-08-17T15:59:27.3623638Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-08-17T15:59:27.3668528Z Entering 'third_party/fbgemm/third_party/cutlass'
2023-08-17T15:59:27.3720836Z Entering 'third_party/fbgemm/third_party/googletest'
2023-08-17T15:59:27.3764528Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-08-17T15:59:27.3808567Z Entering 'third_party/flatbuffers'
2023-08-17T15:59:27.3855609Z Entering 'third_party/fmt'
2023-08-17T15:59:27.3899104Z Entering 'third_party/foxi'
2023-08-17T15:59:27.3942745Z Entering 'third_party/gemmlowp/gemmlowp'
2023-08-17T15:59:27.3988312Z Entering 'third_party/gloo'
2023-08-17T15:59:27.4031698Z Entering 'third_party/googletest'
2023-08-17T15:59:27.4076826Z Entering 'third_party/ideep'
2023-08-17T15:59:27.4120207Z Entering 'third_party/ideep/mkl-dnn'
2023-08-17T15:59:27.4173425Z Entering 'third_party/ios-cmake'
2023-08-17T15:59:27.4218295Z Entering 'third_party/ittapi'
2023-08-17T15:59:27.4261720Z Entering 'third_party/kineto'
2023-08-17T15:59:27.4306149Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2023-08-17T15:59:27.4349593Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2023-08-17T15:59:27.4395383Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2023-08-17T15:59:27.4439519Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2023-08-17T15:59:27.4483927Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2023-08-17T15:59:27.4526248Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2023-08-17T15:59:27.4572064Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2023-08-17T15:59:27.4615954Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2023-08-17T15:59:27.4660215Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2023-08-17T15:59:27.4704740Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2023-08-17T15:59:27.4749409Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-08-17T15:59:27.4792019Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-08-17T15:59:27.4837967Z Entering 'third_party/mimalloc'
2023-08-17T15:59:27.4881577Z Entering 'third_party/nccl/nccl'
2023-08-17T15:59:27.4926155Z Entering 'third_party/neon2sse'
2023-08-17T15:59:27.4970360Z Entering 'third_party/nlohmann'
2023-08-17T15:59:27.5015855Z Entering 'third_party/onnx'
2023-08-17T15:59:27.5076865Z Entering 'third_party/onnx/third_party/benchmark'
2023-08-17T15:59:27.5123095Z Entering 'third_party/onnx/third_party/pybind11'
2023-08-17T15:59:27.5170284Z Entering 'third_party/onnx-tensorrt'
2023-08-17T15:59:27.5213206Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-08-17T15:59:27.5262066Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-08-17T15:59:27.5307332Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-08-17T15:59:27.5350986Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-08-17T15:59:27.5399989Z Entering 'third_party/pocketfft'
2023-08-17T15:59:27.5444105Z Entering 'third_party/protobuf'
2023-08-17T15:59:27.5491645Z Entering 'third_party/protobuf/third_party/benchmark'
2023-08-17T15:59:27.5535871Z Entering 'third_party/protobuf/third_party/googletest'
2023-08-17T15:59:27.5581320Z Entering 'third_party/psimd'
2023-08-17T15:59:27.5626147Z Entering 'third_party/pthreadpool'
2023-08-17T15:59:27.5670331Z Entering 'third_party/pybind11'
2023-08-17T15:59:27.5715392Z Entering 'third_party/python-enum'
2023-08-17T15:59:27.5759341Z Entering 'third_party/python-peachpy'
2023-08-17T15:59:27.5803844Z Entering 'third_party/python-six'
2023-08-17T15:59:27.5848390Z Entering 'third_party/sleef'
2023-08-17T15:59:27.5893578Z Entering 'third_party/tbb'
2023-08-17T15:59:27.5941031Z Entering 'third_party/tensorpipe'
2023-08-17T15:59:27.5986365Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-08-17T15:59:27.6030862Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-08-17T15:59:27.6073932Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-08-17T15:59:27.6117228Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-08-17T15:59:27.6159405Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-08-17T15:59:27.6206970Z Entering 'third_party/zstd'
2023-08-17T15:59:27.6269056Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2023-08-17T15:59:27.6301252Z http.https://github.com/.extraheader
2023-08-17T15:59:27.6314063Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2023-08-17T15:59:27.6354342Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2023-08-17T15:59:27.6623786Z Entering 'android/libs/fbjni'
2023-08-17T15:59:27.6784512Z http.https://github.com/.extraheader
2023-08-17T15:59:27.6816284Z Entering 'third_party/FP16'
2023-08-17T15:59:27.6842411Z http.https://github.com/.extraheader
2023-08-17T15:59:27.6876906Z Entering 'third_party/FXdiv'
2023-08-17T15:59:27.6903019Z http.https://github.com/.extraheader
2023-08-17T15:59:27.6937052Z Entering 'third_party/NNPACK'
2023-08-17T15:59:27.6962988Z http.https://github.com/.extraheader
2023-08-17T15:59:27.6997003Z Entering 'third_party/QNNPACK'
2023-08-17T15:59:27.7023968Z http.https://github.com/.extraheader
2023-08-17T15:59:27.7058930Z Entering 'third_party/VulkanMemoryAllocator'
2023-08-17T15:59:27.7084375Z http.https://github.com/.extraheader
2023-08-17T15:59:27.7118812Z Entering 'third_party/XNNPACK'
2023-08-17T15:59:27.7143938Z http.https://github.com/.extraheader
2023-08-17T15:59:27.7192921Z Entering 'third_party/benchmark'
2023-08-17T15:59:27.7218223Z http.https://github.com/.extraheader
2023-08-17T15:59:27.7253503Z Entering 'third_party/cpuinfo'
2023-08-17T15:59:27.7278985Z http.https://github.com/.extraheader
2023-08-17T15:59:27.7314538Z Entering 'third_party/cub'
2023-08-17T15:59:27.7340141Z http.https://github.com/.extraheader
2023-08-17T15:59:27.7374552Z Entering 'third_party/cudnn_frontend'
2023-08-17T15:59:27.7399332Z http.https://github.com/.extraheader
2023-08-17T15:59:27.7441578Z Entering 'third_party/cutlass'
2023-08-17T15:59:27.7466626Z http.https://github.com/.extraheader
2023-08-17T15:59:27.7509649Z Entering 'third_party/eigen'
2023-08-17T15:59:27.7534761Z http.https://github.com/.extraheader
2023-08-17T15:59:27.7571599Z Entering 'third_party/fbgemm'
2023-08-17T15:59:27.7596889Z http.https://github.com/.extraheader
2023-08-17T15:59:27.7630766Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-08-17T15:59:27.7656298Z http.https://github.com/.extraheader
2023-08-17T15:59:27.7690149Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-08-17T15:59:27.7715247Z http.https://github.com/.extraheader
2023-08-17T15:59:27.7749052Z Entering 'third_party/fbgemm/third_party/cutlass'
2023-08-17T15:59:27.7775741Z http.https://github.com/.extraheader
2023-08-17T15:59:27.7818195Z Entering 'third_party/fbgemm/third_party/googletest'
2023-08-17T15:59:27.7843069Z http.https://github.com/.extraheader
2023-08-17T15:59:27.7876343Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-08-17T15:59:27.7901403Z http.https://github.com/.extraheader
2023-08-17T15:59:27.7937527Z Entering 'third_party/flatbuffers'
2023-08-17T15:59:27.7963027Z http.https://github.com/.extraheader
2023-08-17T15:59:27.7999408Z Entering 'third_party/fmt'
2023-08-17T15:59:27.8025144Z http.https://github.com/.extraheader
2023-08-17T15:59:27.8059950Z Entering 'third_party/foxi'
2023-08-17T15:59:27.8085017Z http.https://github.com/.extraheader
2023-08-17T15:59:27.8117767Z Entering 'third_party/gemmlowp/gemmlowp'
2023-08-17T15:59:27.8143207Z http.https://github.com/.extraheader
2023-08-17T15:59:27.8177002Z Entering 'third_party/gloo'
2023-08-17T15:59:27.8202733Z http.https://github.com/.extraheader
2023-08-17T15:59:27.8236489Z Entering 'third_party/googletest'
2023-08-17T15:59:27.8262259Z http.https://github.com/.extraheader
2023-08-17T15:59:27.8296653Z Entering 'third_party/ideep'
2023-08-17T15:59:27.8322863Z http.https://github.com/.extraheader
2023-08-17T15:59:27.8354807Z Entering 'third_party/ideep/mkl-dnn'
2023-08-17T15:59:27.8379401Z http.https://github.com/.extraheader
2023-08-17T15:59:27.8423458Z Entering 'third_party/ios-cmake'
2023-08-17T15:59:27.8448815Z http.https://github.com/.extraheader
2023-08-17T15:59:27.8482453Z Entering 'third_party/ittapi'
2023-08-17T15:59:27.8507743Z http.https://github.com/.extraheader
2023-08-17T15:59:27.8542026Z Entering 'third_party/kineto'
2023-08-17T15:59:27.8568730Z http.https://github.com/.extraheader
2023-08-17T15:59:27.8601875Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2023-08-17T15:59:27.8628048Z http.https://github.com/.extraheader
2023-08-17T15:59:27.8661696Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2023-08-17T15:59:27.8687502Z http.https://github.com/.extraheader
2023-08-17T15:59:27.8723068Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2023-08-17T15:59:27.8747578Z http.https://github.com/.extraheader
2023-08-17T15:59:27.8782766Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2023-08-17T15:59:27.8808534Z http.https://github.com/.extraheader
2023-08-17T15:59:27.8842044Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2023-08-17T15:59:27.8866978Z http.https://github.com/.extraheader
2023-08-17T15:59:27.8899397Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2023-08-17T15:59:27.8924747Z http.https://github.com/.extraheader
2023-08-17T15:59:27.8960201Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2023-08-17T15:59:27.8985535Z http.https://github.com/.extraheader
2023-08-17T15:59:27.9019833Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2023-08-17T15:59:27.9044351Z http.https://github.com/.extraheader
2023-08-17T15:59:27.9077725Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2023-08-17T15:59:27.9102521Z http.https://github.com/.extraheader
2023-08-17T15:59:27.9138541Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2023-08-17T15:59:27.9163391Z http.https://github.com/.extraheader
2023-08-17T15:59:27.9199184Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-08-17T15:59:27.9224172Z http.https://github.com/.extraheader
2023-08-17T15:59:27.9258830Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-08-17T15:59:27.9283893Z http.https://github.com/.extraheader
2023-08-17T15:59:27.9319861Z Entering 'third_party/mimalloc'
2023-08-17T15:59:27.9345124Z http.https://github.com/.extraheader
2023-08-17T15:59:27.9379817Z Entering 'third_party/nccl/nccl'
2023-08-17T15:59:27.9406395Z http.https://github.com/.extraheader
2023-08-17T15:59:27.9440233Z Entering 'third_party/neon2sse'
2023-08-17T15:59:27.9465541Z http.https://github.com/.extraheader
2023-08-17T15:59:27.9499252Z Entering 'third_party/nlohmann'
2023-08-17T15:59:27.9525572Z http.https://github.com/.extraheader
2023-08-17T15:59:27.9561275Z Entering 'third_party/onnx'
2023-08-17T15:59:27.9586676Z http.https://github.com/.extraheader
2023-08-17T15:59:27.9638087Z Entering 'third_party/onnx/third_party/benchmark'
2023-08-17T15:59:27.9663284Z http.https://github.com/.extraheader
2023-08-17T15:59:27.9698827Z Entering 'third_party/onnx/third_party/pybind11'
2023-08-17T15:59:27.9723526Z http.https://github.com/.extraheader
2023-08-17T15:59:27.9760835Z Entering 'third_party/onnx-tensorrt'
2023-08-17T15:59:27.9785500Z http.https://github.com/.extraheader
2023-08-17T15:59:27.9818681Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-08-17T15:59:27.9843470Z http.https://github.com/.extraheader
2023-08-17T15:59:27.9883052Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-08-17T15:59:27.9907797Z http.https://github.com/.extraheader
2023-08-17T15:59:27.9943165Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-08-17T15:59:27.9968480Z http.https://github.com/.extraheader
2023-08-17T15:59:28.0002228Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-08-17T15:59:28.0027666Z http.https://github.com/.extraheader
2023-08-17T15:59:28.0067299Z Entering 'third_party/pocketfft'
2023-08-17T15:59:28.0093337Z http.https://github.com/.extraheader
2023-08-17T15:59:28.0126871Z Entering 'third_party/protobuf'
2023-08-17T15:59:28.0153046Z http.https://github.com/.extraheader
2023-08-17T15:59:28.0190532Z Entering 'third_party/protobuf/third_party/benchmark'
2023-08-17T15:59:28.0215789Z http.https://github.com/.extraheader
2023-08-17T15:59:28.0250228Z Entering 'third_party/protobuf/third_party/googletest'
2023-08-17T15:59:28.0274488Z http.https://github.com/.extraheader
2023-08-17T15:59:28.0311160Z Entering 'third_party/psimd'
2023-08-17T15:59:28.0336950Z http.https://github.com/.extraheader
2023-08-17T15:59:28.0371695Z Entering 'third_party/pthreadpool'
2023-08-17T15:59:28.0397363Z http.https://github.com/.extraheader
2023-08-17T15:59:28.0431810Z Entering 'third_party/pybind11'
2023-08-17T15:59:28.0458103Z http.https://github.com/.extraheader
2023-08-17T15:59:28.0494208Z Entering 'third_party/python-enum'
2023-08-17T15:59:28.0519829Z http.https://github.com/.extraheader
2023-08-17T15:59:28.0553057Z Entering 'third_party/python-peachpy'
2023-08-17T15:59:28.0578941Z http.https://github.com/.extraheader
2023-08-17T15:59:28.0614035Z Entering 'third_party/python-six'
2023-08-17T15:59:28.0639061Z http.https://github.com/.extraheader
2023-08-17T15:59:28.0673540Z Entering 'third_party/sleef'
2023-08-17T15:59:28.0700397Z http.https://github.com/.extraheader
2023-08-17T15:59:28.0734796Z Entering 'third_party/tbb'
2023-08-17T15:59:28.0760382Z http.https://github.com/.extraheader
2023-08-17T15:59:28.0797951Z Entering 'third_party/tensorpipe'
2023-08-17T15:59:28.0822861Z http.https://github.com/.extraheader
2023-08-17T15:59:28.0857062Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-08-17T15:59:28.0881358Z http.https://github.com/.extraheader
2023-08-17T15:59:28.0916456Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-08-17T15:59:28.0940905Z http.https://github.com/.extraheader
2023-08-17T15:59:28.0974565Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-08-17T15:59:28.0999933Z http.https://github.com/.extraheader
2023-08-17T15:59:28.1033708Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-08-17T15:59:28.1058632Z http.https://github.com/.extraheader
2023-08-17T15:59:28.1091813Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-08-17T15:59:28.1116519Z http.https://github.com/.extraheader
2023-08-17T15:59:28.1153598Z Entering 'third_party/zstd'
2023-08-17T15:59:28.1179201Z http.https://github.com/.extraheader
2023-08-17T15:59:28.1321741Z A job completed hook has been configured by the self-hosted runner administrator
2023-08-17T15:59:28.1375439Z ##[group]Run '/home/weiwangmeta/post-job.sh'
2023-08-17T15:59:28.1393344Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-08-17T15:59:28.1393658Z ##[endgroup]
2023-08-17T15:59:28.1432037Z The approximate cost for this shard is shown below (in $): 
2023-08-17T15:59:28.1459053Z 21.564
2023-08-17T15:59:28.1666688Z Cleaning up orphan processes